Human body posture estimation method and device

Abstract

The invention discloses a human body posture estimation method and device, relates to the technical field of video processing, and is used for realizing real-time adjustment and optimization of a model based on an event camera and improving generalization ability. According to the human body posture estimation method, in the process of reasoning based on the posture estimation model of the event camera, the network parameters of the model are supported to be adjusted and optimized. The attitude estimation result of the RGB image synchronously collected by the RGB camera is used as a label to execute adjustment and optimization of the attitude estimation model based on the event camera, and the accuracy of the attitude estimation model of the event camera can be improved.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a method and device for estimating human body posture. Background Art

[0002] Human posture estimation is a classic problem in the field of computer vision. It refers to the problem of inferring the key points of the human body through signals collected by sensors, so as to provide accurate human posture for intelligent interaction.

[0003] The most commonly used data source for human pose estimation is RGB images acquired by RGB cameras. However, due to the limitations of their working principles, RGB cameras can only output the entire RGB image at a fixed frame rate. In application scenarios that require ultra-low latency, such as autonomous driving, the latency cannot be further reduced. Human pose estimation can also use the event data stream acquired by event cameras. The human pose estimation based on event camera models has lower latency than that based on RGB images, but currently models based on event cameras cannot be optimized and have low generalization capabilities. Summary of the invention

[0004] The embodiments of the present application provide a method and device for estimating human posture, which can optimize the event camera-based model and improve the generalization capability.

[0005] In a first aspect, the present application provides a method for estimating human posture. The method is applied to an image processing device, which is coupled to a red, green, and blue (RGB) camera and an event camera. The method may include: using a first posture estimation model to perform human posture estimation on first event stream data collected by the event camera to obtain a first estimation result, wherein the collection time of the first event stream data is the same as the exposure time of an RGB image collected by a frame of the RGB camera; using a training sample set to update the network parameters of the first posture estimation model; the training sample set includes multiple samples and labels corresponding to each of the multiple samples; the first sample includes the second event stream data collected by the event camera within the exposure time of the first RGB image or the feature information of the second event stream data; the first sample is any sample of the multiple samples; the first RGB image is a frame of image collected by the RGB camera within a historical set time length; the label corresponding to the first sample is a 2D human posture obtained by using the second posture estimation model to perform human posture estimation on the first RGB image; using the updated first posture estimation model to perform human posture estimation on the third event stream data collected by the event camera to obtain a second estimation result.

[0006] The embodiment of the present application supports the adjustment of network parameters, i.e., model tuning, during the use of the event camera-based posture estimation model (i.e., during the inference process), thereby improving the generalization ability. Although RGB cameras are easily affected by changes in lighting conditions, the estimation accuracy of the RGB posture estimation model is relatively high, so the accuracy of the posture estimation model of the event camera can be improved by using the RGB camera to guide the tuning of the event camera's posture estimation model.

[0007] In a possible implementation, using the updated first posture estimation model to perform human posture estimation on the third event stream data collected by the event camera to obtain a second estimation result includes:

[0008] Performing feature extraction on the third event stream data to obtain feature information of the third event stream data, wherein the feature information of the third event stream data includes M event point cloud features and / or octree voxel features, where M is a positive integer;

[0009] The M event point cloud features are located in n time periods, and the n time periods are located within the exposure time of the first RGB image; the first event point cloud feature among the M event point cloud features is used to characterize the pixel position of an event item generated in the first time period and the number of event items generated by the pixel position in the first time period; the first time period is any time period among the n time periods;

[0010] The octree voxel feature is obtained by performing feature extraction after voxelizing a plurality of event items included in the third event stream data using an octree;

[0011] The feature information is input into the updated first posture estimation model to obtain the second estimation result.

[0012] The embodiment of the present application adopts the above-mentioned method to construct multi-level spatial occupancy features of event stream data: when constructing online training samples, the event data is expressed as high-level spatial occupancy situations at multiple different levels, and then the feature is used together with the event point cloud to train the first posture estimation model of the event camera, thereby improving the 2D posture accuracy.

[0013] In some scenarios, the event stream data can be compressed to generate a two-dimensional image similar to a grayscale image, and then a deep learning model in the image field can be used to extract features from the compressed event frames.

[0014] Compared with the event point cloud features and / or octree voxel features, the frame compression method has a larger data volume. There are a large number of pixels without event points in the compressed image, which takes up a large storage space. By using event point cloud features and / or octree voxel features, the sparse characteristics of event stream data can be maintained, and the data volume is small.

[0015] In a possible implementation, the octree voxel features included in the feature information of the third event stream data are generated in the following manner:

[0016] Using the three-dimensional space represented by the event items included in the third event stream data as the nodes of the first level of the octree structure;

[0017] For each child node included in the first level and the i-th level in the octree structure, execute:

[0018] If the number of event items included in the child node is less than or equal to the first quantity threshold, the voxel value of the child node is set to 0; if the number of event items included in the child node is greater than the first quantity threshold, the voxel value of the child node is set to 1, and the child node is further divided into eight equal parts to obtain eight child nodes of the i+1th level; until the number of event items included in all child nodes included in the last level is less than or equal to the first quantity threshold, so as to obtain the voxel values ​​of the child nodes respectively included in K levels, where i is an integer greater than 1 and less than K, and K is an integer greater than 1;

[0019] Feature extraction is performed on the voxel values ​​of the child nodes included in the K levels to obtain the octree voxel features.

[0020] In a possible implementation, feature extraction is performed on the voxel values ​​of the child nodes included in the K levels to obtain the octree voxel features, including:

[0021] Performing feature extraction on a one-dimensional vector composed of voxel values ​​of the child nodes included in the K levels through a two-layer fully connected neural network to obtain the octree voxel feature;

[0022] The voxel values ​​of the sub-nodes included in the K levels in the one-dimensional vector are arranged in the order of the sub-nodes in the three-dimensional space.

[0023] The one-dimensional vector composed of octree voxels is extracted through a double-layer fully connected neural network to obtain the octree voxel features, which is simple to implement and helps to improve resource utilization.

[0024] In a possible implementation manner, updating the network parameters of the first posture estimation model using the training sample set includes:

[0025] Using the first sample to input the first posture estimation model, obtaining a posture prediction value of the first sample output by the first posture estimation model;

[0026] Determining a loss value of the first posture estimation model according to the posture prediction value of the first sample and the label corresponding to the first sample;

[0027] The network parameters of the first posture estimation model are adjusted according to the loss value.

[0028] In a possible implementation, before using the updated first posture estimation model to perform human posture estimation on the third event stream data collected by the event camera to obtain a second estimation result, the method further includes:

[0029] Determining that the image processing device is in a first working mode;

[0030] Wherein, in the first working mode, the event camera is in an on state, and the RGB camera is in an off state;

[0031] The second estimation result is used as the 2D posture of the human body in the collection time period of the third event stream data.

[0032] In the above embodiment, the working mode is determined, and the on states of the event camera and the RGB camera are adjusted according to the working mode. In the first working mode, the RGB camera does not need to be in the on state, which can reduce power consumption.

[0033] In some embodiments, the working mode of the image processing device can be determined based on at least one of the human body movement speed, current light intensity, power consumption requirement, or frame rate requirement, such as determining whether it is in the first working mode.

[0034] In a possible implementation, in the first working mode, the image processing device satisfies any of the following:

[0035] The image processing device is in a battery-powered state; or,

[0036] The image processing device is in a battery-powered state and the remaining power of the battery is less than a power threshold; or;

[0037] The frame rate required by the image processing device is greater than a frame rate threshold; or,

[0038] The number of event items included in the event stream data collected by the event camera within a unit time is greater than a second number threshold; or,

[0039] The light intensity detected by the image processing device is less than an intensity threshold; or;

[0040] The frame rate required by the image processing device is less than or equal to the frame rate threshold and the number of event items collected by the event camera per unit time is greater than a second number threshold; or,

[0041] The image processing device is in a power supply state and the number of event items collected by the event camera per unit time is greater than the second number threshold; or,

[0042] The image processing device is in a power supply state and the number of event items collected by the event camera per unit time is greater than the second number threshold; or,

[0043] The image processing device is in a power supply state, the number of event items collected by the event camera per unit time is greater than the second number threshold, and the light intensity detected by the image processing device is less than the intensity threshold.

[0044] In a possible implementation, the method further includes:

[0045] Before obtaining the second estimation result, determining that the image processing device is in a second working mode; wherein, in the second working mode, the event camera is in an on state, and the RGB camera is in an on state;

[0046] Use the second posture estimation model to perform human posture estimation on the second RGB image to obtain a third estimation result; the exposure time of the second RGB image is the collection time period of the event camera collecting the third event stream data;

[0047] After obtaining the second estimation result, an average value of the second estimation result and the third estimation result is used as the 2D posture of the human body in the collection time period of the third event stream data.

[0048] In a possible implementation, in the second working mode, the image processing device satisfies any of the following:

[0049] The number of event items included in the event stream data collected by the event camera within a unit time is less than or equal to the second number threshold and the light intensity detected by the image processing device is less than the intensity threshold; or,

[0050] The number of event items collected by the event camera within a unit time is greater than the second number threshold and the light intensity detected by the image processing device is greater than the intensity threshold; or,

[0051] The frame rate required by the image processing device is less than or equal to the frame rate threshold and the number of event items collected by the event camera per unit time is less than or equal to the second number threshold, and the light intensity detected by the image processing device is less than the intensity threshold; or,

[0052] The frame rate required by the image processing device is less than or equal to the frame rate threshold and the number of event items collected by the event camera per unit time is greater than the second number threshold, and the light intensity detected by the image processing device is greater than the intensity threshold; or,

[0053] The image processing device is in a power supply state and the number of event items collected by the event camera per unit time is less than or equal to the second number threshold, and the light intensity detected by the image processing device is less than the intensity threshold; or,

[0054] The image processing device is in a power supply state, the number of event items collected by the event camera per unit time is greater than the second number threshold, and the light intensity detected by the image processing device is greater than the intensity threshold.

[0055] In the above method, when the lighting conditions are poor and the motion speed is low, the credibility of the event camera and the RGB camera is low. The accuracy of posture estimation can be improved by taking the average of the two data sources.

[0056] In a possible implementation, the method further includes:

[0057] Determining that the image processing device is in a third working mode, wherein in the third working mode, the RGB camera is in an on state and the event camera is in an off state;

[0058] The second posture estimation model is used to perform human posture estimation on the third RGB image captured by the RGB camera to obtain a 2D human posture of the exposure time of the third RGB image.

[0059] In a possible implementation, in the third working mode, the image processing device satisfies any of the following:

[0060] The image processing device is in a power supply state and the number of event items included in the event stream data collected by the event camera within a unit time is less than or equal to the second number threshold, and the light intensity detected by the image processing device is greater than the intensity threshold;

[0061] The frame rate required by the image processing device is less than or equal to the frame rate threshold, the number of event items included in the event stream data collected by the event camera per unit time is less than or equal to the second number threshold, and the light intensity detected by the image processing device is greater than the intensity threshold.

[0062] In a second aspect, the present application further provides an image processing device, which has the function of implementing the method of the first aspect or any possible implementation of the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0063] In a third aspect, an embodiment of the present application provides a human posture estimation method, which is applied to an image processing device, wherein the image processing device is coupled with an RGB camera and an event camera, and includes:

[0064] Determining the working mode of the image processing device;

[0065] When the image processing device is in the first working mode, the first event stream data collected by the event camera is used to estimate the human body posture to obtain the 2D human body posture in the collection time period of the first event stream data; or

[0066] When the image processing device is in the second working mode, the first event stream data collected by the event camera is used to estimate the human body posture to obtain a first estimation result, and the first RGB image collected by the RGB camera is used to estimate the human body posture to obtain a second estimation result, and the average value of the first estimation result and the second estimation result is used as the human body 2D posture in the collection time period of the first event stream data; wherein the exposure time of the first RGB image is the same as the collection time period of the first event stream data; or,

[0067] When the image processing device is in the third working mode, the first RGB image captured by the RGB camera is used to perform human posture estimation to obtain a 2D human posture of the exposure time of the first RGB image.

[0068] The embodiment of the present application can determine the data source to be used according to the requirements, thereby improving the scene adaptation degree and improving the accuracy of human body posture estimation.

[0069] In a possible implementation, in the first working mode, the image processing device satisfies any of the following:

[0070] The image processing device is in a battery-powered state; or,

[0071] The image processing device is in a battery-powered state and the remaining power of the battery is less than a power threshold; or;

[0072] The frame rate required by the image processing device is greater than a frame rate threshold; or,

[0073] The number of event items included in the event stream data collected by the event camera within a unit time is greater than a second number threshold; or,

[0074] The light intensity detected by the image processing device is less than an intensity threshold; or;

[0075] The frame rate required by the image processing device is less than or equal to the frame rate threshold and the number of event items included in the event stream data collected by the event camera within a unit time is greater than a second number threshold; or,

[0076] The image processing device is in a power supply state and the number of event items included in the event stream data collected by the event camera within a unit time is greater than the second number threshold; or,

[0077] The image processing device is in a power supply state and the number of event items included in the event stream data collected by the event camera within a unit time is greater than the second number threshold; or,

[0078] The image processing device is in a power-supply state, the number of event items included in the event stream data collected by the event camera per unit time is greater than the second number threshold, and the light intensity detected by the image processing device is less than the intensity threshold.

[0079] In a possible implementation, in the second working mode, the image processing device satisfies any of the following:

[0080] The number of event items included in the event stream data collected by the event camera within a unit time is less than or equal to the second number threshold and the light intensity detected by the image processing device is less than the intensity threshold; or,

[0081] The number of event items included in the event stream data collected by the event camera within a unit time is greater than the second number threshold and the light intensity detected by the image processing device is greater than the intensity threshold; or,

[0082] The frame rate required by the image processing device is less than or equal to the frame rate threshold, the number of event items included in the event stream data collected by the event camera per unit time is less than or equal to the second number threshold, and the light intensity detected by the image processing device is less than the intensity threshold; or,

[0083] The frame rate required by the image processing device is less than or equal to the frame rate threshold, the number of event items included in the event stream data collected by the event camera per unit time is greater than the second number threshold, and the light intensity detected by the image processing device is greater than the intensity threshold; or,

[0084] The image processing device is in a power supply state and the number of event items included in the event stream data collected by the event camera per unit time is less than or equal to the second number threshold, and the light intensity detected by the image processing device is less than the intensity threshold; or,

[0085] The image processing device is in a power-supply state, the number of event items included in the event stream data collected by the event camera per unit time is greater than the second number threshold, and the light intensity detected by the image processing device is greater than the intensity threshold.

[0086] In a possible implementation, in the third working mode, the image processing device satisfies any of the following:

[0087] The image processing device is in a power supply state and the number of event items included in the event stream data collected by the event camera within a unit time is less than or equal to the second number threshold, and the light intensity detected by the image processing device is greater than the intensity threshold;

[0088] The frame rate required by the image processing device is less than or equal to the frame rate threshold, the number of event items included in the event stream data collected by the event camera per unit time is less than or equal to the second number threshold, and the light intensity detected by the image processing device is greater than the intensity threshold.

[0089] In a fourth aspect, the present application further provides an image processing device, which has the function of implementing the method of the third aspect or any possible implementation of the third aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0090] In a fifth aspect, an embodiment of the present application provides an image processing device, comprising a processor and a memory, wherein the processor is coupled to the memory, and the memory is used to store programs; the processor is used to execute the programs in the memory, so that the image processing device executes the method described in any implementation method of the first aspect or the third aspect.

[0091] In a sixth aspect, an embodiment of the present application provides an electronic device, including an image processing device, an event camera, and a red, green, and blue (RGB) camera;

[0092] The event camera is used to collect event stream data; the RGB camera is used to collect RGB images;

[0093] The image processing device is used to execute the method described in any implementation of the first aspect or the third aspect based on the event stream data and the RGB image.

[0094] In the seventh aspect, an embodiment of the present application provides an image processing device, which may also be referred to as a digital processing chip or chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are executed by the processing unit. The processing unit is used to perform processing-related functions in any optional implementation method of the first aspect or the third aspect mentioned above.

[0095] In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the method described in any optional implementation of the first aspect or the third aspect.

[0096] In a ninth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method described in any one of the optional implementations in the first aspect or the third aspect.

[0097] Based on the implementations provided in the above aspects, the present application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] Figure 1 A schematic diagram of the structure of an image processing system provided in an embodiment of the present application;

[0099] Figure 2 A schematic diagram of a human body posture estimation method provided in an embodiment of the present application;

[0100] Figure 3 A schematic diagram of octree voxel feature extraction provided in an embodiment of the present application;

[0101] Figure 4 A schematic diagram of the posture estimation model optimization provided in the embodiment of the present application;

[0102] Figure 5 A schematic diagram of another method for estimating human body posture provided in an embodiment of the present application;

[0103] Figure 6 A schematic diagram of the joint posture estimation process of an RGB camera and an event camera provided in an embodiment of the present application;

[0104] Figure 7 A flow chart of an overall method for performing human posture estimation on an electronic device provided in an embodiment of the present application;

[0105] Figure 8 A schematic diagram of a flow chart of another method for estimating a human body posture provided in an embodiment of the present application;

[0106] Fig. 9A schematic diagram of the structure of an image processing device provided in this embodiment;

[0107] Fig.10 A schematic diagram of another image processing device structure provided in this application example;

[0108] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0109] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0110] Among them, in the description of this application, unless otherwise specified, "multiple" means two or more than two. In addition, " / " indicates that the objects associated before and after are in an "or" relationship, for example, A / B can represent A or B; "and / or" in this application is only a kind of association relationship describing the associated objects, indicating that there can be three relationships, for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first" and "second" are used to distinguish the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not limit them to be different. It should also be noted that, unless otherwise specified, the specific description of some technical features in one embodiment can also be used to explain the corresponding technical features mentioned in other embodiments.

[0111] In an embodiment of the present application, an event camera may be a device that outputs an asynchronous signal by measuring the dynamic light intensity change of each pixel. For example, an event camera may be a camera including one or more event camera sensors (also referred to as event sensors). Unlike a traditional camera that captures a complete image, an event camera captures an "event" (or event item), and the event camera regards the brightness change of a pixel in a real scene as an event. Event sensors include, but are not limited to, dynamic vision sensors (DVS), asynchronous time based image sensors (ATIS), dynamic and active pixel vision sensors (DAVIS), etc.

[0112] The event camera captures dynamic changes in the scene based on an event-driven approach, which can be understood as capturing changes in the brightness of pixels in the scene, that is, the event camera outputs changes in pixel brightness. Pixels can also be referred to as pixels. Specifically, when the brightness of a pixel in a real scene changes, the event camera generates an event at the pixel. The event data corresponding to an event can include four parts (t, x, y, p), where x and y are the pixel coordinates of the event in two-dimensional space, that is, the row and column values ​​of the light sensor corresponding to the pixel whose brightness changes, or the coordinates of the pixel whose brightness changes; t is the timestamp of the event, that is, the time when the brightness of the pixel changes; p is the polarity of the event, representing whether the brightness changes up or down. The value of p is 1 or -1, 1 means an increase in brightness, and -1 means a decrease in brightness.

[0113] This application can be applied to application scenarios such as human-computer interaction, autonomous driving, robots, smart homes, augmented reality (AR) / virtual reality (VR), and sports health.

[0114] See also Figure 1 The figure shows a schematic diagram of the structure of the image processing system provided by an embodiment of the present application. The image processing system includes an image processing device, a red, green, and blue (RGB) camera, and an event camera. The RGB camera is used to collect an RGB image stream, and the RGB image stream (or RGB video stream) includes multiple frames of RGB images. The event camera is used to collect an event stream, and the event stream includes event data. The image processing device is used to perform human posture estimation based on the RGB image stream collected by the RGB camera and / or the event stream collected by the event camera.

[0115] The image processing system can be deployed in an electronic device, and the various components of the image processing system can also be deployed separately. In one possible scenario, the image processing device can be a server, and the server is connected and communicates with the RGB camera and the event camera. The server can be a physical server or a cloud server. For example, the RGB camera and the event camera can be deployed on a terminal device, and the server communicates with the terminal device, such as the terminal device sends an RGB image stream and an event stream to the server, so that the server estimates the human body posture. For another example, the camera includes an RGB camera and an event camera, and the server can be coupled with the camera, and the camera sends an RGB image stream and an event stream to the server, so that the server estimates the human body posture. For another example, the RGB camera and the event camera can be set independently, and the image processing device is coupled with the RGB camera and the event camera respectively, such as the image processing device is a cloud server.

[0116] In another possible scenario, the above-mentioned image processing system is deployed on an electronic device. The electronic devices in this application may include, but are not limited to: smart mobile phones, tablet computers, bracelets, head mounted display devices (Head Mount Display, HMD), augmented reality (augmented reality, AR) devices, mixed reality (mixed reality, MR) devices, vehicle-mounted electronic devices, laptop computers (laptop computers), personal computers (personal computers, PCs), monitoring equipment, robots, vehicle-mounted terminals, autonomous driving vehicles, smart home devices, etc. Of course, in the embodiments of this application, there is no restriction on the specific form of the electronic device.

[0117] Exemplarily, electronic devices, such as cars, mobile phones, AR / VR glasses, security monitoring equipment, cameras or other smart home terminals, can access the cloud platform through a wired or wireless network. A server is provided in the cloud platform, and the server may include a centralized server or a distributed server. The electronic device can communicate with the server of the cloud platform through a wired or wireless network to achieve data transmission. For example, after the electronic device collects the data, it can save or back up the data on the cloud platform to prevent data loss. The electronic device can access an access point or a base station to achieve wireless or wired access to the cloud platform. For example, the access point can be a base station, and a SIM card is provided in the electronic device, and the operator's network authentication is achieved through the SIM card, thereby accessing the wireless network. Alternatively, the access point may include a router, and the electronic device accesses the router through a 2.4GHz or 5GHz wireless network, thereby accessing the cloud platform through the router.

[0118] In addition, electronic devices can process data independently or in collaboration with the cloud, and the specific details can be adjusted according to the actual application scenario. For example, an event camera can be set in the electronic device, and the event camera can work in collaboration with the RGB camera or other sensors in the electronic device, or it can work independently. The processor set in the event camera or the processor set in the electronic device processes the data collected by the event camera or other sensors, and can also cooperate with the cloud device to process the data collected by the event camera or other sensors.

[0119] The electronic device includes an image processing device, an RGB camera and an event camera. The image processing device may be a processor or a processing chip or a processing chip system in an electronic device. The processor may include one or more processing units. For example, the processor may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Among them, different processing units may be independent devices or integrated in one or more processors. The controller may generate an operation control signal based on the instruction opcode and the timing signal to complete the control of fetching and executing instructions.

[0120] A memory may also be provided in the processor for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. The memory may store instructions or data that the processor has just used or is cyclically used. If the processor needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor, and thus improves the efficiency of the system.

[0121] In some embodiments, the electronic device may include other devices, such as a display screen, an external memory interface, an internal memory, a bus interface, buttons, a speaker, an antenna, etc. It should be noted that the electronic device provided in this application may include more or fewer components than those in the structure described above. The above description of the structure of the electronic device is only an exemplary description. Those skilled in the art may add or reduce components in the electronic device as required, and this application does not limit this.

[0122] For example, in an autonomous driving scenario, an event camera and an RGB camera are deployed on an autonomous driving vehicle. The electronic device may be an autonomous driving vehicle, and the event camera and the RGB camera capture the scene around the vehicle. The solution provided in this application is used to estimate the posture of pedestrians around the vehicle, judge the actions and intentions of the surrounding pedestrians, and provide a basis for autonomous driving decisions. In other words, the image processing device may be a vehicle-mounted terminal, and the event camera, RGB camera, and vehicle-mounted terminal deployed on the autonomous driving vehicle constitute an image processing system.

[0123] Another example is AR / VR scenario: the event camera and RGB camera are located on AR / VR glasses / helmets. The electronic device can be AR / VR glasses / helmets. The event camera and RGB camera capture the wearer's surrounding environment and estimate the posture of the person in the environment, providing a basis for the AR / VR system to interact with the surrounding environment and people.

[0124] Another example is a smart home scenario: an event camera and an RGB camera are located at an indoor monitoring position, and the electronic device can be a smart home device. The event camera and the RGB camera shoot users in the indoor scene, and the software and hardware architecture of the present application is used to perform human posture estimation by integrating the event camera and the RGB camera, record the posture of indoor users, and intelligently infer the behavior category and purpose based on the human action, and make intelligent decisions and control the indoor equipment. Alternatively, the event camera, the RGB camera, and the control device form an image processing system. The control device can be a device used to perform human posture estimation in a smart home system.

[0125] At present, in the scheme of human posture estimation using event camera-based model estimation, the event data of the event camera is generally compressed into images and then input into the estimation model. The output of the model is a heat map of the distribution of each key point of the human body in space, and the position of the key point is calculated using the soft-argmax function. On the one hand, the human posture estimation model adopted by the above scheme is trained offline on the training data set. After the training is completed, the weight parameters of the model are fixed values, and the weights cannot be updated after the model is deployed. When the distribution of the training data is different from that of the data collected after deployment or there are pattern differences, the generalization ability of the model is limited, resulting in a decrease in the accuracy of human body estimation. On the other hand, in the above scheme, when the human body is stationary or moves slowly, the event data generated by the event camera is significantly reduced, and the information quality of the image data after the event data is compressed is poor, resulting in a decrease in the accuracy of posture estimation.

[0126] In response to the above-mentioned first aspect, the present application provides a first implementation method of human posture estimation, which is used to update the network parameters of the event camera-based model according to historical data, thereby improving the generalization ability of the model. In response to the second aspect, the present application provides a second implementation method of human posture estimation, which can adjust the working mode of the device or equipment as needed, and determine the use of the RGB camera-based posture estimation model and / or the event camera-based posture estimation model according to the needs to improve the posture estimation accuracy.

[0127] The first implementation method is described below.

[0128] In the first implementation method, the RGB camera guides the online update of the attitude estimation model of the event camera: the attitude estimation model of the RGB camera is used to perform high-precision 2D attitude estimation, the attitude estimation results of the RGB camera are used to construct online training samples, and the weights (or network parameters) of the attitude estimation model of the event camera are updated online.

[0129] See also Figure 2 As shown, it is a schematic flow chart of a method for estimating a human posture provided by an embodiment of the present application. The method flow can be executed by the above-mentioned image processing device, and the image processing device can be a chip or a chip system, or a server or an electronic device. The image processing device is coupled with an RGB camera and an event camera. The shooting viewing angle range of the RGB camera and the event camera is the same or the viewing angle overlaps, or the viewing angle range of the RGB camera includes the viewing angle of the event camera, or the viewing angle of the event camera includes the viewing angle of the RGB camera.

[0130] In order to facilitate the distinction between the posture estimation model corresponding to the RGB camera and the posture estimation model corresponding to the event camera, in the subsequent description, the posture estimation model corresponding to the event camera is referred to as the first posture estimation model, and the posture estimation model corresponding to the RGB camera is referred to as the second posture estimation model. The RGB camera is used to collect an RGB image stream, and the RGB image stream includes multiple RGB images. The event camera is used to collect an event stream. The image processing device can perform human posture estimation based on the event stream and the RGB image stream.

[0131] 201, an image processing device uses a first posture estimation model to perform human posture estimation on first event stream data collected by an event camera to obtain a first estimation result. The collection time of the first event stream data is the same as the exposure time of a frame of RGB image collected by the RGB camera.

[0132] 202. The image processing apparatus updates network parameters of a first posture estimation model using a training sample set.

[0133] The training sample set includes multiple samples and labels corresponding to each of the multiple samples; the first sample includes second event stream data or feature information of the second event stream data collected by the event camera within the exposure time of the first RGB image; the first sample is any sample among the multiple samples; the first RGB image is a frame of image collected by the RGB camera within a historical set time length; the label corresponding to the first sample is a 2D human posture obtained by estimating the human posture of the first RGB image using the second posture estimation model.

[0134] In some embodiments, the second posture estimation model can be pre-trained using an image data set available on a public channel. The detection accuracy of the second posture estimation model trained using the image data set obtained on a public channel is relatively high. However, the RGB camera is easily affected by changes in lighting conditions, which significantly reduces the imaging quality and leads to low detection accuracy. Therefore, an event camera is used to collect event streams to detect human body postures. The event camera is not affected by lighting conditions and can improve detection accuracy.

[0135] The exposure time of the first RGB image and the acquisition time of the second event stream data are the same. It can be understood that the RGB camera and the time camera are synchronized in acquisition time. It can be understood that each sample corresponds to an event stream data, and feature extraction is performed on the event stream data to obtain feature information of the event stream data. The event stream data is acquired by the event camera within the exposure time of the RGB camera acquiring one frame of RGB image.

[0136] The above-mentioned second event stream data can be understood as the motion trajectory generated when the target object moves within the monitoring range of the event camera during the exposure time of the first RGB image. The first RGB image is captured by the RGB camera during the exposure time, and the first RGB image also includes the target object. It should be understood that the target object can be a person, and the human body posture of the person can be estimated. In some possible application scenarios, the target object can also be other moving entities such as moving animals, and this application does not make specific limitations on this.

[0137] The update of the network parameters of the first posture estimation model can be triggered by the user, or it can be performed periodically, or it can be triggered after a certain number of training samples are counted, or the estimation accuracy of the first posture estimation model can be detected periodically, and the update is triggered when the estimation accuracy does not meet the requirements. For example, the estimation result of the human body posture of the collected RGB image by the second posture estimation model can be used to determine the estimation accuracy of the first posture estimation model by using the first posture estimation model to determine the estimation result of the human body posture of the event stream data collected by the event camera at intervals. The embodiment of the present application does not specifically limit the triggering timing of the update of the network parameters of the first posture estimation model.

[0138] 203 , the image processing device uses the updated first posture estimation model to perform human posture estimation on the third event stream data collected by the event camera to obtain a second estimation result.

[0139] After the image processing device obtains the event stream data, it constructs a sample based on the obtained event stream data. In some possible implementations, after obtaining the event stream data, samples can be constructed for a fixed number of event items in the event stream data. For example, each sample corresponds to 7,000 event items, that is, the sample includes 7,000 event points. The 7,000 event points constitute a sample. In another possible implementation, a sample can be constructed based on feature information obtained by feature extraction of the event stream data. For example, feature information obtained by feature extraction of a fixed number of event items in the event stream data constitutes a sample.

[0140] The input of the first posture estimation model is the feature information of the event stream data, and the output is the 2D posture of the human body corresponding to the event stream data (which can also be understood as the 2D posture estimation result of the human body).

[0141] In a possible implementation, the feature information of the event stream data may include event point cloud features and / or octree voxel features. In one embodiment, the feature information of the event stream data includes M event point cloud features and octree voxel features. In another embodiment, the feature information of the event stream data includes only event point cloud features. In yet another embodiment, the feature information of the event stream data includes only octree voxel features.

[0142] Taking the third event stream data as an example, when the third event stream data collected by the event camera is used to estimate the human body posture using the updated first posture estimation model, the third event stream data can be firstly feature extracted to obtain the feature information of the third event stream data. Then, the feature information of the third event stream data is input into the updated first posture estimation model, so that the first posture estimation model outputs the estimation result of the human body 2D posture of the third event stream data, that is, the second estimation result.

[0143] Taking the third event stream data mentioned above as an example, the feature information of the third event stream data includes M event point cloud features. The M event point cloud features are located in n time periods, and the n time periods are located within the exposure time of the first RGB image. The event point cloud features can be represented by (x, y, T, n). (x, y) represents the pixel position, that is, the pixel coordinates of the event. T represents the time period. n represents the number of event items generated by the (x, y) pixel position within the T time period. The event point cloud features may include the pixel positions that generate event items within a time period and the number of event items generated by the pixel positions within the event segment. For example, the first event point cloud feature among the M event point cloud features includes multiple pixel positions that generate event items in the first time period and the number of event items for each pixel position among the multiple pixel positions; the first time period is any time period among the M time periods.

[0144] Exemplarily, after collecting the third event stream data, a fixed number of event items, such as 7000 event points, can be obtained from the third event stream data. The generation time range of these event points is evenly divided into N segments on the time axis, and then the time period T is normalized. The time period range can be adjusted to 0~N by rounding down. Then the number of events at the same pixel position in each time period is accumulated to obtain the event point cloud features of each pixel position in each time period, expressed as (x, y, T, n). In this way, events occurring in similar times can be merged to reduce the amount of data. Further, M event point cloud features are selected from the merged event point cloud features. Thus, the M event point cloud features are located in n time periods, and n is less than or equal to N. That is, the input data size of the first posture estimation model is M*4. The feature extraction of other event stream data can adopt the above method, which will not be repeated here.

[0145] The octree voxel feature is described as follows: Taking the octree voxel feature of the third event stream data as an example, the octree voxel feature of the third event stream data is obtained by performing feature extraction after voxelization of multiple event items included in the third event stream data using an octree.

[0146] As an example, the image processing device may generate the octree voxel features included in the feature information of the third event stream data in the following manner:

[0147] The three-dimensional space represented by the event items included in the third event stream data is used as the node of the first level of the octree structure. Each node of the octree structure represents a volume element of a cube. If each node is further divided, it can be divided into eight child nodes. For each child node included in the first level and the i-th level in the octree structure, execute: if the number of event items included in the child node is less than or equal to the first quantity threshold, the voxel value of the child node is set to 0; if the number of event items included in the child node is greater than the first quantity threshold, the voxel value of the child node is set to 1, and the child node is continued to be divided into eight equal parts to obtain eight child nodes of the i+1th level; until the number of event items included in all child nodes included in the last level is less than or equal to the first quantity threshold, so as to obtain the voxel values ​​of the child nodes included in K levels respectively, i is an integer greater than 1 and less than K, and K is an integer greater than 1; feature extraction is performed on the voxel values ​​of the child nodes included in the K levels to obtain the octree voxel features.

[0148] It is understandable that in the above method, the judgment starts from the first level, see Figure 3As shown, if the number of event items included in the node is less than or equal to the first quantity threshold, the voxel value of the node is set to 0, otherwise it is set to 1, and the node is further divided into eight child nodes, and for each child node, it is judged whether the number of event items included in the child node is greater than the first quantity threshold. If it is greater than the first quantity threshold, the child node is set to 1, otherwise it is set to 0, thereby obtaining the child nodes of the second level, and then the second level child nodes whose number of event items is greater than the first quantity threshold continue to be divided. Until the division reaches the Kth level, the number of event items included in all child nodes included in the Kth level is less than or equal to the first quantity threshold.

[0149] In some embodiments, when performing feature extraction on the voxel values ​​of the child nodes included in the K levels, a neural network model may be used, such as a two-layer fully connected neural network. Specifically, the voxel values ​​of the child nodes included in the K levels may be arranged into a one-dimensional vector in the order of the child nodes in the three-dimensional space, and then a two-layer fully connected neural network may be used to perform feature extraction on the one-dimensional vector composed of the voxel values ​​of the child nodes included in the K levels to obtain the octree voxel feature.

[0150] In the present application examples, see Figure 4 As shown, an RGB image captured by an RGB camera is used, and then the 2D posture of the human body obtained by the second posture estimation model is used as a label in the online training sample set, and then the event stream data (or feature information of the event stream data) collected synchronously with the RGB image is used as an online training sample, and input into the first posture estimation model to obtain the posture prediction value of the sample. Then, the loss value of the first posture estimation model is determined according to the posture prediction value of the sample and the label corresponding to the sample. Then, the network parameters of the first posture estimation model are adjusted according to the loss value. For example, taking the first sample in the training sample set as an example, when the network parameters of the first posture estimation model are updated using the training sample set, the first sample can be used to input the first posture estimation model to obtain the posture prediction value of the first sample output by the first posture estimation model; the loss value of the first posture estimation model is determined according to the posture prediction value of the first sample and the label corresponding to the first sample; and the network parameters of the first posture estimation model are adjusted according to the loss value. In some scenarios, the training sample set can be divided into mini-batches, and then the back propagation and gradient descent algorithms are used to update the network parameters (weights) of the first posture estimation model. Batch refers to a batch of samples in the training sample set. In this scenario, the weights can be updated once for each mini-batch. In some other scenarios, the weights may be updated once for each sample using a gradient descent algorithm. In some other scenarios, the weights may be updated once for a training sample set and the training sample set may be used for multiple iterations.

[0151] The embodiment of the present application adopts the above-mentioned method to construct multi-level spatial occupancy features of event stream data: when constructing online training samples, the event data is expressed as high-level spatial occupancy situations at multiple different levels, and then the feature is used together with the event point cloud to train the first posture estimation model of the event camera, thereby improving the 2D posture accuracy.

[0152] The second implementation is described as follows. Event cameras have the advantages of low power consumption, low latency and high dynamic range, but event cameras can only output events when the brightness changes in the field of view. When performing posture estimation tasks, if the human body is stationary or the movement amplitude is small, the number of events output by the event camera per unit time will be significantly reduced, making it difficult to provide sufficient information for the posture estimation model, resulting in a decrease in the accuracy of posture estimation. When only RGB images are used, if the human body moves too fast or the light on the scene is dim, the imaging quality of the RGB camera will be significantly reduced, which will also affect the accuracy of posture estimation. Therefore, in the second implementation of the present application, the image processing device can determine the working mode according to the needs (such as user needs or scene needs, etc.), so as to determine whether to use the event data stream collected by the event camera for human body posture estimation, or to use the RGB image collected by the RGB camera for human body posture estimation, or to use the event data stream collected by the event camera for human body posture estimation The average value of the result and the result of the RGB image collected by the RGB camera for human body posture estimation are used as the final result.

[0153] In some embodiments, the user demand may be to determine the working mode in response to the user's operation. In other embodiments, the scene demand may be, for example, at least one of the status information or environmental information of the current electronic device. Exemplarily, the status information of the electronic device may include at least one of the power supply status, remaining power, required frame rate, etc. of the electronic device. Environmental information may include information related to the light intensity or moving objects within the shooting range of the RGB camera and the event camera. For example, the environmental information may include changes in the light intensity within the shooting range of the RGB sensor or the DVS sensor, or the movement of the object within the shooting range, such as the movement speed of the object. The movement speed can be determined by the number of event items collected by the event camera per unit time. The faster the movement speed, the more event items are collected by the event camera per unit time, and vice versa.

[0154] See also Figure 5 As shown, it is a schematic diagram of the human posture estimation method flow provided by the second implementation mode in the embodiment of the present application. The method flow can be executed by the above-mentioned image processing device, which can be a chip or a chip system, or a server or an electronic device. The image processing device is coupled with an RGB camera and an event camera.

[0155] 501, determine the working mode of the image processing device. The working mode may include a first working mode, a second working mode and a third working mode.

[0156] In some embodiments, in the first working mode, only the event camera is in the enabled state. In the second working mode, both the RGB camera and the event camera are in the enabled state. In the third working mode, only the RGB camera is in the enabled state.

[0157] In some other embodiments, in which working mode, both the event camera and the RGB camera are turned on, but in the first working mode, only the event stream data collected by the event camera is used to estimate the human posture. In the third working mode, only the RGB image collected by the RGB camera is used to estimate the human posture.

[0158] 502 , when the image processing device is in the first working mode, the first event stream data collected by the event camera is used to estimate the human body posture to obtain the 2D posture of the human body in the collection time period of the first event stream data.

[0159] 503, when the image processing device is in the second working mode, the first event stream data collected by the event camera is used to estimate the human body posture to obtain a first estimation result, and the first RGB image collected by the RGB camera is used to estimate the human body posture to obtain a second estimation result, and the average value of the first estimation result and the second estimation result is used as the human body 2D posture in the collection time period of the first event stream data; wherein the exposure time of the first RGB image is the same as the collection time period of the first event stream data. It should be noted that the average value here can be the average value obtained by simple averaging, the average value obtained by weighted averaging, or the average value obtained by geometric averaging, etc.

[0160] 504. When the image processing device is in the third working mode, the first RGB image acquired by the RGB camera is used to estimate the human body posture to obtain the 2D human body posture of the exposure time of the first RGB image.

[0161] In a possible example, in the first working mode, the image processing device satisfies any one of the following 1-1 to 1-9.

[0162] 1-1, the image processing device is in a battery-powered state. In some application scenarios, the image processing device is integrated with the RGB camera and the event camera in an electronic device, and the image processing device is in a battery-powered state, or the electronic device is in a battery-powered state. In other application scenarios, the image processing device is independently deployed in an electronic device, and the image processing device is in a battery-powered state, that is, the electronic device is in a battery-powered state.

[0163] 1-2, the image processing device is in a battery-powered state and the remaining battery power is less than a power threshold.

[0164] 1-3, the frame rate required by the image processing device is greater than the frame rate threshold. For example, in some application scenarios, the image processing device is an AR / VR device, or the image processing device is deployed in an AR / VR device, and the frame rate required by the AR / VR device is generally higher (i.e., a high frame rate is required).

[0165] 1-4, the number of event items collected by the event camera in a unit time is greater than the second number threshold. In this case, the current movement speed of the target object (such as a person) is relatively fast.

[0166] 1-5, the light intensity detected by the image processing device is less than the intensity threshold. In this case, it can be understood that the current ambient light is dim. In some application scenarios, a light intensity sensor can also be deployed in the image processing device or in the electronic device where the image processing device is located, and data is obtained through the light intensity sensor and transmitted to the image processing device, so that the image processing device determines the comparison result between the light intensity and the intensity threshold.

[0167] 1-6, the frame rate required by the image processing device is less than or equal to the frame rate threshold and the number of event items included in the event stream data collected by the event camera per unit time is greater than the second number threshold.

[0168] 1-7, the image processing device is in a power supply state and the number of event items included in the event stream data collected by the event camera per unit time is greater than a second quantity threshold. The power supply state can be understood as using a three-phase power supply or a two-phase power supply for power supply.

[0169] 1-8, the image processing device is in a power supply state and the number of event items included in the event stream data collected by the event camera within a unit time is greater than a second number threshold.

[0170] 1-9 The image processing device is in a power supply state and the number of event items included in the event stream data collected by the event camera per unit time is greater than a second quantity threshold, and the light intensity detected by the image processing device is less than an intensity threshold.

[0171] In another possible example, in the second working mode, the image processing device satisfies any one of the following 2-1 to 2-6:

[0172] 2-1. The number of event items included in the event stream data collected by the event camera within a unit time is less than or equal to a second number threshold and the light intensity detected by the image processing device is less than an intensity threshold.

[0173] 2-2. The number of event items included in the event stream data collected by the event camera within a unit time is greater than a second number threshold and the light intensity detected by the image processing device is greater than an intensity threshold.

[0174] 2-3, the frame rate required by the image processing device is less than or equal to the frame rate threshold and the number of event items included in the event stream data collected by the event camera per unit time is less than or equal to the second number threshold, and the light intensity detected by the image processing device is less than the intensity threshold.

[0175] 2-4, the frame rate required by the image processing device is less than or equal to the frame rate threshold and the number of event items included in the event stream data collected by the event camera per unit time is greater than the second number threshold, and the light intensity detected by the image processing device is greater than the intensity threshold.

[0176] 2-5, the image processing device is in a power-supply state and the number of event items included in the event stream data collected by the event camera per unit time is less than or equal to a second quantity threshold, and the light intensity detected by the image processing device is less than an intensity threshold.

[0177] 2-6, the image processing device is in a power supply state and the number of event items included in the event stream data collected by the event camera per unit time is greater than a second quantity threshold, and the light intensity detected by the image processing device is greater than an intensity threshold.

[0178] In another possible example, in the third working mode, the image processing device satisfies the following 3-1 or 3-2.

[0179] 3-1, the image processing device is in a power supply state and the number of event items included in the event stream data collected by the event camera per unit time is less than or equal to a second quantity threshold, and the light intensity detected by the image processing device is greater than the intensity threshold.

[0180] 3-2, the frame rate required by the image processing device is less than or equal to the frame rate threshold and the number of event items included in the event stream data collected by the event camera per unit time is less than or equal to the second number threshold, and the light intensity detected by the image processing device is greater than the intensity threshold.

[0181] The embodiment of the present application adopts a fusion posture estimation method of an RGB camera and an event camera. In the inference stage, based on power consumption, frame rate, motion speed, and lighting conditions, a suitable processing method is selected or the data collected by a suitable camera is selected to obtain the final 2D posture result, which can improve the accuracy of 2D posture estimation.

[0182] Taking the aforementioned electronic device as a mobile phone as an example, the default configuration is that the event camera (i.e., DVS sensor) is turned on. When using the camera, the camera is called according to the current frame rate requirement. For example, if a high frame rate is required, the event camera can be called to obtain a 2D posture estimation result through the event stream data collected by the event camera and the first posture estimation model.

[0183] Optionally, you can also select a working mode based on the current status of the device. For example, when the current battery power is below a certain threshold, the user may activate the power saving mode and cannot take photos normally. You can only turn on the event camera. Although the photos taken by the event camera are not clear, the power consumption is low, and high-definition imaging is not required when estimating the 2D posture of the human body.

[0184] Optionally, the device can sense the surrounding environment to decide whether to switch the working mode. For example, in night scenes, or when the current device is in high-speed motion, the event camera can be turned on. If it is a static scene, the event camera may not be turned on.

[0185] Based on the above application types, environmental information, and device status, the working mode is determined. During operation, a decision can be made whether to trigger a work switch, thereby starting different cameras in different scenarios, with strong adaptability.

[0186] In some embodiments, the working modes include three types: only turning on the RGB camera, only turning on the event camera, and turning on both the RGB and event cameras. In addition, for different products, the reference factors for the detection application type and the environmental detection may be different.

[0187] For example, a camera in a security scenario has a motion detection function. The camera stores the video only when a moving object is detected, thereby reducing storage space and extending the hard disk storage time. Specifically, when the event camera and RGB camera are used in home or security cameras, only the event camera is enabled by default for human posture estimation.

[0188] For another example, when event cameras and RGB cameras are used in vehicle-mounted assistance / autonomous driving, during driving, when in night scenes or when entering tunnels, the RGB camera may not be able to capture effective scene information. At this time, although the event camera cannot obtain texture information, it can detect the motion posture of people within the shooting range, which is of great auxiliary value for the driver's judgment. If the target object moves quickly, only the event camera can be used to estimate the 2D posture of the human body. If the target object moves slowly, the event camera and RGB camera can be used together to estimate the 2D posture of the human body.

[0189] For example, when applied to AR / VR glasses, the frame rate requirement is high, and event cameras can be used to estimate human posture.

[0190] The following takes an electronic device as an example, in which the electronic device includes an image processing device, an RGB camera, and an event camera, to illustrate the joint attitude estimation solution of the RGB camera and the event camera provided in this application. Figure 6 shown.

[0191] 601, the electronic device determines whether there is a low power consumption requirement or a high frame rate requirement. If so, execute 602, if not, execute 603.

[0192] 602, the electronic device operates in a first operating mode and performs human posture estimation through event stream data collected by an event camera.

[0193] 603 , the electronic device counts whether the number of event items collected by the event camera in a unit time is greater than a second number threshold; if so, execute 604 , if not, execute 605 .

[0194] 604 , the electronic device determines whether the current light intensity is less than the intensity threshold, if so, executes 602 , if not, executes 606 .

[0195] 605 , the electronic device determines whether the current light intensity is less than the intensity threshold, if so, executes 606 , if not, executes 607 .

[0196] 606, the electronic device operates in the second operating mode, and performs human posture estimation based on the event stream data collected by the event camera and the RGB image collected by the RGB camera. For example, the average value of the human posture estimation result based on the event stream data collected by the event camera and the human posture estimation result based on the RGB camera collected by the RGB camera is used as the final human 2D posture.

[0197] Here, the movement speed is determined by the number of event items collected per unit time. When the number of event items is lower than the second threshold, the credibility of the event camera pose estimation decreases. In the case of good lighting conditions, the RGB image collected by the RGB camera is used to estimate the human pose. If the lighting conditions are poor, the credibility of the event camera and the RGB camera is low, so the human pose estimation is performed by fusion.

[0198] 607, the electronic device operates in a third operating mode and performs human posture estimation through the RGB image collected by the RGB camera.

[0199] In some possible implementations, the first possible implementation and the second possible implementation can be implemented in combination. Figure 7As shown, the electronic device can determine the working mode according to the current environment or device status (for example, frame rate / power consumption requirements, motion speed, lighting), and support the update of the first posture estimation model. For example, the update of the first posture estimation model can be triggered at intervals of a certain length of time. Between triggering updates, the training sample set is counted and obtained. It should be noted that the acquired training sample set is collected when both the RGB camera and the event camera are turned on. The input of the posture estimation model of the event camera uses event point cloud features and / or octree voxel features to improve the accuracy of posture estimation.

[0200] Combine the following Figure 2 The corresponding embodiment describes a solution that combines the first possible implementation method with the second possible implementation method. Figure 8 As shown, it is a schematic diagram of another method for estimating human posture provided by an embodiment of the present application. Take the electronic device including an image processing device, an RGB camera and an event camera as an example. Figure 8 In the embodiment, before the electronic device uses the updated first posture estimation model to perform human posture estimation on the third event stream data collected by the event camera to obtain the second estimation result, it executes 801. 801, determines the working mode of the electronic device (or image processing device), and executes 203 when it is determined to be in the first working mode. If it is determined to be in the second working mode, execute 203+802. 802, uses the second posture estimation model to perform human posture estimation on the second RGB image to obtain a third estimation result; the average value of the second estimation result and the third estimation result is used as the 2D posture of the human body in the collection time period of the third event stream data. After determining the third working mode, execute 803, uses the second posture estimation model to perform human posture estimation on the third RGB image collected by the RGB camera to obtain the 2D posture of the human body during the exposure time of the third RGB image.

[0201] In the above Figure 2 , Figure 5 and Figure 8 On the basis of the corresponding embodiments, in order to better implement the above solutions of the embodiments of the present application, the following also provides related devices or equipment for implementing the above solutions. Fig. 9 An image processing device is provided in an embodiment of the present application. The image processing device is coupled with a red, green, and blue (RGB) camera and an event camera. The image processing device includes:

[0202] An acquisition module 910 is used to acquire a first event stream captured by the event camera;

[0203] A processing module 920 is used to use a first posture estimation model to perform human posture estimation on first event stream data collected by the event camera to obtain a first estimation result, where the collection time of the first event stream data is the same as the exposure time of an RGB image collected by a frame of the RGB camera;

[0204] A training module 930 is used to update the network parameters of the first posture estimation model using a training sample set; the training sample set includes multiple samples and labels corresponding to each of the multiple samples; the first sample includes feature information of the second event stream data collected by the event camera within the exposure time of the first RGB image; the first sample is any sample of the multiple samples; the first RGB image is a frame of image collected by the RGB camera within a historical set time length; the label corresponding to the first sample is a 2D human posture obtained by using the second posture estimation model to estimate the human posture of the first RGB image;

[0205] The acquisition module 910 is further used to acquire a third event stream collected by the event camera;

[0206] The processing module 920 is further configured to use the updated first posture estimation model to perform human posture estimation on the third event stream data collected by the event camera to obtain a second estimation result.

[0207] In a possible implementation, the processing module 920 is specifically configured to:

[0208] Performing feature extraction on the third event stream data to obtain feature information of the third event stream data, wherein the feature information of the third event stream data includes M event point cloud features and / or octree voxel features, where M is a positive integer;

[0209] The M event point cloud features are located in n time periods, and the n time periods are located within the exposure time of the first RGB image; the first event point cloud feature among the M event point cloud features is used to characterize the pixel position of an event item generated in the first time period and the number of event items generated by the pixel position in the first time period; the first time period is any time period among the n time periods;

[0210] The octree voxel feature is obtained by performing feature extraction after voxelizing a plurality of event items included in the third event stream data using an octree;

[0211] The feature information is input into the updated first posture estimation model to obtain the second estimation result.

[0212] In a possible implementation, the processing module 920 is specifically configured to generate the octree voxel features included in the feature information of the third event stream data in the following manner:

[0213] Using the three-dimensional space represented by the event items included in the third event stream data as the nodes of the first level of the octree structure;

[0214] For each child node included in the first level and the i-th level in the octree structure, execute:

[0215] If the number of event items included in the child node is less than or equal to the first quantity threshold, the voxel value of the child node is set to 0; if the number of event items included in the child node is greater than the first quantity threshold, the voxel value of the child node is set to 1, and the child node is further divided into eight equal parts to obtain eight child nodes of the i+1th level; until the number of event items included in all child nodes included in the last level is less than or equal to the first quantity threshold, so as to obtain the voxel values ​​of the child nodes respectively included in K levels, where i is an integer greater than 1 and less than K, and K is an integer greater than 1;

[0216] Feature extraction is performed on the voxel values ​​of the child nodes included in the K levels to obtain the octree voxel features.

[0217] In a possible implementation, the processing module, when extracting features from voxel values ​​of child nodes included in the K levels to obtain the octree voxel features, is specifically used to:

[0218] Performing feature extraction on a one-dimensional vector composed of voxel values ​​of the child nodes included in the K levels through a two-layer fully connected neural network to obtain the octree voxel feature;

[0219] The voxel values ​​of the sub-nodes included in the K levels in the one-dimensional vector are arranged in the order of the sub-nodes in the three-dimensional space.

[0220] In a possible implementation, the training module 930 is specifically configured to:

[0221] Using the first sample to input the first posture estimation model, obtaining a posture prediction value of the first sample output by the first posture estimation model;

[0222] Determining a loss value of the first posture estimation model according to the posture prediction value of the first sample and the label corresponding to the first sample;

[0223] The network parameters of the first posture estimation model are adjusted according to the loss value.

[0224] In a possible implementation, the method further includes:

[0225] A detection module 940 is used to determine that the image processing device is in the first working mode before using the updated first posture estimation model to perform human posture estimation on the third event stream data collected by the event camera to obtain a second estimation result;

[0226] Wherein, in the first working mode, the event camera is in an on state, and the RGB camera is in an off state;

[0227] The second estimation result is used as the 2D posture of the human body in the collection time period of the third event stream data.

[0228] In a possible implementation, the method further includes:

[0229] A detection module 940 is used to determine, before obtaining a second estimation result, whether the image processing device is in a second working mode; wherein, in the second working mode, the event camera is in an on state, and the RGB camera is in an on state;

[0230] The processing module 920 is also used to use the second posture estimation model to perform human posture estimation on the second RGB image to obtain a third estimation result; the exposure time of the second RGB image is the collection time period of the event camera collecting the third event stream data; after obtaining the second estimation result, the average value of the second estimation result and the third estimation result is used as the 2D posture of the human body in the collection time period of the third event stream data.

[0231] In a possible implementation, the detection module 940 is further configured to determine that the image processing device is in a third operating mode, wherein in the third operating mode, the RGB camera is in an on state and the event camera is in an off state;

[0232] The processing module 920 is further configured to use the second posture estimation model to perform human posture estimation on the third RGB image captured by the RGB camera to obtain a 2D human posture of the exposure time of the third RGB image.

[0233] See also Fig.10 As shown, another image processing device provided in an embodiment of the present application is coupled with an RGB camera and an event camera, and includes:

[0234] A detection module 1010, used to determine the working mode of the image processing device;

[0235] The processing module 1020 is used to:

[0236] When the image processing device is in the first working mode, the first event stream data collected by the event camera is used to estimate the human body posture to obtain the 2D human body posture in the collection time period of the first event stream data; or

[0237] When the image processing device is in the second working mode, the first event stream data collected by the event camera is used to estimate the human body posture to obtain a first estimation result, and the first RGB image collected by the RGB camera is used to estimate the human body posture to obtain a second estimation result, and the average value of the first estimation result and the second estimation result is used as the human body 2D posture in the collection time period of the first event stream data; wherein the exposure time of the first RGB image is the same as the collection time period of the first event stream data; or,

[0238] When the image processing device is in the third working mode, the first RGB image captured by the RGB camera is used to perform human posture estimation to obtain a 2D human posture of the exposure time of the first RGB image.

[0239] See also Fig.11 , an embodiment of the present application provides a schematic diagram of the structure of an electronic device, including a processor 1110, a memory 1120, an RGB camera 1130, and an event camera 1140. The processor 1110, the RGB camera 1130, and the motion camera 1140 are interconnected through lines.

[0240] The processor 1110 is coupled to the memory 1120, and the memory is used to store programs. Figure 2 , Figure 5 and Figure 8 The processor 1110 is used to execute the program in the memory so that the electronic device Figure 2 , Figure 5 and Figure 8 The method steps performed by the electronic device (or image processing device) in any embodiment.

[0241] The embodiment of the present application also provides an image processing device, which can also be called a digital processing chip or chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface. The program instructions are executed by the processing unit. The processing unit is used to execute the aforementioned Figure 2 , Figure 5 or Figure 8 The method steps are performed by the electronic device (or image processing device) shown in any one of the embodiments.

[0242] The embodiment of the present application also provides a digital processing chip. The digital processing chip integrates a circuit and one or more interfaces for implementing the above-mentioned processor 1110 or the function of the processor 1110. When the digital processing chip integrates a memory, the digital processing chip can complete the method steps of any one or more embodiments in the above-mentioned embodiments. When the digital processing chip does not integrate a memory, it can be connected to an external memory through a communication interface. The digital processing chip implements the actions performed by the electronic device in the above-mentioned embodiment according to the program code stored in the external memory.

[0243] The image processing device provided in the embodiment of the present application may be a chip, and the chip includes: a processing unit and a communication unit, the processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin or a circuit. The processing unit may execute the computer execution instructions stored in the storage unit, so that the chip in the server executes the above Figure 2 , Figure 5 or Figure 8 The human posture estimation method described in the illustrated embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc., and the storage unit can also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0244] Specifically, the aforementioned processing unit or processor may be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0245] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0246] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0247] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0248] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)), etc.

[0249] Finally, it should be noted that the above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for estimating a human body posture, characterized in that: Applied to an image processing device, the image processing device is coupled with a red, green, and blue (RGB) camera and an event camera, comprising: Using a first posture estimation model to perform human posture estimation on first event stream data collected by the event camera to obtain a first estimation result, wherein the collection time of the first event stream data is the same as the exposure time of an RGB image collected by a frame of the RGB camera; The network parameters of the first posture estimation model are updated using a training sample set; the training sample set includes multiple samples and labels corresponding to each of the multiple samples; the first sample includes feature information of the second event stream data collected by the event camera within the exposure time of the first RGB image; the first sample is any sample of the multiple samples; the first RGB image is a frame of image collected by the RGB camera within a historical set time length; the label corresponding to the first sample is a 2D human posture obtained by using the second posture estimation model to estimate the human posture of the first RGB image; The updated first posture estimation model is used to perform human posture estimation on the third event stream data collected by the event camera to obtain a second estimation result.

2. The method according to claim 1, characterized in that Using the updated first posture estimation model to perform human posture estimation on the third event stream data collected by the event camera to obtain a second estimation result, including: Performing feature extraction on the third event stream data to obtain feature information of the third event stream data, wherein the feature information of the third event stream data includes M event point cloud features and / or octree voxel features, where M is a positive integer; The M event point cloud features are located in n time periods, and the n time periods are located within the exposure time of the first RGB image; the first event point cloud feature among the M event point cloud features is used to characterize the pixel position of an event item generated in the first time period and the number of event items generated by the pixel position in the first time period; the first time period is any time period among the n time periods; The octree voxel feature is obtained by performing feature extraction after voxelizing a plurality of event items included in the third event stream data using an octree; The feature information is input into the updated first posture estimation model to obtain the second estimation result.

3. The method according to claim 2, characterized in that The octree voxel features included in the feature information of the third event stream data are generated in the following manner: Using the three-dimensional space represented by the event items included in the third event stream data as the nodes of the first level of the octree structure; For each child node included in the first level and the i-th level in the octree structure, execute: If the number of event items included in the child node is less than or equal to the first quantity threshold, the voxel value of the child node is set to 0; if the number of event items included in the child node is greater than the first quantity threshold, the voxel value of the child node is set to 1, and the child node is further divided into eight equal parts to obtain eight child nodes of the i+1th level; until the number of event items included in all child nodes included in the last level is less than or equal to the first quantity threshold, so as to obtain the voxel values ​​of the child nodes respectively included in K levels, where i is an integer greater than 1 and less than K, and K is an integer greater than 1; Feature extraction is performed on the voxel values ​​of the child nodes included in the K levels to obtain the octree voxel features.

4. The method according to claim 3, characterized in that Feature extraction is performed on the voxel values ​​of the child nodes included in the K levels to obtain the octree voxel features, including: Performing feature extraction on a one-dimensional vector composed of voxel values ​​of the child nodes included in the K levels through a two-layer fully connected neural network to obtain the octree voxel feature; The voxel values ​​of the sub-nodes included in the K levels in the one-dimensional vector are arranged in the order of the sub-nodes in the three-dimensional space.

5. The method according to any one of claims 1 to 4, characterized in that: Updating the network parameters of the first posture estimation model using the training sample set includes: Using the first sample to input the first posture estimation model, obtaining a posture prediction value of the first sample output by the first posture estimation model; Determining a loss value of the first posture estimation model according to the posture prediction value of the first sample and the label corresponding to the first sample; The network parameters of the first posture estimation model are adjusted according to the loss value.

6. The method according to any one of claims 1 to 5, characterized in that: Before using the updated first posture estimation model to perform human posture estimation on the third event stream data collected by the event camera to obtain a second estimation result, the method further includes: Determining that the image processing device is in a first working mode; Wherein, in the first working mode, the event camera is in an on state, and the RGB camera is in an off state; The second estimation result is used as the 2D posture of the human body in the collection time period of the third event stream data.

7. The method according to claim 6, characterized in that In the first working mode, the image processing device satisfies any of the following conditions: The image processing device is in a battery-powered state; or, The image processing device is in a battery-powered state and the remaining power of the battery is less than a power threshold; or; The frame rate required by the image processing device is greater than a frame rate threshold; or, The number of event items included in the event stream data collected by the event camera within a unit time is greater than a second number threshold; or, The light intensity detected by the image processing device is less than an intensity threshold; or; The frame rate required by the image processing device is less than or equal to the frame rate threshold and the number of event items collected by the event camera per unit time is greater than a second number threshold; or, The image processing device is in a power supply state and the number of event items collected by the event camera per unit time is greater than the second number threshold; or, The image processing device is in a power supply state and the number of event items collected by the event camera per unit time is greater than the second number threshold; or, The image processing device is in a power supply state, the number of event items collected by the event camera per unit time is greater than the second number threshold, and the light intensity detected by the image processing device is less than the intensity threshold.

8. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Before obtaining the second estimation result, determining that the image processing device is in a second working mode; wherein, in the second working mode, the event camera is in an on state, and the RGB camera is in an on state; Use the second posture estimation model to perform human posture estimation on the second RGB image to obtain a third estimation result; the exposure time of the second RGB image is the collection time period of the event camera collecting the third event stream data; After obtaining the second estimation result, an average value of the second estimation result and the third estimation result is used as the 2D posture of the human body in the collection time period of the third event stream data.

9. The method according to claim 8, characterized in that In the second working mode, the image processing device satisfies any of the following conditions: The number of event items included in the event stream data collected by the event camera within a unit time is less than or equal to the second number threshold and the light intensity detected by the image processing device is less than the intensity threshold; or, The number of event items collected by the event camera within a unit time is greater than the second number threshold and the light intensity detected by the image processing device is greater than the intensity threshold; or, The frame rate required by the image processing device is less than or equal to the frame rate threshold and the number of event items collected by the event camera per unit time is less than or equal to the second number threshold, and the light intensity detected by the image processing device is less than the intensity threshold; or, The frame rate required by the image processing device is less than or equal to the frame rate threshold and the number of event items collected by the event camera per unit time is greater than the second number threshold, and the light intensity detected by the image processing device is greater than the intensity threshold; or, The image processing device is in a power supply state and the number of event items collected by the event camera per unit time is less than or equal to the second number threshold, and the light intensity detected by the image processing device is less than the intensity threshold; or, The image processing device is in a power supply state, the number of event items collected by the event camera per unit time is greater than the second number threshold, and the light intensity detected by the image processing device is greater than the intensity threshold.

10. The method according to any one of claims 6 to 9, characterized in that: The method further comprises: Determining that the image processing device is in a third working mode, wherein in the third working mode, the RGB camera is in an on state and the event camera is in an off state; The second posture estimation model is used to perform human posture estimation on the third RGB image captured by the RGB camera to obtain a 2D human posture of the exposure time of the third RGB image.

11. The method according to claim 10, characterized in that In the third working mode, the image processing device satisfies any of the following conditions: The image processing device is in a power supply state and the number of event items included in the event stream data collected by the event camera within a unit time is less than or equal to the second number threshold, and the light intensity detected by the image processing device is greater than the intensity threshold; The frame rate required by the image processing device is less than or equal to the frame rate threshold, the number of event items included in the event stream data collected by the event camera per unit time is less than or equal to the second number threshold, and the light intensity detected by the image processing device is greater than the intensity threshold.

12. An image processing device, characterized in that: The image processing device is coupled with a red, green, and blue (RGB) camera and an event camera, and the image processing device comprises: An acquisition module, used for acquiring a first event stream captured by the event camera; A processing module, configured to use a first posture estimation model to perform human posture estimation on first event stream data collected by the event camera to obtain a first estimation result, wherein the collection time of the first event stream data is the same as the exposure time of an RGB image collected by a frame of the RGB camera; A training module, used to update the network parameters of the first posture estimation model using a training sample set; the training sample set includes multiple samples and labels corresponding to each of the multiple samples; the first sample includes feature information of second event stream data collected by the event camera within the exposure time of the first RGB image; the first sample is any sample of the multiple samples; the first RGB image is a frame of image collected by the RGB camera within a historical set time length; the label corresponding to the first sample is a 2D human posture obtained by using the second posture estimation model to estimate the human posture of the first RGB image; The acquisition module is further used to acquire a third event stream collected by the event camera; The processing module is further used to use the updated first posture estimation model to perform human posture estimation on the third event stream data collected by the event camera to obtain a second estimation result.

13. The device according to claim 12, characterized in that The processing module is specifically used for: Performing feature extraction on the third event stream data to obtain feature information of the third event stream data, wherein the feature information of the third event stream data includes M event point cloud features and / or octree voxel features, where M is a positive integer; The M event point cloud features are located in n time periods, and the n time periods are located within the exposure time of the first RGB image; the first event point cloud feature among the M event point cloud features is used to characterize the pixel position of an event item generated in the first time period and the number of event items generated by the pixel position in the first time period; the first time period is any time period among the n time periods; The octree voxel feature is obtained by performing feature extraction after voxelizing a plurality of event items included in the third event stream data using an octree; The feature information is input into the updated first posture estimation model to obtain the second estimation result.

14. The device according to claim 13, characterized in that The processing module is specifically used to generate the octree voxel features included in the feature information of the third event stream data in the following manner: Using the three-dimensional space represented by the event items included in the third event stream data as the nodes of the first level of the octree structure; For each child node included in the first level and the i-th level in the octree structure, execute: If the number of event items included in the child node is less than or equal to the first quantity threshold, the voxel value of the child node is set to 0; if the number of event items included in the child node is greater than the first quantity threshold, the voxel value of the child node is set to 1, and the child node is further divided into eight equal parts to obtain eight child nodes of the i+1th level; until the number of event items included in all child nodes included in the last level is less than or equal to the first quantity threshold, so as to obtain the voxel values ​​of the child nodes respectively included in K levels, where i is an integer greater than 1 and less than K, and K is an integer greater than 1; Feature extraction is performed on the voxel values ​​of the child nodes included in the K levels to obtain the octree voxel features.

15. The device according to claim 14, characterized in that The processing module, when extracting features from the voxel values ​​of the child nodes included in the K levels to obtain the octree voxel features, is specifically used to: Performing feature extraction on a one-dimensional vector composed of voxel values ​​of the child nodes included in the K levels through a two-layer fully connected neural network to obtain the octree voxel feature; The voxel values ​​of the sub-nodes included in the K levels in the one-dimensional vector are arranged in the order of the sub-nodes in the three-dimensional space.

16. The device according to any one of claims 13 to 15, characterized in that: The training module is specifically used for: Using the first sample to input the first posture estimation model, obtaining a posture prediction value of the first sample output by the first posture estimation model; Determining a loss value of the first posture estimation model according to the posture prediction value of the first sample and the label corresponding to the first sample; The network parameters of the first posture estimation model are adjusted according to the loss value.

17. The device according to any one of claims 13 to 16, characterized in that: Also includes: a detection module, configured to determine that the image processing device is in the first working mode before using the updated first posture estimation model to perform human posture estimation on the third event stream data collected by the event camera to obtain a second estimation result; Wherein, in the first working mode, the event camera is in an on state, and the RGB camera is in an off state; The second estimation result is used as the 2D posture of the human body in the collection time period of the third event stream data.

18. The device according to claim 17, characterized in that In the first working mode, the image processing device satisfies any of the following conditions: The image processing device is in a battery-powered state; or, The image processing device is in a battery-powered state and the remaining power of the battery is less than a power threshold; or; The frame rate required by the image processing device is greater than a frame rate threshold; or, The number of event items included in the event stream data collected by the event camera within a unit time is greater than a second number threshold; or, The light intensity detected by the image processing device is less than an intensity threshold; or; The frame rate required by the image processing device is less than or equal to the frame rate threshold and the number of event items collected by the event camera per unit time is greater than a second number threshold; or, The image processing device is in a power supply state and the number of event items collected by the event camera per unit time is greater than the second number threshold; or, The image processing device is in a power supply state and the number of event items collected by the event camera per unit time is greater than the second number threshold; or, The image processing device is in a power supply state, the number of event items collected by the event camera per unit time is greater than the second number threshold, and the light intensity detected by the image processing device is less than the intensity threshold.

19. The device according to any one of claims 14 to 18, characterized in that: Also includes: A detection module, used for determining, before obtaining a second estimation result, that the image processing device is in a second working mode; wherein, in the second working mode, the event camera is in an on state, and the RGB camera is in an on state; The processing module is also used to use the second posture estimation model to perform human posture estimation on the second RGB image to obtain a third estimation result; the exposure time of the second RGB image is the collection time period of the event camera collecting the third event stream data; after obtaining the second estimation result, the average value of the second estimation result and the third estimation result is used as the 2D posture of the human body in the collection time period of the third event stream data.

20. The device according to claim 19, characterized in that In the second working mode, the image processing device satisfies any of the following conditions: The number of event items included in the event stream data collected by the event camera within a unit time is less than or equal to the second number threshold and the light intensity detected by the image processing device is less than the intensity threshold; or, The number of event items collected by the event camera within a unit time is greater than the second number threshold and the light intensity detected by the image processing device is greater than the intensity threshold; or, The frame rate required by the image processing device is less than or equal to the frame rate threshold and the number of event items collected by the event camera per unit time is less than or equal to the second number threshold, and the light intensity detected by the image processing device is less than the intensity threshold; or, The frame rate required by the image processing device is less than or equal to the frame rate threshold and the number of event items collected by the event camera per unit time is greater than the second number threshold, and the light intensity detected by the image processing device is greater than the intensity threshold; or, The image processing device is in a power supply state and the number of event items collected by the event camera per unit time is less than or equal to the second number threshold, and the light intensity detected by the image processing device is less than the intensity threshold; or, The image processing device is in a power supply state, the number of event items collected by the event camera per unit time is greater than the second number threshold, and the light intensity detected by the image processing device is greater than the intensity threshold.

21. The device according to any one of claims 17 to 20, characterized in that The detection module is further used to determine that the image processing device is in a third working mode, wherein in the third working mode, the RGB camera is in an on state and the event camera is in an off state; The processing module is further used to use the second posture estimation model to perform human posture estimation on the third RGB image collected by the RGB camera to obtain the 2D posture of the human body at the exposure time of the third RGB image.

22. The device according to claim 21, characterized in that In the third working mode, the image processing device satisfies any of the following conditions: The image processing device is in a power supply state and the number of event items included in the event stream data collected by the event camera within a unit time is less than or equal to the second number threshold, and the light intensity detected by the image processing device is greater than the intensity threshold; The frame rate required by the image processing device is less than or equal to the frame rate threshold, the number of event items included in the event stream data collected by the event camera per unit time is less than or equal to the second number threshold, and the light intensity detected by the image processing device is greater than the intensity threshold.

23. An image processing device, comprising a processor and a memory, wherein the processor is coupled to the memory, wherein: The memory is used to store programs; the processor is used to execute the programs in the memory, so that the image processing device executes the method as described in any one of claims 1-11.

24. An electronic device, characterized in that: It includes an image processing device, an event camera and a red, green and blue (RGB) camera; The event camera is used to collect event stream data; the RGB camera is used to collect RGB images; The image processing device is used to execute the method according to any one of claims 1 to 11 based on the event stream data and the RGB image.

25. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed on a computer, causes the computer to execute the method according to any one of claims 1 to 11.

26. A computer program product comprising instructions, characterized in that When the method is executed on a computer, the computer is enabled to execute the method according to any one of claims 1 to 11.

Images