Human body posture estimation model, model training method and human body posture estimation method

Abstract

The present application discloses a human body posture estimation model, a model training method and a human body posture estimation method. The training method includes inputting a previous frame image of a current frame image into a preset preceding feature model, determining a preceding feature map of the previous frame image through the preset preceding feature model; inputting the preceding feature map and the current frame image into a human body posture estimation model to be trained to obtain a predicted feature map; training the human body posture estimation model to be trained based on the predicted feature map and the true feature map to obtain a human body posture estimation model. This embodiment supervises human body posture estimation by using a preset preceding feature model obtained through training based on a single-frame image frame, and can utilize the data richness characteristics of a single-frame image frame. At the same time, the preceding feature maps of the current frame image and the previous frame image are used as input items of the human body posture estimation model, and the spatiotemporal information between adjacent image frames can be learned, thereby improving the model performance of the trained human body posture estimation model.

Description

Technical Field

[0001] The present application relates to the technical field of human body posture estimation, and in particular to a human body posture estimation model, a model training method and a human body posture estimation method. Background Art

[0002] Human posture estimation is an algorithm that predicts the position of key points of the human body based on pictures or videos. It can be widely used in human interaction, security, health, enhanced display and other fields.

[0003] In general, common human pose estimation models usually use monocular images as training images, and output heat map information of human key points based on monocular images, and then calculate the distance loss between them and the true value heat map information to perform model training. However, in the case of complex backgrounds (e.g., occlusion, etc.) and blur caused by rapid human movement, the accuracy of the human key points detected by the human pose model based on monocular image training is not high (e.g., objects displayed in the background are identified as human key points), which can easily cause false detection of key points and thus affect the stability of human pose estimation.

[0004] Furthermore, although some researchers currently use videos as training objects, use recurrent neural networks such as LSTM to determine the heat map information of human key points, and then calculate the distance loss between it and the true value heat map information to train the model to avoid the inaccuracy of human key points caused by occlusion and rapid movement of the human body. However, the recurrent neural network that uses videos as training objects consumes a lot of computing resources and running time, which limits the application of the trained human posture estimation model, for example, it cannot be applied to edge devices.

[0005] Therefore the prior art still needs to be improved and enhanced. Summary of the invention

[0006] The technical problem to be solved by the present application is to provide a human body posture estimation model, a model training method and a human body posture estimation method in view of the deficiencies in the prior art.

[0007] In order to solve the above technical problems, the first aspect of the embodiment of the present application provides a training method for a human posture estimation model, the training method comprising:

[0008] Obtaining a current frame image, a true feature map corresponding to the current frame image, and a previous feature map of a previous frame image in a training sample set, wherein the previous frame image and the current frame image are adjacent image frames, and the previous feature map is obtained based on a preset previous feature model, and the preset previous feature model is a model constructed based on a single frame image to optimize the human body posture estimation model;

[0009] Inputting the previous feature map and the current frame image into the human body posture estimation model to be trained to obtain a predicted feature map corresponding to the current frame image;

[0010] The human body posture estimation model to be trained is iteratively optimized according to the predicted feature map and the true feature map corresponding to each of the current frame images to obtain a human body posture estimation model.

[0011] A second aspect of an embodiment of the present application provides a human body posture estimation model, characterized in that it is obtained by training using the training method of the human body posture estimation model as described above, wherein the human body posture estimation model includes a second feature module and an information fusion module:

[0012] The second feature module is used to obtain a training feature map of a current frame image; the structure of the second feature module is the same as the model structure of the first feature module of the preset preceding feature model, and the preset preceding feature model is used to obtain a preceding feature map of a previous frame image;

[0013] The information fusion module is used to fuse the training feature map with the previous feature map to obtain a predicted feature map corresponding to the current frame image.

[0014] A third aspect of the present application provides a method for estimating a human body posture, which uses the human body posture estimation model described above; the method includes:

[0015] Obtaining a previous feature map of a current frame image and a previous frame image; wherein the current frame image and the previous frame image are adjacent image frames, and the previous feature map is obtained by inputting the previous frame image into the human body posture estimation model;

[0016] Fusing the current frame image with the previous feature map through the human body posture estimation model to obtain a target feature map of the current frame image;

[0017] The target human body posture corresponding to the current frame image is determined according to the target feature map.

[0018] A fourth aspect of an embodiment of the present application provides a training device for a human posture estimation model, comprising:

[0019] A first acquisition module is configured to acquire a current frame image, a true feature map corresponding to the current frame image, and a previous feature map of a previous frame image in a training sample set, wherein the previous frame image and the current frame image are adjacent image frames; wherein the previous feature map is obtained based on a preset previous feature model, and the preset previous feature model is a model constructed based on a single frame image to optimize the human body posture estimation model;

[0020] A second acquisition module inputs the previous feature map and the current frame image into the human body posture estimation model to be trained to obtain a predicted feature map corresponding to the current frame image;

[0021] A training module iteratively optimizes the human body posture estimation model to be trained according to the predicted feature map and the true feature map corresponding to each current frame image in the training sample set to obtain a human body posture estimation model; wherein the human body posture estimation model is a model obtained by training based on the adjacent image frames.

[0022] A fifth aspect of an embodiment of the present application provides a human body posture estimation device, which uses a human body posture estimation model trained by the training method of the human body posture estimation model, including:

[0023] A third acquisition module is configured to acquire a preceding feature map of a current frame image and a previous frame image; wherein the current frame image and the previous frame image are adjacent image frames, and the preceding feature map is obtained by inputting the previous frame image into the human body posture estimation model;

[0024] A fourth acquisition module, fusing the current frame image with the previous feature map through the human posture estimation model to acquire a target feature map of the current frame image;

[0025] The estimation module determines the target human body posture corresponding to the current frame image according to the target feature map.

[0026] The sixth aspect of an embodiment of the present application provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the training method of the human body posture estimation model as described above, and / or to implement the human body posture estimation method as described above.

[0027] A seventh aspect of the embodiments of the present application provides an electronic device, comprising: a processor and a memory; wherein:

[0028] The memory stores a computer-readable program executable by the processor;

[0029] When the processor executes the computer-readable program, it implements the steps in the training method of the human body posture estimation model as described above, and / or implements the human body posture estimation method as described above.

[0030] The beneficial effects of the present application are as follows: the present embodiment supervises human posture estimation by using a preceding feature model obtained through training based on a single-frame image frame, and can utilize the data richness characteristics of a single-frame image frame. At the same time, the preceding feature graphs of the current frame image and the previous frame image are used as input items of the human posture estimation model, and the temporal and spatial information between adjacent image frames can be learned, thereby improving the model performance of the trained human posture estimation model. In addition, the present embodiment directly uses the preceding feature graph as an input item, and there is no need to extract features from the preceding feature graph, thereby not increasing the amount of calculation of the human posture estimation model and the time spent on human posture estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without inventive work.

[0032] Figure 1 A flowchart of a method for training a human posture estimation model provided in this application.

[0033] Figure 2 A flow chart showing the principles of the training method for the human body posture estimation model provided in this application.

[0034] Figure 3 A structural principle diagram of the human body pose estimation model in the training method of the human body pose estimation model provided in this application.

[0035] Figure 4 A flowchart of the human body posture estimation method provided in this application.

[0036] Figure 5 This is a flow chart of the principles of the human body posture estimation method provided in this application.

[0037] Figure 6 A schematic diagram of the structure of a training device for a human posture estimation model provided in this embodiment.

[0038] Figure 7 This is a schematic diagram of the structure of the human body posture estimation device provided in this embodiment.

[0039] Figure 8 This is a schematic diagram of the structure of the electronic device provided in this application. DETAILED DESCRIPTION

[0040] The present application provides a human body posture estimation model, a model training method and a human body posture estimation method. In order to make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0041] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0042] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0043] It should be understood that the sequence numbers and sizes of the steps in this embodiment do not mean the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0044] The application content is further explained below through the description of embodiments in conjunction with the accompanying drawings.

[0045] This embodiment provides a training method for a human posture estimation model, which can be performed by a training device or electronic device for a human posture estimation model. The device or electronic device can be implemented by software and / or hardware, and the device or electronic device can be integrated in any intelligent device with network communication function.

[0046] like Figure 1 and Figure 2 As shown, the training method of the human body posture estimation model provided in this embodiment specifically includes:

[0047] S10, obtaining a current frame image, a true feature map corresponding to the current frame image, and a previous feature map of a previous frame image in a training sample set, wherein the previous frame image and the current frame image are adjacent image frames.

[0048] In one implementation, a training sample set may include several training image groups, each of the several training image groups includes a current frame image, a preceding feature map of a previous frame image, and a true feature map of the current frame image, the previous frame image and the current frame image are adjacent image frames in a video frame sequence, that is, the previous frame image and the current frame image are included in the same video frame sequence, and according to the image frame acquisition order, the previous frame image is located before the current frame image, and the previous frame image is adjacent to the current frame image.

[0049] In one implementation, a method for acquiring a preceding feature map of a previous frame image includes: inputting a previous frame image of a current frame image into a preset preceding feature model, and acquiring the preceding feature map of the previous frame image through the preset preceding feature model.

[0050] More specifically, the preset pre-order feature model may be a trained network model or a network model to be trained, the input item of the preset pre-order feature model is the previous frame image, and the output item is the pre-order feature map of the previous frame image, wherein the preset pre-order feature model is obtained by training based on a single frame image. Specifically, if the preset pre-order feature model is a trained network model, it uses a single frame image as a training input item to obtain the thermal map information of the key points of the human body, and then calculates the loss item (for example, distance loss, etc.) with the true value thermal map information and performs iterative optimization to obtain the best model.

[0051] It should be understood that the preset preceding feature model is used as a tool network in this embodiment, which is independently set from the human body posture estimation model and is used to optimize the parameters of the human body posture estimation model according to the preceding feature map.

[0052] In one implementation, if the preset preceding feature model is a network model to be trained, before step S10, the following is further included:

[0053] Obtaining a training sample set corresponding to a preset pre-order feature model, wherein the training sample set includes a plurality of single-frame image frames;

[0054] The pre-order feature model to be trained is trained based on the training sample set to obtain a preset pre-order feature model.

[0055] Specifically, the model structure of the pre-order feature model to be trained is the same as the model structure of the preset pre-order feature model, and the difference between the two is that the model parameters of the pre-order feature model to be trained are initial model parameters, and the model parameters of the preset pre-order feature model are model parameters obtained through training based on a training sample set. The training sample set includes a number of single-frame image frames, each of which carries a labeled feature map, and the labeled feature map is used as a supervisory label to train the pre-order feature model to obtain the best preset pre-order feature model. This implementation method uses a training sample set including single-frame image frames to train the pre-order feature model to be trained, so that the training process of the preset pre-order feature model takes advantage of the large number of training sample sets of single-frame image frames and the rich scenes, thereby improving the model accuracy of the trained preset pre-order feature model.

[0056] In one implementation, the plurality of single-frame image frames may be monocular images acquired by a monocular camera, or image frames randomly selected from a video frame sequence, or some of the single-frame image frames may be monocular images acquired by a monocular camera, and some of the single-frame image frames may be image frames randomly selected from a video frame sequence, etc. Of course, when the training sample set includes image frames randomly selected from a video frame sequence, the image frames randomly selected from a video frame sequence included in the training sample set may be image frames selected from one video frame sequence, or may be image frames selected from multiple video frame sequences.

[0057] It should be noted that when the current frame image is the first frame image of a video frame sequence, the current frame image is used as the previous frame image of the current frame image, that is, if the current frame image is the first frame image of a video frame sequence, the current frame image is input into a preset preceding feature model, and the feature map of the current frame image is determined by the preset preceding feature model, and the feature map of the current frame image is used as the preceding feature map corresponding to the current frame image.

[0058] S20, inputting the previous feature map and the current frame image into the human body posture estimation model to be trained to obtain a predicted feature map corresponding to the current frame image.

[0059] Specifically, the human body posture estimation model to be trained is a pre-built neural network model, and the human body posture estimation model to be trained is configured with initial model parameters. The initial model parameters of the human body posture estimation model to be trained are iteratively optimized to obtain optimal parameters.

[0060] In one implementation, the preceding feature map and the current frame image are input into the human body posture estimation model to be trained, and the predicted feature map corresponding to the current frame image is obtained, which specifically includes:

[0061] Extract features of the current frame image through the human posture estimation model to be trained, and obtain a training feature map corresponding to the current frame image;

[0062] The training feature map is fused with the previous feature map to obtain the predicted feature map corresponding to the current frame image.

[0063] Specifically, the image size of the preceding feature map is the same as the image size of the training feature map, for example, the image size of the preceding feature map is 224*224, and the image size of the training feature map is 224*244, wherein the training feature map is obtained by extracting features from the current frame image by the human body posture estimation model to be trained. In other words, the human body posture estimation model to be trained only extracts features from the current frame image to obtain the training feature map, and fuses the obtained training feature map with the preceding feature map, so that the preceding feature map is only involved in the calculation when the preceding feature map is fused with the training feature map, thereby reducing the amount of calculation caused by adding the preceding feature map, and at the same time, the spatiotemporal information between the previous frame image and the current frame image can be learned, thereby improving the model accuracy of the trained human body posture estimation model without increasing the amount of calculation.

[0064] In one implementation, the preset pre-order feature model includes a first feature module, and the human body posture estimation model to be trained includes a second feature module, wherein the model structure of the first feature module is the same as the model structure of the second feature module. That is, the preset pre-order feature model and the human body posture estimation model to be trained include the same feature module, so that the pre-order feature map extracted by the preset pre-order feature model and the training feature map extracted based on the human body posture estimation model to be trained have the same image scale, that is, the image size of the pre-order feature map is the same as the image size of the training feature map, and the number of channels of the pre-order feature map is the same as the number of channels of the training feature map. For example, Figure 3 As shown, both the first feature module and the second feature module include several cascaded residual units, a first convolution unit and a second convolution unit, wherein the first convolution unit and the second convolution unit are in parallel and are both connected to the last residual unit in the several cascaded residual units; the first convolution unit and the second convolution unit have the same model structure, both include 3*3 convolution blocks, 3*3 convolution blocks, 1*1 convolution blocks and 1*1 convolution blocks cascaded in sequence, the convolution block includes a convolution layer, a normalization layer and an activation layer cascaded in sequence, the n*n before the convolution block indicates that the convolution kernel of the convolution layer in the convolution block is an n*n convolution kernel, n=1,3, the normalization layer adopts a BN layer, and the activation layer is configured with a relu activation function.

[0065] In one implementation, the feature map output by the preset preceding feature model includes a key point position distribution map and a key point connection relationship map, and the feature map output by the human body posture estimation model to be trained includes a key point position distribution map and a key point connection relationship map, that is, the preceding feature map determined based on the preset preceding feature model includes a preceding key point position distribution map and a preceding key point connection relationship map, and the training feature map determined based on the human body posture estimation model to be trained includes a training key point position distribution map and a training key point connection relationship map. Based on this, the above-mentioned fusion of the training feature map with the preceding feature map to obtain the predicted feature map corresponding to the current frame image specifically includes:

[0066] splicing the training key point position distribution map with the previous key point position distribution map, and the training key point connection relationship map with the previous key point connection relationship map by channel, respectively, to obtain a spliced ​​key point position distribution map and a spliced ​​key point connection relationship map;

[0067] According to the splicing key point position distribution map and the splicing key point connection relationship map, the predicted key point position distribution map and the predicted key point connection relationship map are respectively determined to obtain the predicted feature map corresponding to the current frame image.

[0068] In one implementation, the human body posture estimation model to be trained also includes an information fusion module, which is used to splice the training key point position distribution map with the previous key point position distribution map, and the training key point connection relationship map with the previous key point connection relationship map by channel, and determine the predicted feature map based on the spliced ​​key point connection relationship map and the key point connection relationship map, wherein the predicted feature map includes a predicted key point connection relationship map and a predicted key point connection relationship map. This embodiment realizes the fusion of feature maps of adjacent image frames through the information fusion module, so that the predicted feature map fuses the feature information carried by the adjacent image frames, thereby improving the accuracy of the key point information carried by the predicted feature map.

[0069] In one implementation, the information fusion module includes a first fusion unit and a second fusion unit, the first fusion unit is used to fuse the key point position distribution map and determine the predicted key point position distribution map; the second fusion unit is used to fuse the key point connection relationship map and determine the predicted key point connection relationship map. Figure 3As shown in the figure, the first fusion unit is connected to the first convolution unit, the second fusion unit is connected to the second convolution unit, the input item of the first fusion unit includes the output item of the first convolution unit and the position distribution map of the preceding key points, and the input item of the second fusion unit includes the output item of the second convolution unit and the connection relationship map of the preceding key points. Among them, the model structure of the first fusion unit is the same as the model structure of the second fusion unit, and the first fusion unit is taken as an example here for explanation. The first fusion unit includes a connection layer, a 3*3 convolution block, a 3*3 convolution block, a 1*1 convolution block and a 1*1 convolution block cascaded in sequence. The convolution block includes a convolution layer, a normalization layer and an activation layer cascaded in sequence. The n*n before the convolution block indicates that the convolution kernel of the convolution layer in the convolution block is an n*n convolution kernel, n=1,3, the normalization layer uses a BN layer, the activation layer is configured with a relu activation function, and the connection layer uses a concat layer.

[0070] S30, iteratively optimizing the human body posture estimation model to be trained according to the predicted feature map and the true feature map corresponding to each current frame image in the training sample set, to obtain a human body posture estimation model.

[0071] Specifically, the true value feature map is used as the true value corresponding to the current frame image, and the loss term is determined based on the true value feature map corresponding to each current frame image in the training sample set and the predicted feature map. The model parameters of the human posture estimation model to be trained are iteratively optimized based on the loss term to obtain the best human posture estimation model, that is, the human posture estimation model, wherein the true value feature map includes a true value key point position distribution map and a true value key point connection relationship map. Correspondingly, the loss term includes a position loss term determined based on the true value key point position distribution map and the predicted key point position distribution map, and a connection relationship loss term determined based on the true value key point connection relationship map and the predicted key point connection relationship map, and the loss term is the sum of the position loss term and the connection relationship loss term. In a typical implementation, the position loss term and the connection relationship loss term can both be distance loss (L1Loss), or the position loss term is distance loss (L1Loss), and the connection relationship loss term is cross entropy loss (CELoss), etc.

[0072] It should be noted that, during the training process of the human body posture estimation model to be trained, it will be detected whether the training of the human body posture estimation model to be trained meets the training requirements. After meeting the training requirements, the training of the human body posture estimation model to be trained will be stopped, and the trained human body posture estimation model to be trained will be used as the human body posture estimation model, wherein the training requirements include a loss item being less than a preset loss threshold and a training times threshold. Meeting the training requirements means that the loss item is less than the preset loss threshold, or the training times reaches the preset times threshold.

[0073] Based on the training method of the human body posture estimation model described above, this embodiment provides a human body posture estimation method, and uses the training method of the human body posture estimation model described in the above embodiment to obtain a human body posture estimation model; Figure 4 and Figure 5 As shown, the method comprises:

[0074] H10, obtaining the previous feature map of the current frame image and the previous frame image; wherein the current frame image and the previous frame image are adjacent image frames, and the previous feature map is obtained by inputting the previous frame image into the human body posture estimation model;

[0075] H20, fusing the current frame image with the previous feature map through the human posture estimation model to obtain the target feature map of the current frame image;

[0076] H30. Determine the target human body posture corresponding to the current frame image according to the target feature map.

[0077] Specifically, the preceding feature map is a feature map of the previous frame image of the current frame image, that is, the preceding feature map is a feature map extracted by the human body posture estimation model by using the preceding image frame of the current frame image as an input item of the human body posture estimation model. It can be understood that when determining the human body posture corresponding to the previous frame image of the current frame image, the extracted preceding feature map will be saved, and when determining the target human body posture corresponding to the current frame image, the preceding feature map and the current frame image will be input into the human body posture estimation model, so that the human body posture estimation model can learn the spatiotemporal information between the preceding image frame and the current frame image, thereby improving the accuracy of the target human body posture corresponding to the current frame image determined subsequently.

[0078] In one implementation, the current frame image and the previous image frame are adjacent image frames in a video frame sequence. For example, if the current frame image is the Nth video frame in video frame training, then the previous image frame is the N-1th video frame in the video frame sequence. In addition, in order to make the human body posture estimation model applicable to each image frame in the video frame sequence, when the current frame image is the first image frame in the video frame sequence, the current frame image is used as its own previous image frame. That is, when the current frame image is the first image frame in the video frame sequence, the current frame image is input into the human body posture estimation model, and the candidate feature map corresponding to the current frame image is determined by the human body posture estimation model, and then the candidate feature map is copied, and the copied candidate feature map is fused with the candidate feature map to obtain the target feature map corresponding to the current frame image.

[0079] In one implementation, the target feature map includes a target key point position distribution map and a target key point connection relationship map. After obtaining the target feature map, the positions of each key point in the current frame image are extracted based on the target key point position distribution map, and then the connection relationship between the key point positions is determined based on the target key point connection relationship map to obtain the key point positions and connection relationships of each target human body in the current frame image, thereby determining the human body posture of the target human body.

[0080] The present application also provides a training device for a human body posture estimation model. For details not described in the device, please refer to the relevant description in the training method embodiment of the human body posture estimation model.

[0081] See also Figure 6 , Figure 6 1 is a schematic block diagram of a training device for a human posture estimation model provided by an embodiment of the present application. The device comprises: a first acquisition module 61 , a second acquisition module 62 and a training module 63 .

[0082] The first acquisition module 61 acquires the current frame image, the true feature map corresponding to the current frame image, and the previous feature map of the previous frame image in the training sample set, wherein the previous frame image and the current frame image are adjacent image frames; wherein the previous feature map is obtained based on a preset previous feature model, and the preset previous feature model is a model constructed based on a single frame image;

[0083] The second acquisition module 62 inputs the previous feature map and the current frame image into the human body posture estimation model to be trained to obtain a predicted feature map corresponding to the current frame image;

[0084] The training module 63 iteratively optimizes the human body posture estimation model to be trained according to the predicted feature map and the true feature map corresponding to each current frame image in the training sample set to obtain the human body posture estimation model; wherein the human body posture estimation model is a model obtained by training based on adjacent image frames.

[0085] The embodiment of the present application also provides a human body posture estimation device. For details not described in the device, please refer to the relevant description in the aforementioned human body posture estimation method embodiment.

[0086] See also Figure 7 , Figure 7 1 is a schematic block diagram of a human body posture estimation device provided by an embodiment of the present application. The device comprises: a third acquisition module 71 , a fourth acquisition module 72 and an estimation module 73 .

[0087] The third acquisition module 71 acquires the previous feature map of the current frame image and the previous frame image; wherein the current frame image and the previous frame image are adjacent image frames, and the previous feature map is obtained by inputting the previous frame image into the human body posture estimation model;

[0088] A fourth acquisition module 72, fusing the current frame image with the previous feature map through a human posture estimation model to obtain a target feature map of the current frame image;

[0089] The estimation module 73 determines the target human body posture corresponding to the current frame image according to the target feature map.

[0090] The present application also provides an electronic device, such as Figure 8 As shown, the electronic device may include one or more processors 800 ( Figure 8 Only one is shown), a memory 810 and a computer program 820 stored in the memory 810 and executable on one or more processors 800, for example, a training method for a human body posture estimation model and / or a program for a human body posture estimation method. When one or more processors 800 execute the computer program 820, the training method for a human body posture estimation model and / or each step in the embodiment of the human body posture estimation method can be implemented. Alternatively, when one or more processors 800 execute the computer program 820, the functions of each module / unit in the training device for a human body posture estimation model and / or the embodiment of the human body posture estimation device can be implemented, which is not limited here.

[0091] Exemplarily, the computer program 820 may be divided into one or more modules / units, one or more modules / units are stored in the memory 810, and are executed by the processor 800 to complete the present application. One or more modules / units may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program 820 in the processing unit.

[0092] For example, the computer program 820 can be divided into the following modules. The specific functions of each module are as follows:

[0093] A first acquisition module acquires a current frame image, a true feature map corresponding to the current frame image, and a previous feature map of a previous frame image in a training sample set, wherein the previous frame image and the current frame image are adjacent image frames; wherein the previous feature map is obtained based on a preset previous feature model, and the preset previous feature model is a model constructed based on a single frame image;

[0094] The second acquisition module inputs the previous feature map and the current frame image into the human body posture estimation model to be trained to obtain the predicted feature map corresponding to the current frame image;

[0095] The training module iteratively optimizes the human body posture estimation model to be trained according to the predicted feature map and the true feature map corresponding to each current frame image in the training sample set to obtain the human body posture estimation model; wherein the human body posture estimation model is a model obtained by training based on adjacent image frames.

[0096] For another example, the computer program 820 can be divided into the following modules. The specific functions of each module are as follows:

[0097] The third acquisition module acquires the previous feature map of the current frame image and the previous frame image; wherein the current frame image and the previous frame image are adjacent image frames, and the previous feature map is obtained by inputting the previous frame image into the human body posture estimation model;

[0098] The fourth acquisition module fuses the current frame image with the previous feature map through the human body posture estimation model to obtain the target feature map of the current frame image;

[0099] The estimation module determines the target human body posture corresponding to the current frame image according to the target feature map.

[0100] Those skilled in the art will understand that Figure 8 The electronic device is merely an example of an electronic device and does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0101] In one embodiment, the processor 800 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0102] In one embodiment, the memory 810 may be an internal storage unit of an electronic device, such as a hard disk or memory of the electronic device. The memory 810 may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device. Further, the memory 810 may also include both an internal storage unit of the electronic device and an external storage device. The memory 810 is used to store computer programs and other programs and data required by the electronic device. The memory 810 may also be used to temporarily store data that has been output or is to be output.

[0103] An embodiment of the present application further provides another preferred embodiment of an electronic device. In this embodiment, the electronic device includes one or more processors. The one or more processors are used to execute the following program modules stored in the memory:

[0104] The first acquisition module is used to acquire the current frame image in the training sample set, the true value feature map corresponding to the current frame image, and the previous feature map of the previous frame image, wherein the previous frame image and the current frame image are adjacent image frames.

[0105] The second acquisition module is used to input the previous feature map and the current frame image into the human body posture estimation model to be trained to obtain the predicted feature map corresponding to the current frame image.

[0106] The training module is used to iteratively optimize the human body posture estimation model to be trained according to the predicted feature map and the true feature map corresponding to each current frame image in the training sample set to obtain the human body posture estimation model.

[0107] and / or,

[0108] The third acquisition module acquires the current frame image and its corresponding previous feature map;

[0109] A fourth acquisition module, inputting the current frame image and the second preceding feature map into a human posture estimation model to acquire a target feature map of the current frame image;

[0110] The estimation module determines the target human body posture corresponding to the current frame image according to the target feature map.

[0111] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0112] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the various steps in the method for obtaining posture data and / or the neural network construction method embodiment.

[0113] An embodiment of the present application also provides a computer program product. When the computer program product is run on an electronic device, the electronic device can implement each step in the method for acquiring posture data and / or the neural network construction method embodiment.

[0114] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A training method for a human posture estimation model, characterized in that: The training method comprises: Obtaining a current frame image in a training sample set, a true feature map corresponding to the current frame image, and a previous feature map of a previous frame image, wherein the previous frame image and the current frame image are adjacent image frames, and the previous feature map is obtained based on a preset previous feature model, and the preset previous feature model is a model constructed based on a single frame image to optimize the human body posture estimation model, wherein the preset previous feature model is used to obtain thermal map information of key points of the human body; Inputting the previous feature map and the current frame image into the human body posture estimation model to be trained to obtain a predicted feature map corresponding to the current frame image; The human body posture estimation model to be trained is iteratively optimized according to the predicted feature map and the true feature map corresponding to each of the current frame images to obtain the human body posture estimation model.

2. The training method of the human body posture estimation model according to claim 1, characterized in that: The preset preceding feature model may be a trained network model or a network model to be trained.

3. The training method of the human body posture estimation model according to claim 2, characterized in that: If the preset preceding feature model is the network model to be trained, before obtaining the current frame image in the training sample set, the true feature map corresponding to the current frame image, and the preceding feature map of the previous frame image, the method includes: Acquire several single-frame images; The pre-order feature model to be trained is trained based on the plurality of single-frame images to obtain the preset pre-order feature model.

4. The training method of the human body posture estimation model according to claim 1, characterized in that: The step of inputting the previous feature map and the current frame image into the human body posture estimation model to be trained to obtain a predicted feature map corresponding to the current frame image includes: Performing feature extraction on the current frame image by using the human body posture estimation model to be trained to obtain a training feature map corresponding to the current frame image; The training feature map is fused with the previous feature map through the human body posture estimation model to be trained to obtain a predicted feature map corresponding to the current frame image.

5. The training method of the human body posture estimation model according to claim 4, characterized in that: The training feature graph includes a training key point position distribution graph and a training key point connection relationship graph, and the preceding feature graph includes a preceding key point position distribution graph and a preceding key point connection relationship graph.

6. The training method of the human body posture estimation model according to claim 5, characterized in that: The step of fusing the training feature map with the previous feature map to obtain a predicted feature map corresponding to the current frame image includes: splicing the training key point position distribution map with the previous key point position distribution map, and the training key point connection relationship map with the previous key point connection relationship map by channel, respectively, to obtain a spliced ​​key point position distribution map and a spliced ​​key point connection relationship map; A predicted key point position distribution map and a predicted key point connection relationship map are respectively determined according to the splicing key point position distribution map and the splicing key point connection relationship map to obtain a predicted feature map corresponding to the current frame image.

7. A human body posture estimation model, characterized in that: The human body posture estimation model is obtained by training using the training method of the human body posture estimation model according to any one of claims 1 to 6, wherein the human body posture estimation model includes a second feature module and an information fusion module: The second feature module is used to obtain a training feature map of the current frame image; the structure of the second feature module is the same as the model structure of the first feature module of the preset preceding feature model, and the first feature module in the preset preceding feature model is used to obtain a preceding feature map of the previous frame image; The information fusion module is used to fuse the training feature map with the previous feature map to obtain a predicted feature map corresponding to the current frame image.

8. A method for estimating a human body posture, characterized in that: The human body posture estimation model as claimed in claim 7 is used; the method described comprises: Obtaining a preceding feature map of a current frame image and a previous frame image; wherein the current frame image and the previous frame image are adjacent image frames, and the preceding feature map is obtained by inputting the previous frame image into the human body posture estimation model; Fusing the current frame image with the previous feature map through the human body posture estimation model to obtain a target feature map of the current frame image; The target human body posture corresponding to the current frame image is determined according to the target feature map.

9. A training device for a human posture estimation model, characterized in that: include: A first acquisition module is configured to acquire a current frame image, a true feature map corresponding to the current frame image, and a previous feature map of a previous frame image in a training sample set, wherein the previous frame image and the current frame image are adjacent image frames; wherein the previous feature map is obtained based on a preset previous feature model, and the preset previous feature model is a model constructed based on a single frame image to optimize the human body posture estimation model; A second acquisition module inputs the previous feature map and the current frame image into the human body posture estimation model to be trained to obtain a predicted feature map corresponding to the current frame image; A training module iteratively optimizes the human body posture estimation model to be trained according to the predicted feature map and the true feature map corresponding to each current frame image in the training sample set to obtain a human body posture estimation model; wherein the human body posture estimation model is a model obtained by training based on the adjacent image frames.

10. A human body posture estimation device, characterized in that: A human body posture estimation model trained by using the training method of the human body posture estimation model according to any one of claims 1 to 6 comprises: A third acquisition module is configured to acquire a preceding feature map of a current frame image and a previous frame image; wherein the current frame image and the previous frame image are adjacent image frames, and the preceding feature map is obtained by inputting the previous frame image into the human body posture estimation model; A fourth acquisition module, fusing the current frame image with the previous feature map through the human posture estimation model to acquire a target feature map of the current frame image; The estimation module determines the target human body posture corresponding to the current frame image according to the target feature map.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the training method of the human body posture estimation model as described in any one of claims 1-6, and / or to implement the human body posture estimation method as described in any one of claim 8.

12. An electronic device, characterized in that: The device comprises a processor and a memory, wherein: The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, it implements the steps in the training method of the human body posture estimation model as described in any one of claims 1-6, and / or implements the human body posture estimation method as described in any one of claim 8.

Images