3D human posture evaluation method, device and electronic equipment

Abstract

The present application provides a 3D human posture evaluation method, device and electronic device, which belongs to the field of image processing technology. The 3D human posture evaluation method includes: inputting the target static feature sequence into the motion timing feature extraction layer and the continuity mapping feature extraction layer of the trained 3D human posture evaluation model respectively, and determining the target timing similarity feature sequence and the target motion continuity feature sequence respectively; inputting the target timing similarity feature sequence and the target motion continuity feature sequence together into the feature fusion layer of the trained 3D human posture evaluation model, and determining the motion continuity characterization feature sequence. The present application fully considers the temporal correlation of human body movements and the correlation of continuous movements, and greatly makes up for the deficiencies of traditional neural networks in the temporal and global feature extraction capabilities of 3D human postures, and enhances the accuracy and continuity of 3D human posture evaluation.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a 3D human posture evaluation method, device, and electronic equipment. Background Art

[0002] With the development of artificial intelligence technology, human body recognition and posture assessment technology for a single image has been widely used in society and is relatively common.

[0003] Currently, convolutional neural networks (CNNs) and recurrent neural networks (RNNs) are primarily used to perform human pose assessment. However, due to the limited methods for annotating 3D data, existing neural network architectures suffer from depth ambiguity when complex human movements occur, making it difficult to effectively perform 3D human pose assessment. Therefore, how to perform 3D human pose assessment in video sequences captured by a monocular camera is a major challenge in current image recognition technology. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a 3D human posture evaluation method, device and electronic equipment, which fully considers the temporal correlation of human body movements and the correlation of continuous movements, and greatly compensates for the shortcomings of traditional neural networks in the temporal and global feature extraction capabilities of 3D human posture, and enhances the accuracy and continuity of 3D human posture evaluation.

[0005] The present invention provides a method for evaluating a 3D human body posture. The method includes:

[0006] Obtain a target video frame sequence of the target 3D human posture to be detected;

[0007] Inputting the target video frame sequence into the static feature extraction layer of the trained 3D human posture assessment model to determine the target static feature sequence;

[0008] Inputting the target static feature sequence into the motion timing feature extraction layer and the continuity mapping feature extraction layer of the trained 3D human posture assessment model respectively, and determining the target timing similarity feature sequence and the target motion continuity feature sequence respectively;

[0009] Inputting the target temporal similarity feature sequence and the target motion continuity feature sequence into the feature fusion layer of the trained 3D human posture assessment model to determine the motion continuity representation feature sequence;

[0010] Based on the motion continuity characterization feature sequence, the target 3D human posture is evaluated on the target video frame sequence.

[0011] Furthermore, the evaluating of the target 3D human posture of the target video frame sequence based on the motion continuity characterization feature sequence includes:

[0012] Inputting the motion continuity characterization feature sequence into the fully connected neural network reinforcement layer of the trained 3D human posture assessment model to determine the motion continuity reinforcement feature sequence;

[0013] Based on the motion continuity enhancement feature sequence, the target 3D human posture is evaluated on the target video frame sequence.

[0014] Furthermore, the step of inputting the motion continuity characterization feature sequence into the fully connected neural network reinforcement layer of the trained 3D human posture assessment model to determine the motion continuity reinforcement feature sequence includes:

[0015] For a target motion continuity characterization feature corresponding to any target video frame in the motion continuity characterization feature sequence, determine a motion continuity enhancement feature corresponding to the target video frame based on the motion continuity characterization features of a preset number of frames before the target motion continuity characterization feature, the motion continuity characterization features of a preset number of frames after the target motion continuity characterization feature, and the target motion continuity characterization feature;

[0016] A motion continuity enhancement feature sequence is determined according to the motion continuity enhancement features corresponding to each video frame.

[0017] Furthermore, the formula for determining the target time series similarity feature sequence is specifically:

[0018]

[0019] Among them, S is used to represent the target temporal similarity feature sequence; ts(Q) is used to represent the target temporal similarity feature; softmax is used to represent the logistic regression function; f is used to represent the similarity calculation function; Q is used to represent the target static feature sequence; "Q'" is used to represent the matrix transpose of the target static feature sequence; "T" is used to represent the preset frame number of the target video frame sequence; "t" is used to represent the current frame number of the target video frame sequence.

[0020] Furthermore, the formula for determining the target motion continuity feature sequence is specifically:

[0021]

[0022] Among them, C is used to characterize the target motion continuity feature sequence; mp(Q) is used to characterize the target motion continuity feature; softmax is used to characterize the logistic regression function; f is used to characterize the similarity calculation function; λ and γ are constants of the convolution operation respectively; Q is used to characterize the target static feature sequence; "Q'" is used to characterize the matrix transpose of the target static feature sequence; λ(Q) and γ(Q) are both used to characterize the learnable convolution kernel parameters; and λ(Q) = W λ Q;γ(Q)=W γ Q; W is used to represent the hyperparameters; and “T” is used to represent the preset frame number of the target video frame sequence; “t” is used to represent the current frame number of the target video frame sequence.

[0023] The embodiment of the present application further provides a 3D human body posture evaluation device, wherein the 3D human body posture evaluation determination device includes:

[0024] An acquisition module is used to acquire a target video frame sequence of a target 3D human posture to be detected;

[0025] A first determination module is configured to input the target video frame sequence into a static feature extraction layer of a trained 3D human posture assessment model to determine a target static feature sequence;

[0026] A second determination module is configured to input the target static feature sequence into the motion timing feature extraction layer and the continuity mapping feature extraction layer of the trained 3D human posture assessment model, respectively, to determine a target timing similarity feature sequence and a target motion continuity feature sequence;

[0027] A third determination module is configured to input the target temporal similarity feature sequence and the target motion continuity feature sequence into a feature fusion layer of the trained 3D human posture assessment model to determine a motion continuity representation feature sequence;

[0028] An evaluation module is used to evaluate the target 3D human posture of the target video frame sequence based on the motion continuity characterization feature sequence.

[0029] Furthermore, the evaluation module is specifically used to:

[0030] a determination unit, configured to input the motion continuity characterization feature sequence into a fully connected neural network reinforcement layer of the trained 3D human posture assessment model to determine a motion continuity reinforcement feature sequence;

[0031] An evaluation unit is configured to evaluate the target 3D human posture on the target video frame sequence based on the motion continuity enhancement feature sequence.

[0032] Furthermore, the determining unit is configured to determine the motion continuity enhancement feature sequence according to the following steps, including:

[0033] For a target motion continuity characterization feature corresponding to any target video frame in the motion continuity characterization feature sequence, determine a motion continuity enhancement feature corresponding to the target video frame based on the motion continuity characterization features of a preset number of frames before the target motion continuity characterization feature, the motion continuity characterization features of a preset number of frames after the target motion continuity characterization feature, and the target motion continuity characterization feature;

[0034] A motion continuity enhancement feature sequence is determined according to the motion continuity enhancement features corresponding to each video frame.

[0035] An embodiment of the present application also provides an electronic device, comprising: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the 3D human body posture evaluation method as described above are performed.

[0036] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the above-mentioned 3D human body posture evaluation method are executed.

[0037] The 3D human posture evaluation method, device and electronic device provided in the embodiments of the present application, compared with the prior art, the embodiments provided by the present application determine the target temporal similarity feature sequence and the target motion continuity feature sequence in the target video frame sequence of the target 3D human posture, and fuse the above-mentioned target temporal similarity feature sequence and target motion continuity feature sequence to determine the motion continuity characterization feature sequence, fully considering the temporal correlation of human movements and the correlation of continuous movements, and greatly compensating for the deficiencies of traditional neural networks in the temporal and global feature extraction capabilities of 3D human posture, enhancing the accuracy and continuity of 3D human posture evaluation, and realizing the effective and efficient recognition of the temporally consistent and smooth target 3D human posture in the target video frame sequence, that is, realizing the ability to capture the target 3D human posture in motion.

[0038] Moreover, the embodiments of the present application greatly make up for the insufficient ability of traditional neural networks to extract the temporal features and global features of 3D human motion, and greatly enhance the accuracy and continuity of 3D human posture assessment for monocular video acquisition tasks.

[0039] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 FIG1 shows one of the flow charts of a 3D human body posture evaluation method provided in an embodiment of the present application;

[0042] Figure 2 FIG2 shows a second flow chart of a 3D human body posture evaluation method provided in an embodiment of the present application;

[0043] Figure 3 FIG1 shows one of the structural schematic diagrams of a 3D human body posture evaluation device provided in an embodiment of the present application;

[0044] Figure 4 FIG2 shows a second structural diagram of a 3D human body posture evaluation device provided in an embodiment of the present application;

[0045] Figure 5 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown.

[0046] In the picture:

[0047] 300 - 3D human posture evaluation device; 310 - acquisition module; 320 - first determination module; 330 - second determination module; 340 - third determination module; 350 - evaluation module; 351 - determination unit; 352 - evaluation unit; 500 - electronic device; 510 - processor; 520 - memory; 530 - bus. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for which protection is claimed, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work falls within the scope of protection of the present application.

[0049] First, the application scenarios to which this application is applicable are introduced. This application can be applied in the field of image processing technology. Research has found that in the existing technology, due to the limited methods of three-dimensional data annotation, the existing neural network architecture technology will have depth ambiguity problems when the human body movements are complex, making it difficult to effectively complete the 3D human posture assessment task. Therefore, how to complete 3D human posture assessment in the video sequence captured by a monocular camera. This is a major problem in current image recognition technology.

[0050] In the existing technology, when you want to complete the task of human posture assessment, you usually need to use a convolutional neural network (CNN) or a recurrent neural network (RNN) to complete the assessment of human posture. When using a monocular camera to capture video, the above two neural networks lack research on the continuity characteristics of human motion in continuous video frames.

[0051] In existing technologies, trained vectorized deep mesh models can input and decode a single image to project 3D human coordinates. Although this technology is very effective for static images, when the input is dynamic video, the generated dynamic human posture assessment is very jittery, incoherent, and has poor temporal continuity.

[0052] In the existing technology, convolutional neural networks or recurrent neural networks are combined with long short-term memory networks to enable convolutional neural networks or recurrent neural networks to obtain the ability to process video input. However, this combination still has the defect of only being able to process local features and cannot process the global correlation of features when the human body performs movements.

[0053] Based on this, the embodiment of the present application provides a 3D human posture evaluation method, which fully considers the temporal correlation of human body movements and the correlation of continuous movements, greatly compensates for the deficiencies of traditional neural networks in the temporal and global feature extraction capabilities of 3D human posture, and enhances the accuracy and continuity of 3D human posture evaluation.

[0054] See also Figure 1 , Figure 1 This is one of the flow charts of a 3D human body posture evaluation method provided in the embodiment of the present application. Figure 1 As shown in , the 3D human posture evaluation method provided in the embodiment of the present application includes:

[0055] S101: Obtain a target video frame sequence of a target 3D human body posture to be detected.

[0056] In this step, the embodiment provided by the present application uses a monocular camera to shoot and collect a target video of the target 3D human posture to be detected in real time, and determines a target video frame sequence of the target video.

[0057] Here, the target video frame sequence in the embodiments provided in this application can be specifically represented by the symbol “V”.

[0058] in, Here, "v t " is a target video frame of any frame in the target video frame sequence, and "T" is used to represent the total number of frames in the target video frame sequence.

[0059] In this way, a monocular camera uses a monocular algorithm to parse real-time unstructured video data into structured data. For example, based on facial features, determining whether an input face image or video contains a face requires locating key facial information, analyzing attributes such as gender and age. A monocular camera can capture accurate facial expressions in real time, which can be used for facial interaction or customer sentiment analysis.

[0060] S102: Input the target video frame sequence into the static feature extraction layer of the trained 3D human posture assessment model to determine a target static feature sequence.

[0061] In this step, the trained 3D human posture evaluation model is trained based on the sample target video frame sequence and is continuously adjusted through a preset loss function. Here, in order to determine the target static feature sequence in the target video frame sequence, the target video frame sequence needs to be input into the static feature extraction layer (i.e., the convolutional network feature extractor "ex(x)") of the trained 3D human posture evaluation model for determination.

[0062] Here, the number of convolution layers of the static feature extraction layer in the embodiment of the present application can be customized according to different application scenarios. For example, in the embodiment provided in the present application, when the target video frame sequence is input into the static feature extraction layer of the trained 3D human posture evaluation model, the number of layers of the static feature extraction layer can be set to 100 layers (resnet100).

[0063] Among them, the target static feature sequence can be specifically represented by the symbol "Q".

[0064] so, Here, "ex(v t )” is the target static feature of any frame of the target static feature sequence, and “T” is used to represent the total number of frames of the target video frame sequence, where here, Used to represent spatial dimensions.

[0065] S103 , inputting the target static feature sequence into the motion timing feature extraction layer and the continuity mapping feature extraction layer of the trained 3D human posture assessment model respectively, and determining a target timing similarity feature sequence and a target motion continuity feature sequence respectively.

[0066] In this step, after determining the target static feature sequence, the above target static feature sequence is input into the motion timing feature extraction layer of the trained 3D human posture evaluation model to determine the target timing similarity feature sequence. Then the above target static feature sequence is input into the continuity mapping feature extraction layer of the trained 3D human posture evaluation model to determine the target motion continuity feature sequence.

[0067] Here, the trained motion timing feature extraction layer of the 3D human posture evaluation model can determine the posture changes and posture clues of the target human body in continuous motion, and can adaptively calibrate the attention range in the target static feature sequence and the target video frame sequence to better capture the similar dependencies between the global and local aspects of the target human body in continuous motion.

[0068] The continuity mapping feature extraction layer of the trained 3D human posture assessment model can use multiple convolutions and self-attention mechanisms to map the target video frames and target static feature sequences, capturing the continuous dependency between different frames of the target human body when it is in motion (or in other states).

[0069] Among them, the similarity dependency relationship between the global and the local includes multiple relationships, for example, the similarity dependency relationship between the hand paddling and the leg extension when the target human body is doing breaststroke, that is, the features corresponding to the continuity sequence can be expressed as the target motion continuity feature sequence.

[0070] Among them, there are many types of continuous dependencies between different frames. For example, taking the above-mentioned target human body in breaststroke as an example, when the target human body is in breaststroke, the time sequence between the hand movement and the leg extension is a continuous dependency between two video frames, that is, the feature corresponding to the time sequence can be expressed as a target temporal similarity feature sequence.

[0071] Optionally, the formula for determining the target time series similarity feature sequence is specifically:

[0072]

[0073] Among them, S is used to represent the target temporal similarity feature sequence; ts(Q) is used to represent the target temporal similarity feature; softmax is used to represent the logistic regression function; f is used to represent the similarity calculation function; Q is used to represent the target static feature sequence; "Q'" is used to represent the matrix transpose of the static feature set; "T" is used to represent the preset frame number of the target video frame sequence; "t" is used to represent the current frame number of the target video frame sequence.

[0074] Optionally, the formula for determining the target motion continuity feature sequence is specifically:

[0075]

[0076] Among them, C is used to characterize the target motion continuity feature sequence; mp(Q) is used to characterize the target motion continuity feature; softmax is used to characterize the logistic regression function; f is used to characterize the similarity calculation function; λ and γ are constants of convolution operation respectively; Q is used to characterize the target static feature sequence; "Q'" is used to characterize the target static feature sequence; λ(Q) and γ(Q) are used to characterize the learnable convolution kernel parameters; and λ(Q) = W λ Q;γ(Q)=W γ Q; W is used to represent the hyperparameters; and “T” is used to represent the preset frame number of the target video frame sequence; “t” is used to represent the current frame number of the target video frame sequence.

[0077] S104: Input the target temporal similarity feature sequence and the target motion continuity feature sequence into the feature fusion layer of the trained 3D human posture assessment model to determine a motion continuity characterization feature sequence.

[0078] In this step, the feature fusion layer of the trained 3D human posture evaluation model is used to fuse the target temporal similarity feature sequence and the target motion continuity feature sequence, that is, the features corresponding to the time series and the features corresponding to the continuity sequence are used as input, and the feature fusion layer outputs the motion continuity characterization feature sequence for the target human posture.

[0079] Here, the motion continuity characterization feature sequence can be specifically represented by the symbol "E".

[0080] so, in,

[0081] The formula for determining the motion continuity characterization feature sequence is as follows:

[0082]

[0083] Among them, "[]" represents the concat operation. and “ψ” are convolution operations, “W co " can be represented as a learnable weight parameter, "+Q" can represent the residual operation, and "+Q" makes the motion continuity representation feature sequence contain the original motion image features.

[0084] Here, the concat operation is used to concatenate two or more arrays. This operation does not change the existing arrays but simply returns a copy of the concatenated arrays.

[0085] S105 : Evaluate the target 3D human body posture on the target video frame sequence based on the motion continuity characterization feature sequence.

[0086] In this step, after determining the motion continuity characterization feature sequence, the preset label similarity of the preset 3D human posture after the motion continuity characterization feature sequence is input into the trained 3D human posture evaluation model through the label information corresponding to the motion continuity characterization feature sequence is compared to determine the target 3D human posture in the target video frame sequence, and the target 3D human posture is evaluated.

[0087] Here, different loss functions can be used to adjust the parameters of the 3D human posture evaluation model to ensure the precision and accuracy of the trained 3D human posture evaluation model.

[0088] The 3D human posture evaluation method provided in the embodiment of the present application, compared with the prior art, the embodiment provided by the present application determines the target temporal similarity feature sequence and the target motion continuity feature sequence in the target video frame sequence of the target 3D human posture, and fuses the above-mentioned target temporal similarity feature sequence and the target motion continuity feature sequence to determine the motion continuity characterization feature sequence, fully considering the temporal correlation of human movements and the correlation of continuous movements, and greatly compensating for the deficiencies of traditional neural networks in the temporal and global feature extraction capabilities of 3D human posture, enhancing the accuracy and continuity of 3D human posture evaluation, and realizing the effective and efficient recognition of the temporally consistent and smooth target 3D human posture in the target video frame sequence, that is, realizing the ability to capture the target 3D human posture in motion.

[0089] Moreover, the embodiments of the present application greatly make up for the insufficient ability of traditional neural networks to extract the temporal features and global features of 3D human motion, and greatly enhance the accuracy and continuity of 3D human posture assessment for monocular video acquisition tasks.

[0090] See also Figure 2 , Figure 2 This is a flow chart of a 3D human body posture evaluation method provided by another embodiment of the present application. Figure 2 As shown in , the 3D human posture evaluation method provided in the embodiment of the present application includes:

[0091] S201: Obtain a target video frame sequence of a target 3D human body posture to be detected.

[0092] S202: Input the target video frame sequence into the static feature extraction layer of the trained 3D human posture assessment model to determine a target static feature sequence.

[0093] S203 , inputting the target static feature sequence into the motion timing feature extraction layer and the continuity mapping feature extraction layer of the trained 3D human posture assessment model respectively, and determining a target timing similarity feature sequence and a target motion continuity feature sequence respectively.

[0094] S204: Input the target temporal similarity feature sequence and the target motion continuity feature sequence into the feature fusion layer of the trained 3D human posture assessment model to determine a motion continuity characterization feature sequence.

[0095] S205 , inputting the motion continuity characterization feature sequence into the fully connected neural network reinforcement layer of the trained 3D human posture assessment model to determine a motion continuity reinforcement feature sequence.

[0096] In this step, the fully connected neural network reinforcement layer uses a segmented attention method to combine the previous preset number of frames and the next preset number of frames adjacent to the current motion continuity representation feature in the time series into a time series group, and merge the motion continuity representation features of the previous preset number of frames and the motion continuity representation features of the next preset number of frames into the current motion continuity representation feature to strengthen the temporal correlation and redefine the current motion continuity representation feature after reinforcement.

[0097] Here, the preset number of frames can be customized according to different application scenarios. For example, the preset number of frames provided in this application includes but is not limited to being set to T / 8. Therefore, the previous preset number of frames is expressed as t~tT / 8, and the subsequent preset number of frames is expressed as t~t+T / 8.

[0098] Optionally, step S205 includes the following sub-steps:

[0099] Sub-step 2051: For the target motion continuity characterization feature corresponding to any target video frame in the motion continuity characterization feature sequence, determine the motion continuity enhancement feature corresponding to the target video frame based on the motion continuity characterization features of a preset number of frames before the target motion continuity characterization feature, the motion continuity characterization features of a preset number of frames after the target motion continuity characterization feature, and the target motion continuity characterization feature.

[0100] In this step, the embodiment provided by the present application selects a motion continuity characterization feature sequence T / 8 before the target motion continuity characterization feature by any target motion continuity characterization feature. And the motion continuity representation feature sequence of T / 8 after the target motion continuity representation feature Characterization of target motion continuity t Perform temporal reinforcement, determine each movement continuity reinforcement feature, and each movement continuity reinforcement feature is represented by a symbol To express.

[0101] The expression of the motion continuity enhancement feature can be specifically:

[0102]

[0103] Here, “n” represents the number of reinforcement layers in the fully connected neural network.

[0104] Sub-step 2052: Determine a motion continuity enhancement feature sequence based on each of the motion continuity enhancement features.

[0105] In this step, a sequence composed of various motion continuity enhancement features is determined as a motion continuity enhancement feature sequence.

[0106] Here, the motion continuity enhancement feature sequence can be specifically characterized as

[0107] S206 : Evaluate the target 3D human body posture on the target video frame sequence based on the motion continuity enhancement feature sequence.

[0108] In this step, after determining the motion continuity enhancement feature sequence, the target 3D human body posture in the target video frame sequence is determined and evaluated by comparing the label information corresponding to the motion continuity enhancement feature sequence with the preset label similarity of the preset 3D human body posture after the motion continuity enhancement feature sequence is input into the trained 3D human body posture evaluation model.

[0109] Here, different loss functions can be used to adjust the parameters of the 3D human posture evaluation model to ensure the precision and accuracy of the trained 3D human posture evaluation model.

[0110] Among them, the description of S201 to S204 can refer to the description of S101 to S104, and can achieve the same technical effect, so it will not be repeated here.

[0111] The 3D human posture evaluation method provided in the embodiment of the present application, compared with the prior art, the embodiment provided by the present application determines the target temporal similarity feature sequence and the target motion continuity feature sequence in the target video frame sequence of the target 3D human posture, and fuses the above-mentioned target temporal similarity feature sequence and the target motion continuity feature sequence to determine the motion continuity characterization feature sequence, fully considering the temporal correlation of human movements and the correlation of continuous movements, and greatly compensating for the deficiencies of traditional neural networks in the temporal and global feature extraction capabilities of 3D human posture, enhancing the accuracy and continuity of 3D human posture evaluation, and realizing the effective and efficient recognition of the temporally consistent and smooth target 3D human posture in the target video frame sequence, that is, realizing the ability to capture the target 3D human posture in motion.

[0112] Moreover, the embodiments of the present application greatly make up for the insufficient ability of traditional neural networks to extract the temporal features and global features of 3D human motion, and greatly enhance the accuracy and continuity of 3D human posture assessment for monocular video acquisition tasks.

[0113] See also Figure 3 , Figure 3 This is a schematic diagram of the structure of a 3D human posture evaluation device provided in an embodiment of the present application. Figure 3 As shown in , the 3D human posture evaluation device 300 includes:

[0114] An acquisition module 310 is configured to acquire a target video frame sequence of a target 3D human body posture to be detected;

[0115] The first determination module 320 is configured to input the target video frame sequence into a static feature extraction layer of a trained 3D human posture assessment model to determine a target static feature sequence.

[0116] The second determination module 330 is used to input the target static feature sequence into the motion timing feature extraction layer and the continuity mapping feature extraction layer of the trained 3D human posture assessment model, and respectively determine the target timing similarity feature sequence and the target motion continuity feature sequence.

[0117] The third determination module 340 is configured to input the target temporal similarity feature sequence and the target motion continuity feature sequence into a feature fusion layer of the trained 3D human posture assessment model to determine a motion continuity representation feature sequence.

[0118] The evaluation module 350 is configured to evaluate the target 3D human posture on the target video frame sequence based on the motion continuity characterization feature sequence.

[0119] Optionally, the formula for determining the target time series similarity feature sequence is specifically:

[0120]

[0121] Among them, S is used to represent the target temporal similarity feature sequence; ts(Q) is used to represent the target temporal similarity feature; softmax is used to represent the logistic regression function; f is used to represent the similarity calculation function; Q is used to represent the target static feature sequence; "Q'" is used to represent the matrix transpose of the target static feature set; "T" is used to represent the preset number of frames of the target video frame sequence; "t" is used to represent the current number of frames of the target video frame sequence.

[0122] Optionally, the formula for determining the target motion continuity feature sequence is specifically:

[0123]

[0124] Among them, C is used to characterize the target motion continuity feature sequence; mp(Q) is used to characterize the target motion continuity feature; softmax is used to characterize the logistic regression function; f is used to characterize the similarity calculation function; λ and γ are constants of convolution operation respectively; Q is used to characterize the target static feature sequence; "Q'" is used to characterize the matrix transpose of the static feature set; λ(Q) and γ(Q) are both used to characterize the learnable convolution kernel parameters; and λ(Q) = W λ Q;γ(Q)=W γ Q; W is used to represent the hyperparameters; and “T” is used to represent the preset frame number of the target video frame sequence; “t” is used to represent the current frame number of the target video frame sequence.

[0125] The 3D human posture evaluation device 300 provided in the embodiment of the present application, compared with the prior art, determines the target temporal similarity feature sequence and the target motion continuity feature sequence in the target video frame sequence of the target 3D human posture, and fuses the above-mentioned target temporal similarity feature sequence and the target motion continuity feature sequence to determine the motion continuity characterization feature sequence, fully considering the temporal correlation of human movements and the correlation of continuous movements, and greatly compensating for the deficiencies of traditional neural networks in temporal and global feature extraction capabilities for 3D human posture, enhancing the accuracy and continuity of 3D human posture evaluation, and realizing the effective and efficient recognition of temporally consistent and smooth target 3D human postures in the target video frame sequence, that is, realizing the ability to capture the target 3D human posture in motion.

[0126] Moreover, the embodiments of the present application greatly make up for the insufficient ability of traditional neural networks to extract the temporal features and global features of 3D human motion, and greatly enhance the accuracy and continuity of 3D human posture assessment for monocular video acquisition tasks.

[0127] See also Figure 4 , Figure 4 This is a second structural diagram of a 3D human posture evaluation device provided in an embodiment of the present application. Figure 4 As shown in , the 3D human posture evaluation device 300 includes:

[0128] An acquisition module 310 is configured to acquire a target video frame sequence of a target 3D human body posture to be detected;

[0129] The first determination module 320 is configured to input the target video frame sequence into a static feature extraction layer of a trained 3D human posture assessment model to determine a target static feature sequence.

[0130] The second determination module 330 is used to input the target static feature sequence into the motion timing feature extraction layer and the continuity mapping feature extraction layer of the trained 3D human posture assessment model, and respectively determine the target timing similarity feature sequence and the target motion continuity feature sequence.

[0131] The third determination module 340 is configured to input the target temporal similarity feature sequence and the target motion continuity feature sequence into a feature fusion layer of the trained 3D human posture assessment model to determine a motion continuity representation feature sequence.

[0132] The evaluation module 350 is configured to evaluate the target 3D human posture on the target video frame sequence based on the motion continuity characterization feature sequence.

[0133] The determination unit 351 is configured to input the motion continuity characterization feature sequence into the fully connected neural network reinforcement layer of the trained 3D human posture assessment model to determine the motion continuity reinforcement feature sequence.

[0134] The evaluation unit 352 is configured to evaluate the target 3D human posture on the target video frame sequence based on the motion continuity enhancement feature sequence.

[0135] Optionally, the determining unit 351 is configured to determine the motion continuity enhancement feature sequence according to the following steps, including:

[0136] For the target motion continuity characterization feature corresponding to any target video frame in the motion continuity characterization feature sequence, each motion continuity enhancement feature is determined based on the motion continuity characterization features of a preset number of frames before the target motion continuity characterization feature, the motion continuity characterization features of the preset number of frames after the target motion continuity characterization feature, and the target motion continuity characterization feature.

[0137] A motion continuity enhancement feature sequence is determined based on each of the motion continuity enhancement features.

[0138] The 3D human posture evaluation device 300 provided in the embodiment of the present application, compared with the prior art, determines the target temporal similarity feature sequence and the target motion continuity feature sequence in the target video frame sequence of the target 3D human posture, and fuses the above-mentioned target temporal similarity feature sequence and the target motion continuity feature sequence to determine the motion continuity characterization feature sequence, fully considering the temporal correlation of human movements and the correlation of continuous movements, and greatly compensating for the deficiencies of traditional neural networks in temporal and global feature extraction capabilities for 3D human posture, enhancing the accuracy and continuity of 3D human posture evaluation, and realizing the effective and efficient recognition of temporally consistent and smooth target 3D human postures in the target video frame sequence, that is, realizing the ability to capture the target 3D human posture in motion.

[0139] Moreover, the embodiments of the present application greatly make up for the insufficient ability of traditional neural networks to extract the temporal features and global features of 3D human motion, and greatly enhance the accuracy and continuity of 3D human posture assessment for monocular video acquisition tasks.

[0140] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown in FIG, the electronic device 500 includes a processor 510, a memory 520 and a bus 530.

[0141] The memory 520 stores machine-readable instructions executable by the processor 510. When the electronic device 500 is running, the processor 510 communicates with the memory 520 via the bus 530. When the machine-readable instructions are executed by the processor 510, the above-mentioned Figure 1 as well as Figure 2 The specific implementation of the steps of the 3D human body posture evaluation device in the method embodiment shown can be found in the method embodiment, and will not be repeated here.

[0142] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 as well as Figure 2 The specific implementation of the steps of the 3D human body posture evaluation device in the method embodiment shown can be found in the method embodiment, and will not be repeated here.

[0143] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0144] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. There may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some communication interface, indirect coupling or communication connection of devices or units, which may be electrical, mechanical or other forms.

[0145] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0146] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0147] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0148] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The scope of protection of the present application is not limited thereto. Although the present application has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-mentioned embodiments within the technical scope disclosed in the present application, or perform equivalent replacements for some of the technical features thereof. These modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A 3D human body posture evaluation method, characterized in that: The 3D human body posture evaluation method includes: Obtain a target video frame sequence of the target 3D human posture to be detected; Inputting the target video frame sequence into the static feature extraction layer of the trained 3D human posture assessment model to determine the target static feature sequence; Inputting the target static feature sequence into the motion timing feature extraction layer and the continuity mapping feature extraction layer of the trained 3D human posture assessment model respectively, and determining the target timing similarity feature sequence and the target motion continuity feature sequence respectively; Inputting the target temporal similarity feature sequence and the target motion continuity feature sequence into the feature fusion layer of the trained 3D human posture assessment model to determine the motion continuity representation feature sequence; Based on the motion continuity characterization feature sequence, evaluating the target 3D human body posture of the target video frame sequence; The formula for determining the target time series similarity feature sequence is specifically: ； Among them, S is used to characterize the target temporal similarity feature sequence; Used to characterize the target time series similarity features; softmax is used to characterize the logistic regression function; It is used to characterize the similarity calculation function; Q is used to characterize the target static feature sequence; "Q ' "Matrix transpose used to represent the target static feature sequence;" "The preset number of frames used to represent the target video frame sequence;" " is used to represent the current frame number of the target video frame sequence; The formula for determining the target motion continuity feature sequence is specifically: ； Among them, C is used to characterize the target motion continuity feature sequence; Used to characterize the continuity of target motion; softmax is used to characterize the logistic regression function; Used to characterize continuous calculation functions; and are constants of convolution operation respectively; Q is used to characterize the target static feature sequence; and are used to characterize the learnable convolution kernel parameters; and ; ; W is used to represent the hyperparameter; and ; ; ;" "The preset number of frames used to represent the target video frame sequence;" " is used to represent the current frame number of the target video frame sequence.

2. The 3D human body posture evaluation method according to claim 1, characterized in that: The evaluating of the target 3D human posture of the target video frame sequence based on the motion continuity characterization feature sequence includes: Inputting the motion continuity characterization feature sequence into the fully connected neural network reinforcement layer of the trained 3D human posture assessment model to determine the motion continuity reinforcement feature sequence; Based on the motion continuity enhancement feature sequence, the target 3D human posture is evaluated on the target video frame sequence.

3. The 3D human posture evaluation method according to claim 2, characterized in that: Inputting the motion continuity characterization feature sequence into the fully connected neural network reinforcement layer of the trained 3D human posture assessment model to determine the motion continuity reinforcement feature sequence includes: For a target motion continuity characterization feature corresponding to any target video frame in the motion continuity characterization feature sequence, determine a motion continuity enhancement feature corresponding to the target video frame based on the motion continuity characterization features of a preset number of frames before the target motion continuity characterization feature, the motion continuity characterization features of a preset number of frames after the target motion continuity characterization feature, and the target motion continuity characterization feature; A motion continuity enhancement feature sequence is determined according to the motion continuity enhancement features corresponding to each video frame.

4. A 3D human body posture evaluation device, characterized in that: The determination device of the 3D human body posture assessment comprises: An acquisition module is used to acquire a target video frame sequence of a target 3D human posture to be detected; A first determination module is configured to input the target video frame sequence into a static feature extraction layer of a trained 3D human posture assessment model to determine a target static feature sequence; A second determination module is configured to input the target static feature sequence into the motion timing feature extraction layer and the continuity mapping feature extraction layer of the trained 3D human posture assessment model, respectively, to determine a target timing similarity feature sequence and a target motion continuity feature sequence; A third determination module is configured to input the target temporal similarity feature sequence and the target motion continuity feature sequence into a feature fusion layer of the trained 3D human posture assessment model to determine a motion continuity representation feature sequence; An evaluation module, configured to evaluate a target 3D human posture on the target video frame sequence based on the motion continuity characterization feature sequence; The formula for determining the target time series similarity feature sequence is specifically: ； Among them, S is used to characterize the target temporal similarity feature sequence; Used to characterize the target time series similarity features; softmax is used to characterize the logistic regression function; It is used to characterize the similarity calculation function; Q is used to characterize the target static feature sequence; "Q ' "Matrix transpose used to represent the target static feature sequence;" "The preset number of frames used to represent the target video frame sequence;" " is used to represent the current frame number of the target video frame sequence; The formula for determining the target motion continuity feature sequence is specifically: ； Among them, C is used to characterize the target motion continuity feature sequence; Used to characterize the continuity of target motion; softmax is used to characterize the logistic regression function; Used to characterize continuous calculation functions; and are constants of convolution operation respectively; Q is used to characterize the target static feature sequence; and are used to characterize the learnable convolution kernel parameters; and ; ; W is used to represent the hyperparameter; and ; ; ;" "The preset number of frames used to represent the target video frame sequence;" " is used to represent the current frame number of the target video frame sequence.

5. The 3D human posture evaluation device according to claim 4, characterized in that: The evaluation module is specifically used to: a determination unit, configured to input the motion continuity characterization feature sequence into a fully connected neural network reinforcement layer of the trained 3D human posture assessment model to determine a motion continuity reinforcement feature sequence; An evaluation unit is configured to evaluate the target 3D human posture on the target video frame sequence based on the motion continuity enhancement feature sequence.

6. The 3D human posture evaluation device according to claim 5, characterized in that: The determining unit is configured to determine a motion continuity enhancement feature sequence according to the following steps: For a target motion continuity characterization feature corresponding to any target video frame in the motion continuity characterization feature sequence, determine a motion continuity enhancement feature corresponding to the target video frame based on the motion continuity characterization features of a preset number of frames before the target motion continuity characterization feature, the motion continuity characterization features of a preset number of frames after the target motion continuity characterization feature, and the target motion continuity characterization feature; A motion continuity enhancement feature sequence is determined according to the motion continuity enhancement features corresponding to each video frame.

7. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate via the bus, and the machine-readable instructions are executed by the processor to execute the steps of the 3D human posture evaluation method as described in any one of claims 1 to 3 above.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the 3D human posture evaluation method according to any one of claims 1 to 3 are executed.

Images