Data processing method, device, equipment and medium

By combining object pose detection and part pose detection results, the key points of missing are compensated, and the problem of insufficient information in pose estimation in mobile scenes is solved, the accuracy and completeness of pose estimation is improved, and it is suitable for fields such as multimedia and virtual reality. .

CN116934848BActive Publication Date: 2024-11-19TENCENT TECHNOLOGY (SHENZHEN) CO LTD

Patent Information

Application Number
CN202210332763.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-31
Publication Date
2024-11-19
Estimated Expiration
2042-03-31

AI Technical Summary

Technical Problem

In posture estimation in mobile scenes, since the image or video may only contain part of the object, the extracted part information is insufficient, affecting the integrity and accuracy of the object's posture.

Method used

By obtaining the object pose detection results and part pose detection results, and combining standard poses, the target object in the target image frame is estimated to compensate for missing part key points, ensuring the integrity and rationality of the global pose.

Benefits of technology

Improves the accuracy and completeness of object pose estimation and is suitable for multimedia applications, virtual reality and autonomous driving.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116934848B_ABST
    Figure CN116934848B_ABST
Patent Text Reader

Abstract

The embodiment of the present application provides a data processing method, device, equipment and medium, which can be applied to the fields of artificial intelligence, assisted driving, etc. The method includes: obtaining an object posture detection result and a part posture detection result corresponding to a target image frame; the object posture detection result is associated with a target object in the target image frame, and the part posture detection result is associated with a first object part of the target object; according to the object posture detection result, the part posture detection result, and the standard posture associated with the target object, the posture of the target object in the target image frame is estimated to obtain a global posture corresponding to the target object. By adopting the embodiment of the present application, the estimation accuracy of the object posture can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a data processing method, device, equipment and medium. Background Art

[0002] Posture estimation can detect the position of each key point in an image or video, and has very wide application value in film animation, assisted driving, virtual reality, motion recognition and other fields.

[0003] In the current posture estimation algorithm, the final object posture can be constructed by detecting key points in images or videos based on the detected key points and object constraints. However, when the image or video is captured in a mobile scene, the image or video may only contain a part of the object. In the process of posture estimation, due to the lack of some parts of the object, the extracted part information is insufficient, resulting in the final object posture result not being the complete posture of the object, affecting the integrity of the object posture. Summary of the invention

[0004] The embodiments of the present application provide a data processing method, apparatus, device and medium, which can improve the estimation accuracy of the object posture.

[0005] On the one hand, an embodiment of the present application provides a data processing method, including:

[0006] Acquire an object posture detection result and a part posture detection result corresponding to a target image frame; the object posture detection result is associated with a target object in the target image frame, and the part posture detection result is associated with a first object part of the target object;

[0007] According to the object posture detection result, the part posture detection result, and the standard posture associated with the target object, the posture of the target object in the target image frame is estimated to obtain the global posture corresponding to the target object.

[0008] An embodiment of the present application provides a data processing device, including:

[0009] A posture detection module, used to obtain an object posture detection result and a part posture detection result corresponding to a target image frame; the object posture detection result is associated with a target object in the target image frame, and the part posture detection result is associated with a first object part of the target object;

[0010] The posture estimation module is used to estimate the posture of the target object in the target image frame according to the object posture detection result, the part posture detection result, and the standard posture associated with the target object, so as to obtain the global posture corresponding to the target object.

[0011] Among them, the posture detection module includes:

[0012] An object detection unit is used to obtain a target image frame from the video data, input the target image frame into an object detection model, and output an object posture detection result corresponding to the target image frame through the object detection model;

[0013] The part detection unit is used to input the target image frame into the part detection model, and output the part posture detection result corresponding to the target image frame through the part detection model.

[0014] Wherein, the object detection unit comprises:

[0015] The part classification subunit is used to input the target image frame into the object detection model, obtain the object posture features corresponding to the target object in the target image frame through the object detection model, and identify the object part classification results corresponding to the object posture features; the object part classification results are used to characterize the object part categories corresponding to the key points of the target object;

[0016] A part map generation subunit, configured to generate a first activation map according to the object part classification result and the object convolution feature of the target image frame;

[0017] A positioning result determination subunit, used to obtain a pixel average value corresponding to the first activation map, and determine a positioning result of a key point in the target object in the target image frame according to the pixel average value;

[0018] The detection result determination subunit is used to determine the object posture detection result corresponding to the target image frame according to the object part category and the positioning result.

[0019] Among them, the part classification subunits include:

[0020] A global classification subunit, used to obtain an object description feature corresponding to a target object in a target image frame in an object detection model, and output a second classification result corresponding to the object description feature according to a classifier in the object detection model;

[0021] A global map acquisition subunit is used to obtain the object convolution feature for the target image frame output by the target convolution layer in the object detection model, and perform a product operation on the second classification result and the object convolution feature to obtain a second activation map corresponding to the target image frame;

[0022] The block processing subunit is used to perform block processing on the target image frame according to the second activation map to obtain M object part region images, and obtain object part description features corresponding to the M object part region images respectively according to the object detection model; M is a positive integer;

[0023] The feature combination subunit is used to combine the object description feature and the object part description features corresponding to the M object part region images into an object posture feature.

[0024] Wherein, the part detection unit includes:

[0025] An object part detection subunit, configured to input a target image frame into a part detection model, and detect a first object part of a target object in the target image frame in the part detection model;

[0026] A part posture estimation subunit is used for, if a first object part is detected in a target image frame, obtaining a regional image containing the first object part from the target image frame, obtaining a position of a part key point corresponding to the first object part according to the regional image, and determining a part posture detection result corresponding to the target image frame based on the position of the part key point;

[0027] The null value determination subunit is used to determine that the part posture detection result corresponding to the target image frame is a null value if the first object part is not detected in the target image frame.

[0028] Among them, the part posture estimation subunit includes:

[0029] An image cropping subunit, configured to crop the target image frame to obtain a regional image containing the first object part if the first object part is detected in the target image frame;

[0030] A part key point determination subunit is used to obtain part contour features corresponding to the regional image, and predict the position of the part key point corresponding to the first object part according to the part contour features;

[0031] The part key point connection subunit is used to connect the key points of the first object part based on the position of the part key points to obtain the part posture detection result corresponding to the target image frame.

[0032] Among them, the posture estimation module includes:

[0033] A key point number counting unit, used to obtain a standard posture associated with the target object, and count the number of first key points corresponding to the standard posture, and the number of second key points corresponding to the object posture detection result;

[0034] A first interpolation processing unit, configured to perform interpolation processing on the object posture detection result according to the standard posture to obtain a first candidate object posture when the number of the first key points is greater than the number of the second key points;

[0035] The second interpolation processing unit is used to perform interpolation processing on the posture of the first candidate object according to the part posture detection result to obtain the global posture corresponding to the target object.

[0036] Wherein, the second interpolation processing unit includes:

[0037] A first direction determination subunit is used to determine the first part direction corresponding to the third object part according to the key point position of the first object part included in the part posture detection result if the object posture detection result includes the posture of the second object part and the object posture detection result does not include the posture of the third object part; the second object part and the third object part are symmetrical parts of the target object;

[0038] A first position determination subunit is used to obtain a first part length of the second object part in the first candidate object posture, and determine a key point position of the third object part according to the first part length and the first part direction;

[0039] The first key point adding subunit is used to add the key point position of the third object part to the first candidate object posture to obtain the global posture corresponding to the target object in the target image frame.

[0040] Among them, the first key point adding subunit is specifically used for:

[0041] Adding the key point position of the third object part to the first candidate object posture, obtaining the second candidate object posture corresponding to the target object in the target image frame, and obtaining the posture offset between the standard posture and the second candidate object posture;

[0042] If the posture offset is greater than the offset threshold, key point correction is performed on the posture of the second candidate object based on the standard posture to obtain a global posture corresponding to the target object in the target image frame.

[0043] Wherein, the target image frame is the i-th image frame in the video data, and i is a positive integer;

[0044] The second interpolation processing unit includes:

[0045] A second direction determination subunit is used to determine a second part direction corresponding to the second object part and a third part direction corresponding to the third object part according to the key point positions of the first object part included in the part posture detection result if the object posture detection result does not include the postures of the second object part and the third object part; the second object part and the third object part are symmetrical parts of the target object;

[0046] A second position determination subunit is used to obtain, in the (i-1)th image frame, a second part length corresponding to the second object part and a third part length corresponding to the third object part, and determine the key point position of the second object part according to the second part length and the second part direction;

[0047] The second key point adding subunit is used to determine the key point position of the third object part according to the length and direction of the third part, add the key point position of the second object part and the key point position of the third object part to the first candidate object posture, and obtain the global posture corresponding to the target object in the target image frame.

[0048] The device further comprises:

[0049] The virtual object construction module is used to construct a virtual object associated with a target object according to a global posture, and play the virtual object with the global posture in a multimedia application.

[0050] On one hand, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory is connected to the processor, the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the method provided in the above aspect of the embodiment of the present application.

[0051] On one hand, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. The computer program is suitable for being loaded and executed by a processor, so that a computer device with a processor executes the method provided in the above aspect of the embodiment of the present application.

[0052] According to one aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the method provided in the above aspect.

[0053] In an embodiment of the present application, by performing object posture estimation and specific part posture estimation on the target object in the target image frame respectively, an object posture detection result for the target object and a part posture detection result for the first object part of the target object can be obtained. Then, based on the object posture detection result, the part posture detection result and the standard posture, the posture of the target object in the target image frame can be estimated, and the missing part key points of the target object in the target image frame can be compensated, so as to ensure the integrity and rationality of the global posture of the target object finally obtained, and thus the estimation accuracy of the global posture can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1 It is a structural diagram of a network architecture provided by an embodiment of the present application;

[0056] Figure 2 This is a schematic diagram of an object posture estimation scene for video data provided by an embodiment of the present application;

[0057] Figure 3 It is a flowchart of a data processing method provided in an embodiment of the present application;

[0058] Figure 4 is a schematic diagram of a standard posture provided in an embodiment of the present application;

[0059] Figure 5 is a schematic diagram of a scene for object posture estimation provided by an embodiment of the present application;

[0060] Figure 6 It is a schematic diagram of an application scenario of a global posture provided by an embodiment of the present application;

[0061] Figure 7 It is a flowchart of another data processing method provided in an embodiment of the present application;

[0062] Figure 8 is a structural diagram of an object detection model provided in an embodiment of the present application;

[0063] Fig. 9 It is a schematic diagram of a process for obtaining an object posture detection result provided by an embodiment of the present application;

[0064] Fig.10 It is a schematic diagram of a process for obtaining a part posture detection result provided by an embodiment of the present application;

[0065] Fig.11 is a schematic diagram of correction of key points of an object provided in an embodiment of the present application;

[0066] Fig.12 is a schematic diagram of a process of object posture estimation provided by an embodiment of the present application;

[0067] Fig.13 is a structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0068] Fig.14 is a structural schematic diagram of another data processing device provided in an embodiment of the present application;

[0069] Fig.15 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0070] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0071] This application relates to computer vision technology (Computer Vision, CV). Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing to make the computer processing into an image that is more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data.

[0072] The present application relates to pose estimation under computer vision technology. Pose estimation is an important task in computer vision and an essential step for computers to understand the actions and behaviors of target objects. Pose estimation can be converted into a prediction problem for key points of an object, such as predicting the position coordinates of key points of each object in an image, and predicting the skeleton of the object in the image based on the positional relationship between the key points of each object. Among them, the pose estimation involved in the present application may include object pose estimation for the target object, and part pose estimation for specific parts of the target object, etc. The target object may include but is not limited to: human body, animal, plant, etc. The specific parts of the target object may be palm, face, animal limbs, plant roots, etc. The present application does not limit the type of the target object.

[0073] See also Figure 1 , Figure 1 Schematic diagram of a network architecture provided by an embodiment of the present application. Figure 1 As shown, the network architecture may include a server 10d and a user terminal cluster, and the user terminal cluster may include one or more user terminals, and the number of user terminals is not limited here. Figure 1 As shown, the user terminal cluster may specifically include a user terminal 10a, a user terminal 10b, and a user terminal 10c.

[0074] Among them, server 10d can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms.

[0075] The user terminal 10a, the user terminal 10b and the user terminal 10c can include: smart phones, tablet computers, laptop computers, PDAs, mobile internet devices (MID), wearable devices (such as smart watches, smart bracelets, etc.), smart voice interaction devices, smart home appliances (such as smart TVs, etc.), vehicle-mounted devices and other electronic devices with object posture estimation functions. Figure 1 As shown, the user terminal 10a, the user terminal 10b and the user terminal 10c can respectively establish a network connection with the server 10d, so that each user terminal can exchange data with the server 10d through the network connection.

[0076] like Figure 1 The user terminal in the user terminal cluster shown (for example, user terminal 10a) is integrated with an application client having an object posture estimation function, and the application client may include but is not limited to: a multimedia client (for example, a short video client, a live video client, a video client), an object management application (for example, a patient care client). The application client in the user terminal 10a can obtain video data, and the video data may refer to a video of a target object captured in a mobile terminal scenario, such as using a camera integrated in the user terminal 10a to capture the target object to obtain video data, or using a camera device connected to the user terminal 10a (for example, a SLR, a camera, etc.) to capture the target object to obtain video data. It should be noted that in mobile scenarios (for example, selfie scenarios), due to the limitations of shooting distance and shooting equipment, the picture in the video data may only contain a part of the target object. For example, when the target object is a human body, the picture in the video data may only contain the upper body of the human body, or only contain the head of the human body, etc.; when estimating the object posture of the target object in the video data, it is necessary to perform posture repair on the target object contained in the video data to obtain the global posture corresponding to the target line. While ensuring the integrity of the global posture of the target object, the accuracy of the global posture can also be improved.

[0077] It should be noted that the object posture estimation process involved in the embodiment of the present application can be performed by a computer device, which can be Figure 1The user terminals in the user terminal cluster shown, or Figure 1 The server 10d is shown; in short, the computer device can be a user terminal, or a server, or a combination of a server and a user terminal, which is not limited in this application.

[0078] See also Figure 2 , Figure 2 is a schematic diagram of an object posture estimation scene of video data provided by an embodiment of the present application. Figure 1 Taking the user terminal 10a shown in the figure as an example, the object posture estimation process in the target video is described; Figure 2 As shown, the user terminal 10a can obtain video data 20a, which can be a video of the target object shot by a camera integrated in the user terminal 10a, or a video about the target object transmitted to the user terminal 10a by other devices; by performing frame processing on the video data 20a, N image frames are obtained, where N is a positive integer, such as N can take the value of 1, 2,... The first image frame (i.e., image frame T1) can be obtained from the N image frames in chronological order, and the image frame T1 can be input into the object detection model 20b as the target image frame. The image frame T1 is subjected to object detection by the object detection model 20b to obtain the object posture detection result 20c corresponding to the image frame T1; the object posture detection result 20c can include the key points of the target object contained in the image frame T1 (for the convenience of description, the key points of the target object are referred to as object key points below), and the positions of these object key points in the image frame T1; the object posture detection result 20c can also include a first confidence level corresponding to each detected object key point, and the first confidence level can be used to characterize the prediction accuracy of the detected object key points. The larger the first confidence level, the more accurate the detected object key points are, and the more likely they are to be the true key points of the target object.

[0079] For example, when the target object in the video data 20a is a human body, the object key points corresponding to the target object can be considered as joints in the human body structure, and the number of key points and the key point categories of the target object can be predefined, such as the human body structure can include multiple object key points of the limbs, brain, waist, chest and other parts. When the image frame T1 contains a complete target object, the image frame T1 may contain all the object key points of the target object; when the image frame T1 contains only a partial structure of the target object, the image frame T1 may contain some object key points of the target object. After detecting the object key points contained in the image frame T1, the detected object key points can be connected according to the key point category and key point position of the target object, and the result of the connection is marked in the image frame T1, that is, the object posture detection result 20c. Among them, the object detection model 20b can be a pre-trained network model with a target object detection function for a video / image; when the target object is a human body, the object detection model 20b can also be called a human posture estimation model.

[0080] The human body posture 20j of the target object in the image frame T1 can be obtained through the object posture detection result 20c. Since the human body posture 20j lacks some object key points (lack of human body joint points), the user terminal 10a can obtain the standard posture 20k corresponding to the target object. Based on the standard posture 20k, the key point compensation can be performed on the human body posture 20j to obtain the human body posture 20m corresponding to the target object in the image frame T1. Among them, the standard posture 20k can also be considered as the default posture of the target object. The standard posture 20k can be pre-constructed based on all the object key points of the target object, such as the posture of the human body when standing normally can be determined as the standard posture 20k.

[0081] The image frame T1 can also be input into the part detection model 20d, and the specific part (for example, the first object part) of the target object in the image frame T1 is detected by the part detection model 20d to obtain the part posture detection result 20e corresponding to the image frame T1. When it is detected that the first object part of the target object does not exist in the image frame T1, it can be determined that the part posture detection result of the image frame T1 is empty; when it is detected that the first object part of the target object exists in the image frame T1, the key points of the first object part and the position of the key points can be continued to be detected, and according to the key point category and key point position of the first object part, the detected key points of the first object part can be connected, and the connected result is marked in the image frame T1, that is, the part posture detection result 20e. Among them, the number of key points and key point categories corresponding to the first object part of the target object can also be pre-defined; when the target object is a human body, the part detection model 20d can be a palm posture estimation model (here the first object part of the target object is the palm), such as the palm can include palm key points and finger key points; the part detection model 20d can be a pre-trained network model with object part detection function for video / image. For the convenience of description, the key points of the first object part are referred to as part key points below.

[0082] like Figure 2 As shown, the part posture detection result 20e carries the second confidence, which can be used to characterize the possibility that the detected object part is the first object part. For example, through the part detection model 20d, it can be determined that the second confidence of region 20f in the image frame T1 is 0.01 for the first object part, the second confidence of region 20g is 0.09 for the first object part, the second confidence of region 20h is 0.86 for the first object part, and the second confidence of region 20i is 0.84 for the first object part. The larger the second confidence, the greater the possibility that the region is the first object part. For example, based on the second confidence, it can be determined that the first object part is contained in region 20h and region 20i, and the posture of the first object part can be marked in region 20h and region 20i.

[0083] Further, the user terminal 10a can combine the object posture detection result 20c and the part posture detection result 20e to interpolate some of the key points of the target object. For example, when the part posture detection result 20e is a palm key point, the object posture detection result 20c and the part posture detection result 20e can be combined to interpolate the wrist, elbow and other parts of the target object to improve the human body posture 20m of the target object and obtain the human body posture 20n (also referred to as the global posture). Similarly, after obtaining the global posture corresponding to the target object in the image frame T1, the object posture estimation can be performed on the subsequent image frames in the video data 20a in the same manner to obtain the global posture corresponding to the target object in each image frame. Based on the global postures corresponding to the N image frames, the behavior of the target object in the video data 20a can be obtained. It can be understood that the video data 20a can also be a video shot in real time, and the user terminal 10a can perform object posture estimation on the image frames in the video data shot in real time to obtain the behavior of the target object in real time.

[0084] In summary, for image frames that only contain part of the target object, the global posture of the target object in the image frame can be estimated through the object detection results output by the object detection model 20b, the part detection results output by the part detection model 20d, and the standard posture 20m, which can ensure the integrity and rationality of the global posture of the target object finally obtained, and thus improve the estimation accuracy of the global posture.

[0085] See also Figure 3 , Figure 3 is a flow chart of a data processing method provided in an embodiment of the present application. Figure 3 As shown, the data processing method may include the following steps S101-S102:

[0086] Step S101, obtaining an object posture detection result and a part posture detection result corresponding to a target image frame; the object posture detection result is associated with a target object in the target image frame, and the part posture detection result is associated with a first object part of the target object.

[0087] Specifically, the computer device may obtain video data of the target object captured in the mobile terminal scene (for example, Figure 2 The video data 20a) or image data in the corresponding embodiment; when performing posture estimation on the video data or image data, the computer device can determine the image frame in the image data or video data as the target image frame, and by performing object detection on the target image frame, an object posture detection result for the target object can be obtained (for example, the above Figure 2At the same time, the target image frame can also be subjected to part detection to obtain a part posture detection result for the first object part of the target object (for example, the above Figure 2 The part posture detection result 20e in the corresponding embodiment). Among them, the target object may refer to an object contained in the video data, such as a human body, an animal, a plant, etc.; the first object part may refer to one or more parts of the target object, such as the face and palm in the human body structure, the limbs, tail, head in the animal structure, the root of the plant, etc., and the present application does not limit the type of the target object and the type of the first object part. It should be noted that, due to the distance between the shooting device and the target object being photographed in the mobile terminal scene, the target object in the video data or image data may be missing parts, that is, the target object may have a part of the object part that is not in the picture of the video data. By combining the object posture detection result and the part posture detection result, the accuracy of the posture estimation of the target object can be improved.

[0088] For the convenience of description, the embodiments of the present application all take the human body as an example to describe the object posture estimation process of video data or image data. If the object posture estimation is performed on the image data in the mobile terminal scene, the image data can be used as the target image frame; if the object posture estimation is performed on the video data in the mobile terminal scene, the video data can be processed by frame division to obtain N image frames corresponding to the video data, where N is a positive integer, and then an image frame sequence containing N image frames can be formed according to the time sequence of the N image frames in the video data, and the N image frames in the image frame sequence can be used as target image frames in sequence for object posture estimation; for example, the first image frame in the image frame sequence can be used as the target image frame, and after completing the object posture estimation of the first image frame, the second image frame can continue to be used as the target image frame until the object posture estimation of the entire video data is completed.

[0089] Among them, the computer device can obtain an object detection model and a part detection model, input the target image frame into the object detection model, and the object detection model can output the object posture detection result corresponding to the target image frame; at the same time, the target image frame can also be input into the part detection model, and the part detection model can output the part posture detection result corresponding to the target image frame. Among them, the object detection model can be used to detect the key points of the target object in the target image frame (such as human key points, which can also be called object key points). At this time, the object detection model can also be called a human posture estimation model; the object detection model can include but is not limited to: DensePose (real-time human posture recognition system, used to realize real-time posture recognition of dense crowds), OpenPose (a framework for real-time estimation of multi-person body, face and hand shapes), Realtime Multi-Person Pose Estimation (real-time multi-person posture estimation model), DeepPose (a posture estimation method based on deep neural network), mobilenetv2 (lightweight deep neural network), and this application does not limit the type of object detection model. The part detection model can be used to detect the key points of the first object part of the target object (such as the palm key points). At this time, the part detection model can also be called a palm posture estimation model; the part detection model can be a detection-based method or a regression-based method. The detection-based method can predict the part key points of the first object part by generating a heat map, and the regression-based method can directly regress the position coordinates of the part key points; the network structure of the part detection model can be the same as or different from the network structure of the object detection model. When the network structures of the part detection model and the object detection model are the same, the network parameters of the two are also different (obtained by training with different data). This application does not limit the type of part detection model.

[0090] Optionally, the object detection model and part detection model mentioned above may be detection models pre-trained using sample data, such as an object detection model may be trained using sample data carrying human body key point label information (such as a three-dimensional human body data set), and a part detection model may be trained using sample data carrying palm key point information (such as a palm data set); alternatively, the object detection model may be an object detection service called from an artificial intelligence cloud service through an application programming interface (API), and the part detection model may be a part detection service called from an artificial intelligence cloud service through an API interface, without specific limitation here.

[0091] Among them, AI cloud services are also generally referred to as AI as a Service (AIaaS). This is a mainstream service mode of AI platforms. Specifically, AIaaS platforms will split several common AI services and provide independent or packaged services in the cloud. This service model is similar to opening an AI theme mall: all developers can access and use one or more AI services provided by the platform through API interfaces. Some senior developers can also use the AI ​​framework and AI infrastructure provided by the platform to deploy and operate their own cloud AI services.

[0092] Optionally, the object detection model used in the embodiment of the present application can be a human body three-dimensional posture estimation model with confidence, such as the object detection model can predict the object key points of the target object in the target image frame, each predicted object key point can correspond to a first confidence, the first confidence can be used to characterize the prediction accuracy of each predicted object key point, the predicted object key point and the corresponding first confidence can be called the object posture detection result corresponding to the target image frame. The part detection model can be a palm three-dimensional posture estimation model with confidence, such as the part detection model can predict the position area of ​​the first object part in the target image frame, and predict the part key points of the first object part in the position area; the part detection model can predict one or more position areas that may be the first object part, a position area can correspond to a second confidence, the second confidence can be used to characterize the prediction accuracy of each predicted position area, the predicted part key points and the second confidence corresponding to the position area can be called the part posture detection result corresponding to the target image frame.

[0093] Step S102 , based on the object posture detection result, the part posture detection result, and the standard posture associated with the target object, the posture of the target object in the target image frame is estimated to obtain a global posture corresponding to the target object.

[0094] Specifically, the computer device may obtain a standard posture corresponding to the target object (for example, Figure 2 The standard posture in the corresponding embodiment is 20m), which can be considered as the complete default posture of the target object (recorded as T-pose); the number of standard postures can be one or more, such as the default standing posture of the human body, the default sitting posture of the human body, the default squatting posture of the human body, etc. This application does not limit the type and number of standard postures.

[0095] See also Figure 4 , Figure 4 Schematic diagram of a standard posture provided by the embodiment of the present application. Figure 4As shown, the model 30a can be represented as an SMPL (Skinned Multi-Person Linear) model. The model 30a is a parameterized human body model that can be applied to different human body structures. The model 30a can include human body joint distribution: 1 root node (node ​​with serial number 0) and 23 joint nodes (nodes represented by serial numbers 1 to 23), wherein the root node is used to transform the entire human body as a complete rigid body (an object whose volume and shape do not change under the action of force), and the 23 joint nodes can be used to describe the deformation of local human body parts. When the target object is a human body, the above 1 root node and 23 joint nodes can be used as object key points of the target object. Based on the category (e.g., wrist joint point, elbow joint point, palm joint point, ankle joint point, etc.) and position of the object key points, the 1 root node and the 23 joint nodes are connected to obtain a standard posture 30b.

[0096] The target image frame may not contain a complete target object. For example, if some parts of the target object (such as the lower limbs of the human body) are not in the target image frame, then a part of the object key points are missing in the object posture detection result corresponding to the target image frame. The object posture detection result corresponding to the target object can be compensated for key points through the standard posture, and the missing object key points can be improved to obtain the first candidate object posture corresponding to the target object. When the part posture detection result includes the part key points of the first object part, the part key points in the part posture detection result and the object key points in the object posture detection result can be combined to adjust the first candidate object posture to obtain the global posture of the target object in the target image frame. After obtaining the global posture corresponding to the current target image frame, the object posture estimation can be continued for the next image frame in the video data to obtain the global posture of the target object in each image frame of the video data.

[0097] Optionally, the computer device can determine the behavior of the target object according to the global posture of the target object in the video data, and the target object can be managed or cared for through these behavior actions, or human-computer interaction can be performed through the behavior of the target object. In short, the global posture of the target object in the video data can be applied to human-computer interaction scenarios (for example, virtual reality, human-computer animation, etc.), content review scenarios, automatic driving scenarios, virtual live broadcast scenarios, game or movie character action design scenarios, etc. In the human-computer interaction scenario, the image (or video) of the user (target object) can be collected. After obtaining the global posture in the image or video, the machine can be controlled based on the global posture, such as executing a specific instruction based on a specific human body action (determined by the global posture). In the game character action design scenario, the human body action is obtained by the global posture corresponding to the target object to replace the expensive motion capture equipment, which can reduce the cost and difficulty of game character action design.

[0098] Among them, the virtual live broadcast scene can refer to that the live broadcast screen of the live broadcast room does not directly play the video of the anchor user (target object), but plays the video of the virtual object with the same behavior as the anchor user in the live broadcast room. For example, the anchor user's behavior can be determined based on the anchor user's global posture, and then the virtual object can be driven by the anchor user's behavior, that is, a virtual object with the same behavior as the anchor user is constructed, and the virtual object is used for live broadcast, which can not only avoid the anchor user from appearing in the public eye, but also achieve the same live broadcast effect as the real anchor user. For example, a computer device can construct a virtual object associated with the target object according to the global posture of the target object in the video data, and play the virtual object with the global posture in a multimedia application (for example, a live broadcast room, a video website, a short video application, etc.), that is, a video about the virtual object can be played in the multimedia application, and the posture of the virtual object is synchronized with the posture of the target object in the video data. Among them, the global posture corresponding to the target object in the video data will be reflected in the virtual object played in the multimedia application. Every time the target object changes its posture, it will drive the virtual object in the multimedia application to change to the same posture (which can be considered as rebuilding a virtual object with a new posture, where the new posture is the posture of the target object after the change), so that the posture of the target object and the virtual object is always consistent.

[0099] See also Figure 5 , Figure 5 Schematic diagram of a scene of object posture estimation provided by an embodiment of the present application. Taking a virtual live broadcast scene as an example, the object posture estimation process of video data is described; Figure 5As shown, when the anchor user 40c (who can be the target object) needs to broadcast live, he can enter the broadcast room (such as the broadcast room with room number 116889). Before starting the live broadcast, the anchor user 40c can choose the real person live broadcast mode or the virtual live broadcast mode. If the anchor user 40c chooses the virtual live broadcast mode, the virtual object can be pulled. When the anchor user 40c starts the live broadcast, the behavior of the anchor user 40c can be used to drive the virtual object, so that the virtual object and the anchor user 40c maintain the same posture.

[0100] After starting the live broadcast, the anchor user 40c can collect his own video data through the user terminal 40a (for example, a smart phone). At this time, the anchor user 40a can be used as a target object, and the user terminal 40a can be fixed using a bracket 40b. After the user terminal 40a collects the video data of the anchor user 40c, it can obtain the image frame 40g (target image frame) from the video data, and input the image frame 40g into the object detection model and the part detection model respectively. The object detection model can predict the part joint points (object key points) of the anchor user 40c contained in the image frame 40g, and these predicted part joint points can be used as the object posture detection result of the image frame 40g; the part detection model can predict the palm key points of the anchor user 40c contained in the image frame 40g (here the first object part is the palm by default, and the palm key points can also be called part key points), and these predicted palm key points can be used as the part posture detection result of the image frame 40g; the object posture detection result and the part posture detection result here can be marked in the image frame 40g (as shown in image 40h), wherein area 40i and area 40j in image 40h represent the above-mentioned part posture detection results.

[0101] like Figure 5 As shown, through the object posture detection results and part posture detection results shown in image 40h, the human body posture 40k of anchor user 40c in image frame 40g can be obtained; obviously, since image frame 40g only contains the upper body of anchor user 40c, human body posture 40k is not the complete human body posture of anchor user 40c. In this case, a standard posture (default human body posture) can be obtained, and joint point interpolation is performed on human body posture 40k through the standard posture to complete the missing part joint points in human body posture 40k, and obtain the overall human body posture 40m (global posture) for anchor user 40c.

[0102] The virtual object in the live broadcast room can be driven by the overall human posture 40m, so that the virtual object 40m in the live broadcast room has the same overall human posture 40k as the anchor user 40c. For users who enter the live broadcast room to watch the live broadcast, the display page of the live broadcast room where the virtual object is located can be displayed in the user terminal 40d used by them. The display page of the live broadcast room may include area 40e and area 40f. Area 40e can be used to play the video of the virtual object (with the same posture as the anchor user 40c), and area 40f can be used to publish barrages, etc. In the virtual live broadcast scene, users who enter the live broadcast room to watch the live broadcast can only see the video of the virtual object and the voice data of the anchor user 40c, but cannot see the video data of the anchor user 40c. This can protect the personal information of the anchor user 40c and achieve the same live broadcast effect as the anchor user 40c through the virtual object.

[0103] Optionally, the global posture of the above-mentioned target object in the video data can be applied in a content review scenario. When the global posture is the same as the target posture in the content review system, the review result of the target object in the content review system can be determined as a passed review result, and access rights to the content review system can be set for the target object; after the global posture passes the review in the content review system, the target object can have the authority to access the content review system.

[0104] See also Figure 6 , Figure 6 Schematic diagram of an application scenario of a global posture provided by an embodiment of the present application. Figure 6 As shown, user A (target object) can send a verification request to server 50d through user terminal 50a. After receiving the verification request sent by user terminal 50a, server 50d can obtain the identity verification method for user A and return the identity verification method to user terminal 50a. Verification box 50b can be displayed on the terminal screen of user terminal 50a. User A can face the verification box 50b in user terminal 50a and make specific actions (for example, raising hands, kicking legs, putting hands on waist, etc.). User terminal 50a can collect the image 50c to be verified in the verification box 50b in real time (which can be considered as the above-mentioned target image frame), and send the collected image 50c to be verified to server 50d.

[0105] The server 50d can obtain the image 50c to be verified sent by the user terminal 50a, and obtain the target posture 50e set by the user A in the content audit system in advance, and the target posture 50e can be used as the verification information of the user A in the content audit system. The server 50d can use the object detection model, the part detection model and the standard posture to estimate the posture of the image 50c to be verified, and obtain the global posture of the user A in the image 50c to be verified; compare the global posture corresponding to the image 50c to be verified with the target posture 50e for similarity. When the similarity between the global posture of the image 50c to be verified and the target posture 50e is greater than or equal to the similarity threshold (for example, the similarity threshold can be set to 90%), it can be determined that the global posture of the image 50c to be verified is the same as the target posture 50e, and the user A has passed the audit in the content audit system. When the similarity between the global posture of the image to be verified 50c and the target posture 50e is less than the similarity threshold, it can be determined that the global posture of the image to be verified 50c is different from the target posture 50e, and the user A has failed the review in the content review system, and an action error prompt message is returned to the user terminal 50a. The action error prompt message is used to prompt user A to redo the action for identity review.

[0106] In an embodiment of the present application, by performing object posture estimation and specific part posture estimation on the target object in the target image frame respectively, an object posture detection result for the target object and a part posture detection result for the first object part of the target object can be obtained. Then, based on the object posture detection result, the part posture detection result and the standard posture, the posture of the target object in the target image frame can be estimated, and the missing part key points of the target object in the target image frame can be compensated, so as to ensure the integrity and rationality of the global posture of the target object finally obtained, and thus the estimation accuracy of the global posture can be improved.

[0107] See also Figure 7 , Figure 7 FIG. 1 is a flow chart of another data processing method provided in an embodiment of the present application. Figure 7 As shown, the data processing method may include the following steps S201 to S208:

[0108] Step S201, input the target image frame into the object detection model, obtain the object posture features corresponding to the target object in the target image frame through the object detection model, and identify the first classification result corresponding to the object posture features; the first classification result is used to characterize the object part category corresponding to the key points of the target object.

[0109] Specifically, after the computer device obtains the video data shot in the mobile terminal scene, it can select an image frame from the video data as the target image frame, and input the target image frame into the trained object detection model. The object detection model can obtain the object posture feature corresponding to the target object in the target image frame, and the classifier of the object detection model can output the first classification result corresponding to the object posture feature, which can be used to characterize the object part category corresponding to the key point (e.g., human body joint) of the target object. The above-mentioned object posture feature can be an object description feature for the target object extracted by the object detection model, or it can be a fusion feature between the object description feature and the part description feature corresponding to the target object. When the object posture feature is the object description feature corresponding to the target object in the target image frame, it indicates that part-aware block learning is not introduced in the process of extracting features from the target image frame using the object detection model; when the object posture feature is the fusion feature between the object description feature and the part description feature corresponding to the target object in the target image frame, it indicates that part-aware block learning is introduced in the process of extracting features from the target image frame using the object detection model; by introducing part-aware block learning, the object posture feature can include both the local posture features (part description features) of each part of the target object contained in the target image frame and the object description features of the target object contained in the target object, which can enhance the fine-grainedness of the object posture feature and thus improve the accuracy of the object posture detection result.

[0110] Optionally, if part-aware block learning is introduced in the process of extracting features from the target image frame using the object detection model, the computer device can input the target image frame into the object detection model, obtain the object description feature corresponding to the target object in the target image frame in the object detection model, and output a second classification result corresponding to the object description feature according to the classifier in the object detection model; obtain the object convolution feature for the target image frame output by the target convolution layer in the object detection model, multiply the second classification result and the object convolution feature to obtain a second activation map corresponding to the target image frame; block-process the target image frame according to the second activation map to obtain M object part area images, and obtain the part description features corresponding to the M object part area images respectively according to the object detection model, where M is a positive integer; and combine the object description feature and the part description features corresponding to the M object part area images into an object posture feature.

[0111] Among them, the object description feature can be considered as a feature representation extracted from the target image frame for characterizing the target object; the second classification result can also be used to represent the object part category corresponding to the object key point contained in the target image frame; the target convolution layer can refer to the last convolution layer in the object detection model, and the object convolution feature can represent the convolution feature for the target image frame output by the last convolution layer of the object detection model; the second activation map can be the class activation map (Class Activation Mapping, CAM) corresponding to the target image frame, and CAM is a tool for visualizing image features. The object convolution feature output by the last convolution layer in the object detection model and the second classification result are weighted (the second classification result can be considered as the weight corresponding to the object convolution feature), and the second activation map can be obtained. The second activation map can be considered as the result of visualizing the object convolution feature output by the target convolution layer, and can be used to characterize the image pixel area concerned by the object detection model.

[0112] The computer device can use the class activation map (second activation map) of each object key point in the target image frame as prior information of the region position, and perform block processing on the target image frame, that is, crop the target image frame according to the second activation map to obtain an object part region image containing a single part; then, feature extraction can be performed on each object part region image through the object detection model to obtain the part description features corresponding to each object part region image, and the aforementioned object description features and the part description features corresponding to each object part region image can be combined into an object posture feature for the target object; the part description feature can be considered as a feature representation extracted from the object part region image for characterizing the object part.

[0113] Step S202: Generate a first activation map according to the first classification result and the object convolution feature of the target image frame.

[0114] Specifically, after obtaining the first classification result, the computer device may multiply the first classification result and the object convolution feature of the target image frame to generate a first activation map. The first activation map and the second activation map are both class activation maps for the target image frame, except that the first activation map uses the first classification result as the weight of the object convolution feature output by the target convolution layer (here it is assumed that the first classification result combines the object description feature and the part description feature), while the second activation map uses the second classification result as the weight of the object convolution feature output by the target convolution layer, and the second classification result is only related to the object description feature.

[0115] Step S203, obtaining the pixel average value corresponding to the first activation map, determining the positioning result of the key point in the target object in the target image frame according to the pixel average value, and determining the object posture detection result corresponding to the target image frame according to the object part category and the positioning result.

[0116] Specifically, the computer device can take the pixel average value of the first activation map, and determine the pixel average value as the positioning result of the key point in the target object in the target image frame. According to the object part category and the positioning result, the object skeleton of the target object in the target image frame can be determined, and the object skeleton can be used as the object posture detection result corresponding to the target object in the target image frame.

[0117] See also Figure 8 , Figure 8 Schematic diagram of the structure of an object detection model provided in an embodiment of the present application. Figure 8 As shown, after acquiring the target image frame 60a, the computer device can input the target image frame 60a into the object detection model, and perform feature extraction on the target image frame 60a through the feature extraction component 60b in the object detection model (for example, the feature extraction network can be a convolutional network), so as to obtain the object description feature 60c corresponding to the target object in the target image frame 60a, and process the object description feature 60c by using global average pooling (the number of object description features can be multiple, and global average pooling refers to converting an object description feature into a numerical value) and an activation function, and classify the processed result to obtain a second classification result; the second classification result is weighted with the object convolution feature output by the last convolutional layer in the feature extraction component 60b to obtain a second activation map.

[0118] The target image frame 60a is processed into blocks based on the second activation map to obtain M object part region images 60f. The M object part region images 60f are sequentially input into the feature extraction component 60b in the object detection model. The part description features 60g corresponding to the M object part region images 60f can be obtained through the feature extraction component 60b. The M part description features 60g are combined with the object description features 60c of the target image frame 60a to obtain the object posture features; by identifying the object posture features, a first classification result 60d can be obtained. The first classification result 60d and the object convolution features output by the last convolution layer in the feature extraction component 60b are weighted to obtain a first activation map 60e. The pixel average value of the first activation map 60e can be used as the positioning result of the target object in the target image frame 60a, and the object posture detection result corresponding to the target object in the target image frame 60a can be obtained.

[0119] It should be noted that Figure 8 The method for obtaining the object posture detection result described in the corresponding embodiment is only an example of the embodiment of the present application. The present application can also use other methods to obtain the object posture detection result, and the present application does not limit this.

[0120] See also Fig. 9 , Fig. 9 FIG. 1 is a flow chart of obtaining an object posture detection result provided by an embodiment of the present application. Fig. 9 As shown, taking the object detection model as a human body 3D posture estimation model as an example, the computer device can input the target image frame 70a into the human body 3D posture estimation model, and the human body 3D key points of the target object (the target object in this case is a human body) in the target image frame 70a can be obtained through the human body 3D posture estimation model. Fig. 9 As shown, if the 16 human 3D key points contained in the target image frame 70a are detected by the human 3D posture estimation model, they are marked as x1 to x16 respectively, and each human 3D key point can correspond to a position coordinate and a first confidence level. Based on the first confidence level, the possibility that the detected human 3D key point is a real human key point can be determined; if the human 3D key point with the first confidence level is greater than the first confidence threshold (which can be set according to actual needs), it can be considered as a real human key point (for example, the human 3D key point represented by x4 to x16). By connecting the real human key points, the human posture 70c (which can also be considered as the object posture detection result) can be obtained. The human 3D key points with the first confidence level less than or equal to the first confidence threshold are abnormal key points, and these abnormal key points can be compensated in subsequent processing to obtain more accurate human key points.

[0121] It can be understood that a spatial coordinate system is constructed using the target image frame, and the position coordinates of the three-dimensional key points of the human body may refer to the spatial coordinates in the spatial coordinate system.

[0122] Step S204: input the target image frame into a part detection model, and detect a first object part of the target object in the target image frame in the part detection model.

[0123] Specifically, the computer device may also input the target image frame into a part detection model, in which the first part detection model detects whether the target image frame contains the first object part of the target object. The part detection model may be used to detect key points of the first object part, so it is necessary to detect the first object part in the target image frame. If the first object part of the target object is not detected in the target image frame, it may be directly determined that the part posture detection result corresponding to the target image frame is a null value, without the need to perform the subsequent step of detecting key points of the first object part.

[0124] Step S205, if the first object part is detected in the target image frame, obtain a regional image containing the first object part from the target image frame, obtain the position of the part key point corresponding to the first object part according to the regional image, and determine the part posture detection result corresponding to the target image frame based on the position of the part key point.

[0125] Specifically, if the first object part is detected in the target image frame, the position area of ​​the first object part in the target image frame can be determined, and based on the position area of ​​the first object part in the target image frame, the target image frame is cropped to obtain a regional image containing the first object part. In the part detection model, feature extraction can be performed on the regional image to obtain the part contour feature corresponding to the first object part in the regional image, and the position of the part key point corresponding to the first object part can be predicted based on the part contour feature; based on the position of the part key point, the key points of the first object part can be connected to obtain the part posture detection result corresponding to the target image frame.

[0126] See also Fig.10 , Fig.10 FIG. 1 is a flow chart of obtaining a part posture detection result provided by an embodiment of the present application. Fig.10 As shown, taking the part detection model as a palm 3D posture estimation model as an example, the computer device can input the target image frame 80a into the palm 3D posture estimation model, in which it can be detected whether the target image frame 80a contains the palm (first object part) of the target object. If the palm is not detected in the target image frame 80a, it can be determined that the part posture detection result corresponding to the target image frame 80a is a null value; if the palm is detected in the target image frame 80a, the area containing the palm can be determined in the target image frame 80a (for example, area 80c and area 80d in image 80b, area 80c contains the right palm of the target object, and area 80d contains the left palm of the target object), and the palm 3D posture estimation model can be used to detect the palm 3D key points in area 80c and the palm 3D key points in area 80d.

[0127] Among them, multiple possible areas can be obtained through the palm three-dimensional posture estimation model, and the second confidence level of the palm contained in each possible area is predicted, and the area with a second confidence level greater than a second confidence threshold (which can be the same as the aforementioned first confidence threshold or different, and is not limited here) is determined as the area containing the palm, such as the second confidence levels corresponding to area 80c and area 80d are both greater than the second confidence threshold. The right palm posture 80e can be obtained by connecting the palm key points detected in area 80c, and the left palm posture 80f can be obtained by connecting the palm key points detected in area 80d. The above-mentioned left palm posture 80f and right palm posture 80e can be called the part posture detection results corresponding to the target image frame 80a.

[0128] Step S206, obtaining a standard posture associated with the target object, and counting the number of first key points corresponding to the standard posture and the number of second key points corresponding to the object posture detection result.

[0129] Specifically, the computer device can obtain the standard posture corresponding to the target object, and count the first key point number of the object key points contained in the standard posture, and the second key point number of the object key points contained in the object posture detection result. The first key point number is known when constructing the standard posture, and the second key point number is the number of object key points predicted by the object detection model.

[0130] Step S207: when the number of the first key points is greater than the number of the second key points, interpolation processing is performed on the object posture detection result according to the standard posture to obtain a first candidate object posture.

[0131] Specifically, when the number of the first key points is greater than the number of the second key points, it indicates that there are missing object key points in the object posture detection result, and the object posture detection result can be key-point compensated (interpolated) by the standard posture to improve the missing object key points, so as to obtain the first candidate object posture corresponding to the target object. Figure 2 As shown, by performing key point compensation on the human body posture 20j (object posture detection result) through the standard posture 20k, the human body posture 20m can be obtained. At this time, the human body posture 20m can be called the first candidate object posture.

[0132] For example, assuming that the target object is a human body, if the object posture detection result predicted by the object detection model is missing key points of the knees, ankles, feet, elbows, etc., the above object posture detection result can be interpolated through the standard posture, such as adding the missing object key points, to obtain a more reasonable first candidate object posture. Interpolating the object posture detection result through the standard posture can improve the integrity and rationality of the object posture.

[0133] Step S208 , interpolating the posture of the first candidate object according to the part posture detection result to obtain a global posture corresponding to the target object.

[0134] Specifically, in actual application scenarios, the change in the posture of the target object depends largely on a few parts of the target object, that is, some specific parts of the target object (for example, the arm part in the human body structure, the arm part can include key points of the palm, wrist and elbow, etc.) play an important role in the final result; therefore, in the embodiment of the present application, the object part associated with the first object part in the first candidate object posture can be interpolated based on the part posture detection result, and the global posture corresponding to the target object can be obtained. Optionally, if the part posture detection result is a null value (that is, the first object part is not included in the target image frame), the first candidate object posture can be directly determined as the global posture corresponding to the target object.

[0135] For example, assuming that the target object is a human body and the first object part is a palm; when the target image frame contains an elbow part, the key points for the elbow part can be predicted by the object detection model. When the target image frame does not contain an elbow part, the key points of the elbow part cannot be predicted by the object detection model. At this time, the elbow key points and wrist key points of the target object can be determined based on the part posture detection result, and the elbow key points and wrist key points can be added to the first candidate object posture to obtain the global posture corresponding to the target object.

[0136] Optionally, the target object includes a second object part and a third object part, and the second object part and the third object part are symmetrical. For example, if the second object part is the right arm of the target object, the third object part is the left arm of the target object; if the second object part is the left leg of the target object, the third object part is the right leg of the target object, and so on.

[0137] In the case where the part posture detection result includes all the part key points of the first object part (if the first object part is the palm, it is assumed here that the part posture detection result includes the left and right palm key points), if the object posture detection result includes the posture of the second object part, and the object posture detection result does not include the posture of the third object part, that is, the target image frame includes the second object part but does not include the third object part, then the first part direction corresponding to the third object part can be determined according to the key point position of the first object part included in the part posture detection result; wherein the second object part and the third object part are symmetrical parts of the target object. Since the second object part and the third object part are symmetrical parts, the length of the second object part is the same as the length of the third object part, so the first part length of the second object part in the first candidate object posture can be obtained, and the key point position of the third object part can be determined according to the first part length and the first part direction; and the key point position of the third object part is added to the first candidate object posture to obtain the global posture corresponding to the target object in the target image frame.

[0138] If the object posture detection result does not include the posture of the second object part and the third object part, that is, the target image frame does not include the second object part or the third object part, then the second part direction corresponding to the second object part and the third part direction corresponding to the third object part can be determined according to the key point position of the first object part included in the part posture detection result; and then the second part length corresponding to the second object part and the third part length corresponding to the third object part can be obtained in the i-1th image frame; in other words, the length of the second object part in the previous image frame can be used as the length of the second object part in the target image frame, and the image of the third object part in the previous image frame can be used as the length of the third object part in the target image frame. That is, the key point position of the second object part can be determined according to the second part length and the second part direction; the key point position of the third object part can be determined according to the third part length and the third part direction, and the key point position of the second object part and the key point position of the third object part are added to the first candidate object posture to obtain the global posture corresponding to the target object in the target image frame. If the i-1th image frame also contains the second object part and the third object part, you can continue to trace back and obtain the lengths of the second object part and the third object part in the i-2th image frame to determine the key point positions of the second object part and the third object part in the target image frame. If the second object part and the third object part are not detected in the image frame before the target image frame, you can set an approximate length for the second object part and the third object part according to the first candidate object posture to determine the key point positions of the second object part and the third object part in the target image frame.

[0139] For example, assuming that the target object is a human body, the first object part is the palm, and the second object part and the third object part are the left and right arms respectively; under the premise that the left and right palms are detected in the target image frame, the direction of the left forearm can be calculated by the key points of the left palm, and the direction of the right forearm can be calculated by the key points of the right palm. The left forearm is part of the left arm, and the right forearm is part of the right arm.

[0140] If the left and right arms are not detected in the target image frame, the lengths of the left and right forearms (the length of the second part and the length of the second part) in the image frame before the target image frame (for example, the i-1th image frame) can be used as the lengths of the left and right forearms in the target image frame. If the left and right arms are not detected in the target image frame and the image frames before it, the reference lengths of the left and right forearms in the target image frame can be assigned with reference to the shoulder length in the target image frame. If any one of the left and right arms (for example, the left arm) is detected in the target image frame, the length of the left forearm (the length of the first part) can be directly assigned to the right forearm. For example, the right wrist point A, the right palm point B, and the right elbow point C are known to be missing; the direction of the right forearm can be expressed as the direction from the right palm point B to the right wrist point A, which can be recorded as vector BA; the length of the left forearm can be expressed as the length from the right wrist point A to the right elbow point C, which can be recorded as L; the position coordinates of the right elbow point C can be calculated through the above information, which can be expressed as: C=A+BA_normal*L, where C represents the position coordinates of the elbow point, A represents the position coordinates of the wrist point, and BA_normal represents the unit vector of vector BA.

[0141] It can be understood that if the left and right arms are detected in the target image frame, the elbow points predicted by the object detection model can be adjusted and updated based on the detected palm key points, which can improve the accuracy of the elbow points and thus improve the rationality of the global posture.

[0142] Optionally, the posture obtained after interpolation processing of the first candidate object posture based on the part posture detection result may have some unreasonable object key points, so the unreasonable object key points can be corrected in combination with the standard posture to obtain the final global posture of the target object. Specifically, assuming that the third object part is not detected in the target image frame, the computer device can determine the first candidate object posture with the key point position of the third object part added as the second candidate object posture; and then the posture offset between the standard posture and the second candidate object posture can be obtained. If the posture offset is greater than the offset threshold (which can be understood as the maximum angle that the target object can offset under normal circumstances), the second candidate object posture is corrected for key points based on the standard posture to obtain the global posture corresponding to the target object in the target image frame. Among them, the above-mentioned posture offset can be understood as the relevant angle between the second candidate object posture and the standard posture. For example, when the target object is a human body, the posture offset can be the angle between the shoulder of the second candidate object posture and the shoulder of the standard posture.

[0143] See also Fig.11 , Fig.11 is a schematic diagram of correction of key points of an object provided in an embodiment of the present application. Fig.11 As shown, for the target image frame 90a, after obtaining the second candidate object posture corresponding to the target image frame 90a, a human body model 90b can be constructed based on the second candidate object posture; limited by the performance of the object detection model, the region 90c (e.g., shoulder region) in the human body model 90b obviously has a collapse problem compared to the normal human body structure (e.g., standard posture), such as the angle between the shoulder of the first candidate object posture and the shoulder of the standard posture is greater than the offset threshold. The computer device can correct the human body model 90c through the standard posture to obtain a human body model 90d; the region 90e in the human body model 90d can be considered as the result after the correction of region 90c, and the human body posture corresponding to the human body model 90d can be called the global posture corresponding to the target object in the target image frame 90a.

[0144] The video data shot in the mobile scenario usually cannot contain the entire target object. The posture of the target object predicted by the object detection model is incomplete. The rationality of the global posture can be improved through key point interpolation, key point correction and other processing; the position of the object key point associated with the first object part can be calculated through the part posture detection result, which can improve the accuracy of the global posture.

[0145] See also Fig.12 , Fig.12 is a flow chart of an object posture estimation process provided by an embodiment of the present application. Fig.12As shown, assuming that the target object is a human body, after the computer device obtains the video data or image data shot in the mobile terminal scene, it can obtain a human body 3D posture estimation model (object detection model) with confidence and a palm 3D posture estimation model (part detection model) with confidence. The human body 3D posture estimation model can predict the human body 3D key points in any image frame (target image frame), and these human body 3D key points can constitute the object posture detection result; the palm 3D posture estimation model can predict the palm 3D key points in any image frame (target image frame), and these palm 3D key points can constitute the part posture detection result. According to the default posture (standard posture) of the human body, the human body 3D key points predicted by the human body 3D posture estimation model can be interpolated to improve the missing human body key points; the palm 3D key points and the human body 3D key points can also be combined to interpolate the elbows and wrists of the human body (target object) to obtain the candidate human body posture (the above-mentioned second candidate object posture). If human key points that do not conform to the normal human body structure are detected in the candidate human body posture (human key points whose posture offset is greater than the offset threshold), these human key points that do not conform to the normal human body structure can be corrected, and finally a reasonable three-dimensional posture estimation result (i.e. the above-mentioned global posture) is obtained.

[0146] It is understandable that in the specific implementation of this application, user video capture may be involved. When the above embodiments of this application are applied to specific products or technologies, the user's permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0147] In an embodiment of the present application, by performing object posture estimation and specific part posture estimation on the target object in the target image frame respectively, an object posture detection result for the target object and a part posture detection result for the first object part of the target object can be obtained. Then, based on the object posture detection result, the part posture detection result and the standard posture, the posture of the target object in the target image frame can be estimated. The missing part key points of the target object in the target image frame can be compensated, and the object key points that do not conform to the standard posture can be corrected. This can ensure the integrity and rationality of the global posture of the target object finally obtained, and thus improve the estimation accuracy of the global posture.

[0148] See also Fig.13 , Fig.13 Schematic diagram of a data processing device provided in an embodiment of the present application. Fig.13 As shown, the data processing device 1 may include: a posture detection module 11, a posture estimation module 12;

[0149] A posture detection module 11 is used to obtain an object posture detection result and a part posture detection result corresponding to a target image frame; the object posture detection result is associated with a target object in the target image frame, and the part posture detection result is associated with a first object part of the target object;

[0150] The posture estimation module 12 is used to estimate the posture of the target object in the target image frame according to the object posture detection result, the part posture detection result, and the standard posture associated with the target object, so as to obtain the global posture corresponding to the target object.

[0151] The specific functional implementation of the posture detection module 11 and the posture estimation module 12 can be found in the above Figure 3 The description of step S101 - step S102 in the corresponding embodiment will not be repeated here.

[0152] In an embodiment of the present application, by performing global object posture estimation and specific part posture estimation on the target object in the target image frame respectively, an object posture detection result for the target object and a part posture detection result for the first object part of the target object can be obtained. Then, based on the object posture detection result, the part posture detection result and the standard posture, the posture of the target object in the target image frame can be estimated, and the missing part key points of the target object in the target image frame can be compensated, so as to ensure the integrity and rationality of the global posture of the target object finally obtained, and thus improve the estimation accuracy of the global posture.

[0153] See also Fig.14 , Fig.14 Schematic diagram of another data processing device provided in an embodiment of the present application. Fig.14 As shown, the data processing device 2 includes: a posture detection module 21, a posture estimation module 22, and a virtual object construction module 23;

[0154] A posture detection module 21 is used to obtain an object posture detection result and a part posture detection result corresponding to a target image frame; the object posture detection result is associated with a target object in the target image frame, and the part posture detection result is associated with a first object part of the target object;

[0155] The posture estimation module 22 is used to estimate the posture of the target object in the target image frame according to the object posture detection result, the part posture detection result, and the standard posture associated with the target object, so as to obtain the global posture corresponding to the target object.

[0156] The virtual object construction module 23 is used to construct a virtual object associated with the target object according to the global gesture, and play the virtual object with the global gesture in the multimedia application.

[0157] Among them, the specific functional implementation methods of the posture detection module 21, the posture estimation module 22, and the virtual object construction module 23 can refer to the description of the aforementioned related steps, which will not be repeated here.

[0158] In one or more embodiments, the posture detection module 21 includes: an object detection unit 211, a part detection unit 212;

[0159] The object detection unit 211 is used to obtain a target image frame from the video data, input the target image frame into the object detection model, and output the object posture detection result corresponding to the target image frame through the object detection model;

[0160] The part detection unit 212 is used to input the target image frame into the part detection model, and output the part posture detection result corresponding to the target image frame through the part detection model.

[0161] The specific functional implementation of the object detection unit 211 and the part detection unit 212 can be found in Figure 3 The step S101 in the corresponding embodiment will not be described in detail here.

[0162] In one or more embodiments, the object detection unit 211 may include: a part classification subunit 2111, a part map generation subunit 2112, a positioning result determination subunit 2113, and a detection result determination subunit 2114;

[0163] The part classification subunit 2111 is used to input the target image frame into the object detection model, obtain the object posture feature corresponding to the target object in the target image frame through the object detection model, and identify the first classification result corresponding to the object posture feature; the first classification result is used to characterize the object part category corresponding to the key point of the target object;

[0164] A part map generation subunit 2112 is used to generate a first activation map according to the first classification result and the object convolution feature of the target image frame;

[0165] A positioning result determination subunit 2113 is used to obtain a pixel average value corresponding to the first activation map, and determine a positioning result of a key point in the target object in the target image frame according to the pixel average value;

[0166] The detection result determination subunit 2114 is used to determine the object posture detection result corresponding to the target image frame according to the object part category and the positioning result.

[0167] The specific functional implementation of the part classification subunit 2111, the part mapping generation subunit 2112, the positioning result determination subunit 2113, and the detection result determination subunit 2114 can be found in Figure 7Steps S201 to S203 in the corresponding embodiment will not be described in detail here.

[0168] In one or more embodiments, the part classification subunit 2111 includes: a global classification subunit 21111, a global map acquisition subunit 21112, a block processing subunit 21113, and a feature combination subunit 21114;

[0169] The global classification subunit 21111 is used to obtain an object description feature corresponding to the target object in the target image frame in the object detection model, and output a second classification result corresponding to the object description feature according to the classifier in the object detection model;

[0170] The global map acquisition subunit 21112 is used to obtain the object convolution feature for the target image frame output by the target convolution layer in the object detection model, and perform a product operation on the second classification result and the object convolution feature to obtain a second activation map corresponding to the target image frame;

[0171] The block processing subunit 21113 is used to perform block processing on the target image frame according to the second activation map to obtain M object part region images, and obtain part description features corresponding to the M object part region images according to the object detection model; M is a positive integer;

[0172] The feature combination subunit 21114 is used to combine the object description feature and the part description features corresponding to the M object part region images into an object posture feature.

[0173] The specific functional implementation of the global classification subunit 21111, the global mapping acquisition subunit 21112, the block processing subunit 21113, and the feature combination subunit 21114 can be found in Figure 7 The step S201 in the corresponding embodiment will not be described in detail here.

[0174] In one or more embodiments, the part detection unit 212 may include: an object part detection subunit 2121, a part posture estimation subunit 2122, and a null value determination subunit 2123;

[0175] The object part detection subunit 2121 is used to input the target image frame into the part detection model, and detect the first object part of the target object in the target image frame in the part detection model;

[0176] The part posture estimation subunit 2122 is used for obtaining a regional image containing the first object part from the target image frame if a first object part is detected in the target image frame, obtaining a position of a part key point corresponding to the first object part according to the regional image, and determining a part posture detection result corresponding to the target image frame based on the position of the part key point;

[0177] The null value determination subunit 2123 is used to determine that the part posture detection result corresponding to the target image frame is a null value if the first object part is not detected in the target image frame.

[0178] The specific functional implementation of the object part detection subunit 2121, the part posture estimation subunit 2122, and the null value determination subunit 2123 can be found in Figure 7 Steps S204 and S205 in the corresponding embodiment will not be described in detail here.

[0179] In one or more embodiments, the part posture estimation subunit 2122 may include: an image clipping subunit 21221, a part key point determination subunit 21222, and a part key point connection subunit 21223;

[0180] The image clipping subunit 21221 is used to clip the target image frame to obtain a regional image including the first object part if the first object part is detected in the target image frame;

[0181] The part key point determination subunit 21222 is used to obtain the part contour features corresponding to the regional image, and predict the position of the part key point corresponding to the first object part according to the part contour features;

[0182] The part key point connection subunit 21223 is used to connect the key points of the first object part based on the position of the part key points to obtain the part posture detection result corresponding to the target image frame.

[0183] The specific functional implementation of the image clipping subunit 21221, the part key point determination subunit 21222, and the part key point connection subunit 21223 can be found in Figure 7 Step S205 in the corresponding embodiment will not be described in detail here.

[0184] In one or more embodiments, the posture estimation module 22 includes: a key point quantity counting unit 221, a first interpolation processing unit 222, and a second interpolation processing unit 223;

[0185] A key point number counting unit 221 is used to obtain a standard posture associated with the target object, and count the number of first key points corresponding to the standard posture, and the number of second key points corresponding to the object posture detection result;

[0186] A first interpolation processing unit 222 is used to perform interpolation processing on the object posture detection result according to the standard posture to obtain a first candidate object posture when the number of the first key points is greater than the number of the second key points;

[0187] The second interpolation processing unit 223 is used to perform interpolation processing on the posture of the first candidate object according to the part posture detection result to obtain the global posture corresponding to the target object.

[0188] The specific functional implementation of the key point number counting unit 221, the first interpolation processing unit 222, and the second interpolation processing unit 223 can be found in Figure 7 Steps S206 to S208 in the corresponding embodiment will not be described in detail here.

[0189] In one or more embodiments, the second interpolation processing unit 223 may include: a first direction determining subunit 2231 , a first position determining subunit 2232 , and a first key point adding subunit 2233 ;

[0190] The first direction determination subunit 2231 is used to determine the first part direction corresponding to the third object part according to the key point positions of the first object part included in the part posture detection result if the object posture detection result includes the posture of the second object part and the object posture detection result does not include the posture of the third object part; the second object part and the third object part are symmetrical parts of the target object;

[0191] The first position determination subunit 2232 is used to obtain the first part length of the second object part in the first candidate object posture, and determine the key point position of the third object part according to the first part length and the first part direction;

[0192] The first key point adding subunit 2233 is used to add the key point position of the third object part to the first candidate object posture to obtain the global posture corresponding to the target object in the target image frame.

[0193] Optionally, the first key point adding subunit 2233 is specifically used for:

[0194] Adding the key point position of the third object part to the first candidate object posture, obtaining the second candidate object posture corresponding to the target object in the target image frame, and obtaining the posture offset between the standard posture and the second candidate object posture;

[0195] If the posture offset is greater than the offset threshold, key point correction is performed on the posture of the second candidate object based on the standard posture to obtain a global posture corresponding to the target object in the target image frame.

[0196] Optionally, the target image frame is the i-th image frame in the video data, i being a positive integer; the second interpolation processing unit 223 may further include: a second direction determining subunit 2234, a second position determining subunit 2235, and a second key point adding subunit 2236;

[0197] The second direction determining subunit 2234 is used to determine a second part direction corresponding to the second object part and a third part direction corresponding to the third object part according to the key point positions of the first object part included in the part posture detection result if the object posture detection result does not include the postures of the second object part and the third object part; the second object part and the third object part are symmetrical parts of the target object;

[0198] The second position determination subunit 2235 is used to obtain, in the (i-1)th image frame, a second part length corresponding to the second object part and a third part length corresponding to the third object part, and determine the key point position of the second object part according to the second part length and the second part direction;

[0199] The second key point adding subunit 2236 is used to determine the key point position of the third object part according to the length and the direction of the third part, and add the key point position of the second object part and the key point position of the third object part to the first candidate object posture to obtain the global posture corresponding to the target object in the target image frame.

[0200] The specific functional implementation of the first direction determining subunit 2231, the first position determining subunit 2232, the first key point adding subunit 2233, the second direction determining subunit 2234, the second position determining subunit 2235, and the second key point adding subunit 2236 can be found in Figure 7 Step S208 in the corresponding embodiment will not be described in detail here. When the first direction determination subunit 2231, the first position determination subunit 2232, and the first key point adding subunit 2233 are performing corresponding operations, the second direction determination subunit 2234, the second position determination subunit 2235, and the second key point adding subunit 2236 all suspend the execution of operations; when the second direction determination subunit 2234, the second position determination subunit 2235, and the second key point adding subunit 2236 are performing corresponding operations, the first direction determination subunit 2231, the first position determination subunit 2232, and the first key point adding subunit 2233 all suspend the execution of operations.

[0201] In an embodiment of the present application, by performing object posture estimation and specific part posture estimation on the target object in the target image frame respectively, an object posture detection result for the target object and a part posture detection result for the first object part of the target object can be obtained. Then, based on the object posture detection result, the part posture detection result and the standard posture, the posture of the target object in the target image frame can be estimated. The missing part key points of the target object in the target image frame can be compensated, and the object key points that do not conform to the standard posture can be corrected. This can ensure the integrity and rationality of the global posture of the target object finally obtained, and thus improve the estimation accuracy of the global posture.

[0202] See also Fig.15 , Fig.15 Schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.15 As shown, the computer device 1000 may be a user terminal, for example, Figure 1 The user terminal 10a in the corresponding embodiment may also be a server, for example, Figure 1 The server 10d in the corresponding embodiment will not be limited here. For ease of understanding, this application takes the computer device as a user terminal as an example. The computer device 1000 may include: a processor 1001, a network interface 1004 and a memory 1005. In addition, the computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. Among them, the user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or it may be a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 1005 may optionally also be at least one storage device located away from the aforementioned processor 1001. As Fig.15 As shown, the memory 1005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.

[0203] The network interface 1004 in the computer device 1000 can also provide a network communication function, and the optional user interface 1003 can also include a display screen (Display) and a keyboard (Keyboard). Fig.15In the computer device 1000 shown, the network interface 1004 can provide a network communication function; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve:

[0204] Acquire an object posture detection result and a part posture detection result corresponding to a target image frame; the object posture detection result is associated with a target object in the target image frame, and the part posture detection result is associated with a first object part of the target object;

[0205] According to the object posture detection result, the part posture detection result, and the standard posture associated with the target object, the posture of the target object in the target image frame is estimated to obtain the global posture corresponding to the target object.

[0206] It should be understood that the computer device 1000 described in the embodiment of the present application can execute the above Figure 3 , Figure 7 The description of the data processing method in any of the embodiments may also be performed as described above. Fig.13 The description of the data processing device 1 in the corresponding embodiment can also be performed as described above. Fig.14 The description of the data processing device 2 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here either.

[0207] In addition, it should be pointed out here that: the embodiment of the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program, and the computer program includes computer instructions. When the processor executes the computer instructions, it can execute the above Figure 3 , Figure 7 The description of the data processing method in any of the embodiments will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application. As an example, computer instructions can be deployed on a computing device for execution, or on multiple computing devices located at one location, or on multiple computing devices distributed at multiple locations and interconnected by a communication network. Multiple computing devices distributed at multiple locations and interconnected by a communication network can constitute a blockchain system.

[0208] In addition, it should be noted that: the embodiment of the present application also provides a computer program product or a computer program, which may include computer instructions, and the computer instructions may be stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor may execute the computer instructions, so that the computer device executes the above Figure 3 , Figure 7 The description of the data processing method in any of the embodiments will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the computer program product or computer program embodiment involved in the present application, please refer to the description of the method embodiment of the present application.

[0209] It should be noted that, for the above-mentioned various method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0210] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs.

[0211] The modules in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.

[0212] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0213] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A data processing method, characterized in that: include: Obtaining object posture detection results and part posture detection results corresponding to the target image frame; The object posture detection result is associated with a target object in the target image frame, and the part posture detection result is associated with a first object part of the target object; Acquire a standard posture associated with the target object, and count the number of first key points corresponding to the standard posture and the number of second key points corresponding to the object posture detection result; When the number of the first key points is greater than the number of the second key points, interpolating the object posture detection result according to the standard posture to obtain a first candidate object posture; The posture of the first candidate object is interpolated according to the part posture detection result to obtain a global posture corresponding to the target object.

2. The method according to claim 1, characterized in that The obtaining of the object posture detection result and the part posture detection result corresponding to the target image frame includes: Acquire the target image frame from the video data, input the target image frame into an object detection model, and output the object posture detection result corresponding to the target image frame through the object detection model; The target image frame is input into a part detection model, and the part detection model outputs a part posture detection result corresponding to the target image frame.

3. The method according to claim 2, characterized in that The step of inputting the target image frame into an object detection model and outputting an object posture detection result corresponding to the target image frame through the object detection model includes: Inputting the target image frame into an object detection model, obtaining the object posture feature corresponding to the target object in the target image frame through the object detection model, and identifying a first classification result corresponding to the object posture feature; the first classification result is used to characterize the object part category corresponding to the key point of the target object; generating a first activation map according to the first classification result and the object convolution feature of the target image frame; Obtaining a pixel average value corresponding to the first activation map, and determining a positioning result of a key point in the target object in the target image frame according to the pixel average value; According to the object part category and the positioning result, the object posture detection result corresponding to the target image frame is determined.

4. The method according to claim 3, characterized in that The acquiring, by means of the object detection model, an object posture feature corresponding to the target object in the target image frame comprises: Acquire an object description feature corresponding to the target object in the target image frame in the object detection model, and output a second classification result corresponding to the object description feature according to a classifier in the object detection model; Obtaining an object convolution feature for the target image frame output by a target convolution layer in the object detection model, and performing a product operation on the second classification result and the object convolution feature to obtain a second activation map corresponding to the target image frame; The target image frame is processed into blocks according to the second activation map to obtain M object part region images, and part description features corresponding to the M object part region images are obtained according to the object detection model; M is a positive integer; The object description feature and the part description features corresponding to the M object part region images are combined into the object posture feature.

5. The method according to claim 2, characterized in that: The step of inputting the target image frame into a part detection model and outputting a part posture detection result corresponding to the target image frame through the part detection model comprises: Inputting the target image frame into a part detection model, and detecting a first object part of the target object in the target image frame in the part detection model; If the first object part is detected in the target image frame, a regional image including the first object part is obtained from the target image frame, a position of a part key point corresponding to the first object part is obtained according to the regional image, and a part posture detection result corresponding to the target image frame is determined based on the position of the part key point; If the first object part is not detected in the target image frame, it is determined that the part posture detection result corresponding to the target image frame is a null value.

6. The method according to claim 5, characterized in that If the first object part is detected in the target image frame, a regional image including the first object part is obtained from the target image frame, a position of a part key point corresponding to the first object part is obtained according to the regional image, and a part posture detection result corresponding to the target image frame is determined based on the position of the part key point, including: If the first object part is detected in the target image frame, cropping the target image frame to obtain a regional image including the first object part; Acquire a part contour feature corresponding to the regional image, and predict a position of a part key point corresponding to the first object part according to the part contour feature; Based on the positions of the key points of the parts, the key points of the first object parts are connected to obtain the part posture detection result corresponding to the target image frame.

7. The method according to claim 1, characterized in that The interpolation processing is performed on the first candidate object posture according to the part posture detection result to obtain the global posture corresponding to the target object, including: If the object posture detection result includes the posture of the second object part, and the object posture detection result does not include the posture of the third object part, then determining the first part direction corresponding to the third object part according to the key point position of the first object part included in the part posture detection result; the second object part and the third object part are symmetrical parts of the target object; Acquire a first part length of the second object part in the first candidate object posture, and determine a key point position of the third object part according to the first part length and the first part direction; The key point positions of the third object part are added to the first candidate object posture to obtain a global posture corresponding to the target object in the target image frame.

8. The method according to claim 7, characterized in that The step of adding the key point position of the third object part to the first candidate object posture to obtain a global posture corresponding to the target object in the target image frame includes: Adding the key point position of the third object part to the first candidate object posture, obtaining the second candidate object posture corresponding to the target object in the target image frame, and acquiring the posture offset between the standard posture and the second candidate object posture; If the posture offset is greater than the offset threshold, key point correction is performed on the posture of the second candidate object based on the standard posture to obtain a global posture corresponding to the target object in the target image frame.

9. The method according to claim 1, characterized in that: The target image frame is the i-th image frame in the video data, where i is a positive integer; The interpolation processing is performed on the first candidate object posture according to the part posture detection result to obtain the global posture corresponding to the target object, including: If the object posture detection result does not include the postures of the second object part and the third object part, determining the second part direction corresponding to the second object part and the third part direction corresponding to the third object part according to the key point positions of the first object part included in the part posture detection result; the second object part and the third object part are symmetrical parts of the target object; In the (i-1)th image frame, obtaining a second part length corresponding to the second object part and a third part length corresponding to the third object part, and determining a key point position of the second object part according to the second part length and the second part direction; According to the length of the third part and the direction of the third part, the key point positions of the third object part are determined, and the key point positions of the second object part and the key point positions of the third object part are added to the first candidate object posture to obtain the global posture corresponding to the target object in the target image frame.

10. The method according to claim 1, characterized in that Also includes: A virtual object associated with the target object is constructed according to the global posture, and the virtual object with the global posture is played in a multimedia application.

11. A data processing device, characterized in that: include: A posture detection module is used to obtain the object posture detection result and the part posture detection result corresponding to the target image frame; The object posture detection result is associated with a target object in the target image frame, and the part posture detection result is associated with a first object part of the target object; A key point number counting unit, used to obtain a standard posture associated with the target object, and count the number of first key points corresponding to the standard posture, and the number of second key points corresponding to the object posture detection result; A first interpolation processing unit, configured to, when the number of the first key points is greater than the number of the second key points, perform interpolation processing on the object posture detection result according to the standard posture to obtain a first candidate object posture; The second interpolation processing unit is used to perform interpolation processing on the posture of the first candidate object according to the part posture detection result to obtain the global posture corresponding to the target object.

12. A computer device, characterized in that: including memory and processor; The memory is connected to the processor, the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 10.

14. A computer program product, characterized in that The method comprises a computer program / instruction, which implements the method according to any one of claims 1 to 10 when executed by a processor.

Citation Information

Patent Citations

  • Evaluation method and system of motion postures

    CN108256433A

  • Human body posture detection method, device and equipment and storage medium

    CN109558832A

  • Hand posture tracking method, device and system

    CN110348359A

  • Attitude estimation method, processing method based on attitude estimation and electronic equipment

    CN110472462A

  • Image generation method and device based on artificial intelligence, equipment and storage medium

    CN111626218A

Cited By

  • Data processing method and apparatus, device and medium

    EP4411641B1

  • Data processing method and apparatus, device and medium

    WO2023185241A1