Data processing method, data processing device, computer device, and computer program

Abstract

Provided is a data processing method, device, equipment, and medium applicable to fields such as artificial intelligence, driving assistance, etc. The method includes the steps of: acquiring an object pose detection result corresponding to an object in an image frame and a part pose detection result corresponding to a first object part of the object in the image frame, where at least one object part of the object is missing in the object pose detection result, and the first object part is one or more parts of the object; and acquiring an overall pose corresponding to the object by performing an interpolation process for the at least one object part missing in the object pose detection result based on the part pose detection result and a reference pose related to the object, where the overall pose is for operating a computer device to realize a service function corresponding to the overall pose.

Description

[Technical Field]

[0001] This application claims priority from a Chinese patent application filed with the China Patent Office on March 31, 2022, bearing application number 2022103327630 and entitled "Data Processing Method, Apparatus, Device, and Medium," the entire contents of which are incorporated herein by reference.

[0002] The present application relates to the technical field of artificial intelligence, and in particular to a data processing method, device, apparatus, and medium. [Background technology]

[0003] Computer vision (CV) technology is the science that studies how machines can "see," or more specifically, how they use cameras and computers to replace the human eye to perform machine vision tasks such as recognition and measurement, and then use graphics processing and computer processing to obtain images that are more suitable for human observation or transmission to instruments for detection. As a branch of science, CV studies related theories and technologies, and aims to build artificial intelligence systems that can extract information from images or multidimensional data.

[0004] Pose estimation can detect the position of each key point in a picture or video, and has a wide range of application value in fields such as movie animation, driving assistance, virtual reality, and action recognition.

[0005] Current pose estimation algorithms can detect keypoints in an image or video and construct a final object pose based on the detected keypoints and object constraint relationships. Summary of the Invention [Problem to be solved by the invention]

[0006] The embodiments of the present application provide a data processing method, device, apparatus, and medium that can improve the accuracy of object pose estimation. [Means for solving the problem]

[0007] In an embodiment of the present application, there is provided a data processing method executed by a computing device, the method comprising: obtaining an object pose detection result corresponding to an object in an image frame and a part pose detection result corresponding to a first object part of the object in the image frame, wherein at least one object part of the object is missing in the object pose detection result, and the first object part is one or more parts of the object; and a step of obtaining an overall posture corresponding to the object by interpolating at least one missing object part in the object posture detection result based on the part posture detection result and a reference posture related to the object, wherein the overall posture is for operating a computer device to realize a service function corresponding to the overall posture.

[0008] In an embodiment of the present application, there is further provided a data processing apparatus, the apparatus comprising: a pose detection module that obtains an object pose detection result corresponding to an object in an image frame and a part pose detection result corresponding to a first object part of the object in the image frame, wherein at least one object part of the object is missing in the object pose detection result, and the first object part is one or more parts of the object; and a posture estimation module that obtains an overall posture corresponding to the object by performing an interpolation process for at least one object part missing in the object posture detection result based on the part posture detection result and a reference posture related to the object, wherein the overall posture is for operating a computer device to realize a service function corresponding to the overall posture.

[0009] In an embodiment of the present application, a computer device is further provided that includes a memory and a processor, the memory being connected to the processor, a computer program being stored in the memory, and the processor calling the computer program causing the computer device to execute the above-mentioned method in the embodiment of the present application.

[0010] In an embodiment of the present application, a computer-readable storage medium storing a computer program is further provided, which, when loaded and executed by a processor, is configured to cause a computing device having the processor to perform the above-described method in an embodiment of the present application.

[0011] In an embodiment of the present application, there is further provided a computer program product or computer program comprising computer instructions stored in a computer-readable storage medium, which are read by a processor of a computing device from the computer-readable storage medium and, when executed by the processor, cause the computing device to perform the above-described method. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram of a network architecture configuration provided in an embodiment of the present application. [Figure 2] FIG. 1 is a schematic diagram of an object pose estimation scenario for video data provided in an embodiment of the present application; [Figure 3]1 is a flow diagram of a data processing method provided in an embodiment of the present application. [Figure 4] FIG. 1 is a schematic diagram of a reference posture provided in an embodiment of the present application. [Figure 5] FIG. 1 is a schematic diagram of an object pose estimation scenario provided in an embodiment of the present application. [Figure 6] 1 is a schematic diagram of an application scenario of a global posture provided in an embodiment of the present application; FIG. [Figure 7] FIG. 2 is a flow diagram of another data processing method provided in an embodiment of the present application. [Figure 8] FIG. 1 is a schematic diagram of the configuration of an object detection model provided in an embodiment of the present application. [Figure 9] FIG. 2 is a schematic diagram of a flow of obtaining an object pose detection result provided in an embodiment of the present application. [Figure 10] FIG. 10 is a schematic diagram of a flow of acquiring a part posture detection result provided in an embodiment of the present application. [Figure 11] FIG. 1 is a schematic diagram of object keypoint correction provided in an embodiment of the present application; [Figure 12] FIG. 2 is a schematic diagram of the object pose estimation flow provided in an embodiment of the present application. [Figure 13] 1 is a schematic diagram of the configuration of a data processing device provided in an embodiment of the present application. [Figure 14] FIG. 1 is a schematic diagram of the configuration of another data processing device provided in an embodiment of the present application. [Figure 15] FIG. 1 is a schematic diagram of the configuration of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, the configuration of the embodiments of the present application will be described clearly and completely with reference to the drawings of the embodiments of the present application. It is clear that the described embodiments are only some of the embodiments of the present application, and not all of the embodiments. All other embodiments that a person skilled in the art can obtain from the embodiments of the present application without creative work fall within the scope of protection of the present application.

[0014] This application relates to pose estimation under computer vision technology. Pose estimation is an important task in computer vision and an essential step for computers to understand the motion and behavior of objects. Pose estimation can be converted into an object keypoint prediction problem. For example, the position coordinates of each object keypoint in an image may be predicted, and the object skeleton in the image may be predicted based on the positional relationship between the object keypoints. Here, pose estimation according to this application may include object pose estimation for an object and part pose estimation for a specific part of an object. Objects may include, but are not limited to, a human body, an animal, a plant, etc. Specific parts of an object may include a palm, a face, an animal's limbs, a plant's roots, etc. This application does not limit the type of object.

[0015] When an image or video is captured in a mobile scenario, the image or video may only contain a part of the object, so during the object pose estimation, some parts of the object are missing, which results in insufficient extracted part information, and the final object pose result is not the complete pose of the object, which affects the completeness of the object pose.

[0016] In the embodiments of the present application, by performing object pose estimation and specific part pose estimation for an object in an image frame, it is possible to obtain an object pose detection result for the object and a part pose detection result for a first object part of the object; further, based on the object pose detection result, part pose detection result, and reference pose, it is possible to perform pose estimation of the object in the image frame, correct missing part keypoints of the object in the image frame, ensure the completeness and rationality of the finally obtained overall pose of the object, and further improve the estimation accuracy of the overall pose.

[0017] Please refer to FIG. 1. FIG. 1 is a schematic diagram of a network architecture configuration provided in an embodiment of the present application. As shown in FIG. 1, the network architecture may include a server 10d and a user terminal cluster. The user terminal cluster may include one or more user terminals. The number of user terminals is not limited here. As shown in FIG. 1, the user terminal cluster may specifically include a user terminal 10a, a user terminal 10b, a user terminal 10c, etc.

[0018] Here, the server 10d may be an independent physical server, or may be a server cluster or distributed system consisting of multiple physical servers, or may be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDNs, and base cloud computing services such as big data and artificial intelligence platforms.

[0019] User terminal 10a, user terminal 10b, user terminal 10c, etc. may each include an electronic device with an object pose estimation function, such as a smartphone, tablet computer, laptop computer, palmtop computer, mobile internet device (MID), wearable device (e.g., smart watch, smart bracelet, etc.), smart voice interaction device, smart home appliance (e.g., smart TV, etc.), in-vehicle device, etc. As shown in FIG. 1, user terminal 10a, user terminal 10b, user terminal 10c, etc. can each be network-connected to server 10d. This allows each user terminal to exchange data with server 10d via this network connection.

[0020] A user terminal (e.g., user terminal 10a) in the user terminal cluster shown in FIG. 1 incorporates an application client with an object pose estimation function. This application client may include, but is not limited to, a multimedia client (e.g., a short video client, a live video streaming client, or a video client) and an object management application (e.g., a patient care client). The application client in user terminal 10a can acquire video data. This video data may be a video of an object captured in a mobile scenario. For example, the object may be captured by a camera built into user terminal 10a, or by a photographing device (e.g., a single-lens reflex camera, camera, etc.) connected to user terminal 10a. Note that in a mobile scenario (e.g., a selfie scenario), due to limitations of the shooting distance and the photographing device, only a portion of the object may be included in the video data. For example, if the object is a human body, the video data may include only the upper body or only the head of the human body. When performing object pose estimation for an object in video data, the pose of the object included in the video data needs to be corrected. This allows obtaining a global pose corresponding to the object, and also improves the accuracy of the global pose while preserving the integrity of the global pose of the object. In the present embodiment, the global pose is also referred to as a complete pose, which refers to a pose that includes all parts of the object, i.e., a pose that corresponds to the complete object.

[0021] It should be noted that the object pose estimation process according to the embodiments of the present application may be performed by a computer device. This computer device may be a user terminal in the user terminal cluster shown in Fig. 1, or may be the server 10d shown in Fig. 1. In other words, the computer device may be a user terminal, a server, or a combined device configured of a server and a user terminal. This application is not limited thereto.

[0022] Please refer to FIG. 2. FIG. 2 is a schematic diagram of an object pose estimation scenario for video data provided in an embodiment of the present application. Taking the user terminal 10a shown in FIG. 1 as an example, the object pose estimation process for video will be described. As shown in FIG. 2, the user terminal 10a can acquire video data 20a. This video data 20a may be a video of an object captured by a camera built into the user terminal 10a, or may be a video of an object transmitted to the user terminal 10a from another device. N image frames are acquired by framing the video data 20a. N is a positive integer, and the value of N may be 1, 2, .... The first image frame in time series (i.e., image frame T1) is acquired from the N image frames, and this image frame T1 is input to an object detection model 20b. Object detection is performed on the image frame T1 using this object detection model 20b, thereby acquiring an object pose detection result 20c corresponding to this image frame T1. The object pose detection result 20c may include keypoints of the object included in the image frame T1 (for convenience of explanation, the keypoints of the object will be referred to as object keypoints hereinafter) and the positions of these object keypoints in the image frame T1. The object pose detection result 20c may further include a first confidence level corresponding to each detected object keypoint. The first confidence level can be used to indicate the prediction accuracy of the detected object keypoint, and a larger first confidence level indicates that the detected object keypoint is more accurate and more likely to be a true keypoint of the object.

[0023] For example, if the object in the video data 20a is a human body, the object keypoints corresponding to the object are considered to be joint points in the human body structure. The number and classification of the object keypoints may be predefined. For example, the human body structure may include multiple object keypoints for limbs, the brain, the waist, the chest, etc. If the image frame T1 contains a complete object, the image frame T1 may include all of the object keypoints for the object. If the image frame T1 contains only a partial structure of the object, the image frame T1 may include only a portion of the object keypoints. After detecting the object keypoints in the image frame T1, the detected object keypoints may be concatenated according to the object keypoint classification and keypoint position, and the concatenated result, i.e., the object pose detection result 20c, may be marked in the image frame T1. Here, the object detection model 20b may be a pre-trained network model with object detection capabilities for videos / images. If the object is a human body, the object detection model 20b may also be referred to as a human body pose estimation model.

[0024] The object pose detection result 20c can be used to obtain a human body pose 20j of the object in the image frame T1. Because some object key points are missing in the human body pose 20j (i.e., some joint points of the human body are missing), the user terminal 10a may obtain a reference pose 20k corresponding to the object. Based on this reference pose 20k, the key points of the human body pose 20j can be corrected to obtain a human body pose 20m corresponding to the object in the image frame T1. Here, the reference pose 20k can also be considered as a default pose of the object and is also called a reference pose. This reference pose 20k may be constructed in advance based on all object key points of the object. For example, the pose of a human body when normally standing (e.g., the overall pose) may be determined as the reference pose 20k.

[0025] The image frame T1 may be input to a part detection model 20d, which may then detect a specific part (e.g., a first object part) of the object in the image frame T1 to obtain a part pose detection result 20e corresponding to the image frame T1. If it is detected that the first object part of the object does not exist in the image frame T1, the part pose detection result for the image frame T1 may be determined as null. If it is detected that the first object part of the object exists in the image frame T1, the keypoints and positions of the first object part may be subsequently detected, and the detected first object part keypoints may be connected according to the keypoint classification and keypoint position of the first object part. The connected result, i.e., the part pose detection result 20e, may be marked in the image frame T1. Here, the number and classification of keypoints corresponding to the first object part may be similarly predefined. If the object is a human body, the part detection model 20d may be a palm pose estimation model (here, the first object part is the palm). For example, a palm may include a palm center keypoint and finger keypoints. The body part detection model 20d may be a pre-trained network model with object body part detection capabilities for videos / images. For convenience of explanation, the first object body part keypoint will hereinafter be referred to as the body part keypoint.

[0026] As shown in FIG. 2, a second reliability is assigned to the part orientation detection result 20e. This second reliability can be used to represent the likelihood that the detected object part is the first object part. For example, the part detection model 20d can determine that the second reliability that the region 20f in the image frame T1 is the first object part is 0.01, the second reliability that the region 20g is the first object part is 0.09, the second reliability that the region 20h is the first object part is 0.86, and the second reliability that the region 20i is the first object part is 0.84. A higher second reliability indicates a higher likelihood that the region is the first object part. For example, based on the second reliability, it can be determined that the region 20h and the region 20i include the first object part, and the orientation of the first object part can be marked in the region 20h and the region 20i.

[0027] Furthermore, the user terminal 10a may combine the object pose detection result 20c and the part pose detection result 20e to perform interpolation processing for missing object parts, thereby obtaining reasonable object keypoints through the interpolation processing. For example, if the part pose detection result 20e is a palm keypoint, the object pose detection result 20c and the part pose detection result 20e may be combined to perform interpolation processing for missing parts of the object in the image frame T1, such as the wrist or elbow, to complete the object's human body pose 20m and obtain a human body pose 20n (also referred to as a global pose). Similarly, after obtaining the global pose corresponding to the object in the image frame T1, the same method may be used to estimate the object pose for subsequent image frames in the video data 20a, thereby obtaining the global pose corresponding to the object in each image frame. The behavior of the object in the video data 20a can be obtained based on the global poses corresponding to each of the N image frames. As can be understood, the video data 20a may be a video captured in real time. The user terminal 10a can obtain the behavior of an object in real time by performing object pose estimation for image frames in video data captured in real time.

[0028] In other words, for an image frame that includes only a part of an object, the object detection result output from the object detection model 20b, the part detection result output from the part detection model 20d, and the reference posture 20 k The global pose of the object in the image frame may be estimated by the following equation, which can ensure the completeness and rationality of the finally acquired global pose of the object, and further improve the estimation accuracy of the global pose.

[0029] Please refer to Fig. 3. Fig. 3 is a schematic diagram of the flow of a data processing method provided in an embodiment of the present application. As shown in Fig. 3, this data processing method may include the following steps S101 to S102.

[0030] In step S101, an object pose detection result corresponding to an object in an image frame and a part pose detection result corresponding to a first object part of the object are obtained, and in the object pose detection result, at least one object part of the object is missing, and the first object part is one or more parts of the object.

[0031] Specifically, the computing device may acquire video data (e.g., video data 20a in an embodiment corresponding to FIG. 2) or image data of an object captured in a mobile-side scenario. When performing pose estimation for the video data or image data, the computing device may perform object detection on image frames in the image data or video data to acquire an object pose detection result for the object (e.g., object pose detection result 20c in an embodiment corresponding to FIG. 2 above). Additionally, the computing device may perform part detection on the image frames to acquire a part pose detection result for a first object part of the object (e.g., part pose detection result 20e in an embodiment corresponding to FIG. 2 above). Here, the object may be an object included in the video data, such as a human body, an animal, or a plant, and the first object part may be one or more parts of the object, such as a face or palm in a human body structure, or a limb, tail, head, or root in an animal structure. The present application does not limit both the type of object and the type of the first object part. In addition, due to the limited distance between the camera and the object being photographed in the mobile scenario, there may be cases where parts of the object in the video data or image data are missing, i.e., some object parts of the object may not be present in the video data screen. By combining the object pose detection result and the part pose detection result, the accuracy of object pose estimation can be improved.

[0032] For convenience of explanation, in all embodiments of the present application, the object pose estimation process for video data or image data will be described using an example in which the object is a human body. When performing object pose estimation for image data in a mobile scenario, the image data may be used as image frames. When performing object pose estimation for video data in a mobile scenario, N image frames (N is a positive integer) corresponding to the video data may be obtained by framing the video data. Furthermore, an image frame sequence including N image frames may be formed as a time series of N image frames in the video data. Object pose estimation may be performed sequentially for the N image frames in the image frame sequence. For example, after completing object pose estimation for the first image frame in the image frame sequence, object pose estimation may be performed for the second image frame in the image frame sequence, and this may be repeated until object pose estimation for the entire video data is completed.

[0033] Here, the computer device may obtain an object detection model and a body part detection model, and input the image frame into the object detection model. The object detection model may output an object pose detection result corresponding to the image frame. At the same time, the image frame may be input into the body part detection model. The body part detection model may output a body part pose detection result corresponding to the image frame. Here, the object detection model may be used to detect object keypoints (also called object keypoints, e.g., human body keypoints) in the image frame. In this case, the object detection model may also be called a human body pose estimation model. The object detection model may include, but is not limited to, DensePose (a human body real-time pose recognition system for realizing real-time pose recognition of densely packed people), OpenPose (a framework for estimating the body, face, and hand morphology of multiple people in real time), Realtime Multi-Person Pose Estimation (a real-time multi-person pose estimation model), DeepPose (a pose estimation method based on a deep neural network), and mobilenetv2 (a lightweight deep neural network). This application does not limit the type of object detection model. The body part detection model can be used to detect keypoints (e.g., palm keypoints) of a first object part of an object. In this case, the body part detection model is also called a palm pose estimation model. The body part detection model may be a detection-based method or a regression-based method. In a detection-based method, body part keypoints of the first object part may be predicted by generating a heat map. In a regression-based method, position coordinates of body part keypoints may be directly regressed. The network configuration of the body part detection model and the network configuration of the object detection model may be the same or different. If the network configuration of the body part detection model and the network configuration of the object detection model are the same, the network parameters of the two are different (trained from different data). This application does not limit the type of body part detection model.

[0034] In some embodiments, the object detection model and body part detection model may be pre-trained using sample data. For example, the object detection model may be trained using sample data tagged with human body keypoint tag information (e.g., a 3D human body dataset), and the body part detection model may be trained using sample data tagged with palm keypoint information (e.g., a palm dataset). Alternatively, the object detection model may be an object detection service invoked from an artificial intelligence cloud service via an application program interface (API), and the body part detection model may be a body part detection service invoked from the artificial intelligence cloud service via the API. No specific limitations are provided here.

[0035] AI cloud services are commonly referred to as AIaaS (AI as a Service). This is the service model of the current mainstream AI platforms. Specifically, AIaaS platforms separate several standard AI services and provide them as independent or packaged services on the cloud side. This service model is similar to opening an AI-themed shopping mall. All developers can access and use one or more AI services provided by the platform via APIs. Some experienced developers can even use the AI ​​framework and AI infrastructure provided by the platform to deploy and manage their own cloud AI services.

[0036] In some embodiments, the object detection model used in the embodiments of the present application may be a 3D human body pose estimation model with a confidence level. For example, the object detection model may predict object keypoints of an object in an image frame, and each predicted object keypoint may correspond to a first confidence level. The first confidence level may be used to represent the prediction accuracy of each predicted object keypoint. The predicted object keypoints and the corresponding first confidence level may be referred to as an object pose detection result corresponding to the image frame. The body part detection model may be a 3D palm pose estimation model with a confidence level. For example, the body part detection model may predict a location region of a first object part in an image frame and predict body keypoints of the first object part in the location region. The body part detection model may predict one or more location regions where the first object part may be located. A location region may correspond to a second confidence level. The second confidence level may be used to represent the prediction accuracy of each predicted location region. The predicted body part keypoints and the second confidence level corresponding to the location region may be referred to as a body part pose detection result corresponding to the image frame.

[0037] In step S102, an overall posture corresponding to the object is obtained by performing an interpolation process for at least one missing object part in the object posture detection result based on the part posture detection result and a reference posture related to the object, and the overall posture is used to operate a computer device to realize a service function corresponding to the overall posture.

[0038] Specifically, the computing device may acquire a reference pose corresponding to the object (e.g., reference pose 20 in the embodiment corresponding to FIG. 2). k) may be acquired. This reference pose is considered to be the complete default pose (referred to as T-pose) of the object. The number of reference poses may be one or more, for example, a default standing human body pose, a default sitting human body pose, a default crouching human body pose, etc. The present application does not limit the type and number of reference poses.

[0039] Please refer to FIG. 4. FIG. 4 is a schematic diagram of a reference posture provided in an embodiment of the present application. As shown in FIG. 4, a model 30a can be expressed as an SMPL (Skinned Multi-Person Linear) model. This model 30a is a parameterized human body model and can be applied to different human body structures. This model 30a may include a human body joint distribution with one root node (node ​​0) and 23 joint nodes (nodes numbered 1 to 23). Here, the root node is used to transform the entire human body as a completely rigid body (an object whose volume and shape do not change when a force is applied), and the 23 joint nodes can be used to describe local human body part deformations. When the object is a human body, the one root node and the 23 joint nodes may be used as object keypoints of the object. Based on the classification and position of the object keypoints (e.g., wrist joint point, elbow joint point, palm joint point, ankle joint point, etc.), a reference posture 30b can be obtained by connecting the one root node and the 23 joint nodes.

[0040] An image frame may not contain a complete object. For example, if some parts of an object (e.g., the lower limbs of a human body) are not present in an image frame, some object keypoints are missing from the object pose detection result corresponding to that image frame. A first candidate object pose corresponding to the object may be obtained by correcting the keypoints in the object pose detection result corresponding to the object using the reference pose to complete the missing object keypoints. If the part pose detection result includes part keypoints of a first object part, the part keypoints in the part pose detection result and the object keypoints in the object pose detection result may be combined to adjust the first candidate object pose to obtain a global pose of the object in the image frame. After obtaining the global pose corresponding to the current image frame, object pose estimation may be performed on the next image frame in the video data to obtain the global pose of the object in each image frame of the video data.

[0041] In some embodiments, a computing device can determine the behavior and movement of an object based on the overall pose of the object in the video data, and manage or care for the object based on these behaviors and movements, or perform human-computer interaction based on the object's behavior and movement. In short, the overall pose of an object in the video data can be applied to human-computer interaction scenarios (e.g., virtual reality and human-computer animation), content review scenarios, autonomous driving scenarios, virtual live streaming scenarios, and character movement design scenarios for games and movies. In human-computer interaction scenarios, images (or videos) of a user (object) can be collected. After acquiring the overall pose in the image or video, machine operation can be realized based on the overall pose. For example, a specific command can be executed based on a specific human body movement (determined from the overall pose). In game character movement design scenarios, acquiring human body movements based on the overall pose corresponding to the object instead of using expensive motion capture equipment can reduce the cost and difficulty of game character movement design.

[0042] Here, the virtual live streaming scenario may involve playing a video of a virtual object that behaves and moves in the same way as the streamer user, rather than directly playing a video of the streamer user (object) on the live screen of the live streaming room. For example, the streamer user's behavior and movements may be determined based on the streamer user's overall posture, and the streamer user's behavior and movements may be used to drive the virtual object. That is, a virtual object that behaves and moves in the same way as the streamer user may be constructed and used for live streaming. This allows the streamer user to remain out of the public eye while achieving a live streaming effect similar to that of a real streamer user. For example, a computer device may construct a virtual object related to an object based on the overall posture of the object in video data, and play the virtual object with that overall posture in a multimedia application (e.g., a live streaming room, a video website, a short video application, etc.). That is, a video related to the virtual object may be played in the multimedia application, and the posture of the virtual object may be kept synchronized with the posture of the object in the video data. Here, the overall pose corresponding to the object in the video data is embodied on a virtual object played in the multimedia application, and every time the pose of the object changes, the virtual object in the multimedia application is driven to convert to the same pose (this can be thought of as reconstructing the virtual object with a new pose, where the new pose is the pose of the object after the change), thereby maintaining the poses of the object and the virtual object consistent at all times.

[0043] Please refer to FIG. 5. FIG. 5 is a schematic diagram of an object pose estimation scenario provided in an embodiment of the present application. The object pose estimation process for video data will be described using a virtual live streaming scenario as an example. As shown in FIG. 5, when a streamer user 40c (which may be considered an object) needs to perform a live streaming, he or she can enter a live streaming room (e.g., a live streaming room with room number 116889). Before starting the live streaming, the streamer user 40c can select either a real person live streaming mode or a virtual live streaming mode. If the streamer user 40c selects the virtual live streaming mode, the streamer user 40c may pull a virtual object. Once the streamer user 40c starts the live streaming, the streamer user 40c's actions and behavior may be used to drive the virtual object, thereby allowing the virtual object and the streamer user 40c to maintain the same posture.

[0044] After starting live streaming, the streamer user 40c can collect his / her own video data using the user terminal 40a (e.g., a smartphone). cmay be treated as an object, and the user terminal 40a may be fixed with a bracket 40b. The user terminal 40a may collect video data of a streamer user 40c, obtain an image frame 40g from the video data, and input the image frame 40g into an object detection model and a body part detection model, respectively. The object detection model may predict body part joint points (object keypoints) of the streamer user 40c included in the image frame 40g. These predicted body part joint points may be used as an object pose detection result for the image frame 40g. The body part detection model may predict palm keypoints of the streamer user 40c included in the image frame 40g (here, the first object part is the palm by default, and the palm keypoints are also called body part keypoints). These predicted palm keypoints may be used as a body part pose detection result for the image frame 40g. The object pose detection result and body part pose detection result may be marked on the image frame 40g (as shown in image 40h). Here, the regions 40i and 40j in the image 40h indicate the results of the part orientation detection.

[0045] As shown in FIG. 5, a human body posture 40k of the streamer user 40c in the image frame 40g can be obtained using the object posture detection results and part posture detection results displayed in the image 40h. As is apparent, since the image frame 40g only includes the upper body of the streamer user 40c, the human body posture 40k is not the complete human body posture of the streamer user 40c. In this case, a reference posture (complete default human body posture) can be obtained, and the joint points of the human body posture 40k can be interpolated using the reference posture. This completes the missing part joint points in the human body posture 40k, thereby obtaining the entire human body posture 40m (overall posture) for the streamer user 40c.

[0046] The virtual object in the live streaming room is driven by the whole human body posture of 40m. ToStreamer user 40c and similar overall human body posture 40 m A user who enters the live streaming room and watches the live broadcast may have a virtual object located therein. A user who enters the live streaming room and watches the live broadcast may have an exhibition page of the live streaming room where the virtual object is located displayed on the user's user terminal 40d. The exhibition page of the live streaming room may include areas 40e and 40f. Area 40e can be used to play a video of the virtual object (with a posture similar to that of the streamer user 40c), and area 40f can be used to send barrages, etc. In a virtual live streaming scenario, a user who enters the live streaming room and watches the live broadcast may only view the video of the virtual object and the streamer user's 40c's audio data, but not the streamer user's 40c's video data. In this way, the personal information of the streamer user 40c can be protected, and the virtual object can achieve a live streaming effect similar to that of the streamer user 40c.

[0047] In some embodiments, the overall posture of the object in the video data can be applied to a content screening scenario. If the overall posture is the same as the posture in the content screening system, the screening result of the object in the content screening system can be determined to be a passing result, and the object can be granted access to the content screening system. If the overall posture passes the screening of the content screening system, the object can have access to the content screening system.

[0048] Please refer to FIG. 6. FIG. 6 is a schematic diagram of an application scenario of the whole posture provided in an embodiment of the present application. As shown in FIG. 6, a user A (object) can send a verification request to a server 50d via a user terminal 50a. After receiving the verification request sent from the user terminal 50a, the server 50d may obtain an identity verification method for the user A and return the identity verification method to the user terminal 50a. A verification box 50b may be displayed on the terminal screen of the user terminal 50a. The user A can face the verification box 50b on the user terminal 50a and perform a specific action (for example, raising his / her hand, kicking, or hitting his / her hip). The user terminal 50a may collect a verification target image 50c (considered to be the image frame described above) in the verification box 50b in real time and send the verification target image 50c collected in real time to the server 50d.

[0049] The server 50d may acquire the verification target image 50c transmitted from the user terminal 50a and acquire a pose 50e that user A preset in the content review system. This pose 50e may be used as verification information for user A in the content review system. The server 50d may acquire an overall pose of user A in the verification target image 50c by estimating the pose of the verification target image 50c using an object detection model, a body part detection model, and a reference pose, and compare the similarity between the overall pose corresponding to the verification target image 50c and the pose 50e. If the similarity between the overall pose of the verification target image 50c and the pose 50e is equal to or greater than a similarity threshold (for example, the similarity threshold may be set to 90%), it may be determined that the overall pose of the verification target image 50c is the same as the pose 50e, and that user A has passed the review in the content review system. If the similarity between the overall posture of the verification target image 50c and posture 50e is less than the similarity threshold, it may be determined that the overall posture of the verification target image 50c is not the same as posture 50e and that user A has failed the screening in the content screening system, and action error notification information may be returned to the user terminal 50a. This action error notification information is intended to prompt user A to try the action again for identity screening.

[0050] In the embodiments of the present application, by performing object pose estimation and specific part pose estimation for an object in an image frame, it is possible to obtain an object pose detection result for the object and a part pose detection result for a first object part of the object; further, based on the object pose detection result, part pose detection result, and reference pose, it is possible to perform pose estimation of the object in the image frame, correct missing part keypoints of the object in the image frame, ensure the completeness and rationality of the finally obtained overall pose of the object, and further improve the estimation accuracy of the overall pose.

[0051] Please refer to Fig. 7. Fig. 7 is a schematic diagram of the flow of another data processing method provided in an embodiment of the present application. As shown in Fig. 7, this data processing method may include the following steps S201 to S208.

[0052] In step S201, an image frame is input into an object detection model, and object pose features corresponding to the object in the image frame are obtained through the object detection model, and a first classification result corresponding to the object pose features is recognized, and the first classification result is for representing the object part classification corresponding to the object keypoints.

[0053] Specifically, the computing device may acquire video data captured in a mobile scenario, select an image frame from the video data, and input the image frame into a trained object detection model. The object detection model may acquire an object pose feature corresponding to an object in the image frame. A classifier in the object detection model may output a first classification result corresponding to the object pose feature. The first classification result may be used to represent an object part classification corresponding to a key point of the object (e.g., a joint of a human body). Here, the object pose feature may be an object description feature for the object extracted by the object detection model, or may be a fusion feature of the object description feature and the part description feature corresponding to the object. If the object pose feature is an object description feature corresponding to an object in the image frame, it indicates that part detection-based block learning has not been implemented in the process of extracting features from the image frame using the object detection model. If the object pose feature is a fusion feature of the object description feature and the part description feature corresponding to the object in the image frame, it indicates that part detection-based block learning has been implemented in the process of extracting features from the image frame using the object detection model. By introducing block learning based on part detection, the object pose feature can include the local pose feature (part description feature) of each part of the object included in the image frame, and Image Frame , which can enhance the fine granularity of the object pose features and further improve the accuracy of the object pose detection results.

[0054] In some embodiments, when block learning based on part detection is introduced in the process of extracting features from an image frame using an object detection model, the computer device may input the image frame to the object detection model, obtain object description features corresponding to the object in the image frame from the object detection model, output a second classification result corresponding to the object description features using a classifier in the object detection model, obtain object convolution features for the image frame output from the convolution layer in the object detection model, obtain a second activation mapping corresponding to the image frame by multiplying the second classification result and the object convolution features, obtain M (M is a positive integer) object part region images by performing block segmentation processing on the image frame based on the second activation mapping, obtain part description features corresponding to each of the M object part region images based on the object detection model, and combine the object description features and the part description features corresponding to the M object part region images as object pose features.

[0055] Here, the object description features are considered to be feature representations extracted from the image frame to represent the object. The second classification result can also be used to represent the object part classification corresponding to the object keypoints included in the image frame. The convolutional layer may be the last convolutional layer in the object detection model. The object convolutional features can represent convolutional features for the image frame output from the last convolutional layer of the object detection model. The second activation mapping may be a class activation mapping (CAM) corresponding to the image frame. The CAM is a tool for visualizing image features. The second activation mapping can be obtained by weighting the object convolutional features output from the last convolutional layer in the object detection model and the second classification result (the second classification result can be considered as a weight corresponding to the object convolutional features). This second activation matching can be considered as a visualization result of the object convolutional features output from the convolutional layer and can be used to represent the image pixel point region focused by the object detection model.

[0056] The computer device may perform block segmentation of the image frame using the class activation mapping (second activation mapping) of each object keypoint in the image frame as a priori information of the region location, i.e., by cutting the image frame based on the second activation mapping, obtain an object region region image containing a single region. Furthermore, feature extraction may be performed on each object region region image using an object detection model to obtain region description features corresponding to each object region image. The above-mentioned object description features and region description features corresponding to each object region may be combined as object pose features for the object. The region description features are considered to be feature representations for representing the object region, extracted from the object region region images.

[0057] In step S202, a first activation mapping is generated based on the first classification result and the object convolution features of the image frame output from the object detection model.

[0058] Specifically, after obtaining a first classification result, the computer device may generate a first activation mapping by multiplying the first classification result by object convolutional features of the image frame. Here, both the first activation mapping and the second activation mapping are class activation mappings for the image frame. However, the first activation mapping uses the first classification result as the weight of the object convolutional features output from the convolutional layer (where, by default, the first classification result is a combination of object description features and part description features), while the second activation mapping uses the second classification result as the weight of the object convolutional features output from the convolutional layer. The second classification result relates only to the object description features.

[0059] In step S203, pixel average values ​​corresponding to the first activation mapping are obtained, and based on the pixel average values, localization results of key points on the object in the image frame are determined, and based on the object part classification and localization results, object pose detection results corresponding to the image frame are determined.

[0060] Specifically, the computing device may calculate pixel averages of the first activation mapping and determine the pixel averages as localization results of the key points of the object in the image frame. Based on the object feature classification and localization results, an object skeleton of the object in the image frame may be determined. The object skeleton may be used as an object pose detection result corresponding to the object in the image frame.

[0061] Referring to FIG. 8, FIG. 8 is a schematic diagram of the configuration of an object detection model provided in an embodiment of the present application. As shown in FIG. 8, a computer device may acquire an image frame 60a and then input the image frame 60a into the object detection model. A feature extraction component 60b (for example, the feature extraction network may be a convolutional network) in the object detection model performs feature extraction on the image frame 60a to obtain object description features 60c corresponding to the object in the image frame 60a. The object description features 60c are processed using ensemble average pooling (the number of object description features may be multiple, and ensemble average pooling refers to converting one object description feature into a single numerical value) and an activation function, and the processed result is classified to obtain a second classification result. A second activation mapping is obtained by weighting the second classification result and the object convolution feature output from the last convolutional layer of the feature extraction component 60b.

[0062] Based on the second activation mapping, block division processing of the image frame 60a is performed to obtain M object region images 60f. The M object region images 60f are sequentially input to a feature extraction component 60b in the object detection model, and the feature extraction component 60b can obtain region description features 60g corresponding to each of the M object region images 60f. Object pose features are obtained by combining the M region description features 60g with the object description features 60c of the image frame 60a. Recognizing the object pose features can obtain a first classification result 60d. Weighting the first classification result 60d and the object convolution features output from the last convolution layer of the feature extraction component 60b can obtain a first activation mapping 60e. The pixel average value of this first activation mapping 60e may be used as a result of identifying the object's position in the image frame 60a. This localization result is used to obtain an object pose detection result corresponding to the object in the image frame 60a.

[0063] Note that the method for obtaining the object pose detection result described in the embodiment corresponding to Fig. 8 is merely an example of an embodiment of the present application, and the present application may obtain the object pose detection result using other methods, but the present application is not limited to these.

[0064] Please refer to FIG. 9. FIG. 9 is a schematic diagram of the process of obtaining an object pose detection result provided in an embodiment of the present application. As shown in FIG. 9, taking the case where the object detection model is a human body 3D pose estimation model as an example, a computer device may input an image frame 70a into the human body 3D pose estimation model. This human body 3D pose estimation model can obtain human body 3D keypoints of an object (in this case, the object is a human body) in the image frame 70a. As shown in FIG. 9, if the human body 3D pose estimation model detects 16 human body 3D keypoints, respectively denoted x1 to x16, included in the image frame 70a, each human body 3D keypoint may correspond to one position coordinate and one first confidence level. Based on the first confidence level, it can be determined whether the detected human body 3D keypoint is a real human body keypoint. For example, a human body 3D keypoint with a first confidence level greater than a first confidence threshold (which may be set according to actual needs) is considered to be a real human body keypoint (e.g., a human body 3D keypoint represented by x4 to x16). By concatenating the true human body keypoints, a human body pose 70c (which can also be considered as an object pose detection result) can be obtained. The human body 3D keypoints whose first confidence is equal to or less than the first confidence threshold are abnormal keypoints. In subsequent processing, these abnormal keypoints may be corrected to obtain more accurate human body keypoints.

[0065] As can be understood, if a spatial coordinate system is constructed in the image frame, the position coordinates of the human body 3D key points may be spatial coordinates in this spatial coordinate system.

[0066] In step S204, the image frame is input to a part detection model, and a first object part of the object in the image frame is detected in the part detection model.

[0067] Specifically, the computer device may input an image frame into a body part detection model. In this body part detection model, first, it is detected whether the image frame includes a first object part of the object. Here, since the body part detection model can be used to detect key points of the first object part, it is necessary to detect the first object part in the image frame. If the first object part of the object is not detected from the image frame, it can be directly determined that the body part pose detection result corresponding to the image frame is null, and there is no need to perform a subsequent step of detecting key points of the first object part.

[0068] In step S205, if a first object part is detected from the image frame, an area image including the first object part is obtained from the image frame, and part key point positions corresponding to the first object part are obtained based on the area image, and part posture detection results corresponding to the image frame are determined based on the part key point positions.

[0069] Specifically, when a first object part is searched for in an image frame, a position region of the first object part in the image frame may be determined, and the image frame may be cropped based on the position region of the first object part in the image frame to obtain a region image including the first object part. In a part detection model, feature extraction of the region image may be performed to obtain region contour features corresponding to the first object part in the region image. Based on the region contour features, position of region keypoints corresponding to the first object part may be predicted. Based on the position of the region keypoints, keypoints of the first object part may be linked to obtain a region posture detection result corresponding to the image frame.

[0070] Please refer to FIG. 10. FIG. 10 is a schematic diagram of the flow of obtaining a part posture detection result provided in an embodiment of the present application. As shown in FIG. 10, taking the case where the part detection model is a palm 3D posture estimation model as an example, the computer device may input the image frame 80a to the palm 3D posture estimation model. In this palm 3D posture estimation model, it may detect whether the image frame 80a includes the palm of the object (first object part). If the palm is not detected in the image frame 80a, it may determine that the part posture detection result corresponding to the image frame 80a is a null value. If the palm is detected in the image frame 80a, it may determine the region including the palm in the image frame 80a (for example, region 80c and region 80d in image 80b, where region 80c includes the right palm of the object and region 80d includes the left palm of the object). Furthermore, the palm 3D pose estimation model can detect palm 3D key points in region 80c and palm 3D key points in region 80d.

[0071] Here, multiple possible regions are obtained using the 3D palm posture estimation model, and a second confidence level that a palm is included in each possible region is predicted. Regions for which the second confidence level is greater than a second confidence threshold (which may be the same as or different from the first confidence threshold described above; this is not a limitation) may be determined as regions containing palms. For example, the second confidence levels corresponding to regions 80c and 80d are greater than the second confidence threshold. Right palm posture 80e can be obtained by linking palm keypoints detected in region 80c, and left palm posture 80f can be obtained by linking palm keypoints detected in region 80d. The left palm posture 80f and right palm posture 80e described above can be referred to as part posture detection results corresponding to image frame 80a.

[0072] In step S206, a reference pose associated with the object is obtained, and a number of first key points corresponding to the reference pose and a number of second key points corresponding to the object pose detection result are determined.

[0073] Specifically, the computing device may obtain a reference pose corresponding to the object, and count a first number of object keypoints included in the reference pose and a second number of object keypoints included in the object pose detection result, where the first number of keypoints is known when the reference pose is constructed, and the second number of keypoints is the number of object keypoints predicted by the object detection model.

[0074] In step S207, if the number of first key points is greater than the number of second key points, an interpolation process is performed on the object pose detection result based on the reference pose to obtain a first candidate object pose.

[0075] Specifically, if the number of first keypoints is greater than the number of second keypoints, this indicates that there are missing object keypoints in the object pose detection result, and keypoint correction (interpolation processing) of this object pose detection result can be performed using the reference pose to complete the missing object keypoints and obtain a first candidate object pose corresponding to the object. As shown in Figure 2, human body pose 20m can be obtained by performing keypoint correction of human body pose 20j (object pose detection result) using the reference pose 20k. In this case, human body pose 20m can be called the first candidate object pose.

[0076] For example, if the object is a human body, and keypoints for the knees, ankles, feet, elbows, etc. are missing from the object pose detection result predicted by the object detection model, the object pose detection result may be interpolated using a reference pose to obtain a more reasonable first candidate object pose. For example, the missing object keypoints may be added. Interpolating the object pose detection result using the reference pose can improve the completeness and reasonableness of the object pose.

[0077] In step S208, based on the part orientation detection result, an interpolation process is performed on the object parts related to the first object part in the first candidate object orientation to obtain the overall orientation corresponding to the object.

[0078] Specifically, in practical application scenarios, the pose change of an object depends heavily on a small number of parts of the object, that is, some specific parts of the object (e.g., arm parts in human anatomy, which may include key points of parts such as the palm, wrist, and elbow) play an important role in the final result. Therefore, in embodiments of the present application, an interpolation process of object parts related to the first object part in the first candidate object pose may be performed based on the part pose detection result. This allows obtaining a global pose corresponding to the object. In some embodiments, if the part pose detection result is null (i.e., the first object part is not included in the image frame), the first candidate object pose may be directly determined as the global pose corresponding to the object.

[0079] For example, if the object is a human body and the first object part is a palm, if the image frame includes an elbow part, the keypoints for the elbow part can be predicted using the object detection model. If the image frame does not include an elbow part, the keypoints for the elbow part cannot be predicted using the object detection model. In this case, the elbow keypoints and wrist keypoints of the object can be determined based on the part pose detection result, and the elbow keypoints and wrist keypoints can be added to the first candidate object pose. This allows the overall pose corresponding to the object to be obtained.

[0080] In some embodiments, the object includes a second object part and a third object part, and the second object part and the third object part are symmetrical, for example, if the second object part is the right arm of the object, the third object part is the left arm of the object, if the second object part is the left leg of the object, the third object part is the right leg of the object, etc.

[0081] When the part pose detection result includes all part keypoints of the first object part (assuming the first object part is a palm, the part pose detection result includes the left and right palm keypoints), and the object pose detection result includes the pose of the second object part but not the pose of the third object part, i.e., when the image frame includes the second object part but not the third object part, the first part orientation corresponding to the third object part may be determined based on the keypoint positions of the first object part included in the part pose detection result. Here, the second object part and the third object part are symmetrical parts of the object. Because the second object part and the third object part are symmetrical parts, the lengths of the second object part and the third object part are the same. Therefore, the first part length of the second object part in the first candidate object pose is obtained, and the keypoint positions of the third object part are determined based on the first part length and first part orientation, and the keypoint positions of the third object part are added to the first candidate object pose, thereby obtaining the overall pose corresponding to the object in the image frame.

[0082] If the object posture detection result does not include the postures of the second object part and the third object part, i.e., if neither the second object part nor the third object part is included in the image frame, the second part direction corresponding to the second object part and the third part direction corresponding to the third object part may be determined based on the key point position of the first object part included in the part posture detection result. Furthermore, the second part length corresponding to the second object part and the third part length corresponding to the third object part may be obtained from the (i-1)th image frame. In other words, the length of the second object part in the previous image frame is set as the length of the second object part in the image frame, and the third part length corresponding to the third object part in the previous image frame is set as the length of the second object part in the image frame. lengthmay be set as the length of the third object part in the image frame. Furthermore, key point positions of the second object part may be determined based on the second part length and the second part orientation, and key point positions of the third object part may be determined based on the third part length and the third part orientation, and the key point positions of the second object part and the third object part may be added to the first candidate object pose to obtain an overall pose corresponding to the object in the image frame. Similarly, if the (i-1)th image frame does not include the second object part and the third object part, the respective lengths of the second object part and the third object part in the (i-2)th image frame may be subsequently obtained to determine the respective key point positions of the second object part and the third object part in the image frame. If the second object part and the third object part are not detected in all image frames prior to the (i-1)th image frame, the respective key point positions of the second object part and the third object part in the image frame may be determined by setting approximate lengths for each of the second object part and the third object part based on the first candidate object pose.

[0083] For example, suppose the object is a human body, the first object part is a palm, and the second and third object parts are the left and right arms, respectively. Assuming that the left and right palms are detected from the image frame, the direction of the left forearm may be calculated using key points on the left palm, and the direction of the right forearm may be calculated using key points on the right palm. The left forearm belongs to a part of the left arm, and the right forearm belongs to a part of the right arm.

[0084] If the left and right arms are not detected from the image frame, the left and right forearm lengths (second part lengths and 3The first part length may be used as the left and right forearm lengths in the image frame. If the left and right arms are not detected in either the image frame or the previous image frame, the shoulder length in the image frame may be referenced to assign reference lengths to the left and right forearms in the image frame. If one of the left and right arms (e.g., the left arm) is found in the image frame, the left forearm length (first part length) may be directly assigned to the right forearm. For example, it is known that the right wrist point A, the right palm point B, and the right elbow point C are missing. The direction of the right forearm can be expressed as the direction from the right palm point B to the right wrist point A and can be denoted as vector BA. The left forearm length can be expressed as the length from the right wrist point A to the right elbow point C and can be denoted as L. The position coordinate of the right elbow point C can be calculated from the above information and can be expressed as C = A + BA_normal * L. Here, C represents the position coordinate of the elbow point, A represents the position coordinate of the wrist point, and BA_normal represents the unit vector of vector BA.

[0085] As can be seen, when the left and right arms are detected from an image frame, the elbow points predicted by the object detection model can be adjusted and updated based on the detected palm keypoints, thereby improving the accuracy of the elbow points and further improving the rationality of the overall pose.

[0086] In some embodiments, the pose obtained by interpolating the first candidate object pose based on the part pose detection result may contain some inconsistent object keypoints. Therefore, the final overall pose of the object may be obtained by correcting the inconsistent object keypoints with reference to the reference pose. Specifically, if a third object part is not detected from the image frame, the computing device may determine the first candidate object pose, to which the keypoint position of the third object part is added, as the second candidate object pose. Furthermore, a pose offset between the reference pose and the second candidate object pose may be obtained. If the pose offset is greater than an offset threshold (which may be understood as the maximum angle to which an object can normally be offset), the keypoints of the second candidate object pose may be corrected based on the reference pose to obtain an overall pose of the object in the image frame. Here, the pose offset may be understood as the relative angle between the second candidate object pose and the reference pose. For example, if the object is a human body, the pose offset may be the angle between the shoulder of the second candidate object pose and the shoulder of the reference pose.

[0087] Please refer to FIG. 11. FIG. 11 is a schematic diagram of the object keypoint correction provided in the embodiment of the present application. As shown in FIG. 11, for an image frame 90a, after obtaining a second candidate object pose corresponding to this image frame 90a, a human body model 90b can be constructed based on this second candidate object pose. Due to the limited performance of the object detection model, compared with the normal human body structure (e.g., the reference pose), there is an obvious depression problem in region 90c (e.g., the shoulder region) in this human body model 90b. For example, 2 The angle between the shoulder of the candidate object pose and the shoulder of the reference pose is greater than the offset threshold. Area in 90bA human body model 90d may be obtained by rectifying region 90c. Region 90e in this human body model 90d is considered to be the result of rectifying region 90c. The human body pose corresponding to human body model 90d can be referred to as the overall pose corresponding to the object in image frame 90a.

[0088] Video data captured in mobile scenarios usually cannot include the entire object. The pose of the object predicted by the object detection model is not perfect. The rationality of the overall pose can be improved by processes such as keypoint interpolation and keypoint correction. Based on the part pose detection result, the position of the object keypoints related to the first object part can be calculated, thereby improving the accuracy of the overall pose.

[0089] Please refer to FIG. 12. FIG. 12 is a schematic diagram of the object pose estimation process provided in an embodiment of the present application. As shown in FIG. 12, assuming the object is a human body, a computing device may acquire video data or image data captured in a mobile scenario, and then acquire a 3D human body pose estimation model (object detection model) with a confidence level and a 3D palm pose estimation model (body part detection model) with a confidence level. The 3D human body pose estimation model can predict 3D human body keypoints in any one image frame. These 3D human body keypoints can constitute the object pose detection result. The 3D palm pose estimation model can predict 3D palm keypoints in any one image frame. These 3D palm keypoints can constitute the body part pose detection result. To complete missing human body keypoints, an interpolation process may be performed on the 3D human body pose estimation model based on a default human body pose (reference pose). A candidate human body pose (the second candidate object pose) may be obtained by combining the palm 3D keypoints and the human body 3D keypoints and performing interpolation processing of the elbow and wrist of the human body (object). If any human body keypoints that do not match the normal human body structure (human body keypoints whose pose offset amount is greater than the offset threshold) are detected from the candidate human body pose, these human body keypoints that do not match the normal human body structure may be corrected, and a final reasonable 3D pose estimation result (i.e., the overall pose) may be obtained.

[0090] As will be appreciated, specific embodiments of the present application may involve the collection of user video, and the application of the above examples of the present application to specific products or technologies may require user permission or consent, and the collection, use, and processing of related data may be subject to the relevant laws, regulations, and standards of the relevant country or region.

[0091] In the embodiments of the present application, by performing object pose estimation and specific part pose estimation for an object in an image frame, it is possible to obtain an object pose detection result for the object and a part pose detection result for a first object part of the object; further, based on the object pose detection result, part pose detection result, and reference pose, it is possible to perform pose estimation of the object in the image frame, correct missing part keypoints of the object in the image frame, and correct object keypoints that do not match the reference pose, thereby ensuring the completeness and rationality of the finally obtained overall pose of the object, and further improving the estimation accuracy of the overall pose.

[0092] Please refer to Fig. 13. Fig. 13 is a schematic diagram of the configuration of a data processing device provided in an embodiment of the present application. As shown in Fig. 13, the data processing device 1 may include a posture detection module 11 and a posture estimation module 12.

[0093] The posture detection module 11 obtains an object posture detection result corresponding to an object in an image frame and a part posture detection result corresponding to a first object part of the object in the image frame, wherein at least one object part of the object is missing in the object posture detection result, and the first object part is one or more parts of the object.

[0094] The posture estimation module 12 performs an interpolation process for at least one missing object part in the object posture detection result based on the part posture detection result and a reference posture associated with the object, thereby obtaining an overall posture corresponding to the object, which is used to operate a computer device to realize a service function corresponding to the overall posture.

[0095] Here, for specific methods of realizing the functions of the posture detection module 11 and posture estimation module 12, please refer to the explanation of steps S101 to S102 in the embodiment corresponding to Fig. 3. Further explanation will be omitted here.

[0096] In the embodiments of the present application, by respectively performing overall object pose estimation and specific part pose estimation for an object in an image frame, it is possible to obtain an object pose detection result for the object and a part pose detection result for a first object part of the object; further, based on the object pose detection result, part pose detection result, and reference pose, it is possible to perform pose estimation of the object in the image frame, correct missing part keypoints of the object in the image frame, ensure the completeness and rationality of the finally obtained overall pose of the object, and further improve the estimation accuracy of the overall pose.

[0097] Please refer to Fig. 14. Fig. 14 is a schematic diagram of the configuration of another data processing device provided in an embodiment of the present application. As shown in Fig. 14, this data processing device 2 includes a posture detection module 21, a posture estimation module 22, and a virtual object construction module 23.

[0098] The posture detection module 21 obtains an object posture detection result corresponding to an object in an image frame and a part posture detection result corresponding to a first object part of the object in the image frame, and in the object posture detection result, at least one object part of the object is missing, and the first object part is one or more parts of the object.

[0099] The pose estimation module 22 obtains an overall pose corresponding to the object by interpolating at least one missing object part in the object pose detection result based on the part pose detection result and a reference pose associated with the object.

[0100] The virtual object construction module 23 constructs a virtual object related to the object and controls the attitude of the virtual object based on the overall attitude.

[0101] Here, the specific functional implementation methods of the posture detection module 21, posture estimation module 22, and virtual object construction module 23 can be referred to in the explanation of the related steps above, and further explanation will be omitted here.

[0102] In one or more embodiments, the pose detection module 21 includes an object detection unit 211 and a part detection unit 212 .

[0103] The object detection unit 211 inputs the image frame into an object detection model, and obtains the object pose detection result through the object detection model.

[0104] The part detection unit 212 inputs the image frame into a part detection model, and obtains the part pose detection result through the part detection model.

[0105] Here, for specific implementation methods of the functions of the object detection unit 211 and the part detection unit 212, please refer to step S101 in the embodiment corresponding to Fig. 3. Further explanation will be omitted here.

[0106] In one or more embodiments, the object detection unit 211 may include a part classification subunit 2111, a part mapping generation subunit 2112, a localization result determination subunit 2113, and a detection result determination subunit 2114.

[0107] The part classification subunit 2111 inputs the image frame into an object detection model, obtains object pose features corresponding to the object in the image frame through the object detection model, and recognizes a first classification result corresponding to the object pose features, which is for representing the object part classification corresponding to the key points of the object.

[0108] The part mapping generation subunit 2112 generates a first activation mapping based on the first classification result and the object convolution features of the image frame output from the object detection model.

[0109] The localization result determining subunit 2113 obtains pixel mean values ​​corresponding to the first activation mapping, and determines localization results in the image frame of the key points in the object based on the pixel mean values.

[0110] The detection result determination subunit 2114 determines an object pose detection result corresponding to the image frame based on the object part classification and the localization result.

[0111] Here, for specific functional implementation methods of the part classification subunit 2111, part mapping generation subunit 2112, position identification result determination subunit 2113, and detection result determination subunit 2114, please refer to steps S201 to S203 in the embodiment corresponding to Fig. 7. Further explanation will be omitted here.

[0112] In one or more embodiments, the region classification subunit 2111 includes an overall classification subunit 21111, an overall mapping acquisition subunit 21112, a block division processing subunit 21113, and a feature combination subunit 21114.

[0113] The global classification subunit 21111 obtains object description features corresponding to objects in the image frames from the object detection model, and outputs a second classification result corresponding to the object description features based on the classifier in the object detection model.

[0114] The global mapping acquisition subunit 21112 acquires the object convolution features for the image frame output from the convolution layer in the object detection model, and performs a multiplication operation between the second classification result and the object convolution features to acquire a second activation mapping corresponding to the image frame.

[0115] The block division processing subunit 21113 performs block division processing of the image frame based on the second activation mapping to obtain M (M is a positive integer) object part area images, and obtains part description features corresponding to each of the M object part area images based on the object detection model.

[0116] The feature combination subunit 21114 combines the object description feature and the part description features corresponding to the M object part region images as an object pose feature.

[0117] Here, for specific functional implementation methods of the overall classification subunit 21111, the overall mapping acquisition subunit 21112, the block division processing subunit 21113, and the feature combination subunit 21114, please refer to step S201 in the embodiment corresponding to Figure 7. Further explanation will be omitted here.

[0118] In one or more embodiments, the part detection module 212 may include an object part detection subunit 2121 , a part pose estimation subunit 2122 , and a null value determination subunit 2123 .

[0119] The object part detection subunit 2121 inputs the image frame into a part detection model, and detects a first object part of the object in the image frame in the part detection model.

[0120] When a first object part is detected from an image frame, the part posture estimation subunit 2122 obtains an area image including the first object part from the image frame, obtains part keypoint positions corresponding to the first object part based on the area image, and determines a part posture detection result corresponding to the image frame based on the part keypoint positions.

[0121] The null value determination subunit 2123 determines the part pose detection result corresponding to the image frame as a null value when the first object part is not detected from the image frame.

[0122] Here, for specific functional implementation methods of the object part detection subunit 2121, part posture estimation subunit 2122, and null value determination subunit 2123, refer to steps S204 to S205 in the embodiment corresponding to Fig. 7. Further explanation will be omitted here.

[0123] In one or more embodiments, the part pose estimation subunit 2122 may include an image cropping subunit 21221 , a part keypoint determination subunit 21222 , and a part keypoint connection subunit 21223 .

[0124] When the first object part is detected from the image frame, the image cropping subunit 21221 crops the image frame to obtain a region image including the first object part.

[0125] The region keypoint determination subunit 21222 obtains region contour features corresponding to the region image, and predicts region keypoint positions corresponding to the first object region based on the region contour features.

[0126] The part keypoint concatenation subunit 21223 obtains a part pose detection result corresponding to the image frame by concatenating the keypoints of the first object part according to the part keypoint positions.

[0127] Here, the specific functional implementation of the image cropping subunit 21221, the region keypoint determining subunit 21222, and the region keypoint linking subunit 21223 can be seen in step S205 in the embodiment corresponding to Figure 7. Further explanation will be omitted here.

[0128] In one or more embodiments, the pose estimation module 22 includes a keypoint number determination unit 221, a first interpolation processing unit 222, and a second interpolation processing unit 223.

[0129] The keypoint number determination unit 221 obtains a reference pose associated with an object, and determines a first keypoint number corresponding to the reference pose and a second keypoint number corresponding to the object pose detection result.

[0130] If the number of first key points is greater than the number of second key points, the first interpolation processing unit 222 obtains a first candidate object pose by performing interpolation processing for at least one missing object part in the object pose detection result based on the reference pose.

[0131] The second interpolation processing unit 223 obtains an overall posture corresponding to the object by performing interpolation processing of the object parts related to the first object part in the first candidate object posture based on the part posture detection result.

[0132] Here, the number of keypoints decision For specific implementation methods of the functions of the unit 221, the first interpolation processing unit 222, and the second interpolation processing unit 223, please refer to steps S206 to S208 in the embodiment corresponding to Fig. 7. Further explanation will be omitted here.

[0133] In one or more embodiments, the second interpolation processing unit 223 may include: a first direction determining subunit 2231 , a first position determining subunit 2232 , and a first keypoint adding subunit 2233 .

[0134] When the object posture detection result includes the posture of the second object part and does not include the posture of the third object part, the first direction determination subunit 2231 determines the first part direction corresponding to the third object part based on the keypoint position of the first object part included in the part posture detection result, and the second object part and the third object part are symmetrical parts of the object and are related to the first object part.

[0135] The first position determining subunit 2232 obtains a first part length of the second object part in the first candidate object pose, and determines keypoint positions of a third object part based on the first part length and the first part orientation.

[0136] The first keypoint adding subunit 2233 adds the keypoint positions of the third object part to the first candidate object pose, thereby obtaining a global pose corresponding to the object in the image frame.

[0137] In some embodiments, the first keypoint adding subunit 2233 specifically includes: obtaining a second candidate object pose corresponding to the object in the image frame by adding the keypoint position of the third object part to the first candidate object pose; and obtaining a pose offset between the reference pose and the second candidate object pose; If the pose offset amount is greater than the offset threshold, the keypoints of the second candidate object pose are corrected based on the reference pose to obtain a global pose corresponding to the object in the image frame.

[0138] In some embodiments, the image frame is the i-th image frame (i is a positive integer) in the video data, and the second interpolation processing unit 223 may further include a second direction determination subunit 2234, a second position determination subunit 2235, and a second keypoint addition subunit 2236.

[0139] When the object posture detection result does not include the postures of the second object part and the third object part, the second direction determination subunit 2234 determines a second part direction corresponding to the second object part and a third part direction corresponding to the third object part based on the keypoint position of the first object part included in the part posture detection result, and the second object part and the third object part are symmetrical parts of the object and are related to the first object part.

[0140] The second position determination subunit 2235 obtains a second part length corresponding to the second object part and a third part length corresponding to the third object part from the jth (j is a positive integer less than i) image frame, and determines the keypoint position of the second object part based on the second part length and the second part direction.

[0141] The second keypoint adding subunit 2236 determines the keypoint positions of the third object part based on the third part length and the third part orientation, and obtains a global pose corresponding to the object in the image frame by adding the keypoint positions of the second object part and the keypoint positions of the third object part to the first candidate object pose.

[0142] Here, for specific functional implementation manners of the first direction determination subunit 2231, the first position determination subunit 2232, the first keypoint adding subunit 2233, the second direction determination subunit 2234, the second position determination subunit 2235, and the second keypoint adding subunit 2236, please refer to step S208 in the embodiment corresponding to Figure 7. Further description will be omitted here. Here, when the first direction determination subunit 2231, the first position determination subunit 2232, and the first keypoint adding subunit 2233 perform their respective operations, the second direction determination subunit 2234, the second position determination subunit 2235, and the second keypoint adding subunit 2236 all suspend the execution of their operations. When the second direction determination subunit 2234, the second position determination subunit 2235, and the second keypoint adding subunit 2236 perform their respective operations, the first direction determination subunit 2231, the first position determination subunit 2232, and the first keypoint adding subunit 2233 all pause the execution of their operations.

[0143] In the embodiments of the present application, by performing object pose estimation and specific part pose estimation for an object in an image frame, it is possible to obtain an object pose detection result for the object and a part pose detection result for a first object part of the object; further, based on the object pose detection result, part pose detection result, and reference pose, it is possible to perform pose estimation of the object in the image frame, correct missing part keypoints of the object in the image frame, and correct object keypoints that do not match the reference pose, thereby ensuring the completeness and rationality of the finally obtained overall pose of the object, and further improving the estimation accuracy of the overall pose.

[0144] Please refer to FIG. 15. FIG. 15 is a schematic diagram of a computer device provided in an embodiment of the present application. As shown in FIG. 15, the computer device 1000 may be a user terminal, for example, the user terminal 10a in the embodiment corresponding to FIG. 1 above, or a server, for example, the server 10d in the embodiment corresponding to FIG. 1 above. This is not a limitation. For ease of understanding, the present application will take the case where the computer device is a user terminal as an example. The computer device 1000 may include a processor 1001, a network interface 1004, and a memory 1005. The computer device 1000 may further include a user interface 1003 and at least one communication bus 1002. Here, the communication bus 1002 is used to realize connection and communication between these components. Here, the user interface 1003 may include a standard wired interface or a wireless interface. The network interface 1004 may include a standard wired interface or a wireless interface (e.g., a Wi-Fi interface). The memory 1005 may be a high-speed RAM or a non-volatile memory, such as at least one disk storage device. Optionally, the memory 1005 may be at least one storage device located remotely from the processor 1001. As shown in Figure 15, the memory 1005, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a device control application.

[0145] Here, the network interface 1004 in this computer device 1000 can further provide a network communication function. Optionally, the user interface 1003 may further include a display and a keyboard. In the computer device 1000 shown in FIG. 15, the network interface 1004 can provide a network communication function, the user interface 1003 is mainly used to provide an input interface to the user, and the processor 1001 invokes a device control application stored in the memory 1005 to obtaining an object pose detection result corresponding to an object in an image frame and a part pose detection result corresponding to a first object part of the object in the image frame, wherein at least one object part of the object is missing in the object pose detection result, and the first object part is one or more parts of the object; The method can be used to realize the step of: obtaining an overall posture corresponding to an object by interpolating at least one missing object part in the object posture detection result based on the part posture detection result and a reference posture related to the object, wherein the overall posture is for operating a computer device to realize a service function corresponding to the overall posture.

[0146] It should be understood that the computer device 1000 described in the embodiments of the present application can execute the data processing method described in any one of the embodiments of Figures 3 and 7 above, can execute the data processing device 1 described in the embodiment corresponding to Figure 13 above, and can execute the data processing device 2 described in the embodiment corresponding to Figure 14 above. Further description will be omitted here, and the description of the beneficial effects of the same method will also be omitted.

[0147] In addition, an embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. The computer program includes computer instructions. When a processor executes the computer instructions, the data processing method described in any one of the embodiments of FIG. 3 and FIG. 7 can be performed. Therefore, further description is omitted here. Also, a description of the beneficial effects of the same method is omitted. For technical details not disclosed in the embodiment of the computer-readable storage medium of the present application, please refer to the description of the method embodiment of the present application. For example, the computer instructions may be executed on a single computer device, on multiple computer devices located in a single location, or on multiple computer devices distributed across multiple locations and connected to each other via a communication network. The multiple computer devices distributed across multiple locations and connected to each other via a communication network may constitute a blockchain system.

[0148] In addition, embodiments of the present application further provide a computer program product or a computer program. The computer program product or computer program may include computer instructions. The computer instructions may be stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and when the processor executes the computer instructions, the computer device can perform the data processing method described in any one of the embodiments of Figures 3 and 7 above. Therefore, further description will be omitted here, and beneficial effects of the same method will also be omitted. For technical details not disclosed in the embodiments of the computer program product or computer program of the present application, please refer to the description of the method embodiments of the present application.

[0149] Although the above-described method embodiments are expressed as a combination of a series of operations for ease of explanation, those skilled in the art should understand that the present application is not limited by the order of the operations described, since some steps may be performed in other orders or simultaneously. Next, those skilled in the art should also understand that the embodiments described in the specification belong to preferred embodiments, and that such operations and modules are not necessarily required for the present application.

[0150] The steps in the method according to the embodiment of the present application can be reordered, merged, and deleted according to actual needs.

[0151] The modules in the device according to the embodiment of the present application can be merged, split and deleted according to actual needs.

[0152] As will be understood by those skilled in the art, all or part of the method steps according to the above embodiments may be executed by instructing relevant hardware via a computer program. The computer program may be stored in a computer-readable storage medium. When the program is executed, the steps of each method embodiment as described above are executed. Here, the storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), etc.

[0153] The above are merely preferred embodiments of the present application, and of course, they do not limit the scope of the present application. Therefore, equivalent modifications according to the claims of the present application still fall within the scope of the present application.

Claims

1. 1. A data processing method executed by a computing device, comprising: obtaining an object pose detection result corresponding to an object in an image frame and a part pose detection result corresponding to a first object part of the object in the image frame, wherein at least one object part of the object is missing in the object pose detection result, and the first object part is one or more parts of the object; a step of obtaining a global pose corresponding to the object by interpolating at least one missing object part in the object pose detection result based on the part pose detection result and a reference pose related to the object, the global pose being for operating a computer device to realize a service function corresponding to the global pose; obtaining an object pose detection result corresponding to an object in an image frame and a part pose detection result corresponding to a first object part of the object in the image frame; inputting the image frame into an object detection model and obtaining the object pose detection result through the object detection model; inputting the image frame into a body part detection model and obtaining the body part posture detection result by the body part detection model; The step of inputting the image frame into an object detection model and obtaining the object pose detection result by the object detection model includes: inputting the image frame into an object detection model, obtaining object pose features corresponding to the object through the object detection model, and recognizing a first classification result corresponding to the object pose features, wherein the first classification result is for representing an object part classification corresponding to a key point of the object; generating a first activation mapping based on the first classification result and object convolution features of the image frame output from the object detection model; obtaining pixel mean values ​​corresponding to the first activation mapping and determining localizations of key points in the object in the image frames based on the pixel mean values; determining the object pose detection result based on the object part classification and the localization result; method.

2. The step of obtaining object pose features corresponding to the object using the object detection model includes: obtaining object description features corresponding to the object in the image frame from the object detection model, and outputting a second classification result corresponding to the object description features based on a classifier in the object detection model; obtaining object convolutional features for the image frame output from a convolutional layer in the object detection model, and performing a multiplication operation between the second classification result and the object convolutional features to obtain a second activation mapping corresponding to the image frame; acquiring M object region images (M is a positive integer) by performing block division processing of the image frame based on the second activation mapping, and acquiring region description features corresponding to each of the M object region images based on the object detection model; combining the object description feature and part description features corresponding to the M object part region images as the object pose feature; The method of claim 1.

3. the step of inputting the image frame to a body part detection model and obtaining the body part posture detection result using the body part detection model includes: inputting the image frame into a part detection model and detecting a first object part of the object in the part detection model; When the first object part is detected from the image frame, acquiring an area image including the first object part from the image frame, acquiring part key point positions corresponding to the first object part based on the area image, and determining a part posture detection result corresponding to the image frame based on the part key point positions; and determining a part posture detection result corresponding to the image frame as a null value if the first object part is not detected from the image frame. The method of claim 1.

4. When the first object part is detected from the image frame, the step of acquiring an area image including the first object part from the image frame, acquiring part key point positions corresponding to the first object part based on the area image, and determining a part posture detection result corresponding to the image frame based on the part key point positions includes: When the first object portion is detected from the image frame, a region image including the first object portion is acquired by cutting out the image frame; obtaining a region contour feature corresponding to the region image, and predicting a region keypoint position corresponding to the first object region based on the region contour feature; and acquiring a part pose detection result corresponding to the image frame by connecting keypoints of the first object part based on the part keypoint positions. The method of claim 3.

5. A data processing method executed by a computer device, comprising: obtaining an object pose detection result corresponding to an object in an image frame and a part pose detection result corresponding to a first object part of the object in the image frame, wherein at least one object part of the object is missing in the object pose detection result, and the first object part is one or more parts of the object; and performing interpolation processing for at least one missing object part in the object pose detection result based on the part pose detection result and a reference pose related to the object, thereby obtaining a global pose corresponding to the object, wherein the global pose is for operating a computer device to realize a service function corresponding to the global pose; the step of acquiring a global posture corresponding to the object by performing an interpolation process for at least one object part missing in the object posture detection result based on the part posture detection result and a reference posture related to the object, obtaining a reference pose associated with the object and determining a first number of keypoints corresponding to the reference pose and a second number of keypoints corresponding to the object pose detection; if the first number of key points is greater than the second number of key points, performing an interpolation process for at least one missing object part in the object pose detection result based on the reference pose to obtain a first candidate object pose; and performing an interpolation process of an object part related to the first object part in the first candidate object posture based on the part posture detection result, thereby obtaining an overall posture corresponding to the object. method.

6. the step of acquiring an overall posture corresponding to the object by performing an interpolation process on an object part related to a first object part in the first candidate object posture based on the part posture detection result, When the object pose detection result includes a pose of a second object part and the object pose detection result does not include a pose of a third object part, determining a first part orientation corresponding to the third object part based on a key point position of a first object part included in the part pose detection result, wherein the second object part and the third object part are symmetrical parts of the object and are related to the first object part; obtaining a first part length of the second object part in the first candidate object pose, and determining key point positions of the third object part based on the first part length and the first part orientation; and adding keypoint locations of the third object part to the first candidate object pose to obtain a global pose corresponding to the object in the image frame. The method of claim 5.

7. obtaining a global pose corresponding to the object in the image frame by adding keypoint locations of the third object part to the first candidate object pose; obtaining a second candidate object pose corresponding to the object in the image frame by adding keypoint positions of the third object part to the first candidate object pose, and obtaining a pose offset between the reference pose and the second candidate object pose; if the pose offset is greater than an offset threshold, correcting keypoints of the second candidate object pose based on the reference pose to obtain a global pose corresponding to the object in the image frame. The method of claim 6.

8. the image frame is the i-th (i is a positive integer) image frame in the video data, the step of acquiring an overall posture corresponding to the object by performing an interpolation process on an object part related to a first object part in the first candidate object posture based on the part posture detection result, when the object posture detection result does not include postures of a second object part and a third object part, determining a second part orientation corresponding to the second object part and a third part orientation corresponding to the third object part based on key point positions of a first object part included in the part posture detection result, wherein the second object part and the third object part are symmetrical parts of the object and related to the first object part; obtaining a second part length corresponding to the second object part and a third part length corresponding to the third object part from a j-th image frame (j is a positive integer smaller than i), and determining a key point position of the second object part based on the second part length and the second part orientation; determining key point positions of the third object part based on the third part length and the third part orientation, and adding the key point positions of the second object part and the key point positions of the third object part to the first candidate object pose to obtain a global pose corresponding to the object in the image frame; The method of claim 5.

9. constructing a virtual object related to the object, and controlling a pose of the virtual object based on the overall pose; The method of claim 1.

10. 1. A data processing device, comprising: a pose detection module that obtains an object pose detection result corresponding to an object in an image frame and a part pose detection result corresponding to a first object part of the object in the image frame, the first object part being one or more parts of the object; a posture estimation module that performs interpolation processing for missing object parts in the object posture detection result based on the part posture detection result and a reference posture related to the object, thereby obtaining an overall posture corresponding to the object, the overall posture being used to operate a computer device to realize a service function corresponding to the overall posture; Including, The posture detection module inputting the image frame into an object detection model and obtaining the object pose detection result through the object detection model; a function of inputting the image frame into a part detection model and acquiring the part and posture detection result by the part detection model; The function of inputting the image frame into an object detection model and obtaining the object pose detection result by the object detection model includes: a function of inputting the image frame into an object detection model, obtaining object pose features corresponding to the object through the object detection model, and recognizing a first classification result corresponding to the object pose features, wherein the first classification result is for representing an object part classification corresponding to a key point of the object; and generating a first activation mapping based on the first classification result and object convolution features of the image frame output from the object detection model; obtaining pixel mean values ​​corresponding to the first activation mapping and determining localizations of key points in the object in the image frames based on the pixel mean values; and determining the object pose detection result based on the object part classification and the position identification result. Device.

11. A data processing device, a pose detection module that obtains an object pose detection result corresponding to an object in an image frame and a part pose detection result corresponding to a first object part of the object in the image frame, the first object part being one or more parts of the object; a posture estimation module that performs interpolation processing for missing object parts in the object posture detection result based on the part posture detection result and a reference posture related to the object, thereby obtaining an overall posture corresponding to the object, the overall posture being used to operate a computer device to realize a service function corresponding to the overall posture; Including, The pose estimation module: obtaining a reference pose associated with the object; and determining a first number of key points corresponding to the reference pose and a second number of key points corresponding to the object pose detection result; If the first number of key points is greater than the second number of key points, performing an interpolation process for at least one missing object part in the object pose detection result based on the reference pose to obtain a first candidate object pose; and a function of performing an interpolation process of an object part related to a first object part in the first candidate object posture based on the part posture detection result, thereby acquiring an overall posture corresponding to the object. Device.

12. A computing device comprising a memory and a processor, The memory is connected to the processor, and a computer program is stored in the memory, and the processor invokes the computer program to cause the computer device to execute the method according to any one of claims 1 to 9. Computer equipment.

13. A computer program causing a computer to carry out the method according to any one of claims 1 to 9.

Images