DATA PROCESSING METHOD, DATA PROCESSING APPARATUS, COMPUTER DEVICE, AND COMPUTER PROGRAM

Abstract

Provided is a data processing method, device, equipment, and medium applicable to fields such as artificial intelligence, driving assistance, etc. The method includes the steps of: acquiring an object pose detection result corresponding to an object in an image frame and a part pose detection result corresponding to a first object part of the object in the image frame, where at least one object part of the object is missing in the object pose detection result, and the first object part is one or more parts of the object; and acquiring an overall pose corresponding to the object by performing an interpolation process for the at least one object part missing in the object pose detection result based on the part pose detection result and a reference pose related to the object, where the overall pose is for operating a computer device to realize a service function corresponding to the overall pose.

Description

[Technical field]

[0001] This application claims priority to a Chinese patent application filed with the China Patent Office on March 31, 2022, bearing application number 2022103327630 and titled "Data Processing Method, Apparatus, Device, and Medium," the entire contents of which are incorporated herein by reference.

[0002] The present application relates to the technical field of artificial intelligence, and in particular to a data processing method, device, apparatus, and medium. [Background technology]

[0003] Computer Vision (CV) technology is the science that studies how to make machines "see", more specifically, by using cameras and computers instead of human eyes to perform machine vision such as recognition and measurement, and then through graphic processing and computer processing to obtain images that are more suitable for human eyes to observe or transmit to instruments for detection. As a branch of science, Computer Vision studies related theories and technologies to build artificial intelligence systems that can obtain information from images or multi-dimensional data.

[0004] Pose estimation can detect the position of each key point in a picture or video, and has a wide range of application value in fields such as movie animation, driving assistance, virtual reality, and action recognition.

[0005] Current pose estimation algorithms can detect keypoints in an image or video and construct a final object pose based on the detected keypoints and object constraint relations. Summary of the Invention [Problem to be solved by the invention]

[0006] The embodiments of the present application provide a data processing method, device, apparatus, and medium that can improve the accuracy of estimating an object pose. [Means for solving the problem]

[0007] In an embodiment of the present application, there is provided a data processing method executed by a computing device, the method comprising: obtaining an object pose detection result corresponding to an object in an image frame and a part pose detection result corresponding to a first object part of the object in the image frame, where at least one object part of the object is missing in the object pose detection result, and the first object part is one or more parts of the object; and a step of obtaining an overall posture corresponding to the object by performing an interpolation process for at least one missing object part in the object posture detection result based on the part posture detection result and a reference posture related to the object, the overall posture being for realizing a service function corresponding to the overall posture by operating a computing device.

[0008] In an embodiment of the present application, there is further provided a data processing apparatus, comprising: a pose detection module that obtains an object pose detection result corresponding to an object in an image frame and a part pose detection result corresponding to a first object part of the object in the image frame, where at least one object part of the object is missing in the object pose detection result, and the first object part is one or more parts of the object; and a posture estimation module that obtains an overall posture corresponding to the object by performing an interpolation process for at least one object part missing in the object posture detection result based on the part posture detection result and a reference posture related to the object, the overall posture being for realizing a service function corresponding to the overall posture by operating a computing device.

[0009] In an embodiment of the present application, a computer device is further provided, the computer device comprising a memory and a processor, the memory being connected to the processor, a computer program being stored in the memory, and the processor invoking the computer program to cause the computer device to execute the above-mentioned method in the embodiment of the present application.

[0010] In an embodiment of the present application, a computer-readable storage medium storing a computer program is further provided, which, when loaded and executed by a processor, is configured to cause a computing device having the processor to perform the above-mentioned method in an embodiment of the present application.

[0011] In an embodiment of the present application, there is further provided a computer program product or computer program comprising computer instructions, the computer instructions being stored in a computer readable storage medium, a processor of a computing device reading the computer instructions from the computer readable storage medium, and, when executed by the processor, causing the computing device to perform the above method. [Brief description of the drawings]

[0012] [Figure 1] FIG. 2 is a schematic diagram of a network architecture configuration provided in an embodiment of the present application. [Diagram 2] FIG. 2 is a schematic diagram of an object pose estimation scenario for video data provided in an embodiment of the present application; [Diagram 3]FIG. 2 is a schematic diagram of the flow of a data processing method provided in an embodiment of the present application. [Figure 4] FIG. 2 is a schematic diagram of a reference posture provided in an embodiment of the present application. [Diagram 5] FIG. 2 is a schematic diagram of an object pose estimation scenario provided in an embodiment of the present application. [Figure 6] FIG. 2 is a schematic diagram of an application scenario of a global posture provided in an embodiment of the present application; [Figure 7] FIG. 2 is a schematic diagram of the flow of another data processing method provided in an embodiment of the present application. [Figure 8] FIG. 2 is a schematic diagram of the configuration of an object detection model provided in an embodiment of the present application. [Figure 9] FIG. 2 is a schematic diagram of a flow of obtaining an object pose detection result provided in an embodiment of the present application. [Figure 10] 10 is a schematic diagram of a flow of acquiring a part and posture detection result provided in an embodiment of the present application. [Figure 11] FIG. 2 is a schematic diagram of object keypoint correction provided in an embodiment of the present application; [Figure 12] FIG. 2 is a schematic diagram of the flow of object pose estimation provided in an embodiment of the present application. [Figure 13] 1 is a schematic diagram of the configuration of a data processing device provided in an embodiment of the present application. [Figure 14] FIG. 2 is a schematic diagram of the configuration of another data processing device provided in an embodiment of the present application. [Figure 15] FIG. 2 is a schematic diagram of the configuration of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, the configuration of the embodiment of the present application will be described clearly and completely with reference to the drawings of the embodiment of the present application. It is obvious that the described embodiment is only a part of the embodiment of the present application, not all of the embodiments. All other embodiments that a person skilled in the art can obtain from the embodiment of the present application without creative labor belong to the scope of protection of the present application.

[0014] This application relates to pose estimation under computer vision technology. Pose estimation is an important task in computer vision and is also a necessary step for computers to understand the motion and behavior of objects. Pose estimation can be converted into a prediction problem of object keypoints. For example, the position coordinates of each object keypoint in an image may be predicted, and the object skeleton in the image may be predicted based on the positional relationship between each object keypoint. Here, the pose estimation according to this application may include object pose estimation for an object, part pose estimation for a specific part of an object, etc. The object may include, but is not limited to, a human body, an animal, a plant, etc. The specific part of the object may be a palm, a face, an animal's limbs, a plant's roots, etc. This application does not limit the type of object.

[0015] When an image or video is a captured image in a mobile scenario, the image or video may only contain a part of the object, so that during the object pose estimation, some parts of the object are missing, which leads to insufficient extracted part information, and the final object pose result is not the complete pose of the object, which affects the completeness of the object pose.

[0016] In an embodiment of the present application, by performing object posture estimation and specific part posture estimation for an object in an image frame, an object posture detection result for the object and a part posture detection result for a first object part of the object can be obtained; further, based on the object posture detection result, the part posture detection result, and the reference posture, posture estimation of the object in the image frame can be performed; missing part keypoints of the object in the image frame can be corrected; the completeness and rationality of the finally obtained overall posture of the object can be ensured; and the estimation accuracy of the overall posture can be improved.

[0017] Please refer to FIG. 1. FIG. 1 is a schematic diagram of a network architecture configuration provided in an embodiment of the present application. As shown in FIG. 1, the network architecture may include a server 10d and a user terminal cluster. The user terminal cluster may include one or more user terminals. The number of user terminals is not limited here. As shown in FIG. 1, the user terminal cluster may specifically include a user terminal 10a, a user terminal 10b, and a user terminal 10c, etc.

[0018] Here, server 10d may be an independent physical server, or may be a server cluster or distributed system consisting of multiple physical servers, or may be a cloud server that provides base cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0019] The user terminal 10a, the user terminal 10b, the user terminal 10c, etc. may include electronic devices having an object pose estimation function, such as smartphones, tablet computers, notebook computers, palmtop computers, mobile internet devices (MIDs), wearable devices (e.g., smart watches, smart bracelets, etc.), smart voice interaction devices, smart home appliances (e.g., smart TVs, etc.), and in-vehicle devices. As shown in FIG. 1, the user terminal 10a, the user terminal 10b, the user terminal 10c, etc. can each be network-connected to the server 10d. This allows each user terminal to exchange data with the server 10d via this network connection.

[0020] A user terminal (e.g., user terminal 10a) in the user terminal cluster shown in FIG. 1 incorporates an application client having an object pose estimation function. The application client may include, but is not limited to, a multimedia client (e.g., a short video client, a video live streaming client, a video client) and an object management application (e.g., a patient care client). The application client in the user terminal 10a can acquire video data. The video data may be a video of an object captured in a mobile-side scenario. For example, an object may be captured by a camera incorporated in the user terminal 10a to acquire video data, or an object may be captured by a photographing device (e.g., a single-lens reflex camera, a camera, etc.) connected to the user terminal 10a to acquire video data. Note that in a mobile-side scenario (e.g., a self-portrait scenario), due to restrictions on the shooting distance and the photographing device, only a part of the object may be included in the screen in the video data. For example, if the object is a human body, the screen in the video data may include only the upper body of the human body, or only the head of the human body. When performing object pose estimation of an object in the video data, it is necessary to correct the pose of the object included in the video data. This allows obtaining a global pose corresponding to the object, and also improves the accuracy of the global pose while preserving the integrity of the global pose of the object. In the present embodiment, the global pose is also called a complete pose, which refers to a pose that includes all parts of the object, i.e., a pose that corresponds to the complete object.

[0021] It should be noted that the object pose estimation process according to the embodiment of the present application may be performed by a computer device. The computer device may be a user terminal in the user terminal cluster shown in Fig. 1, or may be the server 10d shown in Fig. 1. In short, the computer device may be a user terminal, a server, or a combination device composed of a server and a user terminal. The present application is not limited thereto.

[0022] Please refer to FIG. 2. FIG. 2 is a schematic diagram of an object pose estimation scenario for video data provided in an embodiment of the present application. Taking the user terminal 10a shown in FIG. 1 as an example, the object pose estimation process in a video will be described. As shown in FIG. 2, the user terminal 10a can acquire video data 20a. This video data 20a may be a video of an object captured by a camera built into the user terminal 10a, or may be a video related to an object transmitted from another device to the user terminal 10a. By performing a framing process on the video data 20a, N image frames are acquired. N is a positive integer, and for example, the value of N may be 1, 2, .... The first image frame in time series (i.e., image frame T1) is acquired from the N image frames, and this image frame T1 is input to an object detection model 20b. By performing object detection on the image frame T1 using this object detection model 20b, an object pose detection result 20c corresponding to this image frame T1 can be acquired. The object pose detection result 20c may include keypoints of an object included in the image frame T1 (for convenience of explanation, hereinafter, the keypoints of the object are referred to as object keypoints) and the positions of these object keypoints in the image frame T1. The object pose detection result 20c may further include a first confidence corresponding to each detected object keypoint. The first confidence can be used to represent the prediction accuracy of the detected object keypoint, and a larger first confidence represents that the detected object keypoint is more accurate and more likely to be a real keypoint of the object.

[0023] For example, if the object in the video data 20a is a human body, the object key points corresponding to the object are considered as joint points in the human body structure, and the number of object key points and the classification of the key points may be predefined, for example, the human body structure may include multiple object key points of limbs, brain, waist, chest, etc. If the image frame T1 includes a complete object, the image frame T1 may include all object key points of the object, and if the image frame T1 includes only a part of the object structure, the image frame T1 may include some object key points of the object. After detecting the object key points included in the image frame T1, the detected object key points may be concatenated according to the classification of the object's key points and the key point positions, and the concatenated result, i.e., the object pose detection result 20c, may be marked on the image frame T1. Here, the object detection model 20b may be a pre-trained network model, and has the function of object detection for videos / images. If the object is a human body, the object detection model 20b is also called a human body pose estimation model.

[0024] The object pose detection result 20c can obtain a human body pose 20j of the object in the image frame T1. Since some object key points are missing in the human body pose 20j (joint points of the human body are missing), the user terminal 10a may obtain a reference pose 20k corresponding to the object, and the human body pose 20m corresponding to the object in the image frame T1 can be obtained by correcting the key points of the human body pose 20j based on the reference pose 20k. Here, the reference pose 20k is also considered as a default pose of the object and is also called a reference pose. The reference pose 20k may be constructed in advance based on all object key points of the object. For example, a pose (e.g., a whole pose) when the human body is normally standing may be determined as the reference pose 20k.

[0025] The image frame T1 may be input to a part detection model 20d, and a specific part (e.g., a first object part) of the object in the image frame T1 may be detected by the part detection model 20d to obtain a part posture detection result 20e corresponding to the image frame T1. If it is detected that the first object part of the object does not exist in the image frame T1, the part posture detection result of the image frame T1 may be determined as null. If it is detected that the first object part of the object exists in the image frame T1, the key points and the positions of the key points of the first object part may be subsequently detected, and the detected first object part key points may be linked according to the key point classification and the key point position of the first object part, and the result after linking, i.e., the part posture detection result 20e, may be marked on the image frame T1. Here, the number of key points and the key point classification corresponding to the first object part may be predefined as well. If the object is a human body, the part detection model 20d may be a palm posture estimation model (here, the first object part is a palm). For example, a palm may include a palm center keypoint and finger keypoints. The part detection model 20d may be a pre-trained network model, and has an object part detection function for a video / image. For convenience of explanation, the first object part keypoint is hereinafter referred to as the part keypoint.

[0026] As shown in FIG. 2, the part posture detection result 20e is assigned with a second reliability. This second reliability can be used to indicate the possibility that the detected object part is the first object part. For example, the part detection model 20d can determine that the second reliability that the region 20f in the image frame T1 is the first object part is 0.01, the second reliability that the region 20g is the first object part is 0.09, the second reliability that the region 20h is the first object part is 0.86, and the second reliability that the region 20i is the first object part is 0.84. The higher the second reliability, the higher the possibility that the region is the first object part. For example, based on the second reliability, it can be determined that the first object part is included in the region 20h and the region 20i, and the posture of the first object part may be marked in the region 20h and the region 20i.

[0027] Furthermore, the user terminal 10a may combine the object pose detection result 20c and the part pose detection result 20e to perform an interpolation process for some missing object parts, and obtain a reasonable object key point through the interpolation process. For example, if the part pose detection result 20e is a palm key point, the object pose detection result 20c and the part pose detection result 20e may be combined to perform an interpolation process for parts such as the wrist and elbow of the object that are missing in the image frame T1, thereby completing the human body pose 20m of the object and obtaining a human body pose 20n (also called a global pose). Similarly, after obtaining a global pose corresponding to the object in the image frame T1, the same method may be used to perform object pose estimation for subsequent image frames in the video data 20a to obtain a global pose corresponding to the object in each image frame. Based on the global pose corresponding to each of the N image frames, the behavior of the object in the video data 20a can be obtained. As can be understood, the video data 20a may be a video shot in real time. The user terminal 10a can obtain the behavior of an object in real time by performing object pose estimation for image frames in video data captured in real time.

[0028] In other words, for an image frame that includes only a part of an object, the object detection result output from the object detection model 20b, the part detection result output from the part detection model 20d, and the reference posture 20 k The global pose of the object in the image frame may be estimated by: , which can ensure the completeness and rationality of the finally obtained global pose of the object, and further improve the estimation accuracy of the global pose.

[0029] Please refer to Fig. 3. Fig. 3 is a schematic diagram of the flow of a data processing method provided in an embodiment of the present application. As shown in Fig. 3, the data processing method may include the following steps S101 to S102.

[0030] In step S101, an object pose detection result corresponding to an object in an image frame and a part pose detection result corresponding to a first object part of the object are obtained, and at least one object part of the object is missing in the object pose detection result, and the first object part is one or more parts of the object.

[0031] Specifically, the computing device may obtain video data (e.g., video data 20a in the embodiment corresponding to FIG. 2) or image data of an object captured in a scenario on the mobile side. When performing pose estimation of the video data or image data, the computing device may obtain an object pose detection result for the object (e.g., object pose detection result 20c in the embodiment corresponding to FIG. 2 above) by performing object detection of an image frame in the image data or video data. In addition, a part pose detection result for a first object part of the object (e.g., part pose detection result 20e in the embodiment corresponding to FIG. 2 above) may be obtained by performing part detection of the image frame. Here, the object may be an object included in the video data, such as a human body, an animal, a plant, etc., and the first object part may be one or more parts of the object, such as a face, a palm in a human body structure, a limb, a tail, a head, a root of a plant, etc. in an animal structure. The present application does not limit both the type of the object and the type of the first object part. In addition, due to the limited distance between the image capture device and the captured object in the mobile scenario, the object in the video data or image data may be missing parts, i.e., some object parts of the object may not be present on the screen of the video data. By combining the object pose detection result and the part pose detection result, the accuracy of the object pose estimation can be improved.

[0032] For convenience of explanation, in all the embodiments of the present application, the object pose estimation process of video data or image data will be described taking the case where the object is a human body as an example. When performing object pose estimation of image data in a mobile-side scenario, the image data may be image frames. When performing object pose estimation of video data in a mobile-side scenario, N image frames (N is a positive integer) corresponding to the video data may be obtained by performing framing processing of the video data. Furthermore, an image frame sequence including N image frames may be configured in a time series of N image frames in the video data. Object pose estimation may be performed sequentially for N image frames in the image frame sequence. For example, after completing object pose estimation of the first image frame in the image frame sequence, object pose estimation may be performed successively for the second image frame in the image frame sequence, and this may be repeated until object pose estimation of the entire video data is completed.

[0033] Here, the computer device may obtain an object detection model and a part detection model, and input the image frame into the object detection model. The object detection model may output an object pose detection result corresponding to the image frame. In addition, the image frame may be input into the part detection model. The part detection model may output a part pose detection result corresponding to the image frame. Here, the object detection model can be used to detect key points of an object in the image frame (also called object key points, for example, human body key points). In this case, the object detection model is also called a human body pose estimation model. The object detection model may include, but is not limited to, DensePose (a human body real-time pose recognition system for realizing real-time pose recognition of a dense crowd of people), OpenPose (a framework for estimating the body, face, and hand shapes of multiple people in real time), Realtime Multi-Person Pose Estimation (a real-time multi-person pose estimation model), DeepPose (a pose estimation method based on a deep neural network), and mobilenetv2 (a lightweight deep neural network). This application does not limit the type of the object detection model. The part detection model can be used to detect keypoints (e.g., palm keypoints) of a first object part of an object. The part detection model in this case is also called a palm pose estimation model. The part detection model may be a detection-based method or a regression-based method. In the detection-based method, part keypoints of the first object part may be predicted by generating a heat map. In the regression-based method, position coordinates of part keypoints may be directly regressed. The network configuration of the part detection model and the network configuration of the object detection model may be the same or different. If the network configuration of the part detection model and the network configuration of the object detection model are the same, the network parameters of the two are different (trained from different data). This application does not limit the type of part detection model.

[0034] In some embodiments, the object detection model and the body part detection model may be detection models pre-trained using sample data. For example, the object detection model may be trained using sample data (e.g., a 3D human body data set) with human body keypoint tag information, and the body part detection model may be trained using sample data (e.g., a palm data set) with palm keypoint information. Alternatively, the object detection model may be an object detection service called from an artificial intelligence cloud service via an application program interface (API), and the body part detection model may be a body part detection service called from an artificial intelligence cloud service via an API. No specific limitations are provided here.

[0035] Here, artificial intelligence cloud services are also commonly referred to as AIaaS (AI as a Service). This is the service mode of the current mainstream artificial intelligence platform. Specifically, the AIaaS platform divides some regular AI services and provides independent or packaged services on the cloud side. This service mode is similar to opening an AI-themed shopping mall. All developers can access and use one or more artificial intelligence services provided by the platform through APIs. Some veteran developers can also deploy and manage their own dedicated cloud artificial intelligence services using the AI ​​framework and AI infrastructure provided by the platform.

[0036] In some embodiments, the object detection model used in the embodiments of the present application may be a human body 3D pose estimation model with a confidence level. For example, the object detection model may predict object keypoints of an object in an image frame, and each predicted object keypoint may correspond to one first confidence level. The first confidence level may be used to represent the prediction accuracy of each predicted object keypoint. The predicted object keypoint and the corresponding first confidence level may be referred to as an object pose detection result corresponding to the image frame. The part detection model may be a palm 3D pose estimation model with a confidence level. For example, the part detection model may predict a location region of a first object part in an image frame, and predict part keypoints of the first object part in the location region. The part detection model may predict one or more location regions where the first object part may be located. One location region may correspond to one second confidence level. The second confidence level may be used to represent the prediction accuracy of each predicted location region. The predicted part keypoint and the second confidence level corresponding to the location region may be referred to as a part pose detection result corresponding to the image frame.

[0037] In step S102, an overall posture corresponding to the object is obtained by performing an interpolation process for at least one missing object part in the object posture detection result based on the part posture detection result and a reference posture related to the object, and the overall posture is used to realize a service function corresponding to the overall posture by operating a computing device.

[0038] Specifically, the computing device may acquire a reference pose corresponding to the object (e.g., reference pose 20 in the embodiment corresponding to FIG. 2). k) may be obtained. This reference pose is considered to be the complete default pose (denoted as T-pose) of the object. The reference pose may be one or more in number, for example, a human body default standing pose, a human body default sitting pose, a human body default crouching pose, etc. The present application does not limit the type and number of reference poses.

[0039] Please refer to FIG. 4. FIG. 4 is a schematic diagram of a reference posture provided in an embodiment of the present application. As shown in FIG. 4, the model 30a can be expressed as a Skinned Multi-Person Linear (SMPL) model. This model 30a is a parameterized human body model and can be applied to different human body structures. This model 30a may include a human body joint distribution with one root node (node ​​number 0) and 23 joint nodes (nodes represented by numbers 1 to 23). Here, the root node is used to transform the entire human body as a perfect rigid body (an object whose volume and shape do not change when a force is applied), and the 23 joint nodes can be used to describe local human body part deformation. When the object is a human body, the above one root node and 23 joint nodes may be taken as object key points of the object. Based on the classification (e.g., wrist joint point, elbow joint point, palm joint point, ankle joint point, etc.) and position of the object key point, the reference posture 30b can be obtained by connecting the one root node and the 23 joint nodes.

[0040] An image frame may not include a complete object. For example, if some parts of an object (e.g., the lower limbs of a human body) are not present in an image frame, some object keypoints are missing in the object pose detection result corresponding to this image frame. A first candidate object pose corresponding to the object may be obtained by performing keypoint correction of the object pose detection result corresponding to the object using the reference pose to complete the missing object keypoints. If the part pose detection result includes part keypoints of the first object part, the part keypoints in the part pose detection result and the object keypoints in the object pose detection result may be combined to adjust the first candidate object pose to obtain a global pose of the object in the image frame. After obtaining the global pose corresponding to the current image frame, the global pose of the object in each image frame of the video data may be obtained by subsequently performing object pose estimation of the next image frame in the video data.

[0041] In some embodiments, the computer device can determine the behavior and motion of the object based on the overall posture of the object in the video data, and can manage or care for the object through these behaviors and motions, or perform human-computer interaction through the behavior and motion of the object. In short, the overall posture of the object in the video data can be applied to human-computer interaction scenarios (such as virtual reality and human-computer animation), content review scenarios, automatic driving scenarios, virtual live streaming scenarios, character motion design scenarios for games and movies, etc. In human-computer interaction scenarios, images (or videos) of a user (object) may be collected. After obtaining the overall posture in the image or video, the operation of a machine can be realized based on the overall posture. For example, a specific command is executed based on a specific human body motion (determined from the overall posture). In game character motion design scenarios, the cost and difficulty of game character motion design can be reduced by obtaining human body motion according to the overall posture corresponding to the object instead of expensive motion capture equipment.

[0042] Here, the virtual live streaming scenario may be to play a video of a virtual object that behaves and moves in the same way as the streamer user in the live streaming room, instead of directly playing a video of the streamer user (object) on the live screen of the live streaming room. For example, the streamer user's behavior and behavior may be determined based on the streamer user's overall posture, and the streamer user's behavior and behavior may be used to drive a virtual object, that is, a virtual object that behaves and moves in the same way as the streamer user may be constructed, and live streaming may be performed using the virtual object. This allows the streamer user to be hidden from the public eye, while achieving a live streaming effect similar to that of an actual streamer user. For example, the computer device may construct a virtual object related to the object based on the overall posture of the object in the video data, and play the virtual object with the overall posture in a multimedia application (e.g., a live streaming room, a video website, a short video application, etc.). That is, a video related to the virtual object may be played in the multimedia application, and the posture of the virtual object may be kept synchronized with the posture of the object in the video data. Here, the overall pose corresponding to the object in the video data is embodied on a virtual object played in the multimedia application, and each time the pose of the object changes, the virtual object in the multimedia application is driven to convert to the same pose (think of this as reconstructing the virtual object with a new pose, where the new pose is the pose of the object after the change), thereby keeping the poses of the object and the virtual object consistent at all times.

[0043] Please refer to FIG. 5. FIG. 5 is a schematic diagram of an object pose estimation scenario provided in an embodiment of the present application. The object pose estimation process of video data will be described by taking a virtual live streaming scenario as an example. As shown in FIG. 5, when a streamer user 40c (which may be an object) needs to perform live streaming, he / she can enter a live streaming room (for example, a live streaming room with a room number of 116889). Before starting live streaming, the streamer user 40c can select a real person live streaming mode or a virtual live streaming mode. If the streamer user 40c selects the virtual live streaming mode, the streamer user 40c may pull a virtual object. When the streamer user 40c starts live streaming, the streamer user 40c may drive the virtual object using the actions and movements of the streamer user 40c, thereby making the virtual object and the streamer user 40c maintain the same posture.

[0044] After starting live distribution, the streamer user 40c can collect his / her own video data using a user terminal 40a (e.g., a smartphone). cmay be regarded as an object, and the user terminal 40a may be fixed by a bracket 40b. The user terminal 40a may collect video data of the streamer user 40c, and then obtain an image frame 40g from the video data, and input the image frame 40g into the object detection model and the body part detection model, respectively. The object detection model can predict body part joint points (object key points) of the streamer user 40c included in the image frame 40g. These predicted body part joint points may be regarded as an object posture detection result of the image frame 40g. The body part detection model can predict palm key points (here, the first object part is the palm by default, and the palm key points are also called body part key points) of the streamer user 40c included in the image frame 40g. These predicted palm key points may be regarded as a body part posture detection result of the image frame 40g. The object posture detection result and the body part posture detection result here may be marked on the image frame 40g (as shown in the image 40h). Here, areas 40i and 40j in image 40h indicate the results of the part and orientation detection.

[0045] As shown in FIG. 5, a human body posture 40k of the streamer user 40c in the image frame 40g can be obtained by the object posture detection result and the part posture detection result displayed in the image 40h. As is clear, since the image frame 40g includes only the upper body of the streamer user 40c, the human body posture 40k is not the complete human body posture of the streamer user 40c. In this case, a reference posture (complete default human body posture) may be obtained, and the joint points of the human body posture 40k may be interpolated by the reference posture. In this way, the missing part joint points in the human body posture 40k are completed to obtain the entire human body posture 40m (whole posture) for the streamer user 40c.

[0046] The whole human body posture of 40m drives the virtual object in the live streaming room, and the virtual object in the live streaming room ToOverall human body posture 40 similar to streamer user 40c m A user who enters the live distribution room and watches the live distribution may have an exhibition page of the live distribution room where the virtual object is located on the user terminal 40d used by the user. The exhibition page of the live distribution room may include an area 40e and an area 40f. The area 40e may be used to play a video of the virtual object (having a similar posture to the streamer user 40c), and the area 40f may be used to send a barrage of bullets, etc. In a virtual live distribution scenario, a user who enters the live distribution room and watches the live distribution may only view the video of the virtual object and the audio data of the streamer user 40c, but cannot view the video data of the streamer user 40c. In this way, the personal information of the streamer user 40c can be protected, and a live distribution effect similar to that of the streamer user 40c can be achieved by the virtual object.

[0047] In some embodiments, the overall posture of the object in the above video data can be applied to a content screening scenario. If the overall posture is the same as the posture in the content screening system, it can be determined that the screening result of the object in the content screening system is a screening pass result, and an access right to the content screening system can be set for the object. If the overall posture passes the screening of the content screening system, the object can have an access right to the content screening system.

[0048] Please refer to FIG. 6. FIG. 6 is a schematic diagram of an application scenario of the whole posture provided in the embodiment of the present application. As shown in FIG. 6, a user A (object) can send a verification request to a server 50d through a user terminal 50a. After receiving the verification request sent from the user terminal 50a, the server 50d may obtain an identity verification method for the user A and send the identity verification method back to the user terminal 50a. A verification box 50b may be displayed on the terminal screen of the user terminal 50a. The user A can face the verification box 50b in the user terminal 50a and perform a specific action (for example, raising a hand, kicking, or hitting a waist). The user terminal 50a may collect a verification target image 50c (considered to be the above image frame) in the verification box 50b in real time, and send the verification target image 50c collected in real time to the server 50d.

[0049] The server 50d may acquire the verification target image 50c transmitted from the user terminal 50a, and acquire the posture 50e that the user A has set in advance in the content review system. This posture 50e may be used as verification information of the user A in the content review system. The server 50d may acquire the overall posture of the user A in the verification target image 50c by performing posture estimation of the verification target image 50c using the object detection model, the part detection model, and the reference posture, and compare the similarity between the overall posture corresponding to the verification target image 50c and the posture 50e. If the similarity between the overall posture of the verification target image 50c and the posture 50e is equal to or greater than a similarity threshold (for example, the similarity threshold may be set to 90%), it may be determined that the overall posture of the verification target image 50c is the same as the posture 50e, and the user A has passed the review in the content review system. If the similarity between the overall posture of the verification target image 50c and the posture 50e is smaller than the similarity threshold, it may be determined that the overall posture of the verification target image 50c is not the same as the posture 50e and that the user A has failed the screening in the content screening system, and a motion error presentation information may be returned to the user terminal 50a. This motion error presentation information is intended to present to the user A that he or she should redo the motion for identity screening.

[0050] In an embodiment of the present application, by performing object posture estimation and specific part posture estimation for an object in an image frame, an object posture detection result for the object and a part posture detection result for a first object part of the object can be obtained; further, based on the object posture detection result, the part posture detection result, and the reference posture, posture estimation of the object in the image frame can be performed; missing part keypoints of the object in the image frame can be corrected; the completeness and rationality of the finally obtained overall posture of the object can be ensured; and the estimation accuracy of the overall posture can be improved.

[0051] Please refer to Fig. 7. Fig. 7 is a schematic diagram of the flow of another data processing method provided in an embodiment of the present application. As shown in Fig. 7, this data processing method may include the following steps S201 to S208.

[0052] In step S201, the image frame is input into an object detection model, and the object detection model obtains object pose features corresponding to the object in the image frame, and recognizes a first classification result corresponding to the object pose features, where the first classification result is for representing an object part classification corresponding to the key points of the object.

[0053] Specifically, the computing device may obtain video data captured in a mobile-side scenario, and then select one image frame from the video data and input the image frame into a trained object detection model. The object detection model may obtain an object pose feature corresponding to an object in the image frame. A classifier of the object detection model may output a first classification result corresponding to the object pose feature. The first classification result may be used to represent an object part classification corresponding to a key point of the object (e.g., a joint of a human body). Here, the object pose feature may be an object description feature for an object extracted by the object detection model, or may be a fusion feature of an object description feature and a part description feature corresponding to the object. If the object pose feature is an object description feature corresponding to an object in the image frame, it indicates that block learning based on part detection is not introduced in the process of extracting features of the image frame using the object detection model. If the object pose feature is a fusion feature of an object description feature and a part description feature corresponding to an object in the image frame, it indicates that block learning based on part detection is introduced in the process of extracting features of the image frame using the object detection model. By introducing block learning based on part detection, the object pose feature can include local pose features (part description features) of each part of the object included in the image frame, and Image Frame , which can enhance the fine granularity of the object pose features and further improve the accuracy of the object pose detection result.

[0054] In some embodiments, when block learning based on part detection is introduced in the process of extracting features of an image frame using an object detection model, the computer device may input the image frame into the object detection model, obtain object description features corresponding to the object in the image frame from the object detection model, output a second classification result corresponding to the object description features by a classifier in the object detection model, obtain object convolution features for the image frame output from the convolution layer in the object detection model, obtain a second activation mapping corresponding to the image frame by performing a multiplication operation between the second classification result and the object convolution features, obtain M (M is a positive integer) object part area images by performing block division processing of the image frame based on the second activation mapping, obtain part description features corresponding to each of the M object part area images based on the object detection model, and combine the object description features and the part description features corresponding to the M object part area images as an object posture feature.

[0055] Here, the object description feature is considered as a feature representation for representing an object extracted from an image frame. The second classification result can also be used to represent an object part classification corresponding to an object keypoint included in the image frame. The convolution layer may be the last convolution layer in the object detection model. The object convolution feature can represent a convolution feature for an image frame output from the last convolution layer of the object detection model. The second activation mapping may be a class activation mapping (CAM) corresponding to the image frame. The CAM is a tool for visualizing image features. The second activation mapping can be obtained by weighting the object convolution feature output from the last convolution layer in the object detection model and the second classification result (the second classification result is considered as a weight corresponding to the object convolution feature). This second activation matching is considered as a result of visualizing the object convolution feature output from the convolution layer, and can be used to represent an image pixel point region focused on by the object detection model.

[0056] The computer device may perform block division processing of the image frame using the class activation mapping (second activation mapping) of each object key point in the image frame as a priori information of the region position, i.e., cut out the image frame based on the second activation mapping to obtain an object region region image including a single region. Furthermore, the object detection model may perform feature extraction for each object region region image to obtain a region description feature corresponding to each object region region image. The object description feature and the region description feature corresponding to each object region region may be combined as an object posture feature for the object. The region description feature is considered to be a feature representation for representing an object region extracted from the object region region image.

[0057] In step S202, a first activation mapping is generated based on the first classification result and the object convolution features of the image frame output from the object detection model.

[0058] Specifically, the computing device may obtain a first classification result, and then generate a first activation mapping by multiplying the first classification result with object convolution features of the image frame, where both the first activation mapping and the second activation mapping are class activation mappings for the image frame, where the first activation mapping takes the first classification result as the weight of the object convolution features output from the convolution layer (where, as a default, the first classification result is a combination of the object description features and the part description features), while the second activation mapping takes the second classification result as the weight of the object convolution features output from the convolution layer. The second classification result is related only to the object description features.

[0059] In step S203, pixel average values ​​corresponding to the first activation mapping are obtained, and based on the pixel average values, localization results of key points in the object in the image frame are determined, and based on the object part classification and localization results, object pose detection results corresponding to the image frame are determined.

[0060] Specifically, the computing device may determine pixel averages of the first activation mapping and determine the pixel averages as localization results of key points of the object in the image frames. Based on the object part classification and localization results, an object skeleton of the object in the image frames may be determined. The object skeleton may be an object pose detection result corresponding to the object in the image frames.

[0061] Please refer to FIG. 8, which is a schematic diagram of the configuration of an object detection model provided in an embodiment of the present application. As shown in FIG. 8, a computer device may obtain an image frame 60a, and then input the image frame 60a into an object detection model. A feature extraction component 60b (for example, the feature extraction network may be a convolutional network) in the object detection model performs feature extraction of the image frame 60a, so as to obtain object description features 60c corresponding to the object in the image frame 60a. Using global average pooling (the number of object description features may be multiple, and global average pooling refers to converting one object description feature into one numerical value) and an activation function, the object description features 60c are processed and the processed result is classified to obtain a second classification result, and the second classification result and the object convolution feature output from the last convolution layer in the feature extraction component 60b are weighted to obtain a second activation mapping.

[0062] Based on the second activation mapping, M object part area images 60f are obtained by performing block division processing of the image frame 60a, and the M object part area images 60f are sequentially input to a feature extraction component 60b in the object detection model, and the feature extraction component 60b can obtain part description features 60g corresponding to each of the M object part area images 60f. The M part description features 60g are combined with the object description features 60c of the image frame 60a to obtain an object posture feature. The object posture feature is recognized to obtain a first classification result 60d. The first classification result 60d and the object convolution feature output from the last convolution layer in the feature extraction component 60b are weighted to obtain a first activation mapping 60e. The pixel average value of the first activation mapping 60e may be used as a position identification result of the object in the image frame 60a. The position identification result is used to obtain an object posture detection result corresponding to the object in the image frame 60a.

[0063] Note that the method of obtaining the object pose detection result described in the embodiment corresponding to Fig. 8 is merely an example of the embodiment of the present application, and the object pose detection result may be obtained by other methods in the present application. The present application is not limited thereto.

[0064] Please refer to FIG. 9. FIG. 9 is a schematic diagram of the flow of obtaining an object pose detection result provided in an embodiment of the present application. As shown in FIG. 9, taking the case where the object detection model is a human body 3D pose estimation model as an example, a computer device may input an image frame 70a into the human body 3D pose estimation model. This human body 3D pose estimation model can obtain human body 3D keypoints of an object (in this case, the object is a human body) in the image frame 70a. As shown in FIG. 9, when 16 human body 3D keypoints, respectively denoted as x1 to x16, included in the image frame 70a are detected by the human body 3D pose estimation model, each human body 3D keypoint may correspond to one position coordinate and one first confidence level. Based on the first confidence level, it can be determined that the detected human body 3D keypoint is a real human body keypoint. For example, a human body 3D keypoint whose first confidence level is greater than a first confidence threshold (which may be set according to practical needs) is considered to be a real human body keypoint (e.g., a human body 3D keypoint represented by x4 to x16). By concatenating the real body keypoints, a body pose 70c (also considered as object pose detection result) can be obtained. The body 3D keypoints whose first confidence is less than or equal to the first confidence threshold are abnormal keypoints. In subsequent processing, these abnormal keypoints may be corrected to obtain more accurate body keypoints.

[0065] As can be understood, if a spatial coordinate system is constructed in the image frame, the position coordinates of the human body 3D key points may be spatial coordinates in this spatial coordinate system.

[0066] In step S204, the image frame is input to a part detection model, and a first object part of the object in the image frame is detected in the part detection model.

[0067] Specifically, the computer device may input the image frame into a part detection model. In the part detection model, first, it is detected whether the image frame includes a first object part of the object. Here, it is necessary to detect the first object part in the image frame, since the part detection model can be used to detect key points of the first object part. If the first object part of the object is not detected from the image frame, it can be directly determined that the part posture detection result corresponding to the image frame is a null value, and there is no need to perform a subsequent step of detecting key points of the first object part.

[0068] In step S205, if a first object part is detected from the image frame, an area image including the first object part is obtained from the image frame, and part key point positions corresponding to the first object part are obtained based on the area image, and a part posture detection result corresponding to the image frame is determined based on the part key point positions.

[0069] Specifically, when the first object part is searched from the image frame, a position region of the first object part in the image frame may be determined, and a region image including the first object part may be obtained by cropping the image frame based on the position region of the first object part in the image frame. In the part detection model, a part contour feature corresponding to the first object part in the region image may be obtained by performing feature extraction of the region image. A part key point position corresponding to the first object part may be predicted based on the part contour feature. A part posture detection result corresponding to the image frame may be obtained by connecting key points of the first object part based on the part key point positions.

[0070] Please refer to FIG. 10. FIG. 10 is a schematic diagram of a flow of obtaining a part posture detection result provided in an embodiment of the present application. As shown in FIG. 10, taking the case where the part detection model is a palm 3D posture estimation model as an example, the computer device may input the image frame 80a to the palm 3D posture estimation model. In this palm 3D posture estimation model, it may be detected whether the image frame 80a includes the palm of the object (first object part). If the palm is not detected from the image frame 80a, it may be determined that the part posture detection result corresponding to the image frame 80a is a null value. If the palm is detected from the image frame 80a, it may be determined that the area including the palm in the image frame 80a (for example, area 80c and area 80d in the image 80b, where area 80c includes the right palm of the object and area 80d includes the left palm of the object). Furthermore, the three-dimensional palm pose estimation model can detect three-dimensional palm key points in region 80c and three-dimensional palm key points in region 80d.

[0071] Here, a plurality of possible regions may be obtained by the palm 3D posture estimation model, a second reliability that the palm is included in each possible region may be predicted, and a region in which the second reliability is greater than a second reliability threshold (which may be the same as or different from the first reliability threshold described above; not limited here) may be determined as a region in which the palm is included. For example, the second reliability corresponding to each of the regions 80c and 80d is greater than the second reliability threshold. By linking the palm key points detected in the region 80c, a right palm posture 80e can be obtained, and by linking the palm key points detected in the region 80d, a left palm posture 80f can be obtained. The above left palm posture 80f and right palm posture 80e can be called part posture detection results corresponding to the image frame 80a.

[0072] In step S206, a reference pose associated with the object is obtained, and a first number of key points corresponding to the reference pose and a second number of key points corresponding to the object pose detection result are determined.

[0073] Specifically, the computing device may obtain a reference pose corresponding to the object, and count a first number of object keypoints included in the reference pose and a second number of object keypoints included in the object pose detection result, where the first number of keypoints is known when constructing the reference pose, and the second number of keypoints is the number of object keypoints predicted by the object detection model.

[0074] In step S207, if the number of first key points is greater than the number of second key points, an interpolation process is performed on the object pose detection result based on the reference pose to obtain a first candidate object pose.

[0075] Specifically, if the number of first keypoints is greater than the number of second keypoints, it indicates that there are missing object keypoints in the object pose detection result, and the first candidate object pose corresponding to the object may be obtained by performing keypoint correction (interpolation process) of this object pose detection result using the reference pose to complete the missing object keypoints. As shown in FIG. 2, a human body pose 20m can be obtained by performing keypoint correction of a human body pose 20j (object pose detection result) using the reference pose 20k. The human body pose 20m in this case can be called the first candidate object pose.

[0076] For example, if the object is a human body, and key points of the knees, ankles, feet, elbows, etc. are missing in the object pose detection result predicted by the object detection model, the object pose detection result may be interpolated by a reference pose to obtain a more reasonable first candidate object pose, for example by adding the missing object key points. Interpolating the object pose detection result by the reference pose can improve the completeness and reasonableness of the object pose.

[0077] In step S208, an overall posture corresponding to the object is obtained by performing an interpolation process of the object parts related to the first object part in the first candidate object posture based on the part posture detection result.

[0078] Specifically, in practical application scenarios, the pose change of an object depends heavily on a few parts of the object, that is, some specific parts of the object (e.g., arm parts in human body structure, which may include key points of parts such as palm, wrist, and elbow) play an important role in the final result. Therefore, in the embodiments of the present application, an interpolation process of object parts related to the first object part in the first candidate object pose may be performed based on the part pose detection result. In this way, a global pose corresponding to the object can be obtained. In some embodiments, if the part pose detection result is a null value (i.e., the image frame does not include the first object part), the first candidate object pose may be directly determined as the global pose corresponding to the object.

[0079] For example, if the object is a human body and the first object part is a palm, if the image frame includes an elbow part, the keypoint for the elbow part can be predicted by the object detection model. If the image frame does not include the elbow part, the keypoint for the elbow part cannot be predicted by the object detection model. In this case, the elbow keypoint and wrist keypoint of the object may be determined based on the part pose detection result, and the elbow keypoint and wrist keypoint may be added to the first candidate object pose. This allows the overall pose corresponding to the object to be obtained.

[0080] In some embodiments, the object includes a second object part and a third object part, and the second object part and the third object part are symmetrical, e.g., if the second object part is the right arm of the object, the third object part is the left arm of the object, if the second object part is the left leg of the object, the third object part is the right leg of the object, etc.

[0081] In a state where all part key points of the first object part are included in the part posture detection result (if the first object part is a palm, the part posture detection result includes the left and right palm key points), if the object posture detection result includes the posture of the second object part and does not include the posture of the third object part, that is, if the image frame includes the second object part but not the third object part, the first part direction corresponding to the third object part may be determined based on the key point position of the first object part included in the part posture detection result. Here, the second object part and the third object part are symmetrical parts of the object. Since the second object part and the third object part are symmetrical parts, the length of the second object part and the length of the third object part are the same. Therefore, the first part length of the second object part in the first candidate object posture is obtained, and the key point position of the third object part is determined based on the first part length and the first part direction, and the key point position of the third object part is added to the first candidate object posture, so that the overall posture corresponding to the object in the image frame can be obtained.

[0082] If the object posture detection result does not include the postures of the second object part and the third object part, i.e., if neither the second object part nor the third object part is included in the image frame, the second part direction corresponding to the second object part and the third part direction corresponding to the third object part may be determined based on the key point position of the first object part included in the part posture detection result. Furthermore, the second part length corresponding to the second object part and the third part length corresponding to the third object part may be obtained from the (i-1)th image frame. In other words, the length of the second object part in the previous image frame is set as the length of the second object part in the image frame, and the length of the third object part in the previous image frame is set as the length of the second object part in the image frame. lengthmay be set as the length of the third object part in the image frame. Furthermore, a key point position of the second object part may be determined based on the second part length and the second part direction, a key point position of the third object part may be determined based on the third part length and the third part direction, and the key point positions of the second object part and the third object part may be added to the first candidate object pose to obtain an overall pose corresponding to the object in the image frame. Similarly, when the i-1th image frame does not include the second object part and the third object part, the respective lengths of the second object part and the third object part in the i-2th image frame may be subsequently looked back to determine the respective key point positions of the second object part and the third object part in the image frame. When the second object part and the third object part are not detected from all image frames prior to the image frame, the respective key point positions of the second object part and the third object part in the image frame may be determined by setting approximate lengths for the second object part and the third object part based on the first candidate object pose.

[0083] For example, suppose that the object is a human body, the first object part is a palm, and the second and third object parts are the left and right arms, respectively. Assuming that the left and right palms are detected from the image frame, the direction of the left forearm may be calculated from key points on the left palm, and the direction of the right forearm may be calculated from key points on the right palm. The left forearm belongs to a part of the left arm, and the right forearm belongs to a part of the right arm.

[0084] If the left or right arm is not detected from the image frame, the left or right forearm length (the second part length and 3The left and right forearm lengths (first part length) may be taken as the left and right forearm lengths in the image frame. If the left and right arms are not detected from either the image frame or the previous image frame, the shoulder length in the image frame may be referenced to provide the left and right forearm lengths in the image frame. If one of the left and right arms (e.g., the left arm) is searched for from the image frame, the left forearm length (first part length) may be directly provided to the right forearm. For example, it is known that the right wrist point A, the right palm point B, and the right elbow point C are missing. The direction of the right forearm can be represented as the direction from the right palm point B to the right wrist point A, and can be denoted as vector BA. The left forearm length can be represented as the length from the right wrist point A to the right elbow point C, and can be denoted as L. The position coordinate of the right elbow point C can be calculated from the above information and can be represented as C=A+BA_normal*L. Here, C represents the position coordinate of the elbow point, A represents the position coordinate of the wrist point, and BA_normal represents the unit vector of the vector BA.

[0085] As can be seen, when left and right arms are detected from an image frame, the elbow points predicted by the object detection model may be adjusted and updated based on the detected palm keypoints, thereby improving the accuracy of the elbow points and further improving the rationality of the overall pose.

[0086] In some embodiments, since some object key points may be present in the pose obtained by performing the interpolation process of the first candidate object pose based on the part pose detection result, the final overall pose of the object may be obtained by correcting the object key points by referring to the reference pose. Specifically, if a third object part is not detected from the image frame, the computing device may determine the first candidate object pose to which the key point position of the third object part is added as the second candidate object pose. Furthermore, a pose offset amount between the reference pose and the second candidate object pose may be obtained, and if the pose offset amount is greater than an offset threshold (which may be understood as the maximum angle that an object can be offset in normal cases), the key points of the second candidate object pose may be corrected based on the reference pose to obtain an overall pose corresponding to the object in the image frame. Here, the pose offset amount may be understood as a relative angle between the second candidate object pose and the reference pose. For example, if the object is a human body, the pose offset amount may be an angle between the shoulder of the second candidate object pose and the shoulder of the reference pose, etc.

[0087] Please refer to FIG. 11. FIG. 11 is a schematic diagram of the correction of object keypoints provided in the embodiment of the present application. As shown in FIG. 11, for an image frame 90a, a second candidate object pose corresponding to the image frame 90a is obtained, and then a human body model 90b can be constructed based on the second candidate object pose. Limited by the performance of the object detection model, compared with the normal human body structure (e.g., the reference pose), there is an obvious depression problem in the region 90c (e.g., the shoulder region) in the human body model 90b. For example, 2 The angle between the shoulder of the candidate object pose and the shoulder of the reference pose is greater than an offset threshold. Areas in 90bA human body model 90d may be obtained by rectifying region 90c. Region 90e in the human body model 90d may be considered as a result of rectifying region 90c. The human body pose corresponding to the human body model 90d may be referred to as the global pose corresponding to the object in the image frame 90a.

[0088] Video data captured in a mobile scenario usually cannot include the entire object. The pose of the object predicted by the object detection model is not perfect. The rationality of the overall pose can be improved by processes such as keypoint interpolation and keypoint correction. According to the part pose detection result, the object keypoint position related to the first object part can be calculated, and the accuracy of the overall pose can be improved.

[0089] Please refer to FIG. 12. FIG. 12 is a schematic diagram of the flow of object pose estimation provided in the embodiment of the present application. As shown in FIG. 12, if the object is a human body, the computer device may obtain a confidence-based human body 3D pose estimation model (object detection model) and a confidence-based palm 3D pose estimation model (part detection model) after obtaining video data or image data captured in a mobile-side scenario. The human body 3D pose estimation model can predict the human body 3D keypoints in any one image frame (image frame). These human body 3D keypoints can constitute the object pose detection result. The palm 3D pose estimation model can predict the palm 3D keypoints in any one image frame (image frame). These palm 3D keypoints can constitute the part pose detection result. In order to complete the missing human body keypoints, an interpolation process of the human body 3D keypoints predicted by the human body 3D pose estimation model may be performed based on the human body default pose (reference pose). A candidate human body pose (the above-mentioned second candidate object pose) may be obtained by combining the palm 3D keypoints and the human body 3D keypoints and performing an interpolation process for the elbow and wrist of the human body (object). If any human body keypoints that do not match the normal human body structure (human body keypoints whose pose offset amount is greater than the offset threshold) are detected from the candidate human body poses, these human body keypoints that do not match the normal human body structure may be corrected to finally obtain a reasonable 3D pose estimation result (i.e., the above-mentioned overall pose).

[0090] As can be appreciated, specific embodiments of the present application may involve the collection of user video, and the application of the above examples of the present application to specific products and technologies may require user permission or consent, and the collection, use and processing of related data may be subject to the relevant laws, regulations and standards of the relevant country or region.

[0091] In an embodiment of the present application, by performing object posture estimation and specific part posture estimation for an object in an image frame, an object posture detection result for the object and a part posture detection result for a first object part of the object can be obtained. Furthermore, based on the object posture detection result, the part posture detection result, and the reference posture, posture estimation of the object in the image frame can be performed, and missing part keypoints of the object in the image frame can be corrected to correct object keypoints that do not match the reference posture. This can ensure the completeness and rationality of the ultimately obtained overall posture of the object, and further improve the estimation accuracy of the overall posture.

[0092] Please refer to Fig. 13. Fig. 13 is a schematic diagram of the configuration of a data processing device provided in an embodiment of the present application. As shown in Fig. 13, the data processing device 1 may include a posture detection module 11 and a posture estimation module 12.

[0093] Pose detection module 11 obtains an object pose detection result corresponding to an object in an image frame and a part pose detection result corresponding to a first object part of the object in the image frame, where at least one object part of the object is missing in the object pose detection result, and the first object part is one or more parts of the object.

[0094] The posture estimation module 12 performs an interpolation process for at least one missing object part in the object posture detection result based on the part posture detection result and a reference posture related to the object, thereby obtaining an overall posture corresponding to the object, which is used to realize a service function corresponding to the overall posture by operating a computing device.

[0095] Here, for specific methods of implementing the functions of posture detection module 11 and posture estimation module 12, refer to the explanation of steps S101 to S102 in the embodiment corresponding to Fig. 3. No further explanation will be given here.

[0096] In an embodiment of the present application, by performing overall object pose estimation and specific part pose estimation for an object in an image frame, an object pose detection result for the object and a part pose detection result for a first object part of the object can be obtained; further, based on the object pose detection result, the part pose detection result, and the reference pose, an object pose estimation can be performed in the image frame; missing part keypoints of the object in the image frame can be corrected; thus, the completeness and rationality of the finally obtained overall pose of the object can be ensured; and the estimation accuracy of the overall pose can be improved.

[0097] Please refer to Fig. 14. Fig. 14 is a schematic diagram of the configuration of another data processing device provided in an embodiment of the present application. As shown in Fig. 14, the data processing device 2 includes a posture detection module 21, a posture estimation module 22, and a virtual object construction module 23.

[0098] Pose detection module 21 obtains an object pose detection result corresponding to an object in an image frame and a part pose detection result corresponding to a first object part of the object in the image frame, where at least one object part of the object is missing in the object pose detection result, and the first object part is one or more parts of the object.

[0099] The pose estimation module 22 obtains an overall pose corresponding to the object by performing an interpolation process for at least one missing object part in the object pose detection result based on the part pose detection result and a reference pose related to the object.

[0100] A virtual object construction module 23 constructs a virtual object associated with the object and controls the pose of the virtual object based on the overall pose.

[0101] Here, the specific function implementation methods of the posture detection module 21, posture estimation module 22, and virtual object construction module 23 can be referred to in the above-mentioned explanation of the related steps, and further explanation will be omitted here.

[0102] In one or more embodiments, the pose detection module 21 includes an object detection unit 211 and a part detection unit 212 .

[0103] The object detection unit 211 inputs the image frame into an object detection model, and obtains the object pose detection result through the object detection model.

[0104] The part detection unit 212 inputs the image frame into a part detection model, and obtains the part pose detection result through the part detection model.

[0105] Here, for specific implementation methods of the functions of the object detection unit 211 and the part detection unit 212, refer to step S101 in the embodiment corresponding to Fig. 3. Further explanation will be omitted here.

[0106] In one or more embodiments, the object detection unit 211 may include a part classification subunit 2111 , a part mapping generation subunit 2112 , a localization result determination subunit 2113 , and a detection result determination subunit 2114 .

[0107] The part classification subunit 2111 inputs the image frame into an object detection model, obtains object pose features corresponding to the object in the image frame through the object detection model, and recognizes a first classification result corresponding to the object pose features, where the first classification result is for representing an object part classification corresponding to the key points of the object.

[0108] The part mapping generation subunit 2112 generates a first activation mapping based on the first classification result and the object convolution features of the image frame output from the object detection model.

[0109] The localization result determining subunit 2113 obtains pixel average values ​​corresponding to the first activation mapping, and determines localization results in the image frame of key points in the object based on the pixel average values.

[0110] The detection result determination subunit 2114 determines an object pose detection result corresponding to the image frame based on the object part classification and the localization result.

[0111] Here, for specific functional implementation methods of the part classification subunit 2111, the part mapping generation subunit 2112, the position identification result determination subunit 2113, and the detection result determination subunit 2114, refer to steps S201 to S203 in the embodiment corresponding to Fig. 7. Here, further explanation is omitted.

[0112] In one or more embodiments, the region classification subunit 2111 includes an overall classification subunit 21111 , an overall mapping acquisition subunit 21112 , a block division processing subunit 21113 , and a feature combination subunit 21114 .

[0113] The global classification subunit 21111 obtains object description features corresponding to objects in the image frames from the object detection model, and outputs a second classification result corresponding to the object description features based on the classifier in the object detection model.

[0114] The global mapping acquisition subunit 21112 obtains object convolution features for the image frame output from the convolution layer in the object detection model, and obtains a second activation mapping corresponding to the image frame by performing a multiplication operation between the second classification result and the object convolution features.

[0115] The block division processing subunit 21113 performs block division processing of the image frame based on the second activation mapping to obtain M (M is a positive integer) object part area images, and obtains part description features corresponding to each of the M object part area images based on the object detection model.

[0116] The feature combination subunit 21114 combines the object description feature and the part description features corresponding to the M object part region images as an object pose feature.

[0117] Here, the specific function implementation manner of the overall classification subunit 21111, the overall mapping acquisition subunit 21112, the block division processing subunit 21113, and the feature combination subunit 21114 can be referred to step S201 in the embodiment corresponding to Fig. 7. Here, further description is omitted.

[0118] In one or more embodiments, the part detection module 212 may include an object part detection subunit 2121 , a part pose estimation subunit 2122 , and a null value determination subunit 2123 .

[0119] The object part detection subunit 2121 inputs the image frame into a part detection model, and detects a first object part of the object in the image frame in the part detection model.

[0120] When a first object part is detected from an image frame, the part posture estimation subunit 2122 obtains an area image including the first object part from the image frame, obtains part key point positions corresponding to the first object part based on the area image, and determines a part posture detection result corresponding to the image frame based on the part key point positions.

[0121] The null value determination subunit 2123 determines the part pose detection result corresponding to the image frame as a null value when the first object part is not detected from the image frame.

[0122] Here, for specific implementation methods of the object part detection subunit 2121, part posture estimation subunit 2122, and null value determination subunit 2123, see steps S204 to S205 in the embodiment corresponding to Fig. 7. Further explanation will be omitted here.

[0123] In one or more embodiments, the part pose estimation subunit 2122 may include an image cropping subunit 21221 , a part keypoint determination subunit 21222 , and a part keypoint connection subunit 21223 .

[0124] When the first object part is detected from the image frame, the image cropping subunit 21221 crops the image frame to obtain a region image including the first object part.

[0125] The part keypoint determination subunit 21222 obtains part contour features corresponding to the region image, and predicts part keypoint positions corresponding to the first object part based on the part contour features.

[0126] The part keypoint connecting subunit 21223 obtains a part pose detection result corresponding to the image frame by connecting the keypoints of the first object part according to the part keypoint positions.

[0127] Here, the specific function implementation manner of the image cropping subunit 21221, the region keypoint determining subunit 21222, and the region keypoint linking subunit 21223 can be referred to step S205 in the embodiment corresponding to Fig. 7. Here, further description is omitted.

[0128] In one or more embodiments, the pose estimation module 22 includes a keypoint number determination unit 221, a first interpolation processing unit 222, and a second interpolation processing unit 223.

[0129] The keypoint number determination unit 221 obtains a reference pose associated with an object, and determines a first keypoint number corresponding to the reference pose and a second keypoint number corresponding to the object pose detection result.

[0130] When the number of first key points is greater than the number of second key points, the first interpolation processing unit 222 obtains a first candidate object pose by performing an interpolation process for at least one missing object part in the object pose detection result based on the reference pose.

[0131] The second interpolation processing unit 223 performs an interpolation process of the object part related to the first object part in the first candidate object posture based on the part posture detection result, thereby obtaining an overall posture corresponding to the object.

[0132] Here, the number of keypoints decision For specific implementation methods of the functions of the unit 221, the first interpolation processing unit 222, and the second interpolation processing unit 223, refer to steps S206 to S208 in the embodiment corresponding to Fig. 7. Further explanation will be omitted here.

[0133] In one or more embodiments, the second interpolation processing unit 223 may include: a first direction determining subunit 2231 , a first position determining subunit 2232 , and a first keypoint adding subunit 2233 .

[0134] When the object posture detection result includes the posture of the second object part and does not include the posture of the third object part, the first direction determination subunit 2231 determines a first part direction corresponding to the third object part based on the key point position of the first object part included in the part posture detection result, where the second object part and the third object part are symmetrical parts of the object and are related to the first object part.

[0135] The first position determination subunit 2232 obtains a first part length of the second object part in the first candidate object pose, and determines a key point position of a third object part based on the first part length and the first part orientation.

[0136] A first keypoint adding subunit 2233 obtains a global pose corresponding to the object in the image frame by adding the keypoint positions of the third object part to the first candidate object pose.

[0137] In some embodiments, the first key point adding subunit 2233 specifically includes: obtaining a second candidate object pose corresponding to the object in the image frame by adding a keypoint position of a third object part to the first candidate object pose; and obtaining a pose offset between the reference pose and the second candidate object pose. If the pose offset amount is greater than the offset threshold, a global pose corresponding to the object in the image frame is obtained by rectifying the keypoints of the second candidate object pose based on the reference pose.

[0138] In some embodiments, the image frame is the i-th (i is a positive integer) image frame in the video data, and the second interpolation processing unit 223 may further include a second direction determination subunit 2234, a second position determination subunit 2235, and a second key point addition subunit 2236.

[0139] When the object posture detection result does not include the postures of the second object part and the third object part, the second direction determination subunit 2234 determines a second part direction corresponding to the second object part and a third part direction corresponding to the third object part based on the key point positions of the first object part included in the part posture detection result, where the second object part and the third object part are symmetrical parts of the object and are related to the first object part.

[0140] The second position determination subunit 2235 obtains a second part length corresponding to the second object part and a third part length corresponding to the third object part from the jth (j is a positive integer smaller than i) image frame, and determines a key point position of the second object part based on the second part length and the second part orientation.

[0141] The second keypoint adding subunit 2236 determines a keypoint position of a third object part based on the third part length and the third part orientation, and obtains an overall pose corresponding to the object in the image frame by adding the keypoint position of the second object part and the keypoint position of the third object part to the first candidate object pose.

[0142] Here, the specific function implementation manner of the first direction determination subunit 2231, the first position determination subunit 2232, the first keypoint adding subunit 2233, the second direction determination subunit 2234, the second position determination subunit 2235, and the second keypoint adding subunit 2236 can be referred to step S208 in the embodiment corresponding to Fig. 7. Here, further description is omitted. Here, when the first direction determination subunit 2231, the first position determination subunit 2232, and the first keypoint adding subunit 2233 perform their respective operations, the second direction determination subunit 2234, the second position determination subunit 2235, and the second keypoint adding subunit 2236 all suspend the execution of their operations. When the second direction determination subunit 2234, the second position determination subunit 2235, and the second key point adding subunit 2236 perform their respective operations, the first direction determination subunit 2231, the first position determination subunit 2232, and the first key point adding subunit 2233 all pause the execution of their operations.

[0143] In an embodiment of the present application, by performing object posture estimation and specific part posture estimation for an object in an image frame, an object posture detection result for the object and a part posture detection result for a first object part of the object can be obtained. Furthermore, based on the object posture detection result, the part posture detection result, and the reference posture, posture estimation of the object in the image frame can be performed, and missing part keypoints of the object in the image frame can be corrected to correct object keypoints that do not match the reference posture. This can ensure the completeness and rationality of the ultimately obtained overall posture of the object, and further improve the estimation accuracy of the overall posture.

[0144] Please refer to FIG. 15. FIG. 15 is a schematic diagram of a configuration of a computer device provided in an embodiment of the present application. As shown in FIG. 15, the computer device 1000 may be a user terminal, for example, the user terminal 10a in the embodiment corresponding to the above FIG. 1, or may be a server, for example, the server 10d in the embodiment corresponding to the above FIG. 1. This is not limited here. For easy understanding, the present application takes the case where the computer device is a user terminal as an example. The computer device 1000 may include a processor 1001, a network interface 1004, and a memory 1005. The computer device 1000 may further include a user interface 1003 and at least one communication bus 1002. Here, the communication bus 1002 is used to realize the connection communication between these components. Here, the user interface 1003 may include a standard wired interface, a wireless interface. The network interface 1004 may include a standard wired interface, a wireless interface (for example, a WI-FI interface). The memory 1005 may be a high-speed RAM or a non-volatile memory, such as at least one disk storage device. Optionally, the memory 1005 may be at least one storage device located remotely from the processor 1001 described above. As shown in Figure 15, the memory 1005, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a device control application.

[0145] Here, the network interface 1004 in the computer device 1000 can further provide a network communication function. Optionally, the user interface 1003 may further include a display and a keyboard. In the computer device 1000 shown in FIG. 15, the network interface 1004 can provide a network communication function, the user interface 1003 is mainly used to provide an input interface to a user, and the processor 1001 executes the following operations by calling a device control application stored in the memory 1005: obtaining an object pose detection result corresponding to an object in an image frame and a part pose detection result corresponding to a first object part of the object in the image frame, where at least one object part of the object is missing in the object pose detection result, and the first object part is one or more parts of the object; The present invention can be used to realize a step of obtaining an overall posture corresponding to an object by performing an interpolation process for at least one missing object part in an object posture detection result based on the part posture detection result and a reference posture related to the object, the overall posture being for realizing a service function corresponding to the overall posture by operating a computing device.

[0146] It should be understood that the computer device 1000 described in the embodiment of the present application can execute the description of the data processing method in any one of the embodiments of Figure 3 and Figure 7 above, can execute the description of the data processing device 1 in the embodiment corresponding to Figure 13 above, and can execute the description of the data processing device 2 in the embodiment corresponding to Figure 14 above. Further description is omitted here. Also, description of the beneficial effects of the same method is omitted.

[0147] In addition, in the embodiment of the present application, a computer-readable storage medium is further provided, in which a computer program is stored, the computer program includes computer instructions, and when the processor executes the computer instructions, the data processing method in any one of the embodiments of FIG. 3 and FIG. 7 can be executed. Therefore, further description is omitted here. In addition, the description of the beneficial effects of the same method is also omitted. For technical details not disclosed in the embodiment of the computer-readable storage medium of the present application, please refer to the description of the method embodiment of the present application. For example, the computer instructions may be arranged to be executed on one computer device, executed on multiple computer devices at one location, or executed on multiple computer devices distributed at multiple locations and connected to each other via a communication network. The multiple computer devices distributed at multiple locations and connected to each other via a communication network may constitute a blockchain system.

[0148] In addition, in the embodiments of the present application, a computer program product or a computer program is further provided. The computer program product or computer program may include computer instructions. The computer instructions may be stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and when the processor executes the computer instructions, the computer device can execute the data processing method described in any one of the embodiments of FIG. 3 and FIG. 7 above. Therefore, further description is omitted here. Also, the description of the beneficial effects of the same method is omitted. For technical details not disclosed in the embodiments of the computer program product or computer program of the present application, please refer to the description of the method embodiments of the present application.

[0149] Although the above-described method embodiments are expressed as a combination of a series of operations for ease of explanation, those skilled in the art should know that the present application is not limited by the order of operations described, since some steps may be performed in other orders or simultaneously according to the present application. Next, those skilled in the art should know that all of the embodiments described in the specification belong to preferred embodiments, and such operations and modules are not necessarily required for the present application.

[0150] The steps in the method according to the embodiment of the present application can be reordered, merged, or deleted according to practical needs.

[0151] Modules in the apparatus according to the embodiment of the present application can be merged, divided and deleted according to actual needs.

[0152] As can be understood by those skilled in the art, all or part of the flow of the method according to the above embodiments may be executed by instructing relevant hardware through a computer program. The computer program may be stored in a computer-readable storage medium. When the program is executed, the flow of each of the above method embodiments is executed. Here, the storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), etc.

[0153] The above are merely preferred embodiments of the present application, and of course, they do not limit the scope of the present application. Therefore, equivalent modifications according to the claims of the present application still belong to the scope of the present application.

Claims

1. 1. A data processing method implemented by a computing device, comprising: obtaining an object pose detection result corresponding to an object in an image frame and a part pose detection result corresponding to a first object part of the object in the image frame, wherein at least one object part of the object is missing in the object pose detection result, and the first object part is one or more parts of the object; a step of obtaining an overall posture corresponding to the object by performing an interpolation process for at least one missing object part in the object posture detection result based on the part posture detection result and a reference posture related to the object, the overall posture being for operating a computing device to realize a service function corresponding to the overall posture; The method includes:

2. The step of obtaining an object pose detection result corresponding to an object in an image frame and a part pose detection result corresponding to a first object part of the object in the image frame includes: inputting the image frame into an object detection model and obtaining the object pose detection result through the object detection model; inputting the image frame into a body part detection model and obtaining the body part posture detection result by the body part detection model; The method of claim 1.

3. The step of inputting the image frame into an object detection model and obtaining the object pose detection result by the object detection model includes: inputting the image frame into an object detection model, obtaining object pose features corresponding to the object through the object detection model, and recognizing a first classification result corresponding to the object pose features, the first classification result being for representing an object part classification corresponding to a key point of the object; generating a first activation mapping based on the first classification result and object convolution features of the image frame output from the object detection model; obtaining pixel averages corresponding to the first activation mapping and determining localizations in the image frames of key points in the object based on the pixel averages; determining the object pose detection result based on the object part classification and the localization result. The method of claim 2.

4. The step of obtaining an object pose feature corresponding to the object by the object detection model comprises: obtaining object description features corresponding to the object in the image frame from the object detection model, and outputting a second classification result corresponding to the object description features based on a classifier in the object detection model; obtaining object convolution features for the image frame output from a convolution layer in the object detection model, and performing a multiplication operation between the second classification result and the object convolution features to obtain a second activation mapping corresponding to the image frame; A step of obtaining M object part area images (M is a positive integer) by performing a block division process of the image frame based on the second activation mapping, and obtaining part description features corresponding to each of the M object part area images based on the object detection model; combining the object description feature and part description features corresponding to the M object part area images as the object pose feature; The method according to claim 3.

5. The step of inputting the image frame to a body part detection model and obtaining the body part posture detection result by the body part detection model includes: inputting the image frame into a body part detection model and detecting a first object part of the object in the body part detection model; When the first object part is detected from the image frame, acquiring an area image including the first object part from the image frame, acquiring part key point positions corresponding to the first object part based on the area image, and determining a part posture detection result corresponding to the image frame based on the part key point positions; and determining a part orientation detection result corresponding to the image frame as a null value when the first object part is not detected from the image frame. The method of claim 2.

6. When the first object part is detected from the image frame, a region image including the first object part is obtained from the image frame, a part key point position corresponding to the first object part is obtained based on the region image, and a part posture detection result corresponding to the image frame is determined based on the part key point position. When the first object portion is detected from the image frame, a region image including the first object portion is acquired by cropping the image frame; obtaining a part contour feature corresponding to the region image, and predicting a part keypoint position corresponding to the first object part based on the part contour feature; and acquiring a part pose detection result corresponding to the image frame by connecting key points of the first object part based on the part key point positions. The method according to claim 5.

7. The step of acquiring an overall posture corresponding to the object by performing an interpolation process for at least one object part missing in the object posture detection result based on the part posture detection result and a reference posture related to the object, obtaining a reference pose associated with the object and determining a first number of keypoints corresponding to the reference pose and a second number of keypoints corresponding to the object pose detection; if the first number of key points is greater than the second number of key points, performing an interpolation process for at least one missing object part in the object pose detection result based on the reference pose to obtain a first candidate object pose; and performing an interpolation process of an object part related to a first object part in the first candidate object posture based on the part posture detection result, thereby obtaining an overall posture corresponding to the object. The method of claim 1.

8. The step of acquiring an overall posture corresponding to the object by performing an interpolation process of an object part related to a first object part in the first candidate object posture based on the part posture detection result, determining a first part orientation corresponding to the third object part based on a key point position of a first object part included in the part orientation detection result when the object pose detection result includes a pose of a second object part and the object pose detection result does not include a pose of a third object part, wherein the second object part and the third object part are symmetrical parts of the object and are related to the first object part; obtaining a first part length of the second object part in the first candidate object pose, and determining key point positions of the third object part based on the first part length and the first part orientation; and adding keypoint locations of the third object part to the first candidate object pose to obtain a global pose corresponding to the object in the image frame. The method according to claim 7.

9. the step of obtaining a global pose corresponding to the object in the image frame by adding keypoint locations of the third object part to the first candidate object pose, comprising: obtaining a second candidate object pose corresponding to the object in the image frame by adding keypoint positions of the third object part to the first candidate object pose, and obtaining a pose offset between the reference pose and the second candidate object pose; if the pose offset is greater than an offset threshold, obtaining a global pose corresponding to the object in the image frame by rectifying key points of the second candidate object pose based on the reference pose. The method according to claim 8.

10. the image frame is an i-th (i is a positive integer) image frame in the video data, The step of acquiring an overall posture corresponding to the object by performing an interpolation process of an object part related to a first object part in the first candidate object posture based on the part posture detection result, if the object pose detection result does not include the poses of the second object part and the third object part, determining a second part orientation corresponding to the second object part and a third part orientation corresponding to the third object part based on key point positions of a first object part included in the part pose detection result, the second object part and the third object part being symmetrical parts of the object and related to the first object part; obtaining a second part length corresponding to the second object part and a third part length corresponding to the third object part from a j-th image frame (j is a positive integer smaller than i), and determining a key point position of the second object part based on the second part length and the second part orientation; determining key point positions of the third object part based on the third part length and the third part orientation, and adding the key point positions of the second object part and the key point positions of the third object part to the first candidate object pose to obtain a global pose corresponding to the object in the image frame; The method according to claim 7.

11. constructing a virtual object associated with the object and controlling a pose of the virtual object based on the global pose. The method of claim 1.

12. 1. A data processing device, comprising: a pose detection module for obtaining an object pose detection result corresponding to an object in an image frame and a part pose detection result corresponding to a first object part of the object in the image frame, the first object part being one or more parts of the object; a posture estimation module that performs an interpolation process for missing object parts in the object posture detection result based on the part posture detection result and a reference posture related to the object, thereby obtaining an overall posture corresponding to the object, the overall posture being for operating a computer device to realize a service function corresponding to the overall posture; An apparatus comprising:

13. A computing device comprising a memory and a processor, The memory is connected to the processor, and a computer program is stored in the memory, and the processor invokes the computer program to cause the computer device to execute the method according to any one of claims 1 to 11. Computer equipment.

14. 12. A computer-readable storage medium having a computer program stored thereon, the computer program being configured, when loaded and executed by a processor, to cause a computing device comprising the processor to perform the method according to any one of claims 1 to 11.

15. A computer program product comprising computer programs / instructions which, when executed by a processor, cause the implementation of a method according to any one of claims 1 to 11.

Images