Action determination method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The invention provides an action determination method and device, electronic equipment, a computer program product and a computer readable storage medium. The embodiment of the invention can be applied to action determination scenes of robots, virtual reality, action capture and the like; the method comprises the following steps: extracting N visual features corresponding to N motion coordination hierarchies of a target object from an input image; wherein N is an integer and is greater than or equal to 2; based on the N visual features, determining corresponding M part sensing features for M target coordination hierarchies in the N motion coordination hierarchies; wherein the part sensing feature is used for representing a shielding part, a visible part and a correlation degree between the visible part and the shielding part of the target object, and M is an integer, Ngt; m > = 1; and based on the M part sensing features, performing regression to obtain action information of the shielded part and action information of the visible part. According to the method and the device, the accuracy of action determination when shielding exists can be improved.

Description

Technical Field

[0001] The present application relates to artificial intelligence technology, and in particular to an action determination method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] Determining the motion of a target object from an image is one of the applications of artificial intelligence. This motion information can be used in various scenarios, such as driving virtual humans and controlling robots. However, the target object is easily obscured by other objects, and when faced with occlusion, the methods used in related technologies often suffer from low motion determination accuracy. Summary of the Invention

[0003] Embodiments of the present application provide an action determination method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the accuracy of action determination when there is occlusion.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] This embodiment of the present application provides an action determination method, the method comprising:

[0006] Extract N visual features corresponding to N motion coordination levels of the target object from the input image; where N is an integer, N ≥ 2;

[0007] Based on the N visual features, determining corresponding M part perception features for M target coordination levels in the N motion coordination levels; wherein the part perception features are used to characterize the occluded part and the visible part of the target object, and the degree of association between the visible part and the occluded part, where M is an integer, and N>M≥1;

[0008] Based on the M part perception features, motion information of the occluded part and motion information of the visible part are obtained by regression.

[0009] An embodiment of the present application provides an action determination device, including:

[0010] An image processing module is used to extract N visual features corresponding to N motion coordination levels of the target object from the input image; where N is an integer, N ≥ 2;

[0011] a feature generation module for determining, based on the N visual features, M corresponding part perception features for M target coordination levels in the N motion coordination levels; wherein the part perception features are used to characterize the occluded parts and visible parts of the target object, and the degree of association between the visible parts and the occluded parts, where M is an integer, N>M≥1;

[0012] The motion regression module is used to regress the motion information of the occluded part and the motion information of the visible part based on the M part perception features.

[0013] In some embodiments of the present application, the feature generation module is also used to determine visual features other than the visual features corresponding to the M target coordination levels from the N visual features as features to be shared; fuse the features to be shared with the visual features of the M target coordination levels respectively to obtain M fused features; and based on the M fused features, determine the corresponding M part perception features for the M target coordination levels in the N motion coordination levels.

[0014] In some embodiments of the present application, the feature generation module is also used to perform the following processing through iteration i, where i is an integer and 1≤i≤M: determine the attention weight for each moving part under the i-th target coordination level; use the attention weight to adjust the sub-features corresponding to each moving part under the i-th target coordination level in the i-th fusion feature to obtain adjusted sub-features; use the adjusted sub-features of each moving part to generate the part perception feature of the i-th target coordination level.

[0015] In some embodiments of the present application, the motion regression module is also used to splice the M part perception features to obtain spliced ​​perception features; based on the spliced ​​perception features, the motion information of the occluded part and the motion information of the visible part are regressed.

[0016] In some embodiments of the present application, the motion regression module is also used to determine M alignment perception features based on the M part perception features; perform cross-level interaction on the M alignment perception features to obtain interactive perception features; and regress the motion information of the occluded part and the motion information of the visible part based on the interactive perception features.

[0017] In some embodiments of the present application, the motion regression module is further used to use the part perception feature with the highest feature dimension among the M part perception features as an extended baseline feature; expand the part perception features among the M part perception features except the extended baseline feature according to the dimension of the extended baseline feature to obtain corresponding extended perception features; and use the extended baseline feature and the extended perception feature to determine the M alignment perception features.

[0018] In some embodiments of the present application, the action regression module is also used to determine the index perception feature, the query perception feature and the perception feature to be weighted from the M aligned perception features; use the index perception feature and the query perception feature to calculate the weighted weight; and complete the cross-level interaction of the M aligned perception features by weighting the perception feature to be weighted with the weighted weight to obtain the interactive perception feature.

[0019] In some embodiments of the present application, the image processing module is further used to extract image features from the input image; perform the following processing through iteration j, where j is an integer and N≥j≥1: extract a two-dimensional feature map sequence and a three-dimensional feature map for the j-th motion coordination level from the image features; and generate the visual features at the j-th motion coordination level based on the three-dimensional feature map and the two-dimensional feature map sequence.

[0020] In some embodiments of the present application, the image processing module is further used to calculate the Hadamard product of the two-dimensional feature map sequence and the three-dimensional feature map, and use the Hadamard product to generate the visual features at the j-th motion coordination level.

[0021] In some embodiments of the present application, the motion determination device further includes: an object control module, configured to perform real-time motion control on the virtual object based on the motion information of the blocked portion and the motion information of the visible portion.

[0022] An embodiment of the present application provides an electronic device, comprising:

[0023] a memory for storing computer-executable instructions;

[0024] The processor is used to implement the action determination method provided in the embodiment of the present application when executing the computer-executable instructions stored in the memory.

[0025] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the action determination method provided in the embodiment of the present application when executed by a processor.

[0026] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the action determination method provided in the embodiment of the present application is implemented.

[0027] The embodiments of the present application have the following beneficial effects: first, visual features are extracted at multiple different motion coordination levels, and then the visual features of multiple different motion coordination levels are combined to construct part perception features that can characterize the occluded parts, visible parts, and the degree of correlation between the occluded parts and the visible parts of the target object. Then, the motion information is regressed, so that the visible parts based on different motion coordination levels and the degree of correlation with the occluded parts can be realized, thereby improving the amount of information when inferring the motion information of the occluded parts, and ultimately improving the accuracy of motion determination when there is occlusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 Schematic diagram of the architecture of the action determination system provided in an embodiment of the present application;

[0029] Figure 2 This embodiment of the present application provides Figure 1 A schematic diagram of the server structure in FIG;

[0030] Figure 3 This is a flow diagram of the action determination method provided in the embodiment of the present application. Figure 1 ;

[0031] Figure 4 is a schematic diagram of the movement coordination level provided by an embodiment of the present application;

[0032] Figure 5 This is a flow diagram of the action determination method provided in the embodiment of the present application. Figure 2 ;

[0033] Figure 6 This is a flow diagram of the action determination method provided in the embodiment of the present application. Figure 3 ;

[0034] Figure 7 This is a flow diagram of the action determination method provided in the embodiment of the present application. Figure 4 ;

[0035] Figure 8 Schematic diagram of the algorithm architecture for determining human body movements provided in an embodiment of the present application;

[0036] Figure 9 This is a comparison diagram of the effects of human motion information provided by the embodiment of the present application;

[0037] Figure 10 3 is a schematic diagram comparing action information obtained by various different methods provided in the embodiments of the present application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0039] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0040] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0041] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0042] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant national laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0043] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0044] 1) Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0045] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, pre-trained models, operating / interaction systems, and mechatronics. Pre-trained models, also known as large models or basic models, can be fine-tuned and widely applied to downstream tasks across various AI disciplines. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0046] 2) Computer vision (CV) is the science of making machines "see." Specifically, it refers to the use of cameras and computers to replace the human eye in identifying, detecting, and measuring objects. Further image processing is performed to transform images into images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems that can extract information from images or multidimensional data. Large model technology has brought significant changes to the development of computer vision technology. Pre-trained models in the field of vision, such as the Swin Transformer, Vit, V-MoE, and MAE, can be fine-tuned to quickly and widely apply to specific downstream tasks. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / action recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies.

[0047] 3) Machine Learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning. Pretrained models are the latest development in deep learning, integrating these techniques.

[0048] 4) Motion capture: This involves capturing and recording the motion of a real-world target (such as a human or object) and using that motion information to control a digitized virtual object, thereby achieving a high degree of fidelity to the target's movements. Motion capture has applications in filmmaking, game development, virtual reality, medical rehabilitation, and other fields.

[0049] 5) Occlusion Handling: In motion capture technology, occlusion handling refers to the process of restoring and predicting the motion information of the occluded part as much as possible through algorithms or models when the target object is occluded by other objects during the motion capture process, thereby improving the accuracy and integrity of motion capture, and ultimately making motion capture technology more reliable and stable in complex environments.

[0050] 6) Movement coordination level refers to the granularity used when segmenting the target object (e.g., a human body or object) based on the coordination and complexity of different movements. Taking the human body as an example, the body can be segmented based on the coordination and complexity of different movements (e.g., squatting, walking, etc.). For example, the body can be segmented into individual body parts (e.g., hands, feet), or into coordinated body parts (e.g., forearms, upper arms).

[0051] In an embodiment of the present application, there may be multiple levels of motion coordination, for example, an independent part level, an interdependence level, a motion pattern level, and a global posture level.

[0052] 7) A virtual scene is a virtual scene displayed (or provided) when an application is running on a terminal. This virtual scene can be a simulated environment of the real world, a virtual environment that is partially simulated and partially imaginary, or a purely imaginary virtual environment. The virtual environment can be any of a two-dimensional virtual environment, a 2.5-dimensional virtual scene, or a three-dimensional virtual scene, and is not limited in this embodiment of the present application.

[0053] 8) Virtual objects are the images of various people and objects that can interact in a virtual scene, or movable objects in a virtual scene. In the embodiments of the present application, virtual objects are virtual characters that can be controlled in a virtual scene, such as virtual characters that a user controls through operations on a client, or virtual characters that an agent controls through instructions.

[0054] 9) End-to-end learning: The entire training and prediction process, from raw data input to task output, is completed within the neural network model without any additional processing. In other words, end-to-end learning eliminates the need for external pre- or post-processing between raw data input and final output.

[0055] With the research and advancement of artificial intelligence technology, artificial intelligence will be studied and applied in multiple fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones, digital twins, virtual humans, robots, artificial intelligence generated content (AIGC), conversational interaction, smart medical care, smart customer service, game AI, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0056] Determining the motion of a target object from an image is one of the application areas of artificial intelligence. The determined motion information can be applied to various scenarios, such as driving virtual humans and controlling robots. This process is known as motion capture. There are generally three methods for determining the motion of a target object from an image, which are illustrated below using the human body as an example. The first method uses multimodal sensors, a host computer, a router, and a camera to collect posture and video data of the human body. The resulting electromyographic signals and inertial information are used as features to input into a neural network to obtain motion information. However, this method cannot collect comprehensive data in the presence of occlusions, making it difficult to output reasonable motion information in the presence of occlusions. The second method is to regress motion poses using optical flow features. This method detects specific light points on the target object to achieve motion capture. Similarly, this method suffers from data loss in the presence of occlusions, making it difficult to output reasonable motion information in the presence of occlusions. The third method is to regress human pose and shape parameters from a single color image. Techniques for regressing pose and shape parameters can be roughly divided into two directions: auxiliary regression and direct regression. Auxiliary regression utilizes prior knowledge from auxiliary domains (such as 3D keypoints or segmentation maps) to enhance the neural network's regression of model parameters. For example, some methods decompose pose parameters into two motion processes involving sway and twist. A recurrent neural network reconstructs a spin representation, while the sway signature is derived from an existing 3D keypoint prediction network. The two are ultimately fused into the output. Other methods use a soft attention mechanism for regression to predict an attention mask and enhance some of the information. However, the outputs of these methods all depend on a single auxiliary prior, which cannot provide reasonable motion information in the face of occlusion. Direct regression requires feature extraction from a single color image to regress model parameters. In some methods, a convolutional neural network (CNN) is used to extract image features, and a multilayer perceptron (MLP) is used to directly regress model parameters. A skinned multi-person linear model (SMPL) and weak perspective projection are used to project meshes onto keypoints to optimize the resulting 2D keypoints, thereby incorporating in-the-wild data for training. Other methods build on the Human Mesh Recovery (HMR) model, utilize bbox information (b bounding boxes) to accurately locate the human body in the original image, and introduce full-image projection to effectively enhance prediction results. However, direct regression methods directly extract features and perform regression on the entire image. Therefore, when there are occlusions in the image, it is easy to obtain incorrect regression results, leading to incorrect determination of motion information.For example, some direct regression methods use extracted features to directly regress the parameters of the 24 joints in the SMPL model, without establishing connections between the joints. Consequently, for occluded parts, they tend to use the joint parameters of adjacent visible parts for inference. This approach ignores the correlation between different moving parts of the human body. For example, when a person swings their arms while walking, if the right arm is occluded, the movement of the right arm can generally be inferred from the left leg taking a step forward. However, the direct regression method directly infers the movement of the occluded part from adjacent visible parts, such as the right leg or right shoulder, and cannot accurately infer the movement of the right arm.

[0057] As can be seen from the above, the methods in the related art are unable to accurately determine the motion information of the target object when facing occlusion, resulting in low accuracy in motion determination when there is occlusion.

[0058] Based on this, the embodiments of the present application provide an action determination method, device, electronic device, computer-readable storage medium, and computer program product, which can improve the accuracy of action determination when there is occlusion. The following describes an exemplary application of the electronic device provided by the embodiment of the present application. The electronic device provided by the embodiment of the present application can be implemented as various types of terminals such as laptops, tablet computers, desktop computers, set-top boxes, smart phones, smart speakers, smart watches, smart TVs, and car terminals, and can also be implemented as servers. Below, an exemplary application of the electronic device when it is implemented as a server will be described.

[0059] See also Figure 1 , Figure 1 : is a schematic diagram of the architecture of the action determination system provided in an embodiment of the present application. In order to support an action determination application, in the action determination system 100, the terminal 400 (terminal 400-1 and terminal 400-2 are shown as examples) is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two. The action determination system 100 is also provided with a database 500 for providing data support to the server 200. Among them, the database 500 can be configured in the server 200 or independently of the server 200.

[0060] The terminal 400 is used to start collecting input images in response to a user triggering an acquisition start indicator displayed in a graphical interface (graphic interface 410 - 1 and graphical interface 410 - 2 are shown as examples), and upload the input images to the server 200 through the network 300 .

[0061] The server 200 is used to extract N visual features corresponding to N motion coordination levels of the target object from the input image; wherein N is an integer, N≥2; based on the N visual features, determine corresponding M part perception features for M target coordination levels in the N motion coordination levels, and the part perception features are used to characterize the occluded parts, visible parts, and the degree of correlation between the visible parts and the occluded parts of the target object, M is an integer, N>M≥1; based on the M part perception features, regress the motion information of the occluded parts and the motion information of the visible parts; control the virtual object based on the motion information of the occluded parts and the motion information of the visible parts, and send the motion picture of the virtual object to the terminal 400.

[0062] The terminal 400 displays the action picture of the virtual object on the graphical interface.

[0063] The embodiments of the present application can be implemented with the help of cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network within a wide area network or a local area network to realize data calculation, storage, processing, and sharing.

[0064] Cloud computing is a general term for network technologies, information technology, integration technologies, management platforms, and application technologies used in the cloud computing business model. These technologies can form resource pools that are used on demand and are flexible and convenient. Cloud computing technology will become a crucial support. System backend services within technical networks require extensive computing and storage resources, which must be implemented through cloud computing.

[0065] Exemplarily, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal 400 and the server may be connected directly or indirectly via wired or wireless communication, which is not limited in the embodiments of the present application.

[0066] See also Figure 2 , Figure 2 This embodiment of the present application provides Figure 1 A schematic diagram of the structure of a server (an implementation of an electronic device) in FIG. Figure 2The server 200 shown includes: at least one processor 210, a memory 250, and at least one network interface 220. The various components in the server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 240 is not described in detail. Figure 2 Various buses are labeled as bus system 240 .

[0067] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0068] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 250 may optionally include one or more storage devices that are physically remote from the processor 210.

[0069] The memory 250 includes a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 250 described in the embodiments of the present application is intended to include any suitable type of memory.

[0070] In some embodiments, the memory 250 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0071] Operating system 251, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0072] A network communication module 252 for reaching other electronic devices via one or more (wired or wireless) network interfaces 220 , exemplary network interfaces 220 including Bluetooth, Wi-Fi, and Universal Serial Bus (USB);

[0073] In some embodiments, the action determination device provided in the embodiments of the present application can be implemented in software. Figure 2 The motion determination device 255 stored in the memory 250 is shown. This device can be software in the form of a program or plug-in, and includes the following software modules: an image processing module 2551, a feature generation module 2552, a motion regression module 2553, and an object control module 2554. These modules are logical and can be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.

[0074] In other embodiments, the action determination device provided in the embodiments of the present application can be implemented in hardware. As an example, the action determination device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the action determination method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0075] In some embodiments, the terminal or server (both electronic devices may be real-time) can implement the action determination method provided by the embodiment of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be commands, machine instructions or software instructions at the microprogram level. The computer program can be a native program or software module in the operating system; it can be a local (Nativ e) application (APPlication, APP), that is, a program that needs to be installed in the operating system to run, such as a game APP; it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0076] Below, the action determination method provided by the embodiment of the present application will be described in conjunction with the exemplary application and implementation of the electronic device provided in the embodiment of the present application. As mentioned above, the electronic device that implements the action determination method of the embodiment of the present application can be a terminal, a server, or a combination of the two. Therefore, the execution entity of each step will not be repeated below.

[0077] See also Figure 3 , Figure 3 This is a flow diagram of the action determination method provided in the embodiment of the present application. Figure 1 , will combine Figure 3 The steps shown are explained, Figure 3 The main body of the step is the electronic device.

[0078] Step 101: Extract N visual features corresponding to N motion coordination levels of a target object from an input image.

[0079] The embodiment of the present application is implemented in a scenario where the motion of the target object is captured to obtain the motion information of the target object. In the embodiment of the present application, after the action determination process starts, the electronic device will perform feature extraction on the input image at N motion coordination levels, and the visual features of the target object can be obtained for each motion coordination level, that is, the electronic device can obtain N visual features corresponding to the N motion coordination levels. It should be noted that N is an integer and N≥2. That is, the electronic device will extract visual features of at least two motion coordination levels.

[0080] The input image can be obtained by capturing an image of the target object using an image acquisition device (e.g., a camera, a video camera, etc.). Thus, the input image contains image information of the target object. It is understood that in embodiments of the present application, the input image can also contain image information of obstructions (e.g., clothing, trees, etc.) to the target object. In other words, the input image can contain obstructions to the target object.

[0081] The input image can be a color image, a grayscale image, or an infrared image, and this embodiment of the present application does not limit this. The input image can be received by the electronic device from another device, collected by the electronic device itself, or obtained by the electronic device from the network, and this embodiment of the present application does not limit this.

[0082] The target object is the target for which motion capture is required. The target object can be a human body, an object, or other animals, robots, etc., and the embodiments of the present application do not limit this.

[0083] The motion coordination level refers to the granularity used when dividing the target object into parts based on the coordination and complexity of the target object in different motions. The N motion coordination levels can be any N of the independent part level, the interdependence level, the motion pattern level, and the global posture level. In some embodiments of the present application, the N motion coordination levels can include: the independent part level, the interdependence level, the motion pattern level, and the global posture level.

[0084] It should be noted that the independent part hierarchy corresponds to the various parts that make up the target object. This hierarchy can provide important information when specific areas are occluded. Taking the human body as an example, the independent part hierarchy has a total of 24 parts, including the hand, forearm, head, and neck. The interdependent hierarchy corresponds to the parts of the target object that require coordination when moving. This hierarchy can be used to infer occluded areas based on adjacent visible areas. Taking the human body as an example, the interdependent hierarchy has a total of 11 parts, such as the forearm, upper arm, head and neck, left leg, and right leg. The motion pattern hierarchy is a hierarchy that can reflect the target object's motion pattern, providing information on the motion pattern even when partially visible. Taking the human body as an example, the motion pattern hierarchy includes six parts, such as the entire left arm and the entire left lower limb. The global pose hierarchy corresponds to the entire target object, providing a comprehensive understanding of the target object's posture and motion.

[0085] For example, Figure 4 It is a schematic diagram of the motion coordination level provided in an embodiment of the present application. Figure 4 There are four levels of motion coordination in the image: independent part level 4-1, interdependence level 4-2, motion pattern level 4-3, and global posture level 4-4. It can be seen that different levels of motion coordination can provide visual cues at different granularities for the target object.

[0086] The visual features can be feature maps or feature vectors, which are not limited in the present embodiment. The visual features can be directly extracted by a deep learning model such as a convolutional neural network, or they can be obtained by further processing the feature maps extracted by a deep learning model such as a convolutional neural network.

[0087] In some embodiments of the present application, Figure 3 Step 101 in the above, i.e., extracting N visual features corresponding to N motion coordination levels of the target object from the input image, can be achieved by the following processing: extracting image features from the input image; performing the following processing by iteration j, where j is an integer and N≥j≥1: extracting a two-dimensional feature map sequence and a three-dimensional feature map for the j-th motion coordination level from the image features; and generating visual features at the j-th motion coordination level based on the three-dimensional feature map and the two-dimensional feature map sequence.

[0088] Electronic devices can extract features from input images through trained convolutional neural networks to obtain image features. Afterwards, for each level of motion coordination, the image features will be extracted twice, once through two-dimensional convolution and once through three-dimensional convolution. It should be noted that a single two-dimensional convolution kernel obtains a single two-dimensional feature map. Here, multiple two-dimensional convolution kernels (each moving part corresponds to a two-dimensional convolution kernel, so the number of two-dimensional convolution kernels is the same as the number of moving parts) can be used to extract image features, thereby obtaining multiple two-dimensional feature maps, and multiple two-dimensional feature maps can constitute a two-dimensional feature map sequence. That is, the number of two-dimensional feature maps in the two-dimensional feature map sequence is the same as the number of channels of the convolution layer.

[0089] In some embodiments of the present application, generating visual features at the j-th motion coordination level based on a three-dimensional feature map and a two-dimensional feature map sequence can be achieved by the following processing: calculating the Hadamard product of the two-dimensional feature map sequence and the three-dimensional feature map, and using the Hadamard product to generate visual features at the j-th motion coordination level.

[0090] The electronic device can directly use the Hadamard product as the visual feature at the jth motion coordination level, or it can continue to extract features from the Hadamard product and use the extracted features as the visual feature at the jth motion coordination level. This embodiment of the present application does not limit this.

[0091] In other embodiments of the present application, generating visual features at the j-th motion coordination level based on a three-dimensional feature map and a two-dimensional feature map sequence can also be achieved through the following processing: superimposing the three-dimensional feature map and the two-dimensional feature map sequence, and using the superimposed feature map as the visual features at the j-th motion coordination level.

[0092] It should be noted that the 2D feature maps in the 2D feature map sequence are arranged along the Z-axis of 3D space. Therefore, the 2D feature map sequence can be viewed as a 3D cube and can be superimposed on the 3D feature map. The electronic device can then directly use the superimposed feature map as the visual feature at the jth motion coordination level, or convert the feature map into a feature vector and use the resulting motion vector as the visual feature at the jth motion coordination level.

[0093] In some embodiments of the present application, Figure 3Step 101 in the above, i.e., extracting N visual features corresponding to the N motion coordination levels of the target object from the input image, can also be achieved by the following processing: utilizing a feature extraction network (e.g., a two-dimensional convolutional neural network) corresponding to each of the N motion coordination levels to perform N feature extractions on the input image, and directly determining the N extracted features as N visual features.

[0094] Step 102: Based on the N visual features, determine corresponding M part perception features for M target coordination levels in the N motion coordination levels.

[0095] The electronic device screens N motion coordination levels to obtain M motion coordination levels for feature fusion. These motion coordination levels are referred to as target coordination levels. M is an integer, and N > M ≥ 1. In other words, the number of target coordination levels must be smaller than the number of motion coordination levels. The electronic device then generates a corresponding part perception feature for each target coordination level, resulting in M ​​part perception features corresponding one-to-one to each of the M target coordination levels.

[0096] It should be noted that part-aware features are used to characterize the occluded and visible parts of a target object, as well as the degree of correlation between the visible and occluded parts. Occluded parts are the occluded moving parts of a target object, while visible parts are the visible moving parts of a target object. In other words, part-aware features contain knowledge about which moving parts of a target object are occluded and which are visible, as well as whether there is a correlation between visible moving parts and invisible occluded parts, and the degree of correlation.

[0097] It should also be noted that the M part perception features are determined by N visual features, and N is greater than M. Therefore, each part perception feature, in addition to containing the visual features corresponding to the corresponding target coordination level, also contains visual features of other motion coordination levels. Therefore, the part perception feature integrates the visual clues of multiple motion coordination levels.

[0098] In embodiments of the present application, the electronic device can arbitrarily extract M motion coordination levels from N motion coordination levels to serve as M target coordination levels. Alternatively, the N motion coordination levels can be arranged in ascending order of the granularity used when dividing the target object into parts within the N motion coordination levels to obtain a coordination level sequence, and the first M motion coordination levels in the coordination level sequence can be selected as the M target coordination levels. In other words, in embodiments of the present application, the M motion coordination levels with the smallest granularity used when dividing the parts within the N motion coordination levels can be selected as the M target coordination levels. For example, in some embodiments of the present application, the N motion coordination levels are: an independent part level, an interdependent level, a motion pattern level, and a global posture level. The electronic device selects the three motion coordination levels with the smallest granularity as the target coordination levels, so that the M target coordination levels can include: an independent part level, an interdependent level, and a motion pattern level. Of course, in other embodiments, the target coordination levels can also be manually specified based on actual needs to better meet the requirements of actual scenarios.

[0099] See also Figure 5 , Figure 5 This is a flow diagram of the action determination method provided in the embodiment of the present application. Figure 2 In some embodiments of the present application, Figure 3 Step 102, i.e., determining corresponding M part perception features for M target coordination levels in the N motion coordination levels based on the N visual features, may include the following steps:

[0100] Step 1021: Determine the visual features other than the visual features corresponding to the M target coordination levels from the N visual features as features to be shared.

[0101] In an embodiment of the present application, there will be M corresponding visual features for M target coordination levels. The electronic device will remove the M visual features corresponding to the M target coordination levels from the N visual features and determine the remaining visual features as the features to be shared.

[0102] Exemplarily, when the N motion coordination levels include four levels: independent part level, interdependence level, motion pattern level and global posture level, and the M target coordination levels include three levels: independent part level, interdependence level and motion pattern level, the features to be shared are the visual features corresponding to the global posture level.

[0103] Step 1022: Fuse the features to be shared with the visual features of the M target coordination levels to obtain M fused features.

[0104] The electronic device fuses the to-be-shared features with the visual features corresponding to each target coordination level to obtain a fused feature for each target coordination level. Thus, the electronic device can obtain M fused features corresponding to the M target coordination levels.

[0105] It should be noted that the electronic device can fuse the features to be shared with the visual features corresponding to each target coordination level by splicing, or can fuse the features to be shared with the visual features corresponding to each target coordination level by weighted summation. The embodiments of the present application do not limit this.

[0106] Step 1023: Based on the M fusion features, determine corresponding M part perception features for the M target coordination levels in the N motion coordination levels.

[0107] After obtaining M fusion features, the electronic device will use the fusion features of each target coordination level to generate part perception features of each target coordination level, thereby obtaining M part perception features.

[0108] It can be understood that by fusing the features to be shared with the visual features of M target coordination levels, it is possible to fuse the information provided by multiple motion coordination levels into one feature, thereby increasing the amount of information in the fused feature, and thus increasing the amount of information in the part perception feature.

[0109] In some embodiments of the present application, Figure 4 Step 1023 in , i.e., determining M corresponding part perception features for M target coordination levels in N motion coordination levels based on M fused features, can be achieved by the following processing: performing the following processing by iteration i, where i is an integer and 1≤i≤M: determining an attention weight for each motion part under the i-th target coordination level; adjusting the sub-features corresponding to each motion part under the i-th target coordination level in the i-th fused feature using the attention weight to obtain adjusted sub-features; and generating the part perception features of the i-th target coordination level using the adjusted sub-features of each motion part.

[0110] That is, the fused features corresponding to each target coordination level are composed of the sub-features corresponding to the moving parts at that target coordination level. The electronic device can determine the corresponding attention weight for each moving part through a self-attention mechanism (using the sub-features of each moving part read in through a self-attention module for calculation. The self-attention module here is a trained network that can perceive occlusion and adjust the weights of different moving parts within the same level). It can also directly obtain the weights specified for each moving part at the target coordination level as the attention weights for each moving part. Next, the electronic device uses the obtained attention weights to weight the sub-features corresponding to each moving part. The weighted features are the adjusted sub-features. Finally, the electronic device re-fuse (e.g., concatenates, sums, etc.) the adjusted sub-features of each moving part into a single feature, which is the part-perception feature for each target coordination level.

[0111] It should be noted that in the embodiments of the present application, whether the moving part is visible, and the importance of the visible moving part (i.e., the visible part) relative to the occluded moving part (i.e., the occluded part), can both be reflected by the attention weight. For example, the attention weight of the occluded part can be less than a preset value, while the attention weight of the visible part can be greater than a preset value. The larger the attention weight of the visible part, the higher the degree of correlation between it and the occluded part.

[0112] In other embodiments of the present application, Figure 4 Step 1023, i.e., determining M corresponding part perception features for M target coordination levels in N motion coordination levels based on M fused features, can also be achieved by the following processing: matching each fused feature with multiple preset features corresponding to each target coordination level, and determining the perception features corresponding to the matched preset features as the part perception features corresponding to each target coordination level.

[0113] Among them, the preset features can be set manually or obtained from the cluster center of historical part perception features at the same target coordination level.

[0114] In other embodiments of the present application, Figure 3 Step 102, i.e., the process of determining corresponding M part perception features for M target coordination levels in N motion coordination levels based on N visual features, can also be achieved by the following processing: utilizing the feature extraction model corresponding to each target coordination level, performing feature extraction on the fusion result of the N visual features, and obtaining the part perception features of each target coordination level.

[0115] Among them, the feature extraction model corresponding to each target coordination level can be a convolutional neural network, a recurrent neural network or other models, which is not limited in the embodiments of the present application.

[0116] Step 103: Based on the M part perception features, regress the motion information of the occluded part and the motion information of the visible part.

[0117] After obtaining M part-sensing features, the electronic device regresses motion information for the visible parts based on the knowledge contained in these features, namely, which moving parts are visible and which are occluded at each of the M target coordination levels. Combined with the features of the visible parts, the electronic device regresses motion information for the visible parts. Simultaneously, the electronic device regresses motion information for the occluded parts based on the occluded parts and the degree of correlation between them and the visible parts. The motion information of the occluded parts and the visible parts together constitute the motion information of the target object.

[0118] When M is greater than 1, the electronic device uses the part perception features of multiple target coordination levels to regress the motion information, and the target coordination level essentially belongs to the motion coordination level. Therefore, the regression of the motion information is essentially achieved based on the information provided by multiple motion coordination levels. When M is 1, although the electronic device only uses the part perception features of one target coordination level to regress the motion information, since each part perception feature integrates the information provided by multiple motion coordination levels, when M is 1, the regression of the motion information is essentially achieved based on the information provided by multiple motion coordination levels. It can be seen that in the embodiment of the present application, regardless of whether M is 1, the motion information is essentially obtained by regression based on the information provided by multiple motion coordination levels.

[0119] It should be noted that regression can be understood as predicting the value of a dependent variable using an independent variable. In this embodiment, the independent variable can be understood as M part-perception features, and the dependent variable can be understood as motion information. Therefore, regressing motion information based on M part-perception features can be understood as predicting motion information of occluded and visible parts based on M part-perception features. This prediction process can be implemented using a trained multi-layer perceptron, a fully connected layer, or other network model.

[0120] It is understandable that, compared to visible parts, the motion information of occluded parts needs to be inferred. When performing motion regression based on the information provided by multiple motion coordination levels, it is actually a comprehensive consideration of the posture and motion of the visible parts of each of the multiple motion coordination levels to determine what posture and motion will appear in the occluded part. This approach is obviously richer and more comprehensive than using only the information provided by a single motion coordination level, thereby increasing the amount of information when regressing motion information. For example, for a walking human body, the occluded part is the left arm. If only the information provided by the visible parts of the independent part level (such as the hand, shoulder) is used, one or more reasonable situations may occur in which the arm is forward, backward, left, or right, resulting in an inability to clearly determine the posture and motion of the arm; if the visible parts of the motion pattern level (such as the right leg is behind and the left leg is in front) are considered at the same time, it is found that the human body is most likely walking, then it is easy to judge that the left arm should be in front of the body trunk. Therefore, it can be seen that combining clues from multiple motion coordination levels can significantly increase the amount of information in regressing the motion information of the occluded part.

[0121] In the embodiments of the present application, motion information refers to parameters that can describe the posture and motion of the moving part. Motion information may include the rotation axis angle, three-dimensional coordinates, etc. of the moving part (e.g., defined human joints, rotating parts of a robot), and is not limited in the embodiments of the present application.

[0122] See also Figure 6 , Figure 6 This is a flow diagram of the action determination method provided in the embodiment of the present application. Figure 3 In some embodiments of the present application, Figure 3 Step 103 in the above, i.e., the process of regressing the motion information of the occluded part and the motion information of the visible part based on the M part perception features, can be implemented by the following processing:

[0123] Step 1031: Determine M alignment perception features based on the M part perception features.

[0124] The dimensions of the perception features of different parts may be different (this is because different target coordination levels have different numbers of moving parts. Each moving part has its corresponding sub-features. When the dimensions of the sub-features are the same but the number is different, it is very likely that the dimensions of the perception features of different parts will be different). The electronic device aligns the M part perception features to the same dimension, so that the dimensions of the M aligned perception features are the same.

[0125] In some embodiments of the present application, Figure 6Step 1031 in, i.e., determining M alignment perception features based on M part-perception features, can be achieved by the following processing: taking the part-perception feature with the highest feature dimension among the M part-perception features as the extended baseline feature; expanding the part-perception features other than the extended baseline feature among the M part-perception features according to the dimension of the extended baseline feature to obtain the corresponding extended perception features; and determining M alignment perception features using the extended baseline feature and the extended perception feature.

[0126] It should be noted that the electronic device can interpolate the remaining part-perception features so that the dimensions of the remaining part-perception features are the same as the dimensions of the extended baseline features, or can copy the sub-features in the remaining part-perception features so that the dimensions of the remaining part-perception features are the same as the dimensions of the extended baseline features.

[0127] For example, when the dimension of the extended baseline feature is 24×256 (256 is the dimension of each sub-feature, and 24 is the number of sub-features), and the dimension of the part perception feature that needs to be dimensionally expanded is 11×256 (11 is the number of sub-features), the electronic device can copy one or more sub-features in the part perception feature until the number of sub-features becomes 24, and use the concatenation result of the copied sub-features and the original sub-features as the extended perception feature.

[0128] After obtaining the extended perception feature, the electronic device can directly determine the extended baseline feature and the extended perception feature as M aligned perception features (at this time, there is 1 extended baseline feature and M-1 extended perception features), or it can perform feature extraction again for the extended baseline feature and the extended perception feature respectively, and use the extracted features as M aligned perception features. This embodiment of the present application is not limited here.

[0129] It can be understood that aligning M part-perception features by dimensional expansion can ensure that the information in the part-perception features with higher dimensions is not lost, that is, the part-perception features are highly redundant, thereby ensuring the information volume of the M part-perception features.

[0130] In other embodiments of the present application, Figure 6 Step 1031 in , i.e., determining M alignment-perceived features based on M part-perceived features, can also be achieved by the following processing: taking the part-perceived feature with the lowest feature dimension among the M part-perceived features as the compressed reference feature; compressing the part-perceived features other than the compressed reference feature among the M part-perceived features according to the dimension of the compressed reference feature to obtain the corresponding compressed perception feature; and determining M alignment-perceived features using the compressed reference feature and the compressed perception feature.

[0131] In addition to aligning the M part-sensing features using an expansion method, the electronic device can also align the M part-sensing features using a compression method. In this case, the electronic device can achieve feature dimension compression by downsampling, or it can achieve feature dimension compression by using a convolutional layer with scale compression capability, which is not limited in this embodiment of the present application.

[0132] After obtaining the compressed reference features and the compressed sensing features, the electronic device can directly determine the compressed reference features and the compressed sensing features as M alignment sensing features, or perform feature extraction again based on the compressed reference features and the compressed sensing features to obtain M alignment sensing features.

[0133] Step 1032: Perform cross-level interaction on the M aligned perception features to obtain interactive perception features.

[0134] Since the M alignment perception features belong to different target coordination levels, after obtaining the M alignment perception features, the electronic device performs cross-level feature interaction on the M alignment perception features, and the features obtained by the interaction are called interactive perception features.

[0135] It should be noted that since the M alignment perception features correspond to M different target coordination levels, the cross-level feature interaction is actually to integrate the knowledge contained in the alignment perception features of each target coordination level with the knowledge contained in the alignment perception features of one or more other target coordination levels, so that the obtained interactive perception features integrate the knowledge of different target coordination levels.

[0136] In some embodiments of the present application, Figure 6 Step 1023 in the above, i.e., performing cross-level interaction on the M aligned perception features to obtain interactive perception features, can be achieved by the following processing: determining index perception features, query perception features, and perception features to be weighted from the M aligned perception features; calculating weighted weights using the index perception features and the query perception features; and completing cross-level interaction on the M aligned perception features by weighting the perception features to be weighted with the weighted weights to obtain interactive perception features.

[0137] The index perception feature, query perception feature, and to-be-weighted perception feature can each be any one of the M aligned perception features. The above process can be repeated multiple times, with the index perception feature, query perception feature, and to-be-weighted perception feature being different each time, thereby obtaining multiple weighted features. In this case, the electronic device can concatenate these multiple weighted features as the interactive perception feature.

[0138] It should be noted that the above processing can be completed through a cross-attention layer, in which case the index perception feature is the Q of the cross-attention layer, the query perception feature is the K of the cross-attention layer, and the perception feature to be weighted is the V of the cross-attention layer. In an embodiment of the present application, multiple cross-attention layers can be set so that the M interactive perception features can be used as index perception features, or as query perception features, or as perception features to be weighted in sequence.

[0139] Below, the setting of the cross-attention layer is explained by taking M target coordination levels including independent part level, interdependence level and motion pattern level as an example.

[0140] In the cross-attention layer in the related technology, K and V come from the same feature, while Q comes from another feature. Therefore, if the cross-attention layer in the related technology is used, 6 cross-attention layers are required to realize cross-level interaction of part perception features at the independent part level, interdependence level, and motion pattern level.

[0141] However, the more cross-attention layers there are, the longer the computation time required. Therefore, in the embodiment of the present application, only three cross-attention layers can be set, and the part perception features of the independent part level, the part perception features of the interdependent level, and the part perception features of the motion pattern level can be used as Q of the cross-attention layer, and any two of the remaining part perception features can be selected as K, and the remaining one can be used as V. In this way, it has been experimentally verified that three cross-attention layers can also achieve the same interactive effect, but the number of cross-attention layers required is smaller, thereby reducing the computation time.

[0142] In other embodiments of the present application, Figure 6 Step 1023 in the above, i.e., performing cross-level interaction on the M aligned perception features to obtain interactive perception features, can also be achieved by superimposing the M aligned perception features and determining the superimposed features as interactive perception features.

[0143] Step 1033: Based on the interactive perception features, regress the motion information of the occluded part and the motion information of the visible part.

[0144] Based on the obtained interactive perception features, the electronic device regresses motion information for the visible and occluded parts. The electronic device can read the interactive perception features through a trained multi-layer perceptron to regress the motion information. The electronic device can also match the interactive perception features with preset features corresponding to different motion information to obtain motion information for the occluded and visible parts.

[0145] In other embodiments of the present application, Figure 3 Step 103, i.e., the process of regressing the motion information of the occluded part and the motion information of the visible part based on the M part perception features, can also be achieved by the following processing: splicing the M part perception features to obtain spliced ​​perception features; and regressing the motion information of the occluded part and the motion information of the visible part based on the spliced ​​perception features.

[0146] The electronic device concatenates the M part-perception features to obtain a spliced ​​perception feature. The electronic device can then use a trained multi-layer perceptron to read the spliced ​​perception feature and perform motion information regression. The electronic device can also match the spliced ​​perception feature with preset features corresponding to different motion information to obtain motion information for both obscured and visible parts.

[0147] It can be understood that compared with the related art, there is a problem that the action information of the target object cannot be accurately determined when facing occlusion, which leads to a low accuracy of action determination when there is occlusion. In the embodiment of the present application, visual features are first extracted at multiple different motion coordination levels, and the visual features of multiple different motion coordination levels are combined to construct part perception features that can characterize the occluded parts, visible parts, and the degree of correlation between the occluded parts and the visible parts of the target object, and then the action information is regressed, so that the visible parts based on different motion coordination levels and the degree of correlation with the occluded parts can be realized, thereby improving the amount of information when inferring the action information of the occluded parts, and ultimately improving the accuracy of action determination when there is occlusion.

[0148] based on Figure 3 , see Figure 7 , Figure 7 This is a flow diagram of the action determination method provided in the embodiment of the present application. Figure 4 In some embodiments of the present application, after step 103, that is, after regressing the motion information of the occluded part and the motion information of the visible part based on the M part perception features, the following steps may also be performed:

[0149] Step 104: Perform real-time motion control on the virtual object based on the motion information of the blocked portion and the motion information of the visible portion.

[0150] It's important to note that the combined motion information of the occluded and visible parts provides the motion information for the entire target object. This allows the virtual object to perform the same movements as the target object. For example, if the target object is walking, the combined motion information of the occluded and visible parts allows the virtual object to also walk.

[0151] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0152] The embodiment of the present application is implemented in a scenario where the motion information of the human body is determined in real time based on color images captured by a monocular camera. The captured motion can be used in fields such as human-computer interaction and real-time driving of virtual humans.

[0153] First, the principle of the embodiment of the present application is explained. Since the human body (referred to as the target object) can exhibit different degrees of coordination levels (referred to as motion coordination levels) in different activities, different coordination levels can provide different visual cues for the occluded parts of the human body, which can more accurately judge the motion of the occluded parts compared to a single coordination level. For example, for a human body in a walking state, if the right arm is occluded, it is difficult to infer the specific posture of the arm based on the fingers and shoulders alone. On the basis of the fingers and shoulders, combined with other levels of body parts, such as the characteristic that the entire left leg is backward, it is possible to more accurately judge that the right arm is in a forward swing state. In this process, it is first necessary to infer the motion state of the human body through the overall general motion posture, and then through the visual cues at the limb level, that is, the left leg is used to infer the posture of the missing right arm, and finally the fingers and shoulders are used to infer that the right arm is in a forward swing state. It can be seen that multiple levels of visual cues can significantly improve the accuracy of inferring the motion information of the occluded part.

[0154] In the embodiment of this application, the human body is divided into four coordination levels: independent level (referred to as independent part level), interdependent level (referred to as interdependent level), fully coordinated level (referred to as motion pattern level), and whole body level (referred to as global posture level). Among them, the independent level involves individual body parts, such as hands and feet, a total of 24 parts, which is crucial for when specific parts are occluded; the interdependent level involves coordinated moving body parts, such as forearms and upper arms, a total of 11 parts, which helps to infer occluded parts based on visible parts; the fully coordinated level involves larger body parts, such as the entire arm, which can undergo complex movements, a total of 6 parts; the whole body level involves the entire body, integrating all levels to fully understand posture and movement.

[0155] The following describes the algorithm architecture of the embodiment of the present application.

[0156] See also Figure 8 , Figure 8Schematic diagram of the algorithm architecture for determining the movement of the human body provided by an embodiment of the present application. First, the image 8-1 (referred to as the input image) is input into the backbone network 8-2 for feature extraction. The feature blocks extracted by the backbone network 8-2 (referred to as image features) are passed through the two-dimensional convolution layer 8-3, thereby generating a 2D attention map (referred to as a two-dimensional feature map) at each coordination level, such as the 2D attention map of the independent level 8-41, the 2D attention map of the interdependent level 8-42, the 2D attention map of the fully coordinated level 8-43, and the 2D attention map of the whole body level 8-44. In addition, the feature blocks extracted by the backbone network 8-2 are also passed through the three-dimensional convolution layer 8-5 to extract 3D feature blocks (referred to as three-dimensional feature maps) at each level, and then the Hadamard product of the 2D attention map and the 3D feature block is calculated to extract feature sequences (referred to as visual features) of multiple coordination levels. Next, the whole-body level 8-44 marker sequence is used as a reference dependency (called the to-be-shared feature). This is done by expanding the whole-body level 8-44 marker sequence and connecting it to the feature channels of the remaining three coordination levels, constructing three sets of feature sequences 8-6 (called fused features) at different coordination levels. Subsequently, the three coordination levels' respective self-attention layers 8-7 learn partial dependencies from the feature sequence 8-6 (taking the independent level as an example, the attention weight matrix between the 24 parts is learned here). This enhances the scale of visual cues at each coordination level, giving each coordination level the ability to perceive holistic information, thereby obtaining optimized feature sequences 8-8 (called part-aware features) at each coordination level.

[0157] Continue to see Figure 8 After obtaining the optimized feature sequence 8-8, there are two processing methods. The first method is to splice the feature sequences 8-8 optimized at each coordination level, and then perform parameter regression on the splicing result (called splicing perception feature) through the multi-layer perceptron 8-9 to obtain action information, thereby driving the virtual object; the second method is to perform cross-level feature interaction on the sequence features 8-8 optimized at the three coordination levels, that is, the sequence features 8-8 optimized at the three coordination levels are respectively used as Q (called index perception feature), K (called query perception feature) and V (called weighted perception feature), and input them into the cross-attention layer 8-10 for interaction, and then perform parameter regression based on the output result of the cross-attention layer 8-10 to obtain action information, which is reflected in the virtual person in the image 8-11.

[0158] It should be noted that the first method does not require the operation of the cross-attention layer, and the computational complexity is small, but each coordination level will have the same weight; the second method requires at least three cross-attention layers, and the optimized sequence features 8-8 of the three coordination levels are used as the Q of the three cross-attention layers respectively. This method can enhance the feature sequences of different coordination levels and give greater weight to the layers that are more important for reasoning about the occluded parts.

[0159] In the second approach, the 8-8 feature sequences optimized at different coordination levels need to be aligned. In this case, the feature sequences optimized at the interdependence level and the fully coordinated level can be expanded to align with the 24 parts at the independent part level. For example, the arm (interdependence level) includes the shoulder, elbow, and wrist (all at the independent part level), so the arm feature sequence can be replicated to achieve alignment.

[0160] Next, the effect of the action determination of the embodiment of the present application is described.

[0161] Figure 9 This is a comparison chart of the effects of human motion information provided by the embodiment of this application. Figure 9 For the input image 9-1, there are two rectangular areas, namely rectangular area 9-11 and rectangular area 9-12. When the rectangular area 9-11 is blocked, the virtual human effect can be obtained based on the relevant technology. Figure 9-2 and error location indication Figure 9-3 Based on the method provided in the embodiment of the present application, a virtual human effect can be obtained. Figure 9-4 and error location indication Figure 9-5 Comparison of virtual human effects Figure 9-2 and virtual human effects Figure 9-4 It can be seen that the virtual human obtained by the method provided in the embodiment of the present application can better fit the original human body, as shown in the figure of the error part. Figure 9-3 and error location indication Figure 9-5 It can be seen that the virtual human obtained based on the embodiment of the present application has fewer and smaller parts (dashed elliptical areas) where errors occur. When there is occlusion in the rectangular area 9-12, the virtual human effect can be obtained based on the relevant technology. Figure 9-6 and error location indication Figure 9-7 Based on the method provided in the embodiment of the present application, a virtual human effect can be obtained. Figure 9-8 and error location indication Figure 9-9 Comparison of virtual human effects Figure 9-6 and virtual human effects Figure 9-8It can be seen that the virtual human obtained based on the related technology has a large difference from the original human body's movements, especially in the right arm area, while the virtual human obtained based on the method provided in the embodiment of the application has a smaller difference from the original human body's movements. Figure 9-7 and error location indication Figure 9-9 It can be seen that the error parts of the virtual human obtained by the method provided by the embodiment of the present application are smaller. It can be seen that the action information of the occluded part can be determined more accurately through the embodiment of the present application.

[0162] Figure 10 10-1 is a comparative diagram of the motion information obtained by the various different methods provided in the embodiments of the present application. Image 10-1 provides four human bodies in different actions: action 10-11 is answering a phone call, action 10-12 is sitting on a chair with one's head in one's arms, action 10-13 is jumping, and action 10-14 is bowling. Image 10-2 is a rendering of the motion information obtained based on related technology 1. As can be seen from image 10-2, the motions of the virtual person in related technology 1 cannot match the motions of the human body, especially actions 10-12 and 10-14, which have a large gap with the human body. Image 10-3 is a rendering of the motion information obtained based on related technology 2. As can be seen from image 10-3, the motions of the virtual person in related technology 2 still cannot match the motions of the human body, especially action 10-13, which has a large gap with the human body. Image 10-4 is a rendering of the motion information obtained based on the method provided in the embodiments of the present application. As can be seen from image 10-4, no matter which of the above four actions is performed, it can match the motions of the human body. It can be seen that through the embodiments of the present application, action information can be determined for the blocked part more accurately.

[0163] Below, a table is used to compare the errors of the motion information obtained by the method provided in the embodiment of the present application and the motion information of the related technology. The data set used is based on the 3D Poses in the Wild (3DPW) posture evaluation set, and the compared indicators are the mean joint position error (MPJPE), the MPJPE after the predicted joints and the true joints are rigidly aligned (P-rocrustes Aligned MPJEP, PA-MPJEP) and the mean vertex error (MPVE).

[0164] Table 1

[0165]

[0166]

[0167] As can be seen from Table 1, the MPJPE and MPVE of the embodiment of the present application are smaller than those of all related technologies, while the PA-MPJPE is smaller than that of most related technologies. It can be seen that the action information obtained by the embodiment of the present application is more accurate.

[0168] It is understandable that in the embodiments of the present application, when user information is involved, such as input images, action information and other related data, when the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0169] The following continues to describe the exemplary structure of the action determination device 255 provided in the embodiment of the present application as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the action determination device 255 of the memory 250 may include:

[0170] The image processing module 2551 is used to extract N visual features corresponding to N motion coordination levels of the target object from the input image; where N is an integer, N ≥ 2;

[0171] A feature generation module 2552 is configured to determine, based on the N visual features, M corresponding part-perception features for M target coordination levels in the N motion coordination levels; wherein the part-perception features are used to characterize the occluded parts and visible parts of the target object, as well as the degree of association between the visible parts and the occluded parts, where M is an integer, N>M≥1;

[0172] The motion regression module 2553 is used to regress the motion information of the occluded part and the motion information of the visible part based on the M part perception features.

[0173] In some embodiments of the present application, the feature generation module 2552 is also used to determine visual features other than the visual features corresponding to the M target coordination levels from the N visual features as features to be shared; fuse the features to be shared with the visual features of the M target coordination levels respectively to obtain M fused features; and based on the M fused features, determine the corresponding M part perception features for the M target coordination levels in the N motion coordination levels.

[0174] In some embodiments of the present application, the feature generation module 2552 is also used to perform the following processing through iteration i, where i is an integer and 1≤i≤M: determine the attention weight for each moving part under the i-th target coordination level; use the attention weight to adjust the sub-features corresponding to each moving part under the i-th target coordination level in the i-th fusion feature to obtain adjusted sub-features; use the adjusted sub-features of each moving part to generate the part perception feature of the i-th target coordination level.

[0175] In some embodiments of the present application, the motion regression module 2553 is also used to splice the M part perception features to obtain spliced ​​perception features; based on the spliced ​​perception features, regress the motion information of the occluded part and the motion information of the visible part.

[0176] In some embodiments of the present application, the motion regression module 2553 is also used to determine M alignment perception features based on the M part perception features; perform cross-level interaction on the M alignment perception features to obtain interactive perception features; and regress the motion information of the occluded part and the motion information of the visible part based on the interactive perception features.

[0177] In some embodiments of the present application, the motion regression module 2553 is also used to use the part perception feature with the highest feature dimension among the M part perception features as an extended baseline feature; expand the part perception features among the M part perception features except the extended baseline feature according to the dimension of the extended baseline feature to obtain corresponding extended perception features; and use the extended baseline feature and the extended perception feature to determine the M alignment perception features.

[0178] In some embodiments of the present application, the action regression module 2553 is also used to determine the index perception feature, the query perception feature and the perception feature to be weighted from the M aligned perception features; use the index perception feature and the query perception feature to calculate the weighted weight; and complete the cross-level interaction of the M aligned perception features by weighting the perception feature to be weighted with the weighted weight to obtain the interactive perception feature.

[0179] In some embodiments of the present application, the image processing module 2551 is also used to extract image features from the input image; perform the following processing through iteration j, where j is an integer and N≥j≥1: extract a two-dimensional feature map sequence and a three-dimensional feature map for the j-th motion coordination level from the image features; and generate the visual features at the j-th motion coordination level based on the three-dimensional feature map and the two-dimensional feature map sequence.

[0180] In some embodiments of the present application, the image processing module 2551 is further used to calculate the Hadamard product of the two-dimensional feature map sequence and the three-dimensional feature map, and use the Hadamard product to generate the visual features at the j-th motion coordination level.

[0181] In some embodiments of the present application, the motion determination device 255 further includes: an object control module 2554, which is used to perform real-time motion control on the virtual object based on the motion information of the blocked part and the motion information of the visible part.

[0182] An embodiment of the present application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the action determination method described above in the embodiment of the present application.

[0183] The embodiment of the present application provides a computer-readable storage medium in which computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will be caused to execute the action determination method provided in the embodiment of the present application, for example, Figure 3 The action determination method shown.

[0184] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0185] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0186] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0187] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0188] To sum up, through the embodiments of the present application, visual features can be first extracted at multiple different motion coordination levels, and the visual features of multiple different motion coordination levels can be combined to construct part perception features that can characterize the occluded parts, visible parts, and the degree of correlation between the occluded parts and the visible parts of the target object. Then, the motion information can be regressed, thereby realizing the visible parts based on different motion coordination levels and the degree of correlation with the occluded parts, improving the amount of information when inferring the motion information of the occluded parts, and ultimately improving the accuracy of motion determination when there is occlusion.

[0189] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A method for determining an action, characterized in that: The method comprises: Extract N visual features corresponding to N motion coordination levels of the target object from the input image; where N is an integer, N ≥ 2; Based on the N visual features, determining corresponding M part perception features for M target coordination levels in the N motion coordination levels; wherein the part perception features are used to characterize the occluded part and the visible part of the target object, and the degree of association between the visible part and the occluded part, where M is an integer, and N>M≥1; Based on the M part perception features, motion information of the occluded part and motion information of the visible part are obtained by regression.

2. The method according to claim 1, characterized in that Determining corresponding M part perception features for M target coordination levels in the N motion coordination levels based on the N visual features includes: Determining visual features other than the visual features corresponding to the M target coordination levels from the N visual features as features to be shared; Fusing the features to be shared with the M visual features of the target coordination level to obtain M fused features; Based on the M fusion features, corresponding M part perception features are determined for the M target coordination levels in the N motion coordination levels.

3. The method according to claim 2, characterized in that Determining corresponding M part perception features for M target coordination levels in the N motion coordination levels based on the M fusion features includes: The following process is performed by iteration i, where i is an integer and 1≤i≤M: Determining an attention weight for each movement part at the i-th target coordination level; Using the attention weight, adjusting the sub-features corresponding to the respective motion parts at the i-th target coordination level in the i-th fusion feature to obtain adjusted sub-features; The part perception feature of the i-th target coordination level is generated by using the adjusted sub-features of each of the motion parts.

4. The method according to any one of claims 1 to 3, characterized in that The step of regressing the motion information of the occluded part and the motion information of the visible part based on the M part perception features includes: Splicing the M part perception features to obtain a spliced ​​perception feature; Based on the splicing perception features, the motion information of the occluded part and the motion information of the visible part are regressed.

5. The method according to any one of claims 1 to 3, characterized in that The step of regressing the motion information of the occluded part and the motion information of the visible part based on the M part perception features includes: Determining M alignment perception features based on the M part perception features; Performing cross-level interaction on the M aligned perception features to obtain interactive perception features; Based on the interactive perception features, the motion information of the occluded part and the motion information of the visible part are regressed to obtain.

6. The method according to claim 5, characterized in that Determining M alignment perception features based on the M part perception features includes: The part-aware feature with the highest feature dimension among the M part-aware features is used as the extended benchmark feature; Expanding the part-perceived features other than the extended baseline features among the M part-perceived features according to the dimensions of the extended baseline features to obtain corresponding extended perception features; M alignment perception features are determined using the extended reference features and the extended perception features.

7. The method according to claim 5, characterized in that The cross-level interaction is performed on the M aligned perception features to obtain the interaction perception features, including: Determining, from the M aligned perceptual features, an index perceptual feature, a query perceptual feature, and a perceptual feature to be weighted; Calculating a weighted weight using the index-aware feature and the query-aware feature; By weighting the to-be-weighted perceptual features with the weighted weights, cross-level interaction of the M aligned perceptual features is completed to obtain the interactive perceptual features.

8. The method according to any one of claims 1 to 3, characterized in that The step of extracting N visual features corresponding to N motion coordination levels of the target object from the input image includes: Extracting image features from the input image; The following process is performed by iteration j, where j is an integer and N ≥ j ≥ 1: Extracting a two-dimensional feature map sequence and a three-dimensional feature map for the j-th motion coordination level from the image features; Based on the three-dimensional feature map and the two-dimensional feature map sequence, the visual features at the j-th motion coordination level are generated.

9. The method according to claim 8, characterized in that Generating the visual features at the j-th motion coordination level based on the three-dimensional feature map and the two-dimensional feature map sequence includes: The Hadamard product of the two-dimensional feature map sequence and the three-dimensional feature map is calculated, and the visual feature at the j-th motion coordination level is generated by using the Hadamard product.

10. The method according to any one of claims 1 to 3, characterized in that After regressing the motion information of the occluded part and the motion information of the visible part based on the M part perception features, the method further includes: The virtual object is controlled in real time based on the motion information of the blocked part and the motion information of the visible part.

11. The method according to any one of claims 1 to 3, characterized in that The N levels of movement coordination include: an independent part level, an interdependence level, a movement pattern level, and a global posture level; The independent part level is the level corresponding to the parts that constitute the target object; the interdependent level is the level corresponding to the parts that need to coordinate with each other when the target object moves; the motion pattern level is the level that can reflect the motion pattern of the target object; and the global posture level is the level corresponding to the entire target object. The M target coordination levels include: the independent part level, the interdependence level and the movement pattern level.

12. A motion determination device, characterized in that: The device comprises: An image processing module is used to extract N visual features corresponding to N motion coordination levels of the target object from the input image; where N is an integer, N ≥ 2; a feature generation module for determining, based on the N visual features, M corresponding part perception features for M target coordination levels in the N motion coordination levels; wherein the part perception features are used to characterize the occluded parts and visible parts of the target object, and the degree of association between the visible parts and the occluded parts, where M is an integer, N>M≥1; The motion regression module is used to regress the motion information of the occluded part and the motion information of the visible part based on the M part perception features.

13. An electronic device, characterized in that: The electronic device comprises: a memory for storing computer-executable instructions; A processor, configured to implement the method according to any one of claims 1 to 11 when executing the computer-executable instructions or computer program stored in the memory.

14. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images