Attitude estimation method and related device therefor
By seizing sub-images from object images and obtaining their position information, and combining with the target model for feature processing, the problem of single pose estimation factors in the prior art is solved, and the accurate description of object poses and high accuracy of pose parameters are achieved.
Patent Information
- Application Number
- PCT/CN2024/095720
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-30
- Filing Date
- 2024-05-28
- Publication Date
- 2024-12-05
AI Technical Summary
In the fields of virtual world, metaverse, VR, AR, etc., the existing pose estimation methods are considered in a single factor when pose estimation is based on object images, resulting in inaccurate pose parameters and inaccurate object pose estimation cannot be accurately completed.
By intercepting the sub-image containing its parts from the image of an object, obtaining the position information of the sub-image in the main image, inputting it to the target model for processing, considering the sub-image and its background and global position information, feature extraction, fusion and Encoding, generate pose parameters.
Accurate estimation of part of the object posture is achieved, the accuracy of posture parameters and the accuracy of posture prediction are improved, the current posture of the object can be accurately described, and the accuracy of the result is evaluated through confidence.
Smart Images

Figure CN2024095720_05122024_PF_FP_ABST
Abstract
Description
A posture estimation method and related equipment
[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on May 30, 2023, with application number 202310627327.0 and invention name “A posture estimation method and related equipment”, the entire contents of which are incorporated by reference into this application. Technical Field
[0002] The embodiments of the present application relate to the field of artificial intelligence (AI) technology, and in particular to a posture estimation method and related equipment. Background Art
[0003] With the rapid development of computer technology, motion capture tasks for objects, that is, tasks requiring object posture, are often required in fields such as virtual worlds, metaverses, virtual reality (VR), augmented reality (AR), and human-computer interaction (HCI). This task can be achieved through neural network models in AI technology.
[0004] In related technologies, when it is necessary to estimate the current posture of an object, an image of the object can be first collected and input into a neural network model, so that the image of the object can be processed in a series of ways through the neural network model to obtain the posture parameters of the object (these posture parameters are used to describe the current posture of the object). The posture parameters of the object can be applied to a variety of scenarios. For example, the posture parameters of the object can be used to deform the standard three-dimensional model of the object (the standard three-dimensional model corresponds to the standard posture of the object) to obtain the deformed three-dimensional model of the object (the deformed three-dimensional model corresponds to the current posture of the object), and display it to the user for viewing and use.
[0005] In the above-mentioned posture estimation process, the neural network model estimates the posture of the object based only on the image of the object, and only considers the content presented by the image itself. The factors considered are relatively simple, which will result in the final posture parameters of the object being incorrect, and the posture estimation of the object cannot be accurately completed.
[0006] Summary of the Invention
[0007] The embodiments of the present application provide a posture estimation method and related equipment. When performing posture estimation on an object, the factors considered are relatively comprehensive, and the posture of the object can be accurately estimated.
[0008] The present invention provides a method for posture estimation, which can be implemented by a target model. The method includes:
[0009] When it is necessary to estimate the current posture of the second object contained in the first object, the second image presenting the second object can be first intercepted from the first image presenting the first object. It can be understood that the second object is usually a part of the first object (for example, the first object is a person, and the second object is the hand of this person). Moreover, the first image not only presents the first object, but also presents the background of the first object. Similarly, the second image not only presents the second object, but also presents the background of the second object. After obtaining the first image and the second image, the position information of the second image in the first image can be obtained, that is, the global position information of the second image. The global position information can be used to accurately indicate whether the second object is specifically a part of the first object (for example, the second object is the left hand of this person).
[0010] After obtaining the second image and its position information within the first image, the second image and its position information within the first image can be input into the target model. The target model can then first perform a series of processing on the second image and its position information within the first image to obtain the position information of the second object within the second image. After obtaining the position information of the second object within the second image, the target model can continue to perform another series of processing on the position information of the second object within the second image to obtain the pose parameters of the second object and output the pose parameters of the second object. These parameters can be used to describe the current pose of the second object. At this point, the pose estimation of the second object has been successfully completed.
[0011] As can be seen from the above method, when pose estimation is required for a second object contained within a first object, a second image indicating the second object can be obtained from a first image indicating the first object, along with the position information of the second image within the first image. The second image and its position information within the first image can then be input into a target model. The target model can then process the second image and its position information within the first image to obtain the position information of the second object within the second image. Finally, the target model can further process the position information of the second object within the second image to obtain the pose parameters of the second object. This completes pose estimation for the second object. In this process, the target model estimates the pose of the second object based on the second image and its position information within the first image. This comprehensive approach takes into account not only the content presented in the second image (including the second object and its background) but also the specific position of the second image within the first image. The pose parameters obtained in this manner are generally correct and accurately describe the current pose of the second object, thus enabling accurate pose estimation for the second object.
[0012] In one possible implementation, processing the second image and its position information within the first image to obtain the position information of the second object within the second image includes: extracting features from the second image and its position information within the first image to obtain features of the second image and features of its position information within the first image; performing feature fusion on the features of the second image and its position information within the first image to obtain a feature fusion result; performing feature extraction on the feature fusion result to obtain a first feature of the position information of the second object within the second image; and classifying the first feature to obtain the position information of the second object within the second image. In the aforementioned implementation, after obtaining the second image and its position information within the first image, the target model may perform feature extraction on the second image and its position information within the first image, respectively, to obtain features of the second image and features of its position information within the first image. After obtaining the features of the two images and features of its position information within the first image, the target model may perform feature fusion on the features of the second image and its position information within the first image to obtain a feature fusion result. After obtaining the feature fusion result, the target model may further perform feature extraction on the feature fusion result to obtain the first feature of the position information of the second object within the second image. After obtaining the first feature of the position information of the second object in the second image, the target model can accurately convert the first feature of the position information of the second object in the second image into the position information of the second object in the second image through pixel-level or sub-pixel-level classification.
[0013] In one possible implementation, processing the position information of the second object in the second image to obtain the posture parameters of the second object includes: encoding the position information of the second object in the second image to obtain a second feature of the position information of the second object in the second image; and performing feature fusion on the feature fusion result and the second feature to obtain the posture parameters of the second object. In the aforementioned implementation, after obtaining the position information of the second object in the second image, the target model can encode the position information of the second object in the second image to obtain the second feature of the position information of the second object in the second image. After obtaining the second feature of the position information of the second object in the second image, the target model can perform feature fusion on the feature fusion result and the second feature, thereby accurately obtaining the posture parameters of the second object and outputting the posture parameters of the second object to the outside.
[0014] In one possible implementation, the method further includes: converting the pose parameters to obtain position information of the second object in a camera coordinate system, where the camera coordinate system is constructed based on the camera that captured the first image; mapping the position information of the second object in the camera coordinate system to obtain projected position information of the second object in the second image; and calculating the position information of the second object in the second image and the projected position information of the second object in the second image to obtain a confidence level. In the aforementioned implementation, after obtaining the pose parameters of the second object, the pose parameters of the second object can be used to deform a standard three-dimensional model of the second object, thereby obtaining a transformed deformed three-dimensional model of the second object. The standard three-dimensional model of the second object is a three-dimensional model preset in a camera coordinate system whose origin is the center point of the camera that captured the first image. Then, the position information of the second object in the camera coordinate system can be extracted from the deformed three-dimensional model of the second object. After obtaining the position information of the second object in the camera coordinate system, the position information of the second object in the camera coordinate system can be mapped to a two-dimensional plane to obtain projected position information of the second object in the second image. Then, the position information of the second object in the second image and the projected position information of the second object in the second image can be calculated to obtain the corresponding confidence level, which can be used to evaluate the accuracy of the entire posture estimation. It can be seen that based on the posture parameters input by the target model, it is also possible to obtain a confidence level used to describe whether the posture estimation is accurate, thereby effectively identifying correct or incorrect posture estimation (motion capture) results. If the image for posture estimation is a video frame in a certain video stream, the problem of jitter and flipping between video frames can be improved, thereby improving the smoothness of video motion capture.
[0015] In a possible implementation, the second image includes multiple image blocks, and the position information of the second image in the first image includes position information of the multiple image blocks in the first image.
[0016] In a possible implementation, the posture parameter of the second object includes a rotation angle of the second object and a shape parameter of the second object.
[0017] A second aspect of an embodiment of the present application provides a model training method, which includes: obtaining a second image from a first image, and obtaining position information of the second image in the first image, the first image is used to indicate a first object containing a second object, and the second image is used to indicate the second object; inputting the second image and the position information of the second image in the first image into a model to be trained to obtain posture parameters of the second object, and the model to be trained is used to: process the second image and the position information of the second image in the first image to obtain position information of the second object in the second image; process the position information of the second object in the second image to obtain posture parameters of the second object; and train the model to be trained based on the posture parameters to obtain a target model.
[0018] The target model trained using the above method has a pose estimation function. Specifically, when pose estimation is required for a second object contained within a first object, a second image indicating the second object can be obtained from a first image indicating the first object, and the position information of the second image within the first image can be obtained. The second image and its position information within the first image can then be input into the target model. The target model can then process the second image and its position information within the first image to obtain the position information of the second object within the second image. Finally, the target model can further process the position information of the second object within the second image to obtain the pose parameters of the second object. Thus, pose estimation for the second object is completed. In the above process, the target model can estimate the pose of the second object based on the second image and its position information within the first image. This takes into account not only the content presented in the second image (including the second object and its background) but also the specific position of the second image within the first image. This comprehensive consideration allows the pose parameters of the second object obtained in this manner to be generally correct and accurately describe the current pose of the second object, thus enabling accurate pose estimation for the second object.
[0019] In one possible implementation, the model to be trained is used to: perform feature extraction on the second image and the position information of the second image in the first image to obtain features of the second image and features of the position information of the second image in the first image; perform feature fusion on the features of the second image and features of the position information of the second image in the first image to obtain a feature fusion result; perform feature extraction on the feature fusion result to obtain a first feature of the position information of the second object in the second image; and classify the first feature to obtain the position information of the second object in the second image.
[0020] In one possible implementation, the model to be trained is used to: encode the position information of the second object in the second image to obtain a second feature of the position information of the second object in the second image; and perform feature fusion on the feature fusion result and the second feature to obtain the posture parameters of the second object.
[0021] In one possible implementation, training the model to be trained based on the posture parameters to obtain the target model includes: converting the posture parameters to obtain position information of the second object in the camera coordinate system, where the camera coordinate system is constructed based on the camera that captures the first image; mapping the position information of the second object in the camera coordinate system to obtain projection position information of the second object in the second image; training the model to be trained based on the position information of the second object in the second image and the projection position information of the second object in the second image to obtain the target model.
[0022] In a possible implementation, the second image includes multiple image blocks, and the position information of the second image in the first image includes position information of the multiple image blocks in the first image.
[0023] In a possible implementation, the posture parameter of the second object includes a rotation angle of the second object and a shape parameter of the second object.
[0024] A third aspect of an embodiment of the present application provides a posture estimation device, which includes a target model and comprises: an acquisition module for acquiring a second image from a first image and acquiring position information of the second image in the first image, wherein the first image is used to indicate a first object containing a second object, and the second image is used to indicate the second object; a first processing module for processing the second image and the position information of the second image in the first image to obtain the position information of the second object in the second image; and a second processing module for processing the position information of the second object in the second image to obtain posture parameters of the second object.
[0025] In one possible implementation, the first processing module is used to: perform feature extraction on the second image and the position information of the second image in the first image to obtain features of the second image and features of the position information of the second image in the first image; perform feature fusion on the features of the second image and features of the position information of the second image in the first image to obtain a feature fusion result; perform feature extraction on the feature fusion result to obtain a first feature of the position information of the second object in the second image; and classify the first feature to obtain the position information of the second object in the second image.
[0026] In one possible implementation, the second processing module is used to: encode the position information of the second object in the second image to obtain a second feature of the position information of the second object in the second image; and perform feature fusion on the feature fusion result and the second feature to obtain the posture parameters of the second object.
[0027] In one possible implementation, the device also includes: a conversion module, used to: convert the posture parameters to obtain the position information of the second object in the camera coordinate system, where the camera coordinate system is constructed based on the camera that captures the first image; map the position information of the second object in the camera coordinate system to obtain the projection position information of the second object in the second image; calculate the position information of the second object in the second image and the projection position information of the second object in the second image to obtain a confidence level.
[0028] In a possible implementation, the second image includes multiple image blocks, and the position information of the second image in the first image includes position information of the multiple image blocks in the first image.
[0029] In a possible implementation, the posture parameter of the second object includes a rotation angle of the second object and a shape parameter of the second object.
[0030] A fourth aspect of an embodiment of the present application provides a model training device, which includes: an acquisition module for acquiring a second image from a first image and obtaining position information of the second image in the first image, wherein the first image is used to indicate a first object containing a second object, and the second image is used to indicate the second object; a processing module for inputting the second image and the position information of the second image in the first image into a model to be trained to obtain posture parameters of the second object, and the model to be trained is used to: process the second image and the position information of the second image in the first image to obtain position information of the second object in the second image; process the position information of the second object in the second image to obtain posture parameters of the second object; and a training module for training the model to be trained based on the posture parameters to obtain a target model.
[0031] The target model trained by the above-mentioned device has a pose estimation function. Specifically, when pose estimation is required for a second object contained within a first object, a second image indicating the second object can be first obtained from a first image indicating the first object, and the position information of the second image within the first image can be obtained. The second image and its position information within the first image can then be input into the target model. The target model can then process the second image and its position information within the first image to obtain the position information of the second object within the second image. Finally, the target model can further process the position information of the second object within the second image to obtain the pose parameters of the second object. Thus, pose estimation for the second object is completed. In the aforementioned process, the target model can estimate the pose of the second object based on the second image and its position information within the first image. This takes into account not only the content presented in the second image itself (including the second object and its background) but also the specific position of the second image within the first image. This comprehensive consideration allows the pose parameters of the second object obtained in this manner to be generally correct and accurately describe the current pose of the second object, thus enabling accurate pose estimation for the second object.
[0032] In one possible implementation, the model to be trained is used to: perform feature extraction on the second image and the position information of the second image in the first image to obtain features of the second image and features of the position information of the second image in the first image; perform feature fusion on the features of the second image and features of the position information of the second image in the first image to obtain a feature fusion result; perform feature extraction on the feature fusion result to obtain a first feature of the position information of the second object in the second image; and classify the first feature to obtain the position information of the second object in the second image.
[0033] In one possible implementation, the model to be trained is used to: encode the position information of the second object in the second image to obtain a second feature of the position information of the second object in the second image; and perform feature fusion on the feature fusion result and the second feature to obtain the posture parameters of the second object.
[0034] In one possible implementation, the training module is used to: convert the posture parameters to obtain the position information of the second object in the camera coordinate system, where the camera coordinate system is constructed based on the camera that captures the first image; map the position information of the second object in the camera coordinate system to obtain the projection position information of the second object in the second image; and train the model to be trained based on the position information of the second object in the second image and the projection position information of the second object in the second image to obtain a target model.
[0035] In a possible implementation, the second image includes multiple image blocks, and the position information of the second image in the first image includes position information of the multiple image blocks in the first image.
[0036] In a possible implementation, the posture parameter of the second object includes a rotation angle of the second object and a shape parameter of the second object.
[0037] A fifth aspect of an embodiment of the present application provides a posture estimation device, which includes a memory and a processor; the memory stores code, and the processor is configured to execute the code. When the code is executed, the posture estimation device executes the method described in the first aspect or any possible implementation method of the first aspect.
[0038] A sixth aspect of an embodiment of the present application provides a model training device, which includes a memory and a processor; the memory stores code, and the processor is configured to execute the code. When the code is executed, the model training device executes the method described in the second aspect or any possible implementation method of the second aspect.
[0039] A seventh aspect of an embodiment of the present application provides a circuit system, which includes a processing circuit, and the processing circuit is configured to execute the method described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.
[0040] An eighth aspect of an embodiment of the present application provides a chip system, which includes a processor for calling a computer program or computer instructions stored in a memory so that the processor executes a method as described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.
[0041] In a possible implementation, the processor is coupled to the memory through an interface.
[0042] In one possible implementation, the chip system further includes a memory in which computer programs or computer instructions are stored.
[0043] A ninth aspect of an embodiment of the present application provides a computer storage medium storing a computer program, which, when executed by a computer, enables the computer to implement the method described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.
[0044] A tenth aspect of an embodiment of the present application provides a computer program product, which stores instructions that, when executed by a computer, enable the computer to implement the method described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.
[0045] In an embodiment of the present application, when it is necessary to perform pose estimation for a second object contained within a first object, a second image indicating the second object can be first obtained from a first image indicating the first object, and the position information of the second image within the first image can be obtained. The second image and its position information within the first image can then be input into a target model. The target model can then process the second image and its position information within the first image to obtain the position information of the second object within the second image. Finally, the target model can further process the position information of the second object within the second image to obtain the pose parameters of the second object. Thus, pose estimation for the second object is completed. In the aforementioned process, the target model can estimate the pose of the second object based on the second image and its position information within the first image. This not only takes into account the content presented in the second image itself (including the second object and its background), but also the specific position of the second image within the first image. This comprehensive consideration allows the pose parameters of the second object obtained in this manner to be generally correct and accurately describe the current pose of the second object, thus enabling accurate pose estimation for the second object. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 is a schematic diagram of the structure of the artificial intelligence main framework;
[0047] FIG2 a is a schematic structural diagram of a posture estimation system provided in an embodiment of the present application;
[0048] FIG2 b is another structural diagram of the posture estimation system provided in an embodiment of the present application;
[0049] FIG2c is a schematic diagram of a device related to posture estimation provided in an embodiment of the present application;
[0050] FIG3 is a schematic diagram of the architecture of the system 100 provided in an embodiment of the present application;
[0051] FIG4 is a schematic diagram of a structure of a target model provided in an embodiment of the present application;
[0052] FIG5 is a flow chart of a posture estimation method provided in an embodiment of the present application;
[0053] FIG6 is another schematic diagram of the structure of the target model provided in an embodiment of the present application;
[0054] FIG7( a ) is a schematic diagram of a comparison result provided in an embodiment of the present application;
[0055] FIG7( b ) is another schematic diagram of the comparison results provided in an embodiment of the present application;
[0056] FIG8 is a schematic diagram of an application example provided by an embodiment of the present application;
[0057] FIG9( a ) is another schematic diagram of the comparison results provided in an embodiment of the present application;
[0058] FIG9( b ) is another schematic diagram of the comparison results provided in an embodiment of the present application;
[0059] FIG10 is a flow chart of a model training method provided in an embodiment of the present application;
[0060] FIG11 is a schematic structural diagram of a posture estimation device provided in an embodiment of the present application;
[0061] FIG12 is a schematic structural diagram of a model training device provided in an embodiment of the present application;
[0062] FIG13 is a schematic structural diagram of an execution device provided in an embodiment of the present application;
[0063] FIG14 is a schematic diagram of the structure of a training device provided in an embodiment of the present application;
[0064] FIG15 is a schematic structural diagram of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION
[0065] The embodiments of the present application provide a posture estimation method and related equipment. When performing posture estimation on an object, the factors considered are relatively comprehensive, and the posture of the object can be accurately estimated.
[0066] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.
[0067] With the rapid development of computer technology, in the fields of virtual world, metaverse, VR, AR and HCI, there are usually motion capture tasks for objects (for example, estimating the movements of human hands), that is, object posture tasks (for example, estimating the posture of human hands) are required. This task can be achieved through the neural network model in AI technology.
[0068] In related technologies, when it is necessary to estimate the current posture of an object, an image of the object can be first collected and input into a neural network model, so that the image of the object can be processed in a series of ways by the neural network model to obtain the posture parameters of the object (these posture parameters are used to describe the current posture of the object). The posture parameters of the object can be applied to a variety of scenarios. For example, after obtaining the posture parameters of a person's hand (these posture parameters are used to describe the person's hand in a fist posture), it can be used to deform a standard three-dimensional model of a person's hand (the standard three-dimensional model of a person's hand is used to indicate that the person's hand is in a posture with five fingers open), thereby obtaining a deformed three-dimensional model of the person's hand (the deformed three-dimensional model of the person's hand is used to indicate that the person's hand is in a fist posture), and displaying it to the user for viewing and use.
[0069] In the above-mentioned posture estimation process, the neural network model estimates the posture of the object based only on the image of the object, and only considers the content presented in the image itself (including the object and the background in which the object is located, etc.). The factors considered are relatively simple, which will result in the final posture parameters of the object being incorrect, and the posture estimation of the object cannot be accurately completed.
[0070] Furthermore, during the training process of the aforementioned neural network model, it is often necessary to convert the object's pose parameters into the object's three-dimensional coordinates in the camera coordinate system, and then further map them into the object's two-dimensional coordinates in the image coordinate system. This allows the training model to be trained based on the two-dimensional coordinates to obtain the final neural network model. Because the factors considered by the trained model during the training process are relatively simple, it is impossible to obtain correct pose parameters. This also reduces the accuracy of the subsequently acquired three-dimensional and two-dimensional coordinates, resulting in even lower accuracy of the trained neural network model.
[0071] In order to solve the above problems, an embodiment of the present application provides a posture estimation method, which can be implemented in combination with artificial intelligence (AI) technology. AI technology is a technical discipline that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence. AI technology obtains the best results by perceiving the environment, acquiring knowledge and using knowledge. In other words, artificial intelligence technology is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Using artificial intelligence for data processing is a common application of artificial intelligence.
[0072] First, let's describe the overall workflow of an AI system. See Figure 1, which shows a schematic diagram of the AI framework. This framework will be explained from two perspectives: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). The "intelligent information chain" reflects the entire process from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. Throughout this process, data undergoes a condensed journey from "data-information-knowledge-wisdom." The "IT value chain," spanning the underlying infrastructure of human intelligence, information (provided and processed by technology), and the system's industrial ecosystem, reflects the value that AI brings to the information technology industry.
[0073] (1) Infrastructure
[0074] Infrastructure provides computing power for AI systems, enabling communication with the outside world and supporting this through a foundational platform. External communication occurs through sensors; computing power is provided by intelligent chips (CPUs, NPUs, GPUs, ASICs, FPGAs, and other hardware accelerators). The foundational platform includes a distributed computing framework and network-related platform guarantees and support, including cloud storage and computing, and interconnected networks. For example, sensors communicate with the outside world to acquire data, which is then fed into the intelligent chips within the distributed computing system provided by the foundational platform for computation.
[0075] (2) Data
[0076] Data above the infrastructure layer represents data sources for AI. This data includes graphics, images, voice, and text, as well as IoT data from traditional devices. This includes business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.
[0077] (3) Data processing
[0078] Data processing generally includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.
[0079] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.
[0080] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.
[0081] Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.
[0082] (4) General ability
[0083] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.
[0084] (5) Smart products and industry applications
[0085] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, smart cities, etc.
[0086] Next, several application scenarios of this application are introduced.
[0087] Figure 2a is a schematic diagram of the structure of a posture estimation system provided in an embodiment of the present application. The posture estimation system includes a user device and a data processing device. The user device includes an intelligent terminal such as a mobile phone, a personal computer, or an information processing center. The user device is the initiator of the posture estimation request, and as the initiator of the posture estimation request, the request is typically initiated by the user through the user device.
[0088] The data processing device can be a device or server with data processing capabilities, such as a cloud server, network server, application server, or management server. The data processing device receives pose estimation requests from smart terminals via an interactive interface and then performs pose estimation using machine learning, deep learning, search, reasoning, and decision-making methods through its memory and data processing processor. The memory in the data processing device is a general term that includes local storage and a database that stores historical data. The database can be located on the data processing device or on other network servers.
[0089] In the posture estimation system shown in FIG2a, a user device can receive a user's instruction. For example, the user device can obtain an image input / selected by the user to indicate (present) a certain object, and then initiate a request to a data processing device, causing the data processing device to perform posture estimation processing on the image from the user device indicating the object, thereby obtaining the posture parameters of the object. For example, the user device can obtain an image input by the user to indicate (present) a certain object, and then the user device can initiate a posture estimation request to the data processing device, causing the data processing device to perform a series of processing on the image indicating the object based on the posture estimation request, thereby obtaining the posture parameters of the object. These posture parameters are used to describe the current state of the object.
[0090] In FIG. 2 a , the data processing device may execute the posture estimation method according to the embodiment of the present application.
[0091] Figure 2b is another structural diagram of the posture estimation system provided in an embodiment of the present application. In Figure 2b, the user device directly serves as a data processing device. The user device can directly obtain input from the user and directly process it by the hardware of the user device itself. The specific process is similar to that of Figure 2a. Please refer to the above description and will not be repeated here.
[0092] In the posture estimation system shown in Figure 2b, the user device can receive user instructions. For example, the user device can obtain an image input by the user to indicate (present) a certain object, and then the user device can perform a series of processing on the image used to indicate the object to obtain the posture parameters of the object. These posture parameters are used to describe the current state of the object.
[0093] In FIG2 b , the user equipment itself can execute the posture estimation method of the embodiment of the present application.
[0094] FIG2c is a schematic diagram of a posture estimation related device provided in an embodiment of the present application.
[0095] The user device in the above Figures 2a and 2b can specifically be the local device 301 or the local device 302 in Figure 2c, and the data processing device in Figure 2a can specifically be the execution device 210 in Figure 2c, wherein the data storage system 250 can store the data to be processed of the execution device 210, and the data storage system 250 can be integrated on the execution device 210, or it can be set on the cloud or other network servers.
[0096] The processors in Figures 2a and 2b can perform data training / machine learning / deep learning through a neural network model or other models (for example, a model based on a support vector machine), and use the model finally trained or learned from the data to perform a posture estimation application on the image, thereby obtaining corresponding processing results.
[0097] Figure 3 is a schematic diagram of the system 100 architecture provided in an embodiment of the present application. In Figure 3, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with an external device. The user can input data to the I / O interface 112 through the client device 140. The input data can include: various tasks to be scheduled, callable resources and other parameters in the embodiment of the present application.
[0098] When the execution device 110 preprocesses the input data, or when the computing module 111 of the execution device 110 performs calculations and other related processing (such as implementing the functions of the neural network in this application), the execution device 110 can call the data, code, etc. in the data storage system 150 for the corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing in the data storage system 150.
[0099] Finally, the I / O interface 112 returns the processing result to the client device 140 so as to provide it to the user.
[0100] It is worth noting that the training device 120 can generate corresponding target models / rules based on different training data for different goals or tasks. The corresponding target models / rules can be used to achieve the above goals or complete the above tasks, thereby providing the user with the desired results. The training data can be stored in the database 130 and come from training samples collected by the data collection device 160.
[0101] In the scenario shown in FIG3 , the user can manually input data, which can be performed through the interface provided by I / O interface 112. Alternatively, client device 140 can automatically send input data to I / O interface 112. If user authorization is required for client device 140 to automatically send input data, the user can set the corresponding permissions in client device 140. The user can view the output of execution device 110 on client device 140, which can be presented in a display, sound, action, or other specific form. Client device 140 can also serve as a data acquisition terminal, collecting input data and output results from I / O interface 112 as new sample data and storing them in database 130. Of course, collection can also be performed without client device 140, with I / O interface 112 directly storing the input data and output results from I / O interface 112 as new sample data in database 130.
[0102] It is worth noting that FIG3 is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in FIG3, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 can also be placed in the execution device 110. As shown in FIG3, a neural network can be trained according to the training device 120.
[0103] The present application also provides a chip including a neural network processor (NPU). The chip can be provided in the execution device 110 shown in FIG3 to complete the computational work of the computation module 111. The chip can also be provided in the training device 120 shown in FIG3 to complete the training work of the training device 120 and output the target model / rule.
[0104] The neural network processor (NPU) is a coprocessor mounted on the host central processing unit (CPU), which assigns tasks to it. The core of the NPU is the arithmetic circuit, which is controlled by a controller to extract data from memory (weight memory or input memory) and perform calculations.
[0105] In some implementations, the arithmetic circuit includes multiple processing engines (PEs). In some implementations, the arithmetic circuit is a two-dimensional systolic array. The arithmetic circuit can also be a one-dimensional systolic array or other electronic circuit capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit is a general-purpose matrix processor.
[0106] For example, consider an input matrix A, a weight matrix B, and an output matrix C. The computational circuit retrieves the corresponding data for matrix B from the weight memory and caches it on each PE within the computational circuit. The computational circuit then performs a matrix operation on the matrix A data from the input memory and matrix B, storing the partial or final matrix results in the accumulator.
[0107] The vector computation unit can further process the output of the arithmetic circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector computation unit can be used for network calculations in non-convolutional / non-FC layers in neural networks, such as pooling, batch normalization, and local response normalization.
[0108] In some implementations, the vector computation unit can store the processed output vectors in a unified buffer. For example, the vector computation unit can apply a nonlinear function to the output of the computation circuit, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit generates normalized values, merged values, or both. In some implementations, the processed output vectors can be used as activation inputs to the computation circuit, such as for use in subsequent layers in a neural network.
[0109] The unified memory is used to store input data and output data.
[0110] The weight data is directly transferred from the external memory to the input memory and / or unified memory through the direct memory access controller (DMAC), the weight data in the external memory is stored in the weight memory, and the data in the unified memory is stored in the external memory.
[0111] The bus interface unit (BIU) is used to implement interaction between the main CPU, DMAC and instruction fetch memory through the bus.
[0112] An instruction fetch buffer connected to the controller, used to store instructions used by the controller;
[0113] The controller is used to call the instructions cached in the memory to control the working process of the computing accelerator.
[0114] Generally, unified memory, input memory, weight memory and instruction fetch memory are all on-chip memories, and external memory is memory outside the NPU, which can be double data rate synchronous dynamic random access memory (DDR SDRAM), high bandwidth memory (HBM) or other readable and writable memory.
[0115] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.
[0116] (1) Neural Network
[0117] A neural network can be composed of neural units. A neural unit can refer to an operation unit with xs and intercept 1 as input. The output of the operation unit can be:
[0118] Where s = 1, 2, ... n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.
[0119] The operation of each layer in a neural network can be described by the mathematical expression y = a(Wx + b). From a physical perspective, the operation of each layer in a neural network can be understood as transforming the input space (a set of input vectors) into the output space (i.e., from the row space to the column space of a matrix) through five operations. These operations include: 1. Dimensionality increase / decrease; 2. Scaling / reduction; 3. Rotation; 4. Translation; and 5. "Bending." Operations 1, 2, and 3 are performed by Wx, 4 by +b, and 5 by a(). The word "space" is used here because the objects being classified are not individual things, but rather a class of things, and space refers to the collection of all individuals within that class. W is the weight vector, each value in which represents the weight of a neuron in that layer of the neural network. This vector W determines the spatial transformation from input space to output space described above. That is, the weights W of each layer control how the space is transformed. The goal of training a neural network is to ultimately obtain the weight matrix for all layers of the trained neural network (a weight matrix formed by the vectors W of many layers). Therefore, the training process of a neural network is essentially about learning how to control spatial transformations, and more specifically, about learning the weight matrix.
[0120] Because we want the output of the neural network to be as close as possible to the value we really want to predict, we can compare the current network's predicted value with the desired target value, and then update the weight vector of each layer of the neural network based on the difference between the two (of course, there is usually an initialization process before the first update, which is to pre-configure the parameters for each layer in the neural network). For example, if the network's predicted value is too high, the weight vector is adjusted to make it predict a lower value, and this adjustment is continued until the neural network can predict the desired target value. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value." This is the loss function or objective function, which is an important equation used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the neural network becomes a process of minimizing this loss as much as possible.
[0121] (2) Backpropagation algorithm
[0122] Neural networks can use the back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during training, reducing the reconstruction error loss of the neural network model. Specifically, the forward propagation of the input signal to the output generates error loss. This error loss information is then backpropagated to update the parameters in the initial neural network model, thereby converging the error loss. The BP algorithm is a backward propagation movement dominated by error loss, aiming to obtain the optimal parameters of the neural network model, such as the weight matrix.
[0123] The method provided in this application is described below from the training side of the neural network and the application side of the neural network.
[0124] The model training method provided in the embodiment of the present application involves the processing of data sequences, and can be specifically applied to methods such as data training, machine learning, and deep learning. The training data (for example, the second image used to indicate the second object and the position information of the second image in the first image in the model training method provided in the embodiment of the present application, and the first image used to indicate the first object containing the second object) is symbolized and formalized for intelligent information modeling, extraction, preprocessing, training, etc., and finally a trained neural network is obtained (for example, the target model in the model training method provided in the embodiment of the present application); and the posture estimation method provided in the embodiment of the present application can use the above-mentioned trained neural network to input input data (for example, the second image used to indicate the second object and the position information of the second image in the first image in the posture estimation method provided in the embodiment of the present application, and the first image used to indicate the first object containing the second object) into the trained neural network to obtain output data (for example, the posture parameters of the second object in the posture estimation method provided in the embodiment of the present application). It should be noted that the model training method and posture estimation method provided in the embodiment of the present application are inventions based on the same concept, and can also be understood as two parts in a system, or two stages of an overall process: such as the model training stage and the model application stage.
[0125] The posture estimation method provided in the embodiment of the present application can be implemented by a target model. Figure 4 is a structural schematic diagram of the target model provided in the embodiment of the present application. As shown in Figure 4, the target model includes: a model-based main module, a sub-pixel classifier and a position encoder, wherein the first input end of the main module serves as the input end of the entire target model, the first output end of the main module is connected to the input end of the sub-pixel classifier, the output end of the sub-pixel classifier is connected to the input end of the position encoder, the output end of the position encoder is connected to the second input end of the main module, and the second output end of the main module serves as the output end of the entire target model. In order to understand the workflow of the target model, the workflow of the target model is introduced below in conjunction with Figure 5. Figure 5 is a flow chart of the posture estimation method provided in the embodiment of the present application. As shown in Figure 5, the method includes:
[0126] 501. Acquire a second image from a first image, and acquire position information of the second image in the first image, wherein the first image is used to indicate a first object containing a second object, and the second image is used to indicate the second object.
[0127] In this embodiment, when it is necessary to estimate the current posture of the second object contained in the first object, the second image used to indicate (present) the second object can be first intercepted from the first image used to indicate (present) the first object. It can be understood that the second object is usually a part of the first object. Moreover, the content presented by the first image includes not only the first object, but also the background of the first object. Similarly, the content presented by the second image includes not only the second object, but also the background of the second object. For example, the first object is a person and the second object is the person's hand. Then, the content presented by the first image includes the person and the environment in which the person is located, and the content presented by the second image includes the person's hand and the environment in which the hand is located.
[0128] After obtaining the first image and the second image, the position information of the second image in the first image can be obtained. The position information can be presented as the two-dimensional (2 dimension) coordinates of the second image in the first image coordinate system. The origin of the first image coordinate system is the center point of the first image.
[0129] Specifically, the second image can be divided into multiple image patches (patches), and the position information of the second image in the first image is the position information of these multiple image patches in the first image. For any one of these multiple image patches, the position information of the image patch in the first image patch can be presented as the 2D coordinates of the image patch in the coordinate system of the first image. The 2D coordinates of the image patch can include the center coordinates and vertex coordinates of the image patch, etc., without limitation herein.
[0130] 502. Process the second image and the position information of the second image in the first image to obtain the position information of the second object in the second image.
[0131] After obtaining the second image and the position information of the second image in the first image, the second image and the position information of the second image in the first image can be input into the target model. Therefore, the target model can first process the second image and the position information of the second image in the first image to obtain the position information of the second object in the second image. The position information can be presented as the 2D coordinates of each joint of the second object in the second image coordinate system, and the origin of the second image coordinate system is the center point of the second image.
[0132] Specifically, the target model can obtain the position information of the second object in the second image in the following manner:
[0133] (1) After obtaining the second image and the position information of the second image in the first image, the main module of the target model can perform feature extraction (for example, convolution operation, etc.) on the second image and the position information of the second image in the first image respectively, thereby obtaining the features of the second image and the features of the position information of the second image in the first image accordingly.
[0134] For example, as shown in Figure 6 (Figure 6 is another structural diagram of the target model provided by an embodiment of the present application), after intercepting a hand image I' from a human body image I, the 2D coordinates of each image block of I' in I can be obtained first. These 2D coordinates constitute the global position information (global position map) P. Then, I' and P can be input into the target model, which includes three modules: a model-based main module, a sub-pixel classifier, and a position encoder.
[0135] After obtaining I` and P, the main module of the target model can first perform multi-level convolution on I` and P respectively, so as to obtain the feature h1 of I` and the feature h2 of P.
[0136] (2) After obtaining the features of the two images and the features of the position information of the second image in the first image, the main module can perform feature fusion (for example, addition operation, etc.) on the features of the second image and the features of the position information of the second image in the first image to obtain a feature fusion result.
[0137] Still as in the above example, after obtaining h1 and h2, the main module can add h1 and h2 to obtain the fused feature h3.
[0138] (3) After obtaining the feature fusion result, the main module can perform feature extraction (for example, convolution operation, etc.) on the feature fusion result, thereby obtaining the first feature of the position information of the second object in the second image (which can also be called the initial feature of the position information of the second object in the second image), and send the first feature of the position information of the second object in the second image to the sub-pixel classifier.
[0139] As in the above example, after obtaining h3, the main module can further convolve h3 to obtain the initial feature h4 of the 2D coordinates of each joint point in the hand, and send h4 to the sub-pixel classifier.
[0140] (4) After obtaining the first feature of the position information of the second object in the second image, the sub-pixel classifier can classify the first feature of the position information of the second object in the second image, thereby obtaining the position information of the second object in the second image, and sending the position information of the second object in the second image to the position encoder.
[0141] It should be noted that, since the first feature of the position information of the second object in the second image includes the first feature of the 2D coordinates of each joint point of the second object in the second image coordinate system (hereinafter referred to as the first feature of the 2D coordinates of each joint point), the sub-pixel classifier can first obtain multiple categories, and these multiple categories correspond one-to-one to multiple 2D coordinates. Then, for the first feature of the 2D coordinate of any joint point, the sub-pixel classifier can estimate it, so as to obtain the probability that the first feature of the 2D coordinate of the joint point belongs to these multiple categories, and determine the category with the highest probability as the category to which the first feature of the 2D coordinate of the joint point belongs. Then, the 2D coordinate corresponding to this category is also the 2D coordinate of the joint point. For the remaining joint points, the sub-pixel classifier can also perform the same operation as that performed on the joint point, so the sub-pixel classifier can eventually obtain the 2D coordinates of all joint points of the second object in the second image coordinate system, that is, the position information of the second object in the second image.
[0142] It should also be noted that, assuming the resolution of the second image is 224×224 and the second object contains 21 joints. If classification is performed at the pixel level, the second image can be regarded as a second image coordinate system containing 224×224 2D coordinates. Therefore, the sub-pixel classifier can obtain a total of 224+224 categories (including 224 categories in the X-axis direction and 224 categories in the Y-axis direction), thereby completing the classification task for the 21 joints in 224+224 categories. If classification is performed at the sub-pixel level, each pixel is divided into 3 sub-pixels. The second image can be regarded as a second image coordinate system containing 672 (224×3)×672 2D coordinates. Therefore, the sub-pixel classifier can obtain a total of 672×672 categories, thereby completing the classification task for the 21 joints in 672×672 categories.
[0143] Still like the above example, after obtaining h4, the sub-pixel classifier can classify h4 to obtain the 2D coordinates of each joint point (joints 2D coordinates), and send the 2D coordinates of each joint point to the position encoder.
[0144] 503. Process the position information of the second object in the second image to obtain posture parameters of the second object.
[0145] After obtaining the position information of the second object in the second image, the target model can further process the position information of the second object in the second image to obtain the pose parameters of the second object and output the pose parameters of the second object. It is understood that the pose parameters of the second object typically include multiple parameters such as the rotation angle of the second object and the shape parameters of the second object. These parameters can be used to describe the current pose of the second object. At this point, the pose estimation of the second object is successfully completed.
[0146] Specifically, the target model can obtain the posture parameters of the second object in the following ways:
[0147] (1) After obtaining the position information of the second object in the second image, the position encoder can encode the position information of the second object in the second image, thereby obtaining a second feature of the position information of the second object in the second image (which can also be called the final feature of the position information of the second object in the second image), and send the second feature of the position information of the second object in the second image to the main module.
[0148] Still as in the above example, after obtaining the 2D coordinates of each joint point, the position encoder can encode the 2D coordinates of each joint point, thereby obtaining the final feature h5 of the 2D coordinates of each joint point, and sending h5 to the main module.
[0149] (2) After obtaining the second feature of the position information of the second object in the second image, the main module can perform feature fusion (for example, addition operation, etc.) on the feature fusion result and the second feature to obtain the posture parameters of the second object and output the posture parameters of the second object to the outside.
[0150] As in the above example, after obtaining h5, the main module can fuse h3 and h5 to obtain the rotation angle θ and shape parameter S of the hand, and output θ and S to the outside.
[0151] Furthermore, in order to evaluate the accuracy of the entire posture estimation, the confidence level can be obtained by:
[0152] (1) After obtaining the posture parameters of the second object, the standard three-dimensional model of the second object can be deformed using the posture parameters of the second object, thereby obtaining a deformed three-dimensional model of the second object for conversion. It should be noted that the standard three-dimensional model of the second object is a three-dimensional model preset in a camera coordinate system (the origin of the camera coordinate system is the center point of the camera that captured the first image), the standard three-dimensional model of the second object is used to indicate the standard posture of the second object, and the deformed three-dimensional model of the second object is used to indicate the current posture of the second object.
[0153] Then, the three-dimensional coordinates (3D) of all joint points of the second object in the camera coordinate system, that is, the position information of the second object in the camera coordinate system, can be extracted from the deformed three-dimensional model of the second object.
[0154] After obtaining the position information of the second object in the camera coordinate system, the position information of the second object in the camera coordinate system can be mapped to a two-dimensional plane to obtain the projection position information of the second object in the second image, that is, the 2D projection coordinates of all joint points of the second object in the second image coordinate system.
[0155] (2) Since the position information of the second object in the second image and the projection position information of the second object in the second image are known, the position information of the second object in the second image and the projection position information of the second object in the second image can be calculated (that is, the 2D coordinates of all joint points of the second object in the second image coordinate system and the 2D projection coordinates of all joint points of the second object in the second image coordinate system are calculated), thereby obtaining the corresponding confidence level, which can be used to evaluate the accuracy of the entire posture estimation.
[0156] As in the previous example, after obtaining θ and S, we can transform and map θ and S to obtain the 2D projected coordinates of each joint. Then, we can perform a series of calculations on the 2D coordinates of each joint and their 2D projected coordinates to obtain the final confidence level.
[0157] In addition, the target model provided in the embodiment of the present application (such as Ours in Table 1) can be compared with the models provided by some related technologies (such as Model-Based and Ours in Table 1). The comparison results are shown in Table 1:
[0158] Table 1
[0159] Based on Table 1, it can be seen that the target model provided by the embodiment of the present application has better performance in processing the same video.
[0160] Furthermore, the target model provided in the embodiment of the present application can be compared with the model provided in another related art. The comparison results are shown in Table 2, Figure 7(a) and Figure 7(b) (Figure 7(a) is a schematic diagram of the comparison results provided in the embodiment of the present application, and Figure 7(b) is another schematic diagram of the comparison results provided in the embodiment of the present application):
[0161] Table 2
[0162] In Table 2, the model of Related Art 1 does not utilize the position information of the input image, the model provided by Related Art 2 utilizes the local position information of the input image, and the target model provided by the embodiment of the present application utilizes the global position information of the input image. It can be seen that the target model provided by the embodiment of the present application achieves better performance in all indicators. For example, as shown in Figure 8 (Figure 8 is a schematic diagram of an application example provided by the embodiment of the present application), a left hand image and a right hand image are captured on a human body image. If the input of the model is the right hand image and the local position information of the right hand image (the image coordinate system is constructed using the right hand image itself, and the position information of the right hand image in the image coordinate system), since the hand postures presented by the left hand image and the right hand image are highly similar, the model may misjudge the left hand posture as the right hand posture. If the input of the model is the right hand image and the global position information of the right hand image (the image coordinate system is constructed using the human body image, and the position information of the right hand image in the image coordinate system), the model can fully recognize that the object to be estimated is the right hand, thereby correctly estimating the posture of the right hand.
[0163] Furthermore, the target model provided by the embodiment of the present application can be compared with the model provided by another related art. The comparison results are shown in Table 3, Figure 9(a) and Figure 9(b) (Figure 9(a) is another schematic diagram of the comparison results provided by the embodiment of the present application, and Figure 9(b) is another schematic diagram of the comparison results provided by the embodiment of the present application):
[0164] Table 3
[0165] In Table 3, compared with the related technology three, the target model provided by the embodiment of the present application utilizes a sub-pixel level classification method to obtain 2D coordinates, so the target model provided by the embodiment of the present application achieves better performance in all indicators.
[0166] In an embodiment of the present application, when it is necessary to perform pose estimation for a second object contained within a first object, a second image indicating the second object can be first obtained from a first image indicating the first object, and the position information of the second image within the first image can be obtained. The second image and its position information within the first image can then be input into a target model. The target model can then process the second image and its position information within the first image to obtain the position information of the second object within the second image. Finally, the target model can further process the position information of the second object within the second image to obtain the pose parameters of the second object. Thus, pose estimation for the second object is completed. In the aforementioned process, the target model can estimate the pose of the second object based on the second image and its position information within the first image. This not only takes into account the content presented in the second image itself (including the second object and its background), but also the specific position of the second image within the first image. This comprehensive consideration allows the pose parameters of the second object obtained in this manner to be generally correct and accurately describe the current pose of the second object, thus enabling accurate pose estimation for the second object.
[0167] Furthermore, in an embodiment of the present application, the position information of the second image in the first image, that is, the global position information of the second image, the target model can effectively identify that the second object is specifically a part of the first object based on the global position information of the second image (for example, the target model can effectively identify the left and right hands with the same posture), thereby accurately estimating the posture of the second object, which can further improve the accuracy of the target model.
[0168] Furthermore, during the training process of the target model in the embodiment of the present application, although it is necessary to convert the posture parameters of the second object into the three-dimensional coordinates of the second object in the camera coordinate system, and then further map them into the two-dimensional coordinates of the second object in the second image coordinate system, the target model is obtained by training the model to be trained based on the two-dimensional coordinates. Since the factors considered by the model to be trained during the training process are relatively comprehensive, correct posture parameters can be obtained, which will further improve the accuracy of the three-dimensional and two-dimensional coordinates obtained by the model to be trained, thereby making the trained target model more accurate.
[0169] Furthermore, the embodiments of the present application can also obtain a confidence level used to describe whether the posture estimation is accurate, thereby effectively identifying correct or incorrect posture estimation (motion capture) results. If the image for posture estimation is a video frame in a certain video stream, the problem of jitter and flipping between video frames can be improved, thereby improving the smoothness of video motion capture.
[0170] The above is a detailed description of the posture estimation method provided by the embodiment of the present application. The following is an introduction to the model training method provided by the embodiment of the present application. Figure 10 is a flow chart of the model training method provided by the embodiment of the present application. As shown in Figure 10, the method includes:
[0171] 1001. Acquire a second image from a first image, and acquire position information of the second image in the first image, wherein the first image is used to indicate a first object containing a second object, and the second image is used to indicate the second object.
[0172] In this embodiment, when it is necessary to train the model to be trained, a batch of training data can be obtained, which includes a first image indicating a first object, a second image indicating a second object, and position information of the second image in the first image (that is, the 2D coordinates of the second image in the first image coordinate system, the first image coordinate system being constructed based on the first image), the second image is captured from the first image, and the second object is a part of the first object. It should be noted that the real position information of the second object in the second image (that is, the real 2D coordinates of all joints of the second object in the second image coordinate system, the second image coordinate system being constructed based on the second image) is known.
[0173] In one possible implementation, the second image includes multiple image blocks, and the position information of the second image in the first image includes the position information of the multiple image blocks in the first image (that is, the 2D coordinates of the multiple images in the first image coordinate system).
[0174] For an introduction to step 1001 , please refer to the relevant description of step 501 in the embodiment shown in FIG5 , which will not be repeated here.
[0175] 1002. Input the second image and the position information of the second image in the first image into the model to be trained to obtain posture parameters of the second object. The model to be trained is used to: process the second image and the position information of the second image in the first image to obtain the position information of the second object in the second image; process the position information of the second object in the second image to obtain the posture parameters of the second object.
[0176] After obtaining the second image and its position information within the first image, the second image and its position information within the first image can be input into the model to be trained. The model to be trained can first process the second image and its position information within the first image to obtain the position information of the second object within the second image (i.e., the 2D coordinates of all joints of the second object in the coordinate system of the second image). The model to be trained can then further process the position information of the second object within the second image to obtain the pose parameters of the second object.
[0177] In one possible implementation, the model to be trained is used to: perform feature extraction on the second image and the position information of the second image in the first image to obtain features of the second image and features of the position information of the second image in the first image; perform feature fusion on the features of the second image and features of the position information of the second image in the first image to obtain a feature fusion result; perform feature extraction on the feature fusion result to obtain a first feature of the position information of the second object in the second image; and classify the first feature to obtain the position information of the second object in the second image.
[0178] In one possible implementation, the model to be trained is used to: encode the position information of the second object in the second image to obtain a second feature of the position information of the second object in the second image; and perform feature fusion on the feature fusion result and the second feature to obtain the posture parameters of the second object.
[0179] In a possible implementation, the posture parameter of the second object includes a rotation angle of the second object and a shape parameter of the second object.
[0180] For an introduction to step 1002 , please refer to the relevant description of steps 502 to 503 in the embodiment shown in FIG5 , which will not be repeated here.
[0181] 1003. Train the to-be-trained model based on the posture parameters to obtain a target model.
[0182] After obtaining the posture parameters of the second object, the posture parameters of the second object can be used to train the training model to obtain the target model in the embodiment shown in FIG5 .
[0183] Specifically, the target model can be trained in the following ways:
[0184] (1) After obtaining the posture parameters of the second object, the posture parameters of the second object can be used to deform the standard three-dimensional model of the second object, thereby obtaining a deformed three-dimensional model of the second object for conversion. It should be noted that the standard three-dimensional model of the second object is a three-dimensional model preset in a camera coordinate system (the origin of the camera coordinate system is the center point of the camera that captured the first image). Then, the 3D coordinates of all joint points of the second object in the camera coordinate system can be extracted from the deformed three-dimensional model of the second object, that is, the position information of the second object in the camera coordinate system.
[0185] After obtaining the position information of the second object in the camera coordinate system, the position information of the second object in the camera coordinate system can be mapped to a two-dimensional plane to obtain the projection position information of the second object in the second image, that is, the 2D projection coordinates of all joint points of the second object in the second image coordinate system.
[0186] (2) Since the position information of the second object in the second image, the projection position information of the second object in the second image, and the real position information of the second object in the second image are known, the position information of the second object in the second image and the projection position information of the second object in the second image can be calculated by a preset first loss function to obtain a first loss (that is, the aforementioned confidence), and the position information of the second object in the second image and the real position information of the second object in the second image can be calculated by a preset second loss function to obtain a second loss, and the projection position information of the second object in the second image and the real position information of the second object in the second image can be calculated by a preset third loss function to obtain a third loss. Then, the first loss, the second loss, and the third loss are superimposed to obtain the target loss.
[0187] It will be understood that target loss can be used to indicate the following: (1) the difference between the position information of the second object in the second image and the projected position information of the second object in the second image; (2) the difference between the position information of the second object in the second image and the actual position information of the second object in the second image; (3) the difference between the projected position information of the second object in the second image and the actual position information of the second object in the second image.
[0188] (3) After obtaining the target loss, the target loss can be used to update the parameters of the to-be-trained model, thereby obtaining the to-be-trained model with updated parameters, and the next batch of training data can be used to continue training the to-be-trained model with updated parameters until the model training conditions are met (for example, the target loss converges, etc.), thereby obtaining the target model.
[0189] The target model trained in the embodiment of the present application has a posture estimation function. Specifically, when it is necessary to perform posture estimation on the second object contained in the first object, a second image indicating the second object can be first obtained from the first image indicating the first object, and the position information of the second image in the first image can be obtained, and the second image and the position information of the second image in the first image can be input into the target model. Then, the target model can first process the second image and the position information of the second image in the first image to obtain the position information of the second object in the second image. Finally, the target model can further process the position information of the second object in the second image to obtain the posture parameters of the second object. At this point, the posture estimation for the second object is completed. In the above process, the target model can estimate the posture of the second object based on the second image and the position information of the second image in the first image, taking into account not only the content presented by the second image itself (including the second object and the background of the second object), but also the specific position of the second image in the first image. The factors considered are relatively comprehensive. The posture parameters of the second object obtained in this way are usually correct posture parameters that can accurately describe the current posture of the second object, so the posture estimation for the second object can be accurately completed.
[0190] The above is a detailed description of the posture estimation method and model training method provided in the embodiment of the present application. The posture estimation device and model training device provided in the embodiment of the present application will be introduced below. Figure 11 is a structural schematic diagram of the posture estimation device provided in the embodiment of the present application. As shown in Figure 11, the device includes:
[0191] The acquisition module 1101 is configured to acquire a second image from a first image and obtain position information of the second image within the first image, wherein the first image is configured to indicate a first object containing a second object, and the second image is configured to indicate the second object.
[0192] The first processing module 1102 is configured to process the second image and the position information of the second image in the first image to obtain the position information of the second object in the second image.
[0193] The second processing module 1103 is configured to process the position information of the second object in the second image to obtain posture parameters of the second object.
[0194] In an embodiment of the present application, when it is necessary to perform pose estimation for a second object contained within a first object, a second image indicating the second object can be first obtained from a first image indicating the first object, and the position information of the second image within the first image can be obtained. The second image and its position information within the first image can then be input into a target model. The target model can then process the second image and its position information within the first image to obtain the position information of the second object within the second image. Finally, the target model can further process the position information of the second object within the second image to obtain the pose parameters of the second object. Thus, pose estimation for the second object is completed. In the aforementioned process, the target model can estimate the pose of the second object based on the second image and its position information within the first image. This not only takes into account the content presented in the second image itself (including the second object and its background), but also the specific position of the second image within the first image. This comprehensive consideration allows the pose parameters of the second object obtained in this manner to be generally correct and accurately describe the current pose of the second object, thus enabling accurate pose estimation for the second object.
[0195] In one possible implementation, the first processing module 1102 is used to: perform feature extraction on the second image and the position information of the second image in the first image to obtain features of the second image and features of the position information of the second image in the first image; perform feature fusion on the features of the second image and features of the position information of the second image in the first image to obtain a feature fusion result; perform feature extraction on the feature fusion result to obtain a first feature of the position information of the second object in the second image; and classify the first feature to obtain the position information of the second object in the second image.
[0196] In one possible implementation, the second processing module 1103 is used to: encode the position information of the second object in the second image to obtain a second feature of the position information of the second object in the second image; and perform feature fusion on the feature fusion result and the second feature to obtain the posture parameters of the second object.
[0197] In one possible implementation, the device also includes: a conversion module, used to: convert the posture parameters to obtain the position information of the second object in the camera coordinate system, where the camera coordinate system is constructed based on the camera that captures the first image; map the position information of the second object in the camera coordinate system to obtain the projection position information of the second object in the second image; calculate the position information of the second object in the second image and the projection position information of the second object in the second image to obtain a confidence level.
[0198] In a possible implementation, the second image includes multiple image blocks, and the position information of the second image in the first image includes position information of the multiple image blocks in the first image.
[0199] In a possible implementation, the posture parameter of the second object includes a rotation angle of the second object and a shape parameter of the second object.
[0200] FIG12 is a schematic diagram of a structure of a model training device provided in an embodiment of the present application. As shown in FIG12 , the device includes:
[0201] An acquisition module 1201 is configured to acquire a second image from a first image and obtain position information of the second image in the first image, wherein the first image is used to indicate a first object containing a second object, and the second image is used to indicate the second object;
[0202] Processing module 1202 is configured to input the second image and the position information of the second image within the first image into the to-be-trained model to obtain posture parameters of the second object, wherein the to-be-trained model is configured to: process the second image and the position information of the second image within the first image to obtain the position information of the second object within the second image; and process the position information of the second object within the second image to obtain the posture parameters of the second object;
[0203] The training module 1203 is used to train the to-be-trained model based on the posture parameters to obtain a target model.
[0204] The target model trained in the embodiment of the present application has a posture estimation function. Specifically, when it is necessary to perform posture estimation on the second object contained in the first object, a second image indicating the second object can be first obtained from the first image indicating the first object, and the position information of the second image in the first image can be obtained, and the second image and the position information of the second image in the first image can be input into the target model. Then, the target model can first process the second image and the position information of the second image in the first image to obtain the position information of the second object in the second image. Finally, the target model can further process the position information of the second object in the second image to obtain the posture parameters of the second object. At this point, the posture estimation for the second object is completed. In the above process, the target model can estimate the posture of the second object based on the second image and the position information of the second image in the first image, taking into account not only the content presented by the second image itself (including the second object and the background of the second object), but also the specific position of the second image in the first image. The factors considered are relatively comprehensive. The posture parameters of the second object obtained in this way are usually correct posture parameters that can accurately describe the current posture of the second object, so the posture estimation for the second object can be accurately completed.
[0205] In one possible implementation, the model to be trained is used to: perform feature extraction on the second image and the position information of the second image in the first image to obtain features of the second image and features of the position information of the second image in the first image; perform feature fusion on the features of the second image and features of the position information of the second image in the first image to obtain a feature fusion result; perform feature extraction on the feature fusion result to obtain a first feature of the position information of the second object in the second image; and classify the first feature to obtain the position information of the second object in the second image.
[0206] In one possible implementation, the model to be trained is used to: encode the position information of the second object in the second image to obtain a second feature of the position information of the second object in the second image; and perform feature fusion on the feature fusion result and the second feature to obtain the posture parameters of the second object.
[0207] In one possible implementation, the training model 1203 is used to: convert the posture parameters to obtain the position information of the second object in the camera coordinate system, where the camera coordinate system is constructed based on the camera that captures the first image; map the position information of the second object in the camera coordinate system to obtain the projection position information of the second object in the second image; and train the model to be trained based on the position information of the second object in the second image and the projection position information of the second object in the second image to obtain a target model.
[0208] In a possible implementation, the second image includes multiple image blocks, and the position information of the second image in the first image includes position information of the multiple image blocks in the first image.
[0209] In a possible implementation, the posture parameter of the second object includes a rotation angle of the second object and a shape parameter of the second object.
[0210] It should be noted that the information interaction, execution process, etc. between the modules / units of the above-mentioned device are based on the same concept as the method embodiment of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the method embodiment shown above in the embodiment of the present application, and no further details will be given here.
[0211] The embodiment of the present application also relates to an execution device, and Figure 13 is a structural diagram of the execution device provided by the embodiment of the present application. As shown in Figure 13, the execution device 1300 can be specifically manifested as a mobile phone, a tablet, a laptop computer, an intelligent wearable device, a server, etc., which are not limited here. Among them, the execution device 1300 can be deployed with the posture estimation device described in the embodiment corresponding to Figure 11, for realizing the function of posture estimation in the embodiment corresponding to Figure 5. Specifically, the execution device 1300 includes: a receiver 1301, a transmitter 1302, a processor 1303 and a memory 1304 (wherein the number of processors 1303 in the execution device 1300 can be one or more, and Figure 13 takes one processor as an example), wherein the processor 1303 may include an application processor 13031 and a communication processor 13032. In some embodiments of the present application, the receiver 1301, the transmitter 1302, the processor 1303 and the memory 1304 may be connected via a bus or other means.
[0212] Memory 1304 may include read-only memory and random access memory, and provides instructions and data to processor 1303. A portion of memory 1304 may also include non-volatile random access memory (NVRAM). Memory 1304 stores processor and operation instructions, executable modules, or data structures, or subsets or extended sets thereof. The operation instructions may include various operation instructions for implementing various operations.
[0213] Processor 1303 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together via a bus system. In addition to a data bus, the bus system may also include a power bus, a control bus, and a status signal bus. However, for clarity, all bus systems are referred to as a bus system in the figure.
[0214] The methods disclosed in the above embodiments of the present application can be applied to or implemented by processor 1303. Processor 1303 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in processor 1303. The above processor 1303 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and can further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 1303 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present application can be directly implemented as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory 1304, and processor 1303 reads the information in memory 1304 and, in conjunction with its hardware, completes the steps of the above method.
[0215] Receiver 1301 can be used to receive input digital or character information and generate signal input related to executing device-related settings and function control. Transmitter 1302 can be used to output digital or character information through the first interface. Transmitter 1302 can also be used to send instructions to the disk pack through the first interface to modify data in the disk pack. Transmitter 1302 can also include a display device such as a display screen.
[0216] In an embodiment of the present application, in one case, the processor 1303 is configured to estimate the posture parameters of an object using the target model in the embodiment corresponding to FIG5 .
[0217] The present application also relates to a training device. FIG14 is a schematic diagram of the structure of the training device provided by the present application. As shown in FIG14 , the training device 1400 is implemented by one or more servers. The training device 1400 may vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1414 (e.g., one or more processors) and memory 1432, and one or more storage media 1430 (e.g., one or more mass storage devices) storing application programs 1442 or data 1444. The memory 1432 and storage medium 1430 may be temporary storage or permanent storage. The program stored in the storage medium 1430 may include one or more modules (not shown), each module may include a series of instruction operations in the training device. Furthermore, the CPU 1414 may be configured to communicate with the storage medium 1430 to execute the series of instruction operations in the storage medium 1430 on the training device 1400.
[0218] The training device 1400 may also include one or more power supplies 1426, one or more wired or wireless network interfaces 1450, one or more input and output interfaces 1458; or, one or more operating systems 1441, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.
[0219] Specifically, the training device can execute the model training method in the embodiment corresponding to Figure 10 to obtain the target model.
[0220] An embodiment of the present application also relates to a computer storage medium, which stores a program for signal processing. When the computer storage medium is run on a computer, it enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.
[0221] An embodiment of the present application also relates to a computer program product, which stores instructions that, when executed by a computer, enable the computer to execute the steps executed by the aforementioned execution device, or enable the computer to execute the steps executed by the aforementioned training device.
[0222] The execution device, training device or terminal device provided in the embodiments of the present application can specifically be a chip, and the chip includes: a processing unit and a communication unit, the processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, a pin or a circuit, etc. The processing unit can execute the computer execution instructions stored in the storage unit, so that the chip in the execution device executes the data processing method described in the above embodiment, or so that the chip in the training device executes the data processing method described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit can also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.
[0223] Specifically, see Figure 15 , which is a schematic diagram of the structure of a chip provided in an embodiment of the present application. The chip may be a neural network processor (NPU) 1500. NPU 1500 is mounted on a host CPU (host CPU) as a coprocessor, with tasks assigned by the host CPU. The core of the NPU is arithmetic circuit 1503, which is controlled by controller 1504 to extract matrix data from memory and perform multiplication operations.
[0224] In some implementations, arithmetic circuit 1503 includes multiple processing units (PEs). In some implementations, arithmetic circuit 1503 is a two-dimensional systolic array. Arithmetic circuit 1503 can also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, arithmetic circuit 1503 is a general-purpose matrix processor.
[0225] For example, assume there are input matrix A, weight matrix B, and output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from weight memory 1502 and caches it on each PE in the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from input memory 1501 and performs a matrix operation on matrix B. The partial or final matrix result is stored in accumulator 1508.
[0226] Unified memory 1506 is used to store input and output data. Weight data is directly transferred to weight memory 1502 through the Direct Memory Access Controller (DMAC) 1505. Input data is also transferred to unified memory 1506 through the DMAC.
[0227] BIU stands for Bus Interface Unit, i.e., bus interface unit 1513 , which is used for interaction between the AXI bus, DMAC, and instruction fetch buffer (IFB) 1509 .
[0228] The bus interface unit 1513 (BIU) is used for the instruction fetch memory 1509 to obtain instructions from the external memory, and is also used for the storage unit access controller 1505 to obtain the original data of the input matrix A or the weight matrix B from the external memory.
[0229] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1506 or transfer weight data to the weight memory 1502 or transfer input data to the input memory 1501.
[0230] The vector calculation unit 1507 includes multiple operation processing units. When necessary, it further processes the output of the operation circuit 1503, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization, pixel-level summation, and upsampling of the predicted label plane.
[0231] In some implementations, the vector calculation unit 1507 can store the processed output vector to the unified memory 1506. For example, the vector calculation unit 1507 can apply a linear function or a nonlinear function to the output of the operation circuit 1503, such as linear interpolation of the predicted label plane extracted by the convolution layer, or another example is a vector of accumulated values to generate an activation value. In some implementations, the vector calculation unit 1507 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1503, for example, for use in subsequent layers in a neural network.
[0232] An instruction fetch buffer 1509 connected to the controller 1504 is used to store instructions used by the controller 1504;
[0233] Unified memory 1506, input memory 1501, weight memory 1502, and instruction fetch memory 1509 are all on-chip memories. External memories are private to the NPU hardware architecture.
[0234] The processor mentioned in any of the above places can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above program.
[0235] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.
[0236] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.
[0237] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.
[0238] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a training device or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).
Claims
1. A posture estimation method, characterized in that: The method is implemented by a target model, and the method comprises: Acquire a second image from a first image, and acquire position information of the second image in the first image, wherein the first image is used to indicate a first object containing a second object, and the second image is used to indicate the second object; Processing the second image and the position information of the second image in the first image to obtain the position information of the second object in the second image; The position information of the second object in the second image is processed to obtain a posture parameter of the second object.
2. The method according to claim 1, characterized in that The processing of the second image and the position information of the second image in the first image to obtain the position information of the second object in the second image includes: Performing feature extraction on the second image and position information of the second image in the first image to obtain features of the second image and features of the position information of the second image in the first image; Performing feature fusion on features of the second image and features of position information of the second image in the first image to obtain a feature fusion result; Performing feature extraction on the feature fusion result to obtain a first feature of the position information of the second object in the second image; The first feature is classified to obtain position information of the second object in the second image.
3. The method according to claim 2, characterized in that The step of processing the position information of the second object in the second image to obtain the posture parameter of the second object includes: encoding the position information of the second object in the second image to obtain a second feature of the position information of the second object in the second image; The feature fusion result and the second feature are subjected to feature fusion to obtain a posture parameter of the second object.
4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Converting the posture parameters to obtain position information of the second object in a camera coordinate system, where the camera coordinate system is constructed based on a camera that captures the first image; Mapping the position information of the second object in the camera coordinate system to obtain projection position information of the second object in the second image; The position information of the second object in the second image and the projection position information of the second object in the second image are calculated to obtain a confidence level.
5. The method according to any one of claims 1 to 4, characterized in that: The second image includes a plurality of image blocks, and the position information of the second image in the first image includes the position information of the plurality of image blocks in the first image.
6. The method according to any one of claims 1 to 5, characterized in that: The posture parameters of the second object include a rotation angle of the second object and a shape parameter of the second object.
7. A model training method, characterized in that: The method comprises: Acquire a second image from a first image, and acquire position information of the second image in the first image, wherein the first image is used to indicate a first object containing a second object, and the second image is used to indicate the second object; Inputting the second image and the position information of the second image in the first image into the model to be trained to obtain the posture parameters of the second object, wherein the model to be trained is used to: process the second image and the position information of the second image in the first image to obtain the position information of the second object in the second image; process the position information of the second object in the second image to obtain the posture parameters of the second object; The model to be trained is trained based on the posture parameters to obtain a target model.
8. The method according to claim 7, characterized in that The model to be trained is used for: Extracting features of the second image and position information of the second image in the first image to obtain features of the second image and features of the position information of the second image in the first image; Performing feature fusion on features of the second image and features of position information of the second image in the first image to obtain a feature fusion result; Performing feature extraction on the feature fusion result to obtain a first feature of the position information of the second object in the second image; The first feature is classified to obtain position information of the second object in the second image.
9. The method according to claim 8, characterized in that The model to be trained is used for: encoding the position information of the second object in the second image to obtain a second feature of the position information of the second object in the second image; The feature fusion result and the second feature are subjected to feature fusion to obtain a posture parameter of the second object.
10. The method according to any one of claims 7 to 9, characterized in that: The training of the to-be-trained model based on the posture parameters to obtain the target model comprises: Converting the posture parameters to obtain position information of the second object in a camera coordinate system, where the camera coordinate system is constructed based on a camera that captures the first image; Mapping the position information of the second object in the camera coordinate system to obtain projection position information of the second object in the second image; Based on the position information of the second object in the second image and the projection position information of the second object in the second image, the model to be trained is trained to obtain a target model.
11. The method according to any one of claims 7 to 10, characterized in that: The second image includes a plurality of image blocks, and the position information of the second image in the first image includes the position information of the plurality of image blocks in the first image.
12. The method according to any one of claims 7 to 11, characterized in that: The posture parameters of the second object include a rotation angle of the second object and a shape parameter of the second object.
13. A posture estimation device, characterized in that: The device comprises a target model, and the device comprises: an acquisition module, configured to acquire a second image from a first image and acquire position information of the second image in the first image, wherein the first image is used to indicate a first object including a second object, and the second image is used to indicate the second object; A first processing module, configured to process the second image and the position information of the second image in the first image to obtain the position information of the second object in the second image; The second processing module is used to process the position information of the second object in the second image to obtain the posture parameters of the second object.
14. A model training device, characterized in that: The device comprises: an acquisition module, configured to acquire a second image from a first image and acquire position information of the second image in the first image, wherein the first image is used to indicate a first object including a second object, and the second image is used to indicate the second object; a processing module, configured to input the second image and the position information of the second image in the first image into a model to be trained to obtain a posture parameter of the second object, wherein the model to be trained is configured to: process the second image and the position information of the second image in the first image to obtain the position information of the second object in the second image; process the position information of the second object in the second image to obtain the posture parameter of the second object; The training module is used to train the to-be-trained model based on the posture parameters to obtain a target model.
15. A posture estimation device, characterized in that: The device includes a memory and a processor; the memory stores codes, and the processor is configured to execute the codes. When the codes are executed, the posture estimation device performs the method according to any one of claims 1 to 12.
16. A computer storage medium, characterized in that: The computer storage medium stores one or more instructions, which, when executed by one or more computers, enable the one or more computers to implement the method of any one of claims 1 to 12.
17. A computer program product, characterized in that The computer program product stores instructions, which, when executed by a computer, enable the computer to implement the method according to any one of claims 1 to 12.
Citation Information
Patent Citations
Human body posture estimation method and device, electronic equipment and storage medium
CN114140831A
Posture recognition method and related equipment thereof
CN114241597A
Attitude detection method and device, electronic equipment and storage medium
CN114241604A
Attitude estimation method and related equipment thereof
CN116912314A
Apparatus and Method for Determining Sharpness
US20210118121A1
