Attitude estimation method, device, equipment, storage medium and program product

Abstract

The present application discloses a posture estimation method, device, equipment, storage medium and program product, which relates to the field of machine learning. The method includes: obtaining a color image and a depth image acquired by a depth camera; acquiring point cloud data corresponding to the depth image; performing cross-feature extraction on the color image and the point cloud data based on a cross-modal cross-attention mechanism, outputting a color feature representation corresponding to the color image and a point cloud feature representation corresponding to the point cloud data, and obtaining an image feature representation; performing posture estimation and prediction on the target entity acquired in the color image based on the image feature representation, and obtaining posture data corresponding to the target entity. The color feature representation and the point cloud feature representation are integrated into the extracted image feature representation, so that the entity posture is identified according to the image feature representation. The interaction of the two modal information integrates the entity features in the process of feature extraction, thereby improving the accuracy of identifying the entity posture from the image.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of machine learning, and in particular to a posture estimation method, apparatus, device, storage medium and program product. Background Art

[0002] In the field of robotics, it is usually necessary to use the robot's mechanical arm to grab and place objects in the real world. For example, the robot's mechanical arm grabs a cup placed on the table and transfers the cup to a cabinet.

[0003] In the related technology, in the process of realizing robotic arm grasping, the robot's visual sensing module is first used to collect images to determine the environmental image within the robot's implementation range, and the environmental image is input into a pre-trained neural network model to output a candidate posture estimation result, thereby combining the point cloud data corresponding to the environmental image and the object model data to obtain the posture data of the object in the environment.

[0004] However, in the above scheme, the accuracy of the candidate pose estimation results obtained by environmental image prediction is low, and the point cloud data is easily affected by neighboring objects around the object, resulting in low accuracy of the final pose data. Summary of the invention

[0005] The embodiments of the present application provide a posture estimation method, device, equipment, storage medium and program product, which can improve the estimation accuracy of the posture of an entity in an image. The technical solution is as follows.

[0006] On the one hand, a posture estimation method is provided, the method comprising:

[0007] Acquire a color image and a depth image acquired by a depth camera, wherein the color image and the depth image are images acquired synchronously by the depth camera;

[0008] Acquire point cloud data corresponding to the depth image, the point cloud data being used to indicate the three-dimensional spatial distribution of pixel points in the depth image;

[0009] Performing cross-feature extraction on the color image and the point cloud data based on a cross-modal cross-attention mechanism, and outputting a color feature representation corresponding to the color image and a point cloud feature representation corresponding to the point cloud data;

[0010] splicing the color feature representation and the point cloud feature representation to obtain an image feature representation;

[0011] Based on the image feature representation, a posture estimation and prediction is performed on the target entity captured in the color image to obtain posture data corresponding to the target entity, where the posture data is used to represent the position and posture of the target entity relative to the depth camera.

[0012] On the other hand, a posture estimation device is provided, the device comprising:

[0013] An acquisition module, used to acquire a color image and a depth image acquired by a depth camera, wherein the color image and the depth image are images acquired synchronously by the depth camera;

[0014] The acquisition module is further used to acquire point cloud data corresponding to the depth image, wherein the point cloud data is used to indicate the three-dimensional spatial distribution of pixel points in the depth image;

[0015] A feature extraction module, configured to perform cross-feature extraction on the color image and the point cloud data based on a cross-modal cross-attention mechanism, output a color feature representation corresponding to the color image and a point cloud feature representation corresponding to the point cloud data; and concatenate the color feature representation and the point cloud feature representation to obtain an image feature representation;

[0016] A prediction module is used to perform posture estimation and prediction on the target entity captured in the color image based on the image feature representation, and obtain posture data corresponding to the target entity, wherein the posture data is used to represent the position and posture of the target entity relative to the depth camera.

[0017] On the other hand, a computer device is provided, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a posture estimation method as described in any of the above-mentioned embodiments of the present application.

[0018] On the other hand, a computer-readable storage medium is provided, in which at least one instruction, at least one program, a code set or an instruction set is stored, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement a posture estimation method as described in any of the above-mentioned embodiments of the present application.

[0019] On the other hand, a computer program product is provided, the computer program product comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the posture estimation method described in any of the above embodiments.

[0020] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0021] In the process of object posture recognition, the process of extracting color feature representation and the process of extracting point cloud feature representation are carried out simultaneously, and there is an interactive relationship between the extraction of the two feature representations, that is, the color feature representation and the point cloud feature representation are cross-extracted through the cross-modal cross-attention mechanism, and the color feature representation and the point cloud feature representation are integrated into the extracted image feature representation, so that the entity posture is recognized according to the image feature representation. The interaction of the two modal information will cause the entity features to merge and influence each other during the feature extraction process, thereby improving the accuracy of identifying the entity posture from the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 is a schematic diagram of the overall process of a posture estimation method provided by an exemplary embodiment of the present application;

[0024] Figure 2 It is a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application;

[0025] Figure 3 is a flow chart of a posture estimation method provided by an exemplary embodiment of the present application;

[0026] Figure 4 is based on Figure 3 A schematic diagram of a feature extraction process provided by the illustrated embodiment;

[0027] Figure 5 is a flow chart of a posture estimation method provided by another exemplary embodiment of the present application;

[0028] Figure 6 is based on Figure 5 A schematic diagram of a feature fusion process provided by the illustrated embodiment;

[0029] Figure 7 is based on Figure 5 A schematic diagram of another feature fusion process provided by the illustrated embodiment;

[0030] Figure 8 is based on Figure 5 A schematic diagram of a feature stitching process provided by the illustrated embodiment;

[0031] Fig. 9 is a flow chart of a posture estimation method provided by another exemplary embodiment of the present application;

[0032] Fig.10 is a structural block diagram of a posture estimation device provided by an exemplary embodiment of the present application;

[0033] Fig.11 is a structural block diagram of a posture estimation device provided by another exemplary embodiment of the present application;

[0034] Fig.12 It is a structural block diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0036] First, a brief introduction is given to the terms involved in the embodiments of the present application.

[0037] Artificial Intelligence (AI): It is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0038] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, extraction technology for large feature representation, operation / interaction systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0039] Machine Learning (ML): It is a multi-disciplinary interdisciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0040] Computer Vision (CV): It is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing so that the computer processing becomes an image that is more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0041] In the embodiments of the present application, a method for posture estimation based on an image is mainly protected, that is, a method for posture estimation of an entity in an image.

[0042] Figure 1 FIG. 1 is a schematic diagram of the overall process of a posture estimation method provided by an exemplary embodiment of the present application. Figure 1 As shown, the process mainly includes a color image 110 and a depth image 120 .

[0043] Among them, the color image 110 and the depth image 120 are images acquired by the depth camera, and the color image 110 and the depth image 120 are image pairs acquired by the depth camera at the same angle at the same time. Among them, the color image 110 is used to represent the pixel values ​​corresponding to the acquired image. Schematically, the color image 110 is implemented as a red, green, and blue channel (RedGreenBlue, RGB) image. The depth image 120 is also called a distance image, which refers to an image that uses the distance (depth) from the image collector (i.e., the depth camera mentioned above) to each point in the scene as a pixel value, which directly reflects the geometric shape of the visible surface of the entity. The depth image 120 can be converted into point cloud data for feature extraction and prediction.

[0044] In the present application embodiment, Figure 1 As shown, a bidirectional fusion network is used to predict the key points of the object on the color image 110 and the depth image 120 and finally obtain the object pose. The color image 110 is processed by the first depth network 111; the depth image is processed by the second depth network 121 after generating point cloud data through the camera intrinsic parameters of the depth camera. The two modal information is interacted at each layer of the encoder and decoder of the first depth network 111 and the second depth network 121, and finally the point-by-point features of the color image and the point cloud data are spliced ​​together to perform point-by-point key point offset estimation and semantic segmentation estimation, and finally the pose estimation result of the entity in the color image 110 is obtained.

[0045] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user alone or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the color images, depth images, camera internal parameters, etc. involved in this application are all obtained with full authorization.

[0046] Secondly, the implementation environment involved in the embodiments of the present application is described. Optionally, the embodiments of the present application can be implemented by a terminal alone, by a server alone, or by a terminal and a server together. In this embodiment, the embodiment of the present application is described by taking the implementation of the terminal and the server together as an example.

[0047] For illustration, please refer to Figure 2 The implementation environment involves a terminal 210 and a server 220 , and the terminal 210 and the server 220 are connected via a communication network 230 .

[0048] In some embodiments, the terminal 210 is used to send image data to the server 220. In some embodiments, an application with an image acquisition prediction function (such as an entity posture prediction function, etc.) is installed in the terminal 210. Schematically, an image entity recognition application is installed in the terminal 210. Optionally, a search engine program, a travel application, a life-assisting application, an instant messaging application, a video program, a game program, etc. are installed in the terminal 210, and this embodiment of the application is not limited to this.

[0049] Optionally, the terminal 210 is configured with a depth camera, which is used to collect color images and depth images, or the depth camera is used to collect color images containing depth information, such as red green blue depth channel (Red Green Blue Depth, RGB-D) images. After collecting the image data, the terminal 210 sends the image data to the server 220 for entity posture estimation.

[0050] The server 220 is used to perform posture prediction based on the color image and depth image collected by the terminal 210. The server 220 performs multimodal cross-feature extraction on the point cloud data corresponding to the color image and the depth image, thereby performing posture estimation on the entity in the image based on the extracted image feature representation.

[0051] The above-mentioned terminal can be a terminal device in various forms such as an intelligent robot, a robotic arm, an augmented reality (AR) device, a mobile phone, a tablet computer, a desktop computer, a portable laptop computer, a smart TV, a vehicle-mounted terminal, a smart home device, etc., and the embodiments of the present application are not limited to this.

[0052] It is worth noting that the above-mentioned servers can be independent physical servers, or they can be server clusters or distributed systems composed of multiple physical servers. They can also be cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0053] Among them, cloud technology refers to a hosting technology that unifies hardware, software, network and other resources in a wide area network or local area network to realize data computing, storage, processing and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which is used on demand and flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, in the future, each item may have its own identification mark, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately. All kinds of industry data require strong system backing support, which can only be achieved through cloud computing.

[0054] In some embodiments, the above server can also be implemented as a node in a blockchain system.

[0055] In combination with the above-mentioned noun introduction and application scenarios, the posture estimation method provided by the present application is described. The method can be executed by a server or a terminal, or can be executed by a server and a terminal together. In the embodiment of the present application, the method is executed by a server as an example. Figure 3 As shown, the method includes the following steps.

[0056] Step 301: Acquire a color image and a depth image captured by a depth camera.

[0057] Among them, the color image and the depth image are images acquired synchronously by the depth camera.

[0058] Optionally, the color image and the depth image are images acquired simultaneously by a depth camera at the same angle.

[0059] Among them, the color image is used to indicate the color distribution of pixels collected by the depth camera from the environment. Schematically, the color image is an RGB image, which is used to represent the RGB pixel values ​​corresponding to the pixels collected from the environment; the depth image is used to indicate the distance distribution of pixels collected by the depth camera from the environment, that is, the depth image is used to indicate the distance between each pixel for the depth camera.

[0060] In some embodiments, the color image and the depth image are implemented as two independent images; or, the pixel values ​​in the color image and the depth distance values ​​in the depth image are implemented in the pixel data of one image, which is not limited in the embodiments of the present application.

[0061] The above-mentioned depth camera is implemented as an RGBD camera, which can simultaneously capture the RGB image and depth image corresponding to the environment. Optionally, in the depth image, the grayscale value is used to represent the distance between a certain point in the environment and the depth camera. Schematically, the higher the grayscale value, the closer the distance between the point in the environment and the depth camera; conversely, the lower the grayscale value, the farther the distance between the point in the environment and the depth camera.

[0062] Step 302: Obtain point cloud data corresponding to the depth image.

[0063] Optionally, point cloud data is used to indicate the three-dimensional spatial distribution of pixel points in the depth image. Point cloud data refers to a set of vectors in a three-dimensional coordinate system. For example, in the process of capturing images by a depth camera, the coordinates of each captured point in the three-dimensional coordinate system are determined based on the distance between each captured point and the current depth camera, thereby determining the three-dimensional spatial distribution between each captured point based on the coordinates of each captured point in the three-dimensional coordinate system.

[0064] Optionally, coordinate transformation is performed on the depth image to obtain point cloud data, wherein each pixel in the depth image represents the distance of the object at the specific pixel from the camera plane in the field of view of the depth sensor.

[0065] In some embodiments, the camera intrinsic parameters corresponding to the depth camera are obtained, and the camera intrinsic parameters are acquisition parameters of the depth camera when performing image acquisition; the pixel point data in the depth image is converted based on the camera intrinsic parameters to generate point cloud data. Optionally, the camera intrinsic parameters include the camera perspective angle, resolution, etc., and the coordinate data of the depth image is converted in combination with the camera intrinsic parameters to obtain point cloud data.

[0066] Step 303, perform cross-feature extraction on the color image and point cloud data based on the cross-modal cross-attention mechanism, and output the color feature representation corresponding to the color image and the point cloud feature representation corresponding to the point cloud data.

[0067] Among them, the cross-modal cross-attention mechanism refers to the feature extraction of data from multiple modalities by fusing the data from other modalities separately, wherein, for the data of one modality, the feature representation is extracted in combination with the features expressed by the data of another modality, thereby fusing the features expressed by the data of multiple modalities and unifying the correlation features within the modality and between the modalities, thereby obtaining the feature representation corresponding to each modality.

[0068] In combination with the above-mentioned cross-modal cross-attention mechanism, the cross-feature extraction in the embodiments of the present application refers to feature extraction that performs cross-fusion on data of multiple modalities, wherein in this embodiment, feature representations are extracted for each modality through multiple feature extraction layers, and after feature extraction is performed at each feature extraction layer, cross-attention calculations are performed on the feature representations extracted from the data of multiple modalities respectively. Optionally, the cross-attention calculation method for each modality is the same or different. After obtaining the feature representation after the cross-attention calculation, the feature representation is used as the input of the next feature extraction layer, so that in each feature extraction layer, the features represented by the data of other modalities can be fused into the feature extraction process of the current modality.

[0069] Optionally, in an embodiment of the present application, when performing feature extraction on color images and point cloud data, a feature representation of fused multimodal data is obtained by cross-feature extraction on color channels and depth channels.

[0070] That is, in this embodiment, at each layer of encoder and decoder for feature extraction of color images and point cloud data, the two modal information are interacted and the next layer of feature extraction is performed.

[0071] Schematically, feature extraction is performed on color images based on a first feature extraction network through multiple feature extraction layers; and feature extraction is performed on point cloud data based on a second feature extraction network through multiple feature extraction layers, wherein at each layer, the feature representation extracted by the first feature extraction network and the feature representation extracted by the second feature extraction network perform a modal information interaction, thereby proceeding to the next feature extraction layer.

[0072] Among them, in the i-th feature extraction layer, feature extraction is performed on the color image based on the first feature extraction network to obtain a first feature representation; in the i-th feature extraction layer, feature extraction is performed on the point cloud data based on the second feature extraction network to obtain a second feature representation, where i is a positive integer; based on the first fusion method, the second feature representation is fused to the first feature representation to obtain a first fused feature representation, and the first fused representation is the input feature of the first feature extraction network in the i+1-th feature extraction layer; based on the second fusion method, the first feature representation is fused to the second feature representation to obtain a second fused feature representation, and the second fused feature representation is the input feature of the second feature extraction network in the i+1-th feature extraction layer; after passing through n layers of feature extraction layers, the color feature representation corresponding to the color image and the point cloud feature representation corresponding to the point cloud data are output, n≥i+1, and n is an integer.

[0073] Indicatively, Figure 4 As shown, after acquiring the color image 410 and the point cloud data 420, feature extraction is performed on the color image 410 through a first feature extraction network 430, wherein the first feature extraction network 430 includes multiple feature extraction layers; feature extraction is performed on the point cloud data 420 through a second feature extraction network 440, wherein the second feature extraction network 440 includes multiple feature extraction layers, and the number of feature extraction layers in the first feature extraction network 430 is the same as or different from the number of feature extraction layers in the second feature extraction network 440. In some embodiments, after each feature extraction layer is extracted, the extraction results of the first feature extraction network 430 and the extraction results of the second feature extraction network 440 are interacted; or, in other embodiments, after the k feature extraction layers in the first feature extraction network 430 perform feature extraction on the color image 410, the feature extraction results of the point cloud data 420 are interacted with one feature extraction layer in the second feature extraction network 440; or, after the k feature extraction layers in the second feature extraction network 440 perform feature extraction on the point cloud data 420, the feature extraction results of the color image 410 are interacted with one feature extraction layer in the first feature extraction network 430, where k is a positive integer. In this embodiment, the example of interacting with the extraction results of the first feature extraction network 430 and the second feature extraction network 440 after each feature extraction layer is extracted is used for explanation.

[0074] After being extracted by multiple feature extraction layers in the first feature extraction network, a color feature representation is obtained; and after being extracted by multiple feature extraction layers in the second feature extraction network, a point cloud feature representation is obtained.

[0075] Step 304: concatenate the color feature representation and the point cloud feature representation to obtain an image feature representation.

[0076] After the color feature representation and the point cloud feature representation are extracted, the color feature representation and the point cloud feature representation are fused by splicing to obtain an image feature representation, which is used in subsequent downstream prediction applications. For example, the pose of an entity in an image is predicted by the image feature representation.

[0077] Optionally, when the color feature representation and the point cloud feature representation are spliced ​​together, the color feature representation and the point cloud feature representation of the pixels are spliced ​​together in sequence based on the corresponding relationship between the pixels to obtain the image feature representation.

[0078] Step 305 , performing posture estimation and prediction on the target entity collected in the color image based on the image feature representation, and obtaining posture data corresponding to the target entity.

[0079] The pose data is used to represent the position and pose of the target entity relative to the depth camera.

[0080] After determining the posture data corresponding to the target entity, the embodiments of the present application include at least one of the following application scenarios:

[0081] First, in the scene of object grasping or sorting, the above solution is used to accurately locate the orientation of the object in the environment and the position in three-dimensional space, and the robot arm grasps the object based on the orientation and position information. Optionally, the robot arm can also place the object at a specified position in the environment and control the posture of the object after placement to avoid collision between the object and other objects in the environment.

[0082] Second, in an AR scene, during the process of observing the environment through AR technology, the posture of the object in the environment is identified through the above method, so as to adjust the position and posture of the special effects displayed on the object according to the posture of the object. For example, a cup in the environment is displayed through AR, and a luminous special effect is added to the handle of the cup, wherein the position and angle of the handle of the cup and the display range are determined according to the posture of the cup, so as to adjust the display area of ​​the luminous special effect.

[0083] In summary, in the method provided by the embodiment of the present application, during the process of object posture recognition, the process of extracting color feature representation and the process of extracting point cloud feature representation are carried out simultaneously, and there is an interactive relationship between the extraction of the two feature representations, that is, the color feature representation and the point cloud feature representation are cross-extracted through a cross-modal cross-attention mechanism, and the color feature representation and the point cloud feature representation are integrated into the extracted image feature representation, so that the entity posture is recognized according to the image feature representation. The interaction of the two modal information causes the entity features to merge and influence each other during the feature extraction process, thereby improving the accuracy of identifying the entity posture from the image.

[0084] The method provided in this embodiment, after acquiring the depth image, first obtains the camera intrinsic parameters corresponding to the depth camera, and performs data conversion on the pixel points in the depth image according to the camera intrinsic parameters, thereby generating point cloud data, thereby improving the accuracy of the point cloud data.

[0085] The method provided in this embodiment cross-extracts the color feature representation and the point cloud feature representation for each feature extraction layer, so that information interaction is generated between the color feature representation and the point cloud feature representation, thereby improving the accuracy of feature extraction and thus improving the accuracy of identifying the entity posture from the image.

[0086] In an optional embodiment, the first fusion method and the second fusion method are implemented by converting the feature representation dimension. Figure 5 is a flowchart of a posture estimation method provided by another exemplary embodiment of the present application. The method can be executed by a server or a terminal, or by both a server and a terminal. In the embodiment of the present application, the method is executed by a server as an example for explanation. Figure 5 As shown, in the above step 303, at the i-th feature extraction layer, feature extraction is performed on the color image based on the first feature extraction network to obtain a first feature representation, and feature extraction is performed on the point cloud data based on the second feature extraction network to obtain a second feature representation, and then feature fusion can be performed through the following steps 3031 to 3036.

[0087] Step 3031, obtaining neighboring points of point cloud pixels in the point cloud data in the color image.

[0088] Optionally, obtain the first image coordinates of the point cloud pixel points in the point cloud data in the depth image; obtain the second image coordinates corresponding to the pixel points in the color image; convert the first image coordinates into the corresponding third image coordinates in the color image, and determine the neighboring points of the point cloud pixel points in the color image based on the distance between the third image coordinates and the second image coordinates corresponding to the pixel points.

[0089] Optionally, the first image coordinates of the point cloud pixel in the point cloud data in the depth image are represented as A(x, y, z), where x represents the coordinate of the point cloud pixel A on the x-axis in the three-dimensional coordinates, y represents the coordinate of the point cloud pixel A on the y-axis in the three-dimensional coordinates, and z represents the coordinate of the point cloud pixel A on the z-axis in the three-dimensional coordinates. Among them, x, y and z can be converted to u, v, d, where u represents the horizontal coordinate of the point cloud pixel A in the depth image, v represents the vertical coordinate of the point cloud pixel A in the depth image, and d represents the depth of the point cloud pixel A.

[0090] Optionally, based on the distance between the third image coordinates and the second image coordinates corresponding to the pixel points, the pixel points corresponding to the m second image coordinates closest to the third image coordinates are taken as neighboring points, where m is a positive integer.

[0091] Step 3032: Obtain neighbor point feature representations corresponding to the neighbor points in the first feature representation.

[0092] Since the depth image is indexed by (u, v, d), and u, v have the same meaning as u, v in the color image, the features of the color image at the pixel in the point cloud can be extracted through u, v.

[0093] Step 3033: Concatenate the point cloud feature representation corresponding to the point cloud pixel in the second feature representation with the neighboring feature representation to obtain a second fused feature representation corresponding to the point cloud pixel.

[0094] Indicatively, Figure 6 As shown, in the point cloud feature representation, for the point cloud pixel 610, the neighboring point 620 in the color feature representation is determined; the point cloud feature representation 611 of the point cloud pixel 610 and the neighboring point feature representation 621 corresponding to the neighboring point 620 are spliced ​​to obtain a second fused feature representation 630 corresponding to the point cloud pixel. After multi-layer fusion, the obtained second fused feature representation 630 is the feature representation corresponding to the point cloud pixel in the image feature representation 640.

[0095] Figure 7 is a schematic diagram corresponding to the second fusion method mentioned above, such as Figure 7 As shown, assuming Figure 6 The dimension of the first feature representation 710, i.e., the color feature representation, after passing through a certain layer of encoder or decoder is H×W×C1, and the dimension of the second feature representation 720, i.e., the point cloud feature representation, is N×C2 (H, W, C1 are RGB feature maps, i.e., the width, height, and dimension of the first feature representation, which may change with downsampling and upsampling; N and C2 are the number and dimension of the point cloud, which also change with the point cloud features, i.e., the upsampling and downsampling of the second feature representation). The operation of fusing RGB features into point cloud features is as follows:

[0096] First, the RGB features are converted into H×W×2C3 dimensions through a multilayer perceptron (MLP) and assigned to K (with dimensions of H×W×C3) and V (with dimensions of H×W×C3); the point cloud features are adjusted to N×C3 dimensions through MLP.

[0097] Determine the nearest neighbor point from the point cloud pixel to the color image (dimension is N×K. K is the K nearest neighbor points, indicating the first K nearest neighbor points from the point cloud to the color image, where the distance is determined by the Euclidean distance of the three-dimensional point), extract the corresponding RGB feature with dimension N×K×C3, and expand it to N×K×1×C3. At the same time, expand the dimension of the point cloud feature N×C3 to N×1×1×C3.

[0098] Perform softmax(QK) on the underlined dimensions in the figure T )V operation, and obtain the feature to be fused N×1×1×C3. Reduce its dimension to N×C3, and concatenate it with the original point cloud feature to obtain N×(C2+C3), and obtain the fused feature N×C2 through MLP.

[0099] After obtaining the fused features, continue to pass through the convolutional layer of the second feature extraction network.

[0100] Step 3034: transform the second feature representation using the feature dimension corresponding to the first feature representation to obtain a transformed feature representation.

[0101] In the process of fusing the second feature representation into the first feature representation, since the number of point cloud pixels is less than the number of color image pixels, only the feature representation of the color image pixels that are the nearest neighbors to the point cloud pixels is used during fusion.

[0102] Figure 8 is a schematic diagram corresponding to the first fusion method mentioned above, such as Figure 8 As shown, assuming Figure 6 The dimension of the first feature representation 810, i.e., the color feature representation, after passing through a certain layer of encoder or decoder is H×W×C1, and the dimension of the second feature representation 820, i.e., the point cloud feature representation, is N×C2 (H, W, C1 are RGB feature maps, i.e., the width, height, and dimension of the first feature representation, which may change with downsampling and upsampling; N and C2 are the number and dimension of the point cloud, which also change with the upsampling and downsampling of the point cloud features, i.e., the second feature representation). The point cloud feature representation 820 is converted into H×W×C2 through a multi-layer perceptron MLP, i.e., a conversion feature representation 830.

[0103] Step 3035, concatenate the first feature representation and the converted feature representation to obtain a concatenated feature representation.

[0104] Indicatively, Figure 8 As shown, the converted feature representation 830 is concatenated with the first feature representation 810 to obtain a concatenated feature representation 840, and the dimension of the concatenated feature representation 840 is H×W×(C1+C2).

[0105] Step 3036, downsample the concatenated feature representation to obtain a first fused feature representation.

[0106] like Figure 8 As shown, the above-mentioned spliced ​​feature representation 840 is downsampled to obtain a first fused feature representation 850, and the dimension of the first fused feature representation 850 is H×W×C1.

[0107] In some embodiments, the embodiments of the present application mainly improve the information fusion part. Schematically, given an RGBD image, the depth information and the camera intrinsic parameter matrix are used to adjust it to a three-dimensional coordinate representation of the XYZ map, which corresponds one-to-one to the pixel coordinates of the RGB image.

[0108] In the process of fusing the first feature representation into the second feature representation, the point cloud processing network can find the pixel in the XYZ map that is closest to the point cloud pixel and extract the RGB features of the corresponding point for fusion. Figure 1 Therefore, as the RGB feature map is downsampled, the XYZ map also needs to be downsampled. Optionally, the XYZ map is downsampled using the nearest interpolation algorithm. Thus, the information of the surrounding points of the RGB image corresponding to the point cloud is introduced during information fusion.

[0109] Optionally, a cross-modal attention mechanism is used to fuse RGB features into point cloud features, as shown in the following formula 1:

[0110]

[0111] As shown in Formula 1, after each encoding or decoding stage, the shared MLP is applied to the point cloud and RGB features to obtain the query Q, key K, and value V respectively. At the same time, the index of the RGB feature corresponding to the point cloud is obtained through the nearest neighbor query from the point cloud to the XYZ map, k n (source, target) means finding the nearest neighbor in target for each element in source, n represents the number of nearest neighbors, that is, the number of RGB features to be fused. After taking out the corresponding RGB features, a cross-attention operation is performed with the point cloud features, that is, Finally, after splicing it with the original point cloud features and passing through MLP, the point cloud features that have been fused with RGB features are obtained. And passed to the next encoding or decoding stage. r2p =k n (pointcloud,map xyz ) refers to finding neighboring points in the color image pixels from the point cloud feature representation.

[0112] In the process of fusing the second feature representation into the first feature representation, as shown in the following formula 2:

[0113]

[0114] Among them, index p2r =k n (map xyz , pointcloud) refers to the color image pixel map xyz Find the point cloud feature representation of the neighboring points in the pointcloud. p2r Represents the point cloud feature representation to be fused into the color feature representation, according to index p2r The index of is obtained. Indicates that the color feature representation in the point cloud feature representation has been fused. cat means concatenating the two inputs together. MLP means multi-layer perceptron.

[0115] Considering that the number of downsampled point cloud features is much smaller than the number of RGB features in network processing, and the distribution is sparse, when fusing the second feature representation into the first feature representation, only the nearest neighbor point cloud information is used, and the surrounding information is not used. n (map xyz , pointcloud) takes n=1.

[0116] In summary, in the method provided by the embodiment of the present application, during the process of object posture recognition, the process of extracting color feature representation and the process of extracting point cloud feature representation are carried out simultaneously, and there is an interactive relationship between the extraction of the two feature representations, that is, the color feature representation and the point cloud feature representation are cross-extracted through a cross-modal cross-attention mechanism, and the color feature representation and the point cloud feature representation are integrated into the extracted image feature representation, so that the entity posture is recognized according to the image feature representation. The interaction of the two modal information causes the entity features to merge and influence each other during the feature extraction process, thereby improving the accuracy of identifying the entity posture from the image.

[0117] The method provided in this embodiment integrates color feature representation into point cloud feature representation, and integrates point cloud feature representation into color feature representation through a cross-modal cross-attention mechanism, thereby improving the accuracy of extracting image feature representation and the accuracy of identifying entity posture from images.

[0118] In an optional embodiment, during the process of posture estimation prediction based on image feature representation, bias is estimated and semantics is segmented. Fig. 9 is a flowchart of a posture estimation method provided by another exemplary embodiment of the present application. The method can be executed by a server or a terminal, or by both a server and a terminal. In the embodiment of the present application, the method is executed by a server as an example for explanation. Fig. 9 As shown, the above step 305 can be implemented as the following steps 3051 to 3053.

[0119] Step 3051, input the image feature representation into the bias estimation model, and output the bias estimation result corresponding to the target entity.

[0120] Optionally, the target entity is a pre-set entity appearing in the color image; or, the target entity is the entity with the largest display area currently in the color image; or, the target entity is the entity currently closest to the depth camera, which is not limited in the embodiments of the present application.

[0121] The bias estimation model is a pre-trained model used to estimate the bias results of entity recognition in the image. The bias estimation result is used to represent the position bias of the target entity relative to the depth camera, that is, based on the bias estimation result, the current position of the target entity in the environment relative to the depth camera can be determined.

[0122] Schematically, the training process of the bias estimation model includes: obtaining a sample image, which is annotated with a bias label, and the bias label is used to indicate the bias of the image acquisition device corresponding to the target entity in the sample image; bias estimation is performed on the sample image through the candidate bias estimation model to be trained, and a predicted bias result is output; according to the difference between the predicted bias result and the bias label, the candidate bias estimation model is iteratively trained until a bias estimation model that meets the training requirements is obtained.

[0123] Step 3052, input the image feature representation into the semantic segmentation model, and output the semantic segmentation result corresponding to the target entity.

[0124] The semantic segmentation result is used to represent the independent segmentation relationship between the target entity and other entities in the color image.

[0125] The semantic segmentation model is a pre-trained model used to estimate the segmentation between entities in an image based on image semantics. The semantic segmentation result is used to represent the segmentation between the target entity and other entities in the image.

[0126] Schematically, the training process of the semantic segmentation model includes: obtaining a sample image, which is annotated with a semantic segmentation label, and the semantic segmentation label is used to indicate the segmentation of the target entity in the sample image corresponding to other entities; performing semantic segmentation on the sample image through the candidate semantic segmentation model to be trained, and outputting a predicted semantic segmentation result; according to the difference between the predicted semantic segmentation result and the semantic segmentation label, iteratively training the candidate semantic segmentation model until a semantic segmentation model that meets the training requirements is obtained.

[0127] It is worth noting that the above-mentioned step 3051 and step 3052 are two parallel steps. Step 3051 can be executed first or step 3052 can be executed first, or step 3051 and step 3052 can be executed at the same time. The embodiment of the present application is not limited to this.

[0128] Step 3053, based on the prior data corresponding to the target entity, the bias estimation result and the semantic segmentation result, the posture data corresponding to the target entity is generated.

[0129] Optionally, the prior data refers to pre-set or pre-acquired data corresponding to the target entity, and includes at least one of the parameters of the target entity, such as length, width, height, three-dimensional model shape, hardness, and touchable position.

[0130] In summary, in the method provided by the embodiment of the present application, during the process of object posture recognition, the process of extracting color feature representation and the process of extracting point cloud feature representation are carried out simultaneously, and there is an interactive relationship between the extraction of the two feature representations, that is, the color feature representation and the point cloud feature representation are cross-extracted through a cross-modal cross-attention mechanism, and the color feature representation and the point cloud feature representation are integrated into the extracted image feature representation, so that the entity posture is recognized according to the image feature representation. The interaction of the two modal information causes the entity features to merge and influence each other during the feature extraction process, thereby improving the accuracy of identifying the entity posture from the image.

[0131] The method provided in this embodiment further accurately identifies the target entity in the image through the bias estimation model and the semantic segmentation model, thereby providing the entity location and interaction scheme that can be interacted with, thereby improving the accuracy of entity interaction.

[0132] The method proposed in this patent improves the object posture estimation effect and realizes end-to-end posture estimation, which can be further applied to the self-supervised task of posture estimation. We use the ADD distance of object posture estimation and ADDAUC to measure the accuracy of posture estimation. Among them, the ADD distance is defined as shown in the following formula 3:

[0133]

[0134] in, and The predicted results and the true annotations are shown respectively. v∈O represents the object point cloud representation in the object coordinate system. For symmetrical objects, ADD-s is used as the evaluation criterion. ADD-s is defined as shown in the following formula 4:

[0135]

[0136] In terms of object pose estimation, the method provided by the embodiment of the present application provides a higher accuracy effect. As shown in Table 1 below, compared with the new convolutional neural network PoseCNN and the Discriminative Correlation Filters (DCF) network for 6D target pose estimation, the solution provided by the embodiment of the present application performs better in terms of accuracy.

[0137] Table 1

[0138]

[0139] Among them, the values ​​in the table represent the accuracy rate, and the higher the value, the higher the accuracy rate.

[0140] Fig.10 is a structural block diagram of a posture estimation device provided by an exemplary embodiment of the present application, such as Fig.10 As shown, the device comprises:

[0141] An acquisition module 1010 is used to acquire a color image and a depth image acquired by a depth camera, wherein the color image and the depth image are images acquired synchronously by the depth camera;

[0142] The acquisition module 1010 is further used to acquire point cloud data corresponding to the depth image, where the point cloud data is used to indicate the three-dimensional spatial distribution of pixels in the depth image;

[0143] The feature extraction module 1020 is used to perform cross-feature extraction on the color image and the point cloud data based on a cross-modal cross-attention mechanism, output a color feature representation corresponding to the color image and a point cloud feature representation corresponding to the point cloud data; and concatenate the color feature representation and the point cloud feature representation to obtain an image feature representation;

[0144] The prediction module 1030 is used to perform posture estimation and prediction on the target entity captured in the color image based on the image feature representation, and obtain posture data corresponding to the target entity, wherein the posture data is used to represent the position and posture of the target entity relative to the depth camera.

[0145] In an optional embodiment, the feature extraction module 1020 is further used to perform feature extraction on the color image based on a first feature extraction network at the i-th feature extraction layer to obtain a first feature representation; perform feature extraction on the point cloud data based on a second feature extraction network at the i-th feature extraction layer to obtain a second feature representation, where i is a positive integer;

[0146] The feature extraction module 1020 is further configured to fuse the second feature representation into the first feature representation based on a first fusion method to obtain a first fused feature representation, wherein the first fused feature representation is an input feature of the first feature extraction network in the i+1th feature extraction layer; and fuse the first feature representation into the second feature representation based on a second fusion method to obtain a second fused feature representation, wherein the second fused feature representation is an input feature of the second feature extraction network in the i+1th feature extraction layer;

[0147] The feature extraction module 1020 is further used to output, through n layers of feature extraction layers, a color feature representation corresponding to the color image and a point cloud feature representation corresponding to the point cloud data, where n≥i+1 and n is an integer.

[0148] In an optional embodiment, the acquisition module 1010 is further used to acquire neighboring points of a point cloud pixel point in the point cloud data in the color image; and acquire a neighboring point feature representation corresponding to the neighboring point in the first feature representation;

[0149] The feature extraction module 1020 is further used to concatenate the point cloud feature representation corresponding to the point cloud pixel in the second feature representation with the neighboring point feature representation to obtain a second fused feature representation corresponding to the point cloud pixel.

[0150] In an optional embodiment, the acquisition module 1010 is also used to obtain the first image coordinates of the point cloud pixel points in the point cloud data in the depth image; obtain the second image coordinates corresponding to the pixel points in the color image in the color image; convert the first image coordinates into the corresponding third image coordinates in the color image; and determine the neighboring points of the point cloud pixel points in the color image based on the distance between the third image coordinates and the second image coordinates corresponding to the pixel points.

[0151] In an optional embodiment, the acquisition module 1010 is further used to acquire a neighbor point feature representation corresponding to the neighbor point in the first dimensional feature representation;

[0152] The feature extraction module 1020 is also used to convert the first feature representation into a first dimensional feature representation through a multi-layer perceptron layer; and to convert the second feature representation into a second dimensional feature representation through the multi-layer perceptron layer; and to concatenate the point cloud feature representation corresponding to the point cloud pixel in the second dimensional feature representation with the neighboring point feature representation to obtain a second fused feature representation corresponding to the point cloud pixel.

[0153] In an optional embodiment, the feature extraction module 1020 is further used to convert the second feature representation with a feature dimension corresponding to the first feature representation to obtain a converted feature representation; splice the first feature representation and the converted feature representation to obtain a spliced ​​feature representation; downsample the spliced ​​feature representation to obtain the first fused feature representation.

[0154] In an optional embodiment, the acquisition module 1010 is also used to obtain camera intrinsic parameters corresponding to the depth camera, where the camera intrinsic parameters are acquisition parameters of the depth camera when acquiring images; and convert the pixel point data in the depth image based on the camera intrinsic parameters to generate the point cloud data.

[0155] In an optional embodiment, if Fig.11 As shown, the prediction module 1030 includes:

[0156] An input unit 1031 is used to input the image feature representation into a bias estimation model, and output a bias estimation result corresponding to the target entity, wherein the bias estimation result is used to represent a relative position bias between the target entity and the depth camera;

[0157] The input unit 1031 is further used to input the image feature representation into a semantic segmentation model, and output a semantic segmentation result corresponding to the target entity, wherein the semantic segmentation result is used to represent an independent segmentation relationship between the target entity and other entities in the color image;

[0158] The generating unit 1032 is used to generate the posture data corresponding to the target entity based on the prior data corresponding to the target entity, the bias estimation result and the semantic segmentation result.

[0159] To summarize, in the device provided by the embodiment of the present application, during the process of object posture recognition, the process of extracting color feature representation and the process of extracting point cloud feature representation are carried out simultaneously, and there is an interactive relationship between the extraction of the two feature representations, that is, the color feature representation and the point cloud feature representation are cross-extracted through a cross-modal cross-attention mechanism, and the color feature representation and the point cloud feature representation are integrated into the extracted image feature representation, so that the entity posture is recognized according to the image feature representation. The interaction of the two modal information causes the entity features to merge and influence each other during the feature extraction process, thereby improving the accuracy of identifying the entity posture from the image.

[0160] It should be noted that the posture estimation device provided in the above embodiment is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the posture estimation device provided in the above embodiment and the posture estimation method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0161] Fig.12 A schematic diagram of the structure of a computer device provided by an exemplary embodiment of the present application is shown. The computer device may be as follows: Figure 2 Computer equipment shown.

[0162] Specifically, the computer device 1200 includes a central processing unit (CPU) 1201, a system memory 1204 including a random access memory (RAM) 1202 and a read-only memory (ROM) 1203, and a system bus 1205 connecting the system memory 1204 and the central processing unit 1201. The computer device 1200 also includes a mass storage device 1206 for storing an operating system 1213, application programs 1214, and other program modules 1215.

[0163] The mass storage device 1206 is connected to the central processing unit 1201 through a mass storage controller (not shown) connected to the system bus 1205. The mass storage device 1206 and its associated computer readable media provide non-volatile storage for the computer device 1200. That is, the mass storage device 1206 may include a computer readable medium (not shown) such as a hard disk or a compact disc read only memory (CD-ROM) drive.

[0164] Without loss of generality, computer-readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other solid-state storage technology, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media are not limited to the above. The above-mentioned system memory 1204 and mass storage device 1206 can be collectively referred to as memory.

[0165] According to various embodiments of the present application, the computer device 1200 can also be connected to a remote computer on the network through a network such as the Internet. That is, the computer device 1200 can be connected to the network 1212 through the network interface unit 1211 connected to the system bus 1205, or the network interface unit 1211 can be used to connect to other types of networks or remote computer systems (not shown).

[0166] The above-mentioned memory also includes one or more programs, and the one or more programs are stored in the memory and configured to be executed by the CPU.

[0167] The embodiment of the present application also provides a computer device, which can be implemented as follows: Figure 2 The computer device includes a processor and a memory, wherein at least one instruction, at least one program, code set or instruction set is stored in the memory, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the posture estimation method provided by the above-mentioned method embodiments.

[0168] An embodiment of the present application also provides a computer-readable storage medium, on which is stored at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the posture estimation method provided by the above-mentioned method embodiments.

[0169] The embodiments of the present application also provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device performs any of the posture estimation methods described in the above embodiments.

[0170] Optionally, the computer readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid state drive (SSD), or an optical disk. Among them, the random access memory may include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM). The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0171] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0172] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A posture estimation method, characterized in that: The method comprises: Acquire a color image and a depth image acquired by a depth camera, wherein the color image and the depth image are images acquired synchronously by the depth camera; Acquire point cloud data corresponding to the depth image, the point cloud data being used to indicate the three-dimensional spatial distribution of pixel points in the depth image; Performing cross-feature extraction on the color image and the point cloud data based on a cross-modal cross-attention mechanism, and outputting a color feature representation corresponding to the color image and a point cloud feature representation corresponding to the point cloud data; splicing the color feature representation and the point cloud feature representation to obtain an image feature representation; Based on the image feature representation, a posture estimation and prediction is performed on the target entity captured in the color image to obtain posture data corresponding to the target entity, where the posture data is used to represent the position and posture of the target entity relative to the depth camera.

2. The method according to claim 1, characterized in that The cross-feature extraction of the color image and the point cloud data based on the cross-modal cross-attention mechanism, and outputting a color feature representation corresponding to the color image and a point cloud feature representation corresponding to the point cloud data, includes: At the i-th feature extraction layer, feature extraction is performed on the color image based on a first feature extraction network to obtain a first feature representation; At the i-th feature extraction layer, feature extraction is performed on the point cloud data based on a second feature extraction network to obtain a second feature representation, where i is a positive integer; Based on a first fusion method, the second feature representation is fused to the first feature representation to obtain a first fused feature representation, where the first fused feature representation is an input feature of the first feature extraction network in the (i+1)th feature extraction layer; Based on a second fusion method, the first feature representation is fused into the second feature representation to obtain a second fused feature representation, where the second fused feature representation is an input feature of the second feature extraction network in the (i+1)th feature extraction layer; Through n layers of feature extraction layers, the color feature representation corresponding to the color image and the point cloud feature representation corresponding to the point cloud data are output, where n≥i+1 and n is an integer.

3. The method according to claim 2, characterized in that The fusing the first feature representation into the second feature representation based on a second fusion mode to obtain a second fused feature representation includes: Obtain neighboring points of a point cloud pixel point in the point cloud data in the color image; Obtaining a neighbor point feature representation corresponding to the neighbor point in the first feature representation; The point cloud feature representation corresponding to the point cloud pixel in the second feature representation is concatenated with the neighboring point feature representation to obtain a second fused feature representation corresponding to the point cloud pixel.

4. The method according to claim 3, characterized in that The obtaining of neighboring points of a point cloud pixel in the point cloud data in the color image includes: Obtaining first image coordinates of a point cloud pixel in the point cloud data in the depth image; Obtaining second image coordinates corresponding to pixel points in the color image in the color image; converting the first image coordinates into corresponding third image coordinates in the color image; Based on the distance between the third image coordinates and the second image coordinates respectively corresponding to the pixel points, the neighboring points of the point cloud pixel point in the color image are determined.

5. The method according to claim 3, characterized in that: The obtaining the neighboring point feature representation corresponding to the neighboring point in the first feature representation includes: Obtaining a neighbor point feature representation corresponding to the neighbor point in the first feature representation; The step of concatenating the point cloud feature representation corresponding to the pixel point of the point cloud in the second feature representation with the neighboring point feature representation to obtain a second fused feature representation corresponding to the point cloud includes: Converting the first feature representation into a first-dimensional feature representation through a multi-layer perceptron layer; and converting the second feature representation into a second-dimensional feature representation through the multi-layer perceptron layer; The point cloud feature representation corresponding to the point cloud pixel in the second dimensional feature representation is concatenated with the neighboring point feature representation to obtain a second fused feature representation corresponding to the point cloud pixel.

6. The method according to claim 2, characterized in that The fusing the second feature representation into the first feature representation based on a first fusion mode to obtain a first fused feature representation includes: Converting the second feature representation using a feature dimension corresponding to the first feature representation to obtain a converted feature representation; Concatenating the first feature representation and the converted feature representation to obtain a concatenated feature representation; The concatenated feature representation is downsampled to obtain the first fused feature representation.

7. The method according to any one of claims 1 to 6, characterized in that: The acquiring point cloud data corresponding to the depth image includes: Obtain camera intrinsic parameters corresponding to the depth camera, where the camera intrinsic parameters are acquisition parameters of the depth camera when performing image acquisition; The pixel data in the depth image is converted based on the camera intrinsic parameters to generate the point cloud data.

8. The method according to any one of claims 1 to 6, characterized in that: The step of performing posture estimation and prediction on the target entity collected in the color image based on the image feature representation to obtain posture data corresponding to the target entity includes: Inputting the image feature representation into a bias estimation model, and outputting a bias estimation result corresponding to the target entity, wherein the bias estimation result is used to represent a relative position bias between the target entity and the depth camera; Inputting the image feature representation into a semantic segmentation model, and outputting a semantic segmentation result corresponding to the target entity, wherein the semantic segmentation result is used to represent an independent segmentation relationship between the target entity and other entities in the color image; Based on the prior data corresponding to the target entity, the bias estimation result and the semantic segmentation result, posture data corresponding to the target entity is generated.

9. A posture estimation device, characterized in that: The device comprises: An acquisition module, used to acquire a color image and a depth image acquired by a depth camera, wherein the color image and the depth image are images acquired synchronously by the depth camera; The acquisition module is further used to acquire point cloud data corresponding to the depth image, wherein the point cloud data is used to indicate the three-dimensional spatial distribution of pixel points in the depth image; A feature extraction module, configured to perform cross-feature extraction on the color image and the point cloud data based on a cross-modal cross-attention mechanism, output a color feature representation corresponding to the color image and a point cloud feature representation corresponding to the point cloud data; and concatenate the color feature representation and the point cloud feature representation to obtain an image feature representation; A prediction module is used to perform posture estimation and prediction on the target entity captured in the color image based on the image feature representation, and obtain posture data corresponding to the target entity, wherein the posture data is used to represent the position and posture of the target entity relative to the depth camera.

10. A computer device, characterized in that: The computer device includes a processor and a memory, wherein at least one program is stored in the memory, and the at least one program is loaded and executed by the processor to implement the posture estimation method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The storage medium stores at least one program, and the at least one program is loaded and executed by the processor to implement the posture estimation method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that The method comprises a computer program or an instruction, wherein the computer program or the instruction is executed by a processor to implement the posture estimation method as claimed in any one of claims 1 to 8.

Images