An object posture detection method, device, equipment and medium

Abstract

The application discloses an object posture detection method and device, equipment and medium, and relates to the technical field of computer vision. The scheme collects the pixel image of the target object through the image acquisition device, and collects the depth image of the target object through the depth information acquisition device, without calculating the depth information of the object through the stereo vision algorithm, thereby reducing the calculation amount. Further, the pixel image and the depth information are simultaneously subjected to dimension reduction processing; when the pixel image and the depth information are used to restore the target object in three dimensions and detect the posture of the target object subsequently, the pixel image and the depth information subjected to the dimension reduction processing retain the complete features of the target object and greatly reduce the data processing amount. Therefore, the scheme can effectively reduce the calculation amount of the object posture recognition process, reduce the time complexity of the object posture recognition process, and improve the detection efficiency.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, apparatus, device, and medium for object pose detection. Background Technology

[0002] With the development of technology and the improvement of living standards, express delivery has become increasingly common in people's lives. Traditional express delivery sorting mainly relies on manual labor, which is not only prone to sorting errors but also inefficient. Therefore, automated express delivery sorting methods using robotic arms and visual recognition technology have emerged. Specifically, visual recognition technology is used to determine the posture of objects, and then the robotic arm grasps and moves the objects according to their posture, thereby achieving automated sorting.

[0003] Currently, binocular vision localization technology is mainly used to calculate object pose. This technology requires matching images from two cameras to find corresponding pixels, and then using stereo vision algorithms to calculate the object's depth information. However, this process involves a large amount of pixel matching and calculation, consuming significant computational resources and time, which is detrimental to the rapid identification and detection of object poses in scenarios such as express delivery sorting.

[0004] Given the above problems, how to solve the shortcomings of the current object pose recognition process, which has a large amount of computation and high time complexity, is an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide an object posture detection method, apparatus, device, and medium to address the shortcomings of current object posture recognition processes, such as high computational load and high time complexity.

[0006] To solve the above-mentioned technical problems, the present invention provides an object pose detection method, comprising:

[0007] Acquire pixel images of the target object captured by the image acquisition device, and acquire depth images of the target object captured by the depth information acquisition device;

[0008] The pixel image is input into the object detection network to obtain the object category, object detection box position, object label, and the score corresponding to the object label;

[0009] Determine whether the score corresponding to the target label is greater than a first threshold;

[0010] If not, return to the step of obtaining the pixel image of the target object and obtaining the depth image of the target object acquired by the depth information acquisition device;

[0011] If so, the depth information of the target object in the target box is determined based on the depth image and the position of the target detection box;

[0012] Dimensionality reduction processing is performed on the pixel image and depth information of the target object;

[0013] The target object is reconstructed in three dimensions based on the target detection box position, the object category, the pixel image after dimensionality reduction, and the depth information to determine the pose coordinates of the target object.

[0014] On one hand, the step of inputting the pixel image into the target detection network to obtain the object category, target detection box position, target label, and the score corresponding to the target label of the target object includes:

[0015] The pixel image is scaled proportionally.

[0016] The pixel image, after being scaled proportionally, is placed into a three-layer convolutional neural network to obtain an initial feature map; wherein each layer of the convolutional neural network contains a convolutional layer, a data batch normalization layer, and a hard tangent activation function;

[0017] The target recommendation region in the initial feature map is determined based on the region generation network;

[0018] The initial feature map and the target recommendation region are input into the region interest alignment network to obtain the target feature map;

[0019] The target feature map is input into the head layer to obtain object category information and target detection region information;

[0020] The object category information is sequentially input into two convolutional layers to obtain the object category;

[0021] The target detection region information is input into the convolutional layer to obtain the target detection box position, the target label, and the score corresponding to the target label.

[0022] On the other hand, the dimensionality reduction processing of the pixel image and depth information of the target object includes:

[0023] The point where the image acquisition centerline of the image acquisition device coincides with the point where the target object is located is determined as the center point;

[0024] The spatial coordinates of the center point are determined based on the pixel coordinates in the pixel image and the depth information value in the depth information.

[0025] Three concentric circles are defined with the center point as the center; wherein each of the concentric circles lies in the same plane;

[0026] Six rays are generated with the center point as the endpoint; wherein the included angles between each ray are equal, and each ray intersects with each of the concentric circles;

[0027] Based on the pixel coordinates in the pixel image, obtain the pixel coordinate values ​​of the projection points generated by projecting each intersection point onto the target object;

[0028] The depth information value of each projection point is determined based on the depth information; wherein, the depth information value of each projection point is specifically the average value of the depth information values ​​of each point within a circular area centered on the projection point and with a preset value as the radius.

[0029] The spatial coordinates of each projection point are determined based on the pixel coordinates and depth information values ​​corresponding to each projection point.

[0030] On the other hand, the step of performing 3D reconstruction of the target object based on the target detection box position, the object category, the dimensionally reduced pixel image, and the depth information to determine the pose coordinates of the target object includes:

[0031] The boundary point coordinates of the target object are determined based on the ground plane coordinates and the boundary points of the target detection box position;

[0032] The plane equation or surface equation of the target object is determined based on the object category, the pixel image after dimensionality reduction, and the depth information.

[0033] Based on the boundary point coordinates of the target object and the plane equation or surface equation of the target object, the target object is reconstructed in three dimensions to generate a three-dimensional model of the target object.

[0034] Generate the pose coordinates of the target object in the real coordinate system based on the three-dimensional model of the target object.

[0035] On the other hand, determining the plane equation or surface equation of the target object based on the object category, the dimension-reduced pixel image, and the depth information includes:

[0036] When the target object is determined to be a rigid object according to the object category, the angle between two adjacent projection points corresponding to the same ray is determined to be less than a second threshold based on the spatial coordinates of the center point and the spatial coordinates of each projection point.

[0037] If it is determined that the angle between two adjacent projection points corresponding to the same ray is less than the second threshold, then the midpoint is selected between the two projection points;

[0038] Based on the intermediate point, additional points other than the two adjacent projection points are determined, and the spatial coordinate values ​​of the additional points are obtained according to the pixel coordinates in the pixel image and the depth information; wherein, the additional points are not on the projection of the rays corresponding to the two adjacent projection points;

[0039] A plane equation is fitted based on the spatial coordinates of the additional point and the spatial coordinates of two adjacent projection points;

[0040] The remaining projection points are adapted according to the plane equation;

[0041] If it is determined that the included angle between two adjacent projection points corresponding to the same ray is not less than the second threshold, then a new projection point is selected between the two adjacent projection points according to the bisection method; wherein, the new projection point corresponds to the same ray as the two adjacent projection points.

[0042] The spatial coordinates of the new projection point are obtained based on the pixel coordinates in the pixel image and the depth information, and it is determined again whether the angle between two adjacent projection points corresponding to the same ray is less than the second threshold.

[0043] If it is determined that the angle between two adjacent projection points corresponding to the same ray is less than the second threshold, then proceed to the step of selecting the midpoint between the two projection points;

[0044] If it is determined that the included angle between two adjacent projection points corresponding to the same ray is not less than the second threshold, then the radius of each concentric circle is reduced to redetermine the spatial coordinate value corresponding to each projection point;

[0045] Return to the step of determining whether the included angle between two adjacent projection points corresponding to the same ray is less than a second threshold based on the spatial coordinates of the center point and the spatial coordinates of each projection point.

[0046] On the other hand, determining the plane equation or surface equation of the target object based on the object category, the dimension-reduced pixel image, and the depth information includes:

[0047] When the target object is determined to be a flexible object based on the object category, a surface equation is fitted based on the spatial coordinates of the center point and the spatial coordinates of each projection point.

[0048] On the other hand, after fitting the plane equation based on the spatial coordinates of the additional point and the spatial coordinates of the two adjacent projection points, the method further includes:

[0049] Obtain the number of plane equations and determine whether the number of plane equations is greater than 3;

[0050] If so, then obtain the similarity between each of the plane equations;

[0051] Fit the plane equation corresponding to the similarity being greater than the third threshold.

[0052] To address the aforementioned technical problems, the present invention also provides an object posture detection device, comprising:

[0053] The first acquisition module is used to acquire pixel images of the target object acquired by the image acquisition device and depth images of the target object acquired by the depth information acquisition device.

[0054] The second acquisition module is used to input the pixel image into the target detection network to obtain the object category, target detection box position, target label, and the score corresponding to the target label of the target object;

[0055] The judgment module is used to determine whether the score corresponding to the target label is greater than a first threshold; if not, the first acquisition module is triggered; if yes, the first determination module is triggered.

[0056] The first determining module is used to determine the depth information of the target object in the target box based on the depth image and the position of the target detection box;

[0057] The dimensionality reduction module is used to perform dimensionality reduction processing on the pixel image and the depth information of the target object;

[0058] The second determining module is used to perform three-dimensional reconstruction of the target object based on the target detection box position, the object category, the dimension-reduced pixel image, and the depth information, so as to determine the pose coordinates of the target object.

[0059] To address the aforementioned technical problems, the present invention also provides an object posture detection device, comprising:

[0060] Memory, used to store computer programs;

[0061] A processor is used to implement the steps of the object pose detection method described above when executing the computer program.

[0062] To address the aforementioned technical problems, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the object pose detection method described above.

[0063] This invention provides an object pose detection method. The method involves acquiring pixel images of a target object using an image acquisition device and depth images of the target object using a depth information acquisition device. The pixel images are input into a target detection network to obtain the object category, target detection box position, target label, and score corresponding to the target label. The method then determines whether the score corresponding to the target label is greater than a first threshold. If not, it returns to the steps of acquiring the pixel images and depth images of the target object using the depth information acquisition device. If yes, it determines the depth information of the target object within the target box based on the depth image and the target detection box position. The pixel images and depth information of the target object are then subjected to dimensionality reduction processing. Finally, the target object is reconstructed in three dimensions based on the target detection box position, object category, and the dimensionality-reduced pixel images and depth information to determine the target object's pose coordinates. The beneficial effect of this invention is that by acquiring pixel images of the target object using an image acquisition device and depth images of the target object using a depth information acquisition device, the depth information of the object does not need to be calculated using a stereo vision algorithm, thus reducing the computational load. Furthermore, dimensionality reduction processing was performed on both the pixel image and depth information. When subsequently using the pixel image and depth information to perform 3D reconstruction and pose detection of the target object, the dimensionality-reduced pixel image and depth information not only preserve the complete features of the target object but also significantly reduce the amount of data processing. Therefore, this scheme can effectively reduce the computational load of the object pose recognition process, lower the time complexity of the object pose recognition process, and improve detection efficiency.

[0064] In addition, the present invention also provides an object posture detection device, equipment and medium, with the same effect as above. Attached Figure Description

[0065] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 A flowchart of an object pose detection method provided in an embodiment of the present invention;

[0067] Figure 2 A schematic diagram illustrating the arrangement and detection of an image acquisition device and a depth information acquisition device according to an embodiment of the present invention;

[0068] Figure 3 This is a schematic diagram of an object posture detection device provided in an embodiment of the present invention;

[0069] Figure 4This is a schematic diagram of an object posture detection device provided in an embodiment of the present invention. Detailed Implementation

[0070] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0071] The core of this invention is to provide an object posture detection method, device, equipment, and medium to solve the shortcomings of the current object posture recognition process, which has high computational load and high time complexity.

[0072] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0073] With the development of technology and the improvement of living standards, express delivery has become increasingly common in people's lives. Specifically, in the process of transporting express packages from one city to another, they need to pass through transit stations, also known as sorting centers. These sorting centers further differentiate the packages based on their destination city, thus distributing them to different cities – this process is called express delivery sorting. Traditional express delivery sorting mainly relies on manual labor. Specifically, personnel use barcode scanners to scan the barcode information on the packages, converting it into easily identifiable destination information, and then placing the packages in different areas according to that information. This method is not only prone to sorting errors but also inefficient.

[0074] Therefore, automated parcel sorting methods have emerged that utilize robotic arms and visual recognition technology. Specifically, visual recognition technology determines the posture of objects, and then a robotic arm grasps and moves the objects based on their posture, thus achieving automated sorting. Accurately determining the posture of objects is the key to achieving automated parcel sorting.

[0075] Currently, object pose calculation is mainly achieved using binocular vision localization technology. Binocular vision localization requires matching images captured by two cameras to find corresponding pixels, and then calculating the object's depth information using stereo vision algorithms. However, this process involves a large amount of pixel matching and calculation, thus consuming significant computational resources and time, which is detrimental to the rapid identification and detection of object poses in scenarios such as express delivery sorting. Based on these problems, this invention provides an object pose detection method aimed at solving the problems of high computational load and high time complexity in current object pose recognition processes.

[0076] Figure 1 This is a flowchart illustrating an object pose detection method provided in an embodiment of the present invention. Figure 1 As shown, the method includes:

[0077] S10: Acquire the pixel image of the target object acquired by the image acquisition device, and acquire the depth image of the target object acquired by the depth information acquisition device.

[0078] Specifically, firstly, a pixel image of the target object is acquired by an image acquisition device, and then a depth image of the target object is acquired by a depth information acquisition device. In this embodiment, the type of image acquisition device is not limited; it can be a digital camera, camcorder, or scanner, etc. The depth information acquisition device is a device capable of acquiring depth images. The depth image contains distance information about objects in the scene. In this embodiment, the type of depth information acquisition device is not limited; it can be a depth camera, or it can be an acoustic ranging radar or a laser ranging radar.

[0079] It is important to note that in some embodiments, an RGB-D camera can simultaneously acquire pixel images and depth images of the target object. An RGB-D camera is a camera with red, green, and blue (RGB) and depth (D) sensors. It can simultaneously capture color pixel images and depth images. The RGB image provides color information of objects in the scene, while the depth image provides distance information of objects in the scene. By combining RGB and depth information, RGB-D cameras can achieve more accurate object detection, tracking, and 3D reconstruction applications. Furthermore, RGB-D cameras typically use technologies such as infrared light or structured light to acquire depth information. Infrared light or structured light emits a series of light rays, and the distance between the object and the camera is calculated by measuring the reflection time or deformation of the light. This technology can acquire the accurate 3D shape of the object without contact. Therefore, using an RGB-D camera to simultaneously acquire pixel images and depth images of the target object has the advantage of greater accuracy.

[0080] S11: Input the pixel image into the object detection network to obtain the object category, object detection box position, object label, and the score corresponding to the object label.

[0081] Furthermore, the pixel image is input into the object detection network to obtain the object category, object detection box position, object label, and the score corresponding to the object label. In this invention, object categories are divided into two types: rigid objects and flexible objects. It can be understood that in the scenario of express delivery posture detection, express boxes and parcels with regular shapes are rigid objects; express bags and parcels without regular shapes are flexible objects. In specific implementations, the information value corresponding to rigid objects is 0, and the information value corresponding to flexible objects is 1.

[0082] The target detection bounding box position refers to the location information of the target detection bounding box. It can be understood that in a pixel image, the pixels of a target object occupy a portion of all pixels in the image. The target detection bounding box selects the entire target object region, indicating its specific location within the pixel image. Correspondingly, the target detection bounding box position is the location information of the region selecting the entire target object. In this embodiment, the shape of the target detection bounding box is not limited; it can be a rectangle or a circle, depending on the specific implementation.

[0083] Target labels, also known as target detection box categories, require the identification of the specific type of the target object in the implementation process, and then assign a corresponding target label. The target label determines the specific type of the target object within the target detection box, such as: type A express box, type B express parcel, or type C express container. Correspondingly, the target label score represents the confidence level corresponding to the target label. For example, when the target label is type A express box, the target label score is the confidence level that the target object is a type A express box.

[0084] In this embodiment, there are no restrictions on the object category, target detection box position, target label, and the specific acquisition process of the target label; these are determined according to the specific implementation situation.

[0085] S12: Determine whether the score corresponding to the target label is greater than the first threshold. If not, return to step S10; if yes, proceed to step S13.

[0086] Determine if the score corresponding to the target label is greater than a first threshold. If the score is not greater than the first threshold, the confidence level of the target label is considered insufficient, indicating an error in target object recognition or failure to identify the target object. The target object's pose cannot be further determined, and the process returns to step S10 to re-acquire the pixel and depth images of the target object and re-evaluate. If the score is greater than the first threshold, the confidence level of the target label is considered sufficient, the target object is successfully identified, and subsequent pose detection can proceed.

[0087] It should be noted that the size of the first threshold is not limited in this embodiment, but depends on the specific implementation.

[0088] S13: Determine the depth information of the target object in the target box based on the depth image and the position of the target detection box.

[0089] Understandably, in a depth image, the depth information of the target object accounts for a portion of the total depth information, while pose detection only requires the depth information of the target object. Therefore, to obtain the depth information of the target object, the depth information of the target object within the bounding box is determined based on the depth image and the location of the target detection box.

[0090] S14: Perform dimensionality reduction processing on the pixel image and depth information of the target object.

[0091] Further dimensionality reduction processing is performed on the obtained pixel images and depth information of the target object. The purpose of dimensionality reduction is to reduce the amount of data, thereby reducing the computational load in subsequent pose detection and recognition calculations. It is important to note that the dimensionality reduction process must preserve the complete features of the target object in the pixel images and depth information to ensure the accuracy of target object pose recognition.

[0092] It should be noted that this embodiment does not limit the specific process of dimensionality reduction of the pixel image and depth information of the target object, and it depends on the specific implementation.

[0093] S15: Based on the target detection box position, object category, and the pixel image and depth information after dimensionality reduction, perform 3D reconstruction of the target object to determine the pose coordinates of the target object.

[0094] Finally, based on the detected object category and target detection box position, combined with the dimensionality-reduced pixel image and depth information, the target object is reconstructed in 3D. The reconstructed target object has a 3D model in the real coordinate system, thus accurately determining the target object's posture, facilitating the robotic arm's grasping of the target object. This embodiment does not limit the specific process of reconstructing the target object in 3D; it depends on the specific implementation.

[0095] In this embodiment, pixel images of the target object acquired by an image acquisition device and depth images of the target object acquired by a depth information acquisition device are obtained. The pixel images are input into a target detection network to obtain the object category, target detection box position, target label, and score corresponding to the target label. It is determined whether the score corresponding to the target label is greater than a first threshold. If not, the process returns to the steps of acquiring pixel images and depth images of the target object acquired by the depth information acquisition device. If yes, the depth information of the target object within the target box is determined based on the depth image and the target detection box position. Dimensionality reduction processing is performed on the pixel images and depth information of the target object. The target object is then reconstructed in three dimensions based on the target detection box position, object category, and the dimensionality-reduced pixel images and depth information to determine its pose coordinates. Therefore, this solution acquires pixel images of the target object using an image acquisition device and depth images using a depth information acquisition device, eliminating the need to calculate the object's depth information using a stereo vision algorithm, thus reducing computational load. Furthermore, dimensionality reduction processing was performed on both the pixel image and depth information. When subsequently using the pixel image and depth information to perform 3D reconstruction and pose detection of the target object, the dimensionality-reduced pixel image and depth information not only preserve the complete features of the target object but also significantly reduce the amount of data processing. Therefore, this scheme can effectively reduce the computational load of the object pose recognition process, lower the time complexity of the object pose recognition process, and improve detection efficiency.

[0096] Based on the above embodiments, in some embodiments, pixel images are input into an object detection network to obtain the object category, object detection box position, object label, and the score corresponding to the object label, including:

[0097] S111: Scale the pixel image proportionally.

[0098] S112: The proportionally scaled pixel image is fed into a three-layer convolutional neural network to obtain the initial feature map; each layer of the convolutional neural network contains a convolutional layer, a data batch normalization layer, and a hard tangent activation function.

[0099] S113: Determine the target recommendation region in the initial feature map based on the region generation network.

[0100] S114: Input the initial feature map and the target recommendation region into the region interest alignment network to obtain the target feature map.

[0101] S115: Input the target feature map into the head layer to obtain object category information and target detection region information.

[0102] S116: Input the object category information into two convolutional layers in sequence to obtain the object category.

[0103] S117: Input the target detection region information into the convolutional layer to obtain the target detection box position, target label, and the score corresponding to the target label.

[0104] To obtain the object category, bounding box location, object label, and corresponding score, the pixel image is first scaled proportionally. For example, if the original resolution of the pixel image is 1280*1080, it is scaled proportionally to 640*640. This reduces the amount of data and facilitates processing by the object detection network. During the scaling process, the aspect ratio of the pixel image should not be changed; that is, the longer side of the pixel image is scaled to 640, and the shorter side is filled with grayscale to achieve 640.

[0105] It is important to note that in order to ensure a one-to-one correspondence between the pixel information in the pixel image and the depth information in the depth image, the depth image must also be scaled proportionally to the pixel image. In practice, when using an RGB-D camera to simultaneously acquire pixel and depth images of a target object, the original image must first be obtained from the video stream captured by the RGB-D camera. The pixel and depth images in the original image are then separated, and then both are scaled proportionally to ensure the consistency of pixel and depth information.

[0106] Furthermore, the proportionally scaled pixel image is fed into a three-layer Convolutional Neural Network (CNN) to obtain the initial feature map. In the specific implementation, each CNN layer contains a 3x3 CNN convolutional layer, a batch normalization (BN) layer, and a hard swish activation function. The target recommendation region in the initial feature map is determined by the Region Proposal Network (RPN). The Region Proposal Network is a network used for object detection and tracking. Its main function is to generate candidate boxes in the image, i.e., to extract regions that may contain the target. The RPN network is a fully convolutional network that can simultaneously predict the object bounding box and object score at each location. It extracts candidate boxes of different sizes and scales on the image using a sliding window approach and predicts an object score for each candidate box to determine whether the candidate box contains the target. The RPN network architecture typically consists of two parallel convolutional layers: one for predicting the bounding box coordinates of the candidate boxes, and the other for predicting the object score of the candidate boxes. These predictions will be used to filter out candidate boxes with high object scores, which will then serve as input for subsequent object detection or object tracking.

[0107] The initial feature map and the target proposed region are simultaneously input into the Region of Interest Align (ROI Align) network to obtain the target feature map. It's important to note that the ROI Align network is a feature extraction method used in object detection tasks that addresses the truncation error problem inherent in Region of Interest Pooling (RoIPooling). In RoIPooling, Regions of Interest (ROIs) of different sizes are mapped onto a fixed-size feature map by dividing the ROI into a fixed-size grid and then selecting the maximum value in each grid to extract features. However, the mismatch between the size of the ROI and the size of the feature map leads to truncation error. ROI Align addresses this problem by using bilinear interpolation. It divides the ROI into smaller sub-regions and performs interpolation in each sub-region to obtain a more accurate feature representation. Specifically, ROI Align divides the ROI into smaller grids and performs bilinear interpolation in each grid to compute the feature value at each location. This preserves more spatial information and improves feature accuracy.

[0108] Furthermore, the target feature map is input into the head layer to obtain object category information and target detection region information. It's important to note that in this embodiment, the head layer has two branches: one for object category detection and the other for target detection computation. Specifically, the object category information is sequentially input into two 1x1 CNN convolutional layers, and the object category is obtained after passing through two 1x1 CNN layers. The target detection region information is input into the CNN convolutional layer to obtain the target detection box position, target label, and the corresponding score.

[0109] In this embodiment, scaling the pixel image proportionally reduces the amount of data and facilitates processing by the object detection network. Utilizing the RPN network to extract target recommendation regions from the initial feature map greatly simplifies the object detection and tracking process, enabling the model to automatically generate candidate boxes, reducing the workload of manually designing candidate boxes, and improving the accuracy and efficiency of detection and tracking. Simultaneously, using the ROI Align network to obtain target feature maps improves the performance of the object detection task, especially for small targets and targets with blurred boundaries. It can better capture the details and spatial information of the target, thereby improving the accuracy and stability of detection. Furthermore, in this embodiment, the head layer of the object detection network can branch the network in two ways: one branch for object category detection and the other for object detection computation, improving data processing efficiency.

[0110] Figure 2This is a schematic diagram illustrating the arrangement and detection of an image acquisition device and a depth information acquisition device according to an embodiment of the present invention. Based on the above embodiments, in some embodiments, such as... Figure 2 As shown, dimensionality reduction processing is performed on the pixel image and depth information of the target object, including:

[0111] S141: The point where the image acquisition centerline of the image acquisition device coincides with the target object is determined as the center point.

[0112] S142: Determine the spatial coordinates of the center point based on the pixel coordinates in the pixel image and the depth information value in the depth information.

[0113] S143: Determine three concentric circles with the center point as the center; wherein each concentric circle lies in the same plane.

[0114] S144: Generate six rays with the center point as the endpoint; the included angles between the rays are equal, and each ray intersects with each concentric circle.

[0115] S145: Obtain the pixel coordinates of the projection points generated by projecting each intersection point onto the target object based on the pixel coordinates in the pixel image.

[0116] S146: Determine the depth information value of each projection point based on the depth information.

[0117] Specifically, the depth information value of the projection point is the average value of the depth information values ​​of all points within a circular area centered on the projection point and with a preset value as the radius.

[0118] S147: Determine the spatial coordinates of each projection point based on the pixel coordinates and depth information of each projection point.

[0119] In practice, the center point is determined by the point where the image acquisition axis of the image acquisition device coincides with the target object. This means the image acquisition axis is a straight line extending from the image acquisition device and perpendicular to the image acquisition plane; the target object is typically placed on the image acquisition plane. Furthermore, to ensure consistency between the pixel image and the depth image, the image acquisition device and the depth information acquisition device are usually located in the same place (the RGB-D camera simultaneously acquires pixel and depth images).

[0120] Further, the spatial coordinates A(x) of the center point A are determined based on the pixel coordinates in the pixel image and the depth information value in the depth information. a ,y a ,d aThree concentric circles (c0, c1, c2) are defined with the center point as the center. It is important to note that all concentric circles lie in the same plane. In this embodiment, the radius of each concentric circle is not limited; for example, it can be determined based on the field of view of the image acquisition device. It is understood that the image acquisition device can acquire images within its field of view, but cannot acquire images outside its field of view. Therefore, the radius of the smallest concentric circle can be specifically set to one-quarter of the field of view of the image acquisition device, the radius of the middle concentric circle to one-half of the field of view, and the radius of the largest concentric circle to three-quarters of the field of view. In this way, the three concentric circles can essentially cover the entire field of view of the image acquisition device.

[0121] Then, six rays (l0, l1, l2, l3, l4, l5) are generated with the center point as the endpoint. It is important to note that the included angles between each ray are equal, and each ray intersects with each concentric circle. That is, the included angles between the six rays are 60°, and the six rays l0, l1, l2, l3, l4, l5 intersect with the three concentric circles c0, c1, c2 at a total of 18 points.

[0122] Furthermore, the pixel coordinates of the projection points generated by the projection of each intersection point onto the target object are obtained based on the pixel coordinates in the pixel image. It can be understood that since the plane containing the three concentric circles is perpendicular to the image acquisition axis, the pixel coordinates in the target object's pixel image correspond to the coordinates of each point on the plane containing the three concentric circles. In other words, the coordinates of the 18 intersection points generated by the six rays intersecting the three concentric circles can be determined based on the pixel coordinates in the pixel image. Correspondingly, the pixel coordinates of the 18 projection points generated by the projection of each intersection point onto the target object are:

[0123] The depth information value d of each projection point is determined based on the depth information. It should be noted that, in order to avoid damage to the surface of the target object or drift during the test, the depth information value of the projection point is specifically the average of the depth information values ​​of all points within a circular area centered on the projection point and with a preset value as the radius.

[0124] Finally, the spatial coordinates of each projection point are determined based on the pixel coordinates and depth information of each projection point:

[0125] In this embodiment, the spatial coordinate values ​​corresponding to each projection point obtained through the above process are the dimensionality-reduced pixel image and depth information, which include the pixel coordinate information and depth information of the target object. This avoids using all the pixel coordinate information and depth information of the target object for subsequent 3D reconstruction, greatly reducing the amount of computation and improving processing efficiency.

[0126] Based on the above embodiments, in some embodiments, the target object is reconstructed in three dimensions according to the target detection box position, object category, and the dimension-reduced pixel image and depth information to determine the pose coordinates of the target object, including:

[0127] S151: Determine the boundary point coordinates of the target object based on the ground plane coordinates and the boundary points of the target detection box position.

[0128] S152: Determine the plane equation or surface equation of the target object based on the object category and the pixel image and depth information after dimensionality reduction.

[0129] S153: Based on the boundary point coordinates of the target object and the plane equation or surface equation of the target object, perform three-dimensional reconstruction of the target object to generate a three-dimensional model of the target object.

[0130] S154: Generate the attitude coordinates of the target object in the real coordinate system based on the 3D model of the target object.

[0131] To determine the pose coordinates of the target object, this embodiment specifically determines the boundary point coordinates of the target object based on the ground plane coordinates and the boundary points of the target detection box, i.e., determining the boundary of the target object. Further, the planar equation or surface equation of the target object is determined based on the object category, the dimensionality-reduced pixel image, and depth information. It is understood that the planar equation or surface equation of the target object represents the shape expression of the target object, and based on this expression, a 3D reconstruction of the target object can be achieved. This embodiment does not limit the specific process for determining the planar equation or surface equation of the target object; it depends on the specific implementation.

[0132] Furthermore, based on the boundary point coordinates of the target object and its planar or surface equations, a 3D reconstruction of the target object is performed to generate a 3D model. Finally, the target object's attitude coordinates in the real coordinate system are generated based on its 3D model, thus achieving attitude detection of the target object.

[0133] In this embodiment, the plane equation or surface equation of the target object is determined by using the pixel image and depth information after dimensionality reduction, and the three-dimensional reconstruction of the target object is achieved by using the plane equation or surface equation, which greatly reduces the amount of data for pixel-by-pixel calculation and improves the running efficiency.

[0134] Based on the above embodiments, in some embodiments, the plane equation or surface equation of the target object is determined according to the object category and the dimension-reduced pixel image and depth information, including:

[0135] S161: When the target object is determined to be a rigid object based on the object category, determine whether the included angle between two adjacent projection points corresponding to the same ray is less than the second threshold based on the spatial coordinates of the center point and the spatial coordinates of each projection point; if yes, proceed to step S162; if no, proceed to step S166.

[0136] S162: Select the midpoint between the two projection points.

[0137] S163: Determine additional points beyond the two adjacent projection points based on the intermediate point, and obtain the spatial coordinate values ​​of the additional points according to the pixel coordinates and depth information in the pixel image.

[0138] The additional point is not on the projection of the ray corresponding to the two adjacent projection points.

[0139] S164: Fit the plane equation based on the spatial coordinates of the additional point and the spatial coordinates of the two adjacent projection points.

[0140] S165: Adapt the remaining projection points according to the plane equation.

[0141] S166: Select a new projection point between two adjacent projection points according to the bisection method.

[0142] The new projection point corresponds to the same ray as the two adjacent projection points.

[0143] S167: Obtain the spatial coordinates of the new projection point based on the pixel coordinates and depth information in the pixel image, and determine again whether the angle between two adjacent projection points corresponding to the same ray is less than the second threshold; if yes, return to step S162; if no, proceed to step S168.

[0144] S168: Reduce the radius of each concentric circle to redetermine the spatial coordinates of each projection point. Return to step S161.

[0145] In practical implementation, since the target object, being a rigid object (such as a parcel box), has a planar structure, it is necessary to obtain its planar equation. Specifically, when the target object is determined to be a rigid object based on its object category, the spatial coordinates of the center point and the spatial coordinates of each projection point are used to determine whether the angle between two adjacent projection points corresponding to the same ray is less than a second threshold. It should be noted that in this embodiment, the size of the second threshold is not limited and depends on the specific implementation. An example is given below:

[0146]

[0147] Where θ0 is the projection point and projection point The angle between them From center point A to projection point The vector, For projection point to projection point The vector. It can be understood that when the angle θ0 in the above formula is less than the second threshold, then the center point A and the projection point are considered... and projection point If the projections are in the same plane, they are considered to be in different planes. The angle between two adjacent projection points corresponding to other rays can be determined using the formula above.

[0148] If the angle between two adjacent projection points corresponding to the same ray is determined to be less than the second threshold, then the midpoint between the two projection points is selected. and projection point For example, projection point That is, the center point A and the projection point. and projection point The midpoint among the three. Based on this midpoint, determine the additional point N, which is not adjacent to the two projection points. It is important to note that the additional point is not on the projection of the rays corresponding to the two adjacent projection points; that is, the additional point N and the projection point... and projection point The two do not correspond to the same ray. Further, the spatial coordinates N(x) of the additional point N are obtained based on the pixel coordinates and depth information in the pixel image. n ,y n ,d n Finally, based on the planar calculation formula, combined with the spatial coordinates N(x) of the additional point N, n ,y n ,d n ) and two adjacent projection points and spatial coordinates and Fit the plane equation. Once the plane is determined, the remaining projection points can be fitted according to the plane equation; if there are projection points on the plane equation, there is no need to calculate the corresponding plane equation separately for those projection points.

[0149] If the angle between two adjacent projection points corresponding to the same ray is determined to be no less than the second threshold, then a new projection point is selected between the two adjacent projection points according to the bisection method; wherein the new projection point corresponds to the same ray as the two adjacent projection points. Using the projection point... and projection point For example, according to the bisection method at the projection point and projection point Select from the options to obtain a new projection point. Further, new projection points are obtained based on the pixel coordinates and depth information in the pixel image. spatial coordinates Next, determine whether the angle between two adjacent projection points corresponding to the same ray is less than the second threshold. It is important to note that the two adjacent projection points selected at this point should be projection points... and new projection points

[0150] If the angle between two adjacent projection points corresponding to the same ray is determined to be less than the second threshold, then the midpoint is selected between the two projection points, and subsequent steps are performed. If the angle between two adjacent projection points corresponding to the same ray is determined to be not less than the second threshold, it indicates that the radius of the concentric circles was set too large during the dimensionality reduction processing of the pixel image and depth information. Therefore, it is necessary to reduce the radius of each concentric circle to redetermine the spatial coordinate values ​​corresponding to each projection point. After redetermining the spatial coordinate values ​​corresponding to each projection point, the process returns to the step of determining whether the angle between two adjacent projection points corresponding to the same ray is less than the second threshold based on the spatial coordinate values ​​of the center point and the spatial coordinate values ​​of each projection point. This process is repeated until all the plane equations of the target object, which is a rigid object, are obtained.

[0151] In this embodiment, after confirming that the target object is a rigid object, the shape expression of the target object is confirmed by calculating the plane equation of the target object, so as to facilitate its subsequent three-dimensional reconstruction.

[0152] Based on the above embodiments, in some embodiments, the plane equation or surface equation of the target object is determined according to the object category and the dimension-reduced pixel image and depth information, including:

[0153] S171: When the target object is determined to be a flexible object based on the object category, the surface equation is fitted based on the spatial coordinates of the center point and the spatial coordinates of each projection point.

[0154] In practical implementation, since the target object, which is a flexible object (such as a courier bag), has a curved surface structure, it is necessary to obtain its surface equation. Specifically, when the target object is determined to be a flexible object based on its object category, the surface equation is fitted based on the spatial coordinates of the center point and the spatial coordinates of each projection point.

[0155] In this embodiment, after confirming that the target object is a flexible object, the surface equation of the target object is calculated to determine its shape expression, which facilitates subsequent 3D reconstruction. Furthermore, since the shape expression of the target object is determined only by the spatial coordinates of the center point and the spatial coordinates of each projection point, the attitude calculation for flexible objects is simplified.

[0156] In practical implementation, for example, a hexahedron like a parcel box, a maximum of three planes can be seen when viewed from one direction. Therefore, to avoid the increased computational load caused by the intersection of multiple planes, based on the above embodiments, in some embodiments, after fitting the plane equation according to the spatial coordinates of the additional point and the spatial coordinates of two adjacent projection points, the following is also included:

[0157] S181: Obtain the number of plane equations and determine whether the number of plane equations is greater than 3; if so, proceed to step S182.

[0158] S182: Obtain the similarity between the equations of each plane.

[0159] S183: The plane equation corresponding to a similarity greater than the third threshold.

[0160] Specifically, the number of obtained planar equations is determined, and it is judged whether the number of planar equations is greater than 3. If the number of planar equations is not greater than 3, the process ends. If the number of planar equations is greater than 3, it is considered that there are redundant planar equations, and it is necessary to further obtain the similarity between the planar equations. After obtaining the similarity between the planar equations, the planar equations with similarity greater than the third threshold are fitted, thereby ensuring that the number of planar equations is not greater than 3.

[0161] It should be noted that the size of the third threshold is not limited in this embodiment and depends on the specific implementation.

[0162] In this embodiment, after obtaining the planar equation of the rigid object, based on the characteristics of the rigid object (box-like hexahedron), namely that a single vision can observe at most three planes at a time, the multiple planes are fitted to ensure that the number of planar equations is no more than 3, thus avoiding the increase in computational load caused by the intersection of multiple planes.

[0163] Furthermore, in practical implementation, multiple sets of image acquisition devices and depth information acquisition devices can be set up to simultaneously acquire pixel images and depth images of the target object to enhance the accuracy of pose detection. In this case, to ensure the smooth progress of the subsequent pose detection process for the target object, it is necessary to perform fitting processing on the plane equations or surface equations obtained based on the data acquired by multiple acquisition devices. An example is given below:

[0164] Assuming three RGB-D cameras (α, β, γ) are set up to simultaneously acquire pixel images and depth images of the target object, then the plane equation or surface equation obtained based on the data acquired by the three RGB-D cameras needs to be fitted:

[0165] P = 0.4C α +0.3C β +0.3C γ ;

[0166] Where P is the final fitted plane equation or surface equation, and C α For the plane equation or surface equation obtained based on the data acquired by camera α, C β C is the equation of a plane or surface obtained based on the data acquired by camera β. γ The equations of a plane or a surface are derived from the data collected by the camera γ.

[0167] It is understandable that, compared to cameras β and γ, camera α has a better field of view of the target object, so camera α should have a greater weight than cameras β and γ.

[0168] In this way, by setting up three RGB-D cameras, the errors caused by single-camera detection are avoided, and different weighting coefficients are set according to the position of the cameras, which increases the stability of the system.

[0169] In the above embodiments, the object posture detection method has been described in detail. The present invention also provides embodiments of the object posture detection device.

[0170] Figure 3 This is a schematic diagram of an object posture detection device provided in an embodiment of the present invention. Figure 3 As shown, the device includes:

[0171] The first acquisition module 10 is used to acquire pixel images of the target object acquired by the image acquisition device and depth images of the target object acquired by the depth information acquisition device.

[0172] The second acquisition module 11 is used to input the pixel image into the object detection network to obtain the object category, object detection box position, object label and the score corresponding to the object label.

[0173] The judgment module 12 is used to determine whether the score corresponding to the target label is greater than the first threshold; if not, the first acquisition module 10 is triggered; if yes, the first determination module 13 is triggered.

[0174] The first determining module 13 is used to determine the depth information of the target object in the target box based on the depth image and the position of the target detection box.

[0175] Dimensionality reduction module 14 is used to perform dimensionality reduction processing on the pixel image and depth information of the target object.

[0176] The second determining module 15 is used to perform three-dimensional reconstruction of the target object based on the target detection box position, object category, and pixel image and depth information after dimensionality reduction, so as to determine the pose coordinates of the target object.

[0177] In some embodiments, the second acquisition module 11 includes:

[0178] The proportional scaling module is used to proportionally scale pixel images;

[0179] The first input submodule is used to feed the proportionally scaled pixel image into a three-layer convolutional neural network to obtain an initial feature map; wherein each layer of the convolutional neural network contains a convolutional layer, a data batch normalization layer, and a hard tangent activation function.

[0180] The first determination submodule is used to determine the target recommendation region in the initial feature map based on the region generation network;

[0181] The second input submodule is used to input the initial feature map and the target recommendation region into the region interest alignment network to obtain the target feature map;

[0182] The third input submodule is used to input the target feature map into the head layer to obtain object category information and target detection region information;

[0183] The fourth input submodule is used to sequentially input object category information into two convolutional layers to obtain the object category;

[0184] The fifth input submodule is used to input the target detection region information into the convolutional layer to obtain the target detection box position, target label, and the score corresponding to the target label.

[0185] In some embodiments, the dimensionality reduction module 14 includes:

[0186] The second determining submodule is used to determine the point where the image acquisition centerline of the image acquisition device coincides with the target object as the center point;

[0187] The third determination submodule is used to determine the spatial coordinates of the center point based on the pixel coordinates in the pixel image and the depth information value in the depth information.

[0188] The fourth submodule is used to determine three concentric circles with the center point as the center; wherein each concentric circle lies in the same plane;

[0189] The first generation submodule is used to generate six rays with the center point as the endpoint; wherein the included angle between each ray is equal, and each ray intersects with each concentric circle;

[0190] The first acquisition submodule is used to obtain the pixel coordinate values ​​of the projection points generated by the projection of each intersection point onto the target object according to the pixel coordinates in the pixel image;

[0191] The fifth determination submodule is used to determine the depth information value of each projection point based on the depth information; wherein, the depth information value of the projection point is specifically the average value of the depth information values ​​of each point in a circular area with the projection point as the center and a preset value as the radius.

[0192] The sixth determination submodule is used to determine the spatial coordinates of each projection point based on the pixel coordinates and depth information of each projection point.

[0193] In some embodiments, the second determining module 15 includes:

[0194] The seventh determination submodule is used to determine the boundary point coordinates of the target object based on the ground plane coordinates and the boundary points of the target detection box position;

[0195] The eighth determination submodule is used to determine the plane equation or surface equation of the target object based on the object category and the pixel image and depth information after dimensionality reduction.

[0196] The 3D reconstruction submodule is used to reconstruct the target object in 3D based on the boundary point coordinates of the target object and the plane equation or surface equation of the target object, so as to generate a 3D model of the target object.

[0197] The second generation submodule is used to generate the pose coordinates of the target object in the real coordinate system based on the 3D model of the target object.

[0198] In some embodiments, the eighth determining submodule includes:

[0199] The first judgment submodule is used to determine whether the angle between two adjacent projection points corresponding to the same ray is less than a second threshold when the target object is determined to be a rigid object based on the spatial coordinates of the center point and the spatial coordinates of each projection point. If yes, the first selection submodule is triggered; if no, the second selection submodule is triggered.

[0200] The first selection submodule is used to select the midpoint between two projection points;

[0201] The second acquisition submodule is used to determine additional points other than the two adjacent projection points based on the intermediate point, and to obtain the spatial coordinate values ​​of the additional points according to the pixel coordinates and depth information in the pixel image.

[0202] The additional point is not on the projection of the ray corresponding to the two adjacent projection points;

[0203] The first fitting submodule is used to fit the plane equation based on the spatial coordinates of the additional point and the spatial coordinates of two adjacent projection points.

[0204] The adaptation submodule is used to adapt the remaining projection points according to the plane equation;

[0205] The second selection submodule is used to select a new projection point between two adjacent projection points according to the bisection method.

[0206] In this case, the new projection point corresponds to the same ray as the two adjacent projection points;

[0207] The third acquisition submodule is used to obtain the spatial coordinates of the new projection point based on the pixel coordinates and depth information in the pixel image, and then trigger the first judgment submodule again; if yes, trigger the first selection submodule; if no, trigger the zoom-out module.

[0208] The shrinking module is used to shrink the radius of each concentric circle in order to redetermine the spatial coordinate values ​​corresponding to each projection point. This triggers the first judgment submodule again.

[0209] In some embodiments, the eighth determining submodule includes:

[0210] The second fitting submodule is used to fit the surface equation based on the spatial coordinates of the center point and the spatial coordinates of each projection point when the target object is determined to be a flexible object according to the object category.

[0211] In some embodiments, it also includes:

[0212] The second judgment submodule is used to obtain the number of plane equations and determine whether the number of plane equations is greater than 3; if so, the fourth acquisition submodule is triggered.

[0213] The fourth submodule is used to obtain the similarity between the equations of each plane;

[0214] The third fitting submodule is used to fit the plane equations corresponding to similarities greater than the third threshold.

[0215] In this embodiment, the object pose detection device includes a first acquisition module, a second acquisition module, a judgment module, a first determination module, a dimensionality reduction module, and a second determination module. The object pose detection device can implement all the steps of the above-described object pose detection method during operation. It acquires pixel images of the target object from an image acquisition device and depth images of the target object from a depth information acquisition device; inputs the pixel images into a target detection network to obtain the object category, target detection box position, target label, and score corresponding to the target label; determines whether the score corresponding to the target label is greater than a first threshold; if not, it returns to the steps of acquiring pixel images of the target object and acquiring depth images of the target object from the depth information acquisition device; if so, it determines the depth information of the target object in the target box based on the depth image and the target detection box position; performs dimensionality reduction processing on the pixel images and depth information of the target object; and performs three-dimensional reconstruction of the target object based on the target detection box position, object category, and the dimensionality-reduced pixel images and depth information to determine the pose coordinates of the target object. Therefore, this scheme acquires pixel images of the target object using an image acquisition device and depth images using a depth information acquisition device, eliminating the need to calculate the object's depth information using stereo vision algorithms, thus reducing computational load. Furthermore, dimensionality reduction processing is applied to both the pixel images and depth information. When subsequently using these images for 3D reconstruction and pose detection, the dimensionality-reduced images and depth information retain the complete features of the target object while significantly reducing data processing volume. Therefore, this scheme effectively reduces the computational load of the object pose recognition process, lowers its time complexity, and improves detection efficiency.

[0216] Figure 4 This is a schematic diagram of an object posture detection device provided in an embodiment of the present invention. Figure 4 As shown, the object pose detection device includes:

[0217] Memory 20 is used to store computer programs.

[0218] The processor 21 is used to implement the steps of the object pose detection method mentioned in the above embodiments when executing a computer program.

[0219] The object posture detection device provided in this embodiment may include, but is not limited to, smartphones, tablets, laptops, or desktop computers.

[0220] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an Artificial Intelligence (AI) processor, which handles computational operations related to machine learning.

[0221] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 20 is used to store at least the following computer program 201, which, after being loaded and executed by the processor 21, is capable of implementing the relevant steps of the object pose detection method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, and the storage method may be temporary or permanent storage. The operating system 202 may include Windows, Unix, Linux, etc. The data 203 may include, but is not limited to, the data involved in the object pose detection method.

[0222] In some embodiments, the object posture detection device may further include a display screen 22, an input / output interface 23, a communication interface 24, a power supply 25, and a communication bus 26.

[0223] Those skilled in the art will understand that Figure 4 The structure shown does not constitute a limitation on the object pose detection device and may include more or fewer components than shown.

[0224] In this embodiment, the object pose detection device includes a memory and a processor. The memory stores a computer program. The processor executes the computer program to implement the steps of the object pose detection method mentioned in the above embodiment. The method involves acquiring pixel images of the target object from an image acquisition device and depth images of the target object from a depth information acquisition device. The pixel images are input into a target detection network to obtain the object category, target detection box position, target label, and score corresponding to the target label. It is then determined whether the score corresponding to the target label is greater than a first threshold. If not, the process returns to the steps of acquiring pixel images of the target object and depth images of the target object from the depth information acquisition device. If yes, the depth information of the target object in the target box is determined based on the depth image and the target detection box position. Dimensionality reduction processing is performed on the pixel images and depth information of the target object. Based on the target detection box position, object category, and the dimensionality-reduced pixel images and depth information, the target object is reconstructed in three dimensions to determine its pose coordinates. Therefore, this solution acquires pixel images of the target object using an image acquisition device and depth images of the target object using a depth information acquisition device, eliminating the need to calculate the object's depth information using a stereo vision algorithm, thus reducing computational load. Furthermore, dimensionality reduction processing was performed on both the pixel image and depth information. When subsequently using the pixel image and depth information to perform 3D reconstruction and pose detection of the target object, the dimensionality-reduced pixel image and depth information not only preserve the complete features of the target object but also significantly reduce the amount of data processing. Therefore, this scheme can effectively reduce the computational load of the object pose recognition process, lower the time complexity of the object pose recognition process, and improve detection efficiency.

[0225] Finally, the present invention also provides an embodiment corresponding to a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, performs the steps described in the above method embodiments.

[0226] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0227] In this embodiment, a computer program is stored on a computer-readable storage medium. When the computer program is executed by a processor, it implements the steps described in the above method embodiment. The process involves acquiring pixel images of the target object from an image acquisition device and depth images of the target object from a depth information acquisition device. The pixel images are input into a target detection network to obtain the object category, target detection box position, target label, and score corresponding to the target label. It is then determined whether the score corresponding to the target label is greater than a first threshold. If not, the process returns to the steps of acquiring pixel images of the target object and depth images of the target object from the depth information acquisition device. If yes, the depth information of the target object in the target box is determined based on the depth image and the target detection box position. Dimensionality reduction processing is performed on the pixel images and depth information of the target object. Based on the target detection box position, object category, and the dimensionality-reduced pixel images and depth information, the target object is reconstructed in three dimensions to determine its pose coordinates. Therefore, this solution acquires pixel images of the target object using an image acquisition device and depth images of the target object using a depth information acquisition device, eliminating the need to calculate the object's depth information using a stereo vision algorithm, thus reducing computational load. Furthermore, dimensionality reduction processing was performed on both the pixel image and depth information. When subsequently using the pixel image and depth information to perform 3D reconstruction and pose detection of the target object, the dimensionality-reduced pixel image and depth information not only preserve the complete features of the target object but also significantly reduce the amount of data processing. Therefore, this scheme can effectively reduce the computational load of the object pose recognition process, lower the time complexity of the object pose recognition process, and improve detection efficiency.

[0228] The foregoing has provided a detailed description of the object posture detection method, apparatus, device, and medium provided by the present invention. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make several improvements and modifications to the present invention without departing from the principles of the invention, and these improvements and modifications also fall within the protection scope of the present invention.

[0229] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for detecting the pose of an object, characterized in that, include: Acquire pixel images of the target object captured by the image acquisition device, and acquire depth images of the target object captured by the depth information acquisition device; The pixel image is input into the object detection network to obtain the object category, object detection box position, object label, and the score corresponding to the object label; Determine whether the score corresponding to the target label is greater than a first threshold; If not, return to the step of obtaining the pixel image of the target object and obtaining the depth image of the target object acquired by the depth information acquisition device; If so, the depth information of the target object in the target box is determined based on the depth image and the position of the target detection box; Dimensionality reduction processing is performed on the pixel image and depth information of the target object; The target object is reconstructed in three dimensions based on the target detection box position, the object category, the pixel image after dimensionality reduction, and the depth information to determine the pose coordinates of the target object; The dimensionality reduction processing of the pixel image and depth information of the target object includes: The point where the image acquisition centerline of the image acquisition device coincides with the point where the target object is located is determined as the center point; The spatial coordinates of the center point are determined based on the pixel coordinates in the pixel image and the depth information value in the depth information. Three concentric circles are defined with the center point as the center; wherein each of the concentric circles lies in the same plane; Six rays are generated with the center point as the endpoint; wherein the included angles between each ray are equal, and each ray intersects with each of the concentric circles; Based on the pixel coordinates in the pixel image, obtain the pixel coordinate values ​​of the projection points generated by projecting each intersection point onto the target object; The depth information value of each projection point is determined based on the depth information; wherein, the depth information value of each projection point is specifically the average value of the depth information values ​​of each point within a circular area centered on the projection point and with a preset value as the radius. The spatial coordinates of each projection point are determined based on the pixel coordinates and depth information values ​​corresponding to each projection point.

2. The object pose detection method according to claim 1, characterized in that, The step of inputting the pixel image into the object detection network to obtain the object category, object detection box position, object label, and the score corresponding to the object label includes: The pixel image is scaled proportionally. The pixel image, after being scaled proportionally, is placed into a three-layer convolutional neural network to obtain an initial feature map; wherein each layer of the convolutional neural network contains a convolutional layer, a data batch normalization layer, and a hard tangent activation function; The target recommendation region in the initial feature map is determined based on the region generation network; The initial feature map and the target recommendation region are input into the region interest alignment network to obtain the target feature map; The target feature map is input into the head layer to obtain object category information and target detection region information; The object category information is sequentially input into two convolutional layers to obtain the object category; The target detection region information is input into the convolutional layer to obtain the target detection box position, the target label, and the score corresponding to the target label.

3. The object pose detection method according to claim 1, characterized in that, The step of performing 3D reconstruction of the target object based on the target detection box position, the object category, the dimensionally reduced pixel image, and the depth information to determine the pose coordinates of the target object includes: The boundary point coordinates of the target object are determined based on the ground plane coordinates and the boundary points of the target detection box position; The plane equation or surface equation of the target object is determined based on the object category, the pixel image after dimensionality reduction, and the depth information. Based on the boundary point coordinates of the target object and the plane equation or surface equation of the target object, the target object is reconstructed in three dimensions to generate a three-dimensional model of the target object. Generate the pose coordinates of the target object in the real coordinate system based on the three-dimensional model of the target object.

4. The object pose detection method according to claim 3, characterized in that, Determining the plane equation or surface equation of the target object based on the object category, the dimension-reduced pixel image, and the depth information includes: When the target object is determined to be a rigid object according to the object category, the angle between two adjacent projection points corresponding to the same ray is determined to be less than a second threshold based on the spatial coordinates of the center point and the spatial coordinates of each projection point. If it is determined that the angle between two adjacent projection points corresponding to the same ray is less than the second threshold, then the midpoint is selected between the two projection points; Based on the intermediate point, additional points other than the two adjacent projection points are determined, and the spatial coordinate values ​​of the additional points are obtained according to the pixel coordinates in the pixel image and the depth information; wherein, the additional points are not on the projection of the rays corresponding to the two adjacent projection points; A plane equation is fitted based on the spatial coordinates of the additional point and the spatial coordinates of two adjacent projection points; The remaining projection points are adapted according to the plane equation; If it is determined that the included angle between two adjacent projection points corresponding to the same ray is not less than the second threshold, then a new projection point is selected between the two adjacent projection points according to the bisection method; wherein, the new projection point corresponds to the same ray as the two adjacent projection points. The spatial coordinates of the new projection point are obtained based on the pixel coordinates in the pixel image and the depth information, and it is determined again whether the angle between two adjacent projection points corresponding to the same ray is less than the second threshold. If it is determined that the angle between two adjacent projection points corresponding to the same ray is less than the second threshold, then proceed to the step of selecting the midpoint between the two projection points; If it is determined that the included angle between two adjacent projection points corresponding to the same ray is not less than the second threshold, then the radius of each concentric circle is reduced to redetermine the spatial coordinate value corresponding to each projection point; Return to the step of determining whether the included angle between two adjacent projection points corresponding to the same ray is less than a second threshold based on the spatial coordinates of the center point and the spatial coordinates of each projection point.

5. The object pose detection method according to claim 3, characterized in that, Determining the plane equation or surface equation of the target object based on the object category, the dimension-reduced pixel image, and the depth information includes: When the target object is determined to be a flexible object based on the object category, a surface equation is fitted based on the spatial coordinates of the center point and the spatial coordinates of each projection point.

6. The object pose detection method according to claim 4, characterized in that, After fitting the plane equation based on the spatial coordinates of the additional point and the spatial coordinates of two adjacent projection points, the method further includes: Obtain the number of plane equations and determine whether the number of plane equations is greater than 3; If so, then obtain the similarity between each of the plane equations; Fit the plane equation corresponding to the similarity being greater than the third threshold.

7. An object posture detection device, characterized in that, include: The first acquisition module is used to acquire pixel images of the target object acquired by the image acquisition device and depth images of the target object acquired by the depth information acquisition device. The second acquisition module is used to input the pixel image into the target detection network to obtain the object category, target detection box position, target label, and the score corresponding to the target label of the target object; The judgment module is used to determine whether the score corresponding to the target label is greater than a first threshold. If not, the first acquisition module is triggered; if yes, the first determination module is triggered. The first determining module is used to determine the depth information of the target object in the target box based on the depth image and the position of the target detection box; The dimensionality reduction module is used to perform dimensionality reduction processing on the pixel image and the depth information of the target object; The second determining module is used to perform three-dimensional reconstruction of the target object based on the target detection box position, the object category, the dimension-reduced pixel image, and the depth information, so as to determine the pose coordinates of the target object; The dimensionality reduction module includes: The second determining submodule is used to determine the point where the image acquisition centerline of the image acquisition device and the target object coincide as the center point; The third determining submodule is used to determine the spatial coordinates of the center point based on the pixel coordinates in the pixel image and the depth information value in the depth information. The fourth determining submodule is used to determine three concentric circles with the center point as the center; wherein each of the concentric circles is in the same plane; The first generation submodule is used to generate six rays with the center point as the endpoint; wherein the included angle between each ray is equal, and each ray intersects with each concentric circle; The first acquisition submodule is used to acquire the pixel coordinate values ​​of the projection points generated by projecting each intersection point onto the target object according to the pixel coordinates in the pixel image; The fifth determining submodule is used to determine the depth information value of each projection point based on the depth information; wherein, the depth information value of the projection point is specifically the average value of the depth information values ​​of each point in a circular area formed by the projection point as the center and a preset value as the radius; The sixth determining submodule is used to determine the spatial coordinate value corresponding to each projection point based on the pixel coordinate value and the depth information value corresponding to each projection point.

8. An object posture detection device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the object pose detection method as described in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the object pose detection method as described in any one of claims 1 to 6.

Images