Point cloud segmentation method, apparatus, device and storage medium for target object

Abstract

This disclosure provides a method, apparatus, device, and storage medium for point cloud segmentation of a target object. It aims to improve the accuracy of point cloud segmentation. The method includes: acquiring point cloud data and an RGB image containing the target object during crane lifting; extracting contours from the RGB image to obtain various contours; for any contour, obtaining a current feature vector of the contour based on the position of each pixel and the RGB value of each pixel; determining the target contour of the target object using the current feature vectors of each contour and the standard contour feature vector of the target object; determining the three-dimensional position coordinates of the target object's center point in a radar coordinate system based on the target contour; determining the circumscribed sphere radius of the target object based on the extreme values ​​of the horizontal and vertical position coordinates of the target contour; and segmenting the point cloud data containing the target object using the three-dimensional position coordinates of the target object's center point in the radar coordinate system and the circumscribed sphere radius to obtain the point cloud data of the target object.

Description

Technical Field

[0001] This invention relates to the field of data processing, and in particular to a point cloud segmentation method, apparatus, device, and storage medium for a target object. Background Technology

[0002] LiDAR and computer vision technologies provide precise perception and measurement capabilities, and their application in crane lifting can significantly improve the intelligence level of lifting operations. LiDAR and RGB cameras can acquire real-time 3D point cloud information and 2D image information of the lifting operation scene. In the complex 3D point cloud of the lifting scene, real-time and accurate segmentation of the target object point cloud cluster is the prerequisite and key to realizing the positioning of the load (the object lifted during crane lifting operations) and the real-time monitoring and tracking of the load's posture during the lifting operation.

[0003] Existing real-time target detection and localization methods based on LiDAR and camera fusion fall into two categories. One involves using deep learning-based methods to obtain 2D target bounding boxes from 2D images. Then, using extrinsic parameters calibrated between the LiDAR and camera, point cloud data corresponding to the objects falling within these 2D boxes is extracted. This point cloud data is then clustered, segmented, and filtered to obtain the point cloud data for target localization. Alternatively, deep learning methods can be used directly to perform 3D target detection on the point cloud data, obtaining 3D target bounding boxes. The point cloud data within these boxes is then clustered and segmented to obtain the point cloud data for the target. However, this approach requires sample labeling and training on the target before the lifting operation to obtain the target detection network model. In actual lifting operations, the type and shape of each load may differ. Therefore, single-class sample training cannot cover all loads, making it impossible to automatically complete the detection and segmentation of all loads using a pre-trained single network model. Another approach involves segmenting by setting distance or density parameters. However, this method is sensitive to noise and parameter values; noise near the target object or improper parameter settings directly affect the accuracy of point cloud segmentation. Summary of the Invention

[0004] The exemplary embodiments of this disclosure provide a point cloud segmentation method, apparatus, device, and storage medium for target objects, which are used to improve the accuracy of point cloud segmentation of target objects.

[0005] A first aspect of this disclosure provides a point cloud segmentation method for a target object, the method comprising:

[0006] During the crane hoisting process, point cloud data and RGB images containing the target object are acquired for a specified duration, wherein the target object is the object that the crane needs to lift;

[0007] Contour extraction is performed on the RGB image to obtain each contour in the RGB image;

[0008] For any given contour, the current feature vector of the contour is obtained based on the position of each pixel within the contour and the RGB value of each pixel.

[0009] The target contour of the target object is determined by using the current feature vector of each contour and the standard contour feature vector of the target object.

[0010] Based on the target outline of the target object, determine the three-dimensional position coordinates of the center point of the target object in the radar coordinate system;

[0011] Based on the extreme values ​​of the horizontal and vertical coordinates in the target contour and the camera intrinsic parameters, the radius of the circumscribed sphere of the target object is determined.

[0012] The point cloud data containing the target object is segmented using the three-dimensional position coordinates of the center point of the target object in the radar coordinate system and the radius of the circumscribed sphere of the target object, to obtain the point cloud data corresponding to the target object.

[0013] In this embodiment, the standard contour feature vector of the target object is matched with each contour in the image to determine the contour of the target object in the image. Furthermore, based on the contour of the target object, the three-dimensional position coordinates of the target object's center point in the radar coordinate system and the radius of the target object's circumsphere are determined. Finally, the three-dimensional position coordinates of the target object's center point in the radar coordinate system and the radius of the target object's circumsphere are used to segment the point cloud data containing the target object, obtaining point cloud data corresponding to the target object. Therefore, in this embodiment, the standard contour feature vector of the target object is used to segment the target object. Based on the standard contour feature vector of the target object, the shape and color of the target object can be accurately identified. It does not require training a target object detection model, nor does it require setting distance or density parameters for clustering segmentation, thus improving the robustness and accuracy of point cloud segmentation of the target object.

[0014] A second aspect of this disclosure provides a point cloud segmentation apparatus for a target object, the apparatus comprising:

[0015] The acquisition module is used to acquire point cloud data and RGB images containing a target object corresponding to a specified time period during the crane hoisting process, wherein the target object is the object that the crane needs to lift;

[0016] The contour extraction module is used to extract contours from the RGB image to obtain each contour in the RGB image;

[0017] The contour vector determination module is used to obtain the current feature vector of any contour based on the position of each pixel in the contour and the RGB value of each pixel.

[0018] The matching module is used to determine the target contour of the target object among the contours by using the current feature vector of each contour and the standard contour feature vector of the target object.

[0019] The center point position determination module is used to determine the three-dimensional position coordinates of the center point of the target object in the radar coordinate system based on the target outline of the target object;

[0020] The circumscribed sphere radius determination module is used to determine the circumscribed sphere radius of the target object based on the extreme values ​​of the horizontal and vertical coordinates in the target contour and the camera intrinsic parameters.

[0021] The segmentation module is used to segment the point cloud data containing the target object using the three-dimensional position coordinates of the center point of the target object in the radar coordinate system and the radius of the circumscribed sphere of the target object, so as to obtain the point cloud data corresponding to the target object.

[0022] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0023] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor; the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described in the first aspect.

[0024] According to a fourth aspect provided in the embodiments of this disclosure, a computer storage medium is provided, the computer storage medium storing a computer program for performing the method as described in the first aspect. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a schematic diagram of the structure of a crane according to one embodiment of the present disclosure;

[0027] Figure 2 This is a schematic diagram illustrating an applicable scenario according to one embodiment of the present disclosure;

[0028] Figure 3 This is one of the flowcharts illustrating a point cloud segmentation method for a target object according to an embodiment of the present disclosure;

[0029] Figure 4 This is a flowchart illustrating the process of determining the current feature vector of a contour according to an embodiment of the present disclosure.

[0030] Figure 5 This is a flowchart illustrating a method for determining the three-dimensional position coordinates of the center point of a target object in a radar coordinate system according to an embodiment of the present disclosure;

[0031] Figure 6 This is a flowchart illustrating the process of determining the height of a target object according to an embodiment of the present disclosure;

[0032] Figure 7 This is a schematic flowchart illustrating the process of determining the depth image of a target object according to an embodiment of the present disclosure;

[0033] Figure 8 This is a flowchart illustrating a method for determining the circumscribed sphere radius of a target object according to an embodiment of the present disclosure;

[0034] Figure 9 This is a schematic diagram of a process for filtering ground point cloud data according to an embodiment of the present disclosure;

[0035] Figure 10 This is a second schematic flowchart of a point cloud segmentation method for a target object according to an embodiment of the present disclosure;

[0036] Figure 11 A point cloud segmentation apparatus for a target object according to an embodiment of the present disclosure;

[0037] Figure 12 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0039] In this disclosure, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0040] The application scenarios described in this disclosure are for the purpose of more clearly illustrating the technical solutions of this disclosure and do not constitute a limitation on the technical solutions provided in this disclosure. Those skilled in the art will understand that with the emergence of new application scenarios, the technical solutions provided in this disclosure are also applicable to similar technical problems. In the description of this disclosure, unless otherwise stated, "multiple" means two or more.

[0041] In existing technologies, deep learning-based methods are used to obtain 2D target bounding boxes from 2D images. Then, extrinsic parameters calibrated between the LiDAR and camera are used to extract point cloud data corresponding to the 2D bounding boxes. The point clouds are then clustered, segmented, and filtered to obtain the point cloud data of the hoisting target, thus achieving target localization. Alternatively, deep learning methods can be used directly to perform 3D target detection on the point cloud data, obtaining 3D target bounding boxes. The point clouds within these boxes are then clustered and segmented to obtain the point cloud data of the hoisting target. However, both of these methods require sample annotation and training on the hoisting target before the hoisting operation to obtain the target detection network model. In actual hoisting operations, the type and shape of each hoisting target may differ. Therefore, single-class sample training cannot cover all hoisting targets, making it impossible to automatically complete the detection and segmentation of all hoisting targets using a pre-trained single network model. Another approach is to achieve inter-class segmentation by setting distance or density parameters. This method is sensitive to noise and parameter values; noise near the target object or improper parameter settings directly affect the accuracy of point cloud segmentation.

[0042] Therefore, this disclosure provides a point cloud segmentation method for a target object. The method matches the standard contour feature vector of the target object with contours in an image to determine the contour of the target object. Based on the contour, the three-dimensional position coordinates of the target object's center point in the radar coordinate system and the radius of its circumsphere are determined. Finally, the point cloud data containing the target object is segmented using the three-dimensional position coordinates of the target object's center point in the radar coordinate system and the radius of its circumsphere, resulting in point cloud data corresponding to the target object. Therefore, this application achieves target object segmentation through the standard contour feature vector of the target object. Based on the standard contour feature vector, the shape and color of the target object can be accurately identified without training a target object detection model or setting distance or density parameters for clustering segmentation, thus improving the robustness and accuracy of point cloud segmentation of the target object.

[0043] Before describing in detail the segmentation method of the target object in this application, the structure of the crane in this application will be explained first. For example... Figure 1As shown, the crane includes a body 11, a boom 12, a camera 13, and a radar 14. From Figure 1 As can be seen from the diagram, the camera and radar in this embodiment are mounted on the top of the boom. Camera 13 is used to acquire RGB images containing the target object. Radar 14 is used to acquire point cloud data containing the target object. Figure 1 The crane structure shown in the illustration is for informational purposes only and does not limit the scope of the crane's structure. The following section, with reference to the accompanying drawings, provides a detailed description of the solutions presented in this disclosure.

[0044] like Figure 2 The image shows an application scenario for a point cloud segmentation method for a target object, which includes a server 110 and a crane 120.

[0045] In one possible application scenario, during the lifting process by the crane 220, the server 210 acquires point cloud data and RGB images containing a target object corresponding to a specified duration from the crane 220, wherein the target object is the object to be lifted by the crane. The server 210 extracts contours from the RGB images to obtain various contours within the RGB images. Then, for any given contour, the server 210 obtains the current feature vector of the contour based on the position of each pixel within the contour and the RGB values ​​of each pixel. Using the current feature vectors of each contour and the standard contour feature vector of the target object, the server 210 determines the target contour of the target object within each contour. Then, based on the target contour of the target object, the server 210 determines the three-dimensional position coordinates of the center point of the target object in the radar coordinate system. Based on the extreme values ​​of the horizontal and vertical position coordinates of the target contour and the camera intrinsic parameters, the server 210 determines the circumscribed sphere radius of the target object. Finally, the server 210 uses the three-dimensional position coordinates of the center point of the target object in the radar coordinate system and the circumscribed sphere radius of the target object to segment the point cloud data containing the target object, obtaining point cloud data corresponding to the target object.

[0046] in, Figure 2 The server 210 and the crane 220 can exchange information through a communication network. The communication network can be either wireless or wired.

[0047] For example, server 210 can access the network via cellular mobile communication technology to communicate with crane 220, wherein the cellular mobile communication technology includes, for example, 5th generation mobile networks (5G) technology.

[0048] Optionally, server 210 can access the network and communicate with crane 220 via short-range wireless communication, such as Wireless Fidelity (Wi-Fi) technology.

[0049] Furthermore, the description in this application focuses only on a single crane 220 and a single server 210. However, those skilled in the art should understand that the crane 220 and server 210 shown are intended to illustrate the operation of the crane 220 and server 210 involved in the technical solutions of this application, and are not intended to imply any limitation on the number, type, or location of the crane 220 and server 210. It should be noted that adding additional modules to or removing individual modules from the illustrated environment will not change the underlying concept of the exemplary embodiments of this application.

[0050] It should be noted that the point cloud segmentation method for the target object proposed in this application is not only applicable to... Figure 2 The application scenarios shown are also applicable to any point cloud segmentation device with a target object.

[0051] The point cloud segmentation method for the target object of this application, in conjunction with the application scenarios described above and with reference to the accompanying drawings, will be described below in an exemplary embodiment. It should be noted that the above application scenarios are only shown to facilitate understanding of the methods and principles of this application, and the implementation of this application is not limited in any way in this respect.

[0052] like Figure 3 The diagram shown is a flowchart of the point cloud segmentation method for the target object of this disclosure, which may include the following steps:

[0053] Step 301: During the crane hoisting process, acquire point cloud data and RGB images containing the target object corresponding to a specified duration, wherein the target object is the object that the crane needs to lift;

[0054] The point cloud data containing the target object obtained in this embodiment includes the three-dimensional position coordinates of each discrete point in the radar coordinate system. In this embodiment, step 301 can be executed once every specified time interval.

[0055] In one embodiment, before performing step 301, the crane's radar and camera are jointly calibrated to obtain camera intrinsic and extrinsic parameters.

[0056] In this embodiment, the camera and radar can be jointly calibrated using existing methods. This embodiment does not limit the joint calibration method between the camera and radar. The specific method of joint calibration in this embodiment can be set according to the actual situation.

[0057] Furthermore, the specified duration in this embodiment can be 0.1 seconds, 1 second, or 10 seconds, etc. The specific specified duration can be set according to the actual situation, and this embodiment does not limit the specific value of the specified duration.

[0058] Step 302: Extract contours from the RGB image to obtain each contour in the RGB image;

[0059] This application embodiment utilizes a contour extraction algorithm to extract contours from RGB images. The contour extraction algorithm in this application embodiment can be an interior hollowing method or a boundary tracing method, etc. However, this application embodiment does not limit the contour extraction algorithm; the contour extraction algorithm in this application embodiment can be set according to actual conditions.

[0060] Step 303: For any contour, based on the position of each pixel within the contour and the RGB value of each pixel, obtain the current feature vector of the contour;

[0061] like Figure 4 The diagram shown illustrates the process of determining the current feature vector of a contour, which may include the following steps:

[0062] Step 401: Determine the Hu moment of the contour based on the position of each pixel in the contour;

[0063] The Hu moments are a set of seven invariant moments obtained from the second and third central moments. The method for determining the Hu moments in this application is the same as that in the prior art, and will not be described again here.

[0064] Step 402: Determine the mean value of the R component, the mean value of the G component, and the mean value of the B component based on the RGB values ​​of each pixel.

[0065] The RGB values ​​include R, G, and B values. In one embodiment, step 402 may be specifically implemented as follows: determining the average value of the R values ​​of each pixel within the contour as the average value of the R component; determining the average value of the G values ​​of each pixel within the contour as the average value of the G component; and determining the average value of the B values ​​of each pixel within the contour as the average value of the component.

[0066] Step 403: Based on the Hu moments of the contour, the mean of the R component, the mean of the G component, and the mean of the B component, obtain the current feature vector of the contour.

[0067] In one embodiment, step 403 may be specifically implemented as: determining the vector composed of the Hu moments of the contour, the mean of the R components, the mean of the G components, and the mean of the B components as the current feature vector of the contour.

[0068] For example, the Hu moment of the profile is , , , , , , The mean of the R components is The mean of the G component is And the mean of component B is Then the current feature vector of the obtained contour is ( , , , , , , , , , ).

[0069] Step 304: Determine the target contour of the target object among the contours by using the current feature vector of each contour and the standard contour feature vector of the target object;

[0070] Before detailing the method in step 304, let's first explain the standard contour feature vector for determining the target object in the embodiments of this application. In one embodiment, before performing step 301,

[0071] In response to a user-sent crane control command, the crane boom is raised to a specified height above the target object, at which point the camera can capture a full view of the target object from above. The camera on the crane captures a single-frame RGB image of the target object while the boom is stationary. A contour extraction algorithm is used to extract all contours, including the target object, from the RGB image. A contour is a set of coordinates of continuous edge pixels of an object in the image. All contours and the RGB image are overlaid and displayed on the screen of the industrial control computer in the crane cab. The standard contour of the target object is then manually confirmed by clicking on the screen. The Hu moments, the mean values ​​of the R, G, and B components are calculated for the manually selected standard contour of the target object. Finally, the standard contour feature vector of the target object is obtained based on the Hu moments, the mean values ​​of the R, G, and B components of the target object's standard contour.

[0072] It should be noted that the method for determining the Hu moment of the standard contour of the target object, the mean value of the R component, the mean value of the G component, and the mean value of the B component in this embodiment is the same as that in steps 401 to 403 described above, and will not be repeated here.

[0073] The specific method for determining the target contour of the target object in step 304 is described below. In one embodiment, step 304 can be specifically implemented as follows: for any contour, based on the current feature vector of the contour and the standard contour feature vector of the target object, the similarity between the contour and the target object is obtained; the contour with the largest similarity value among the contours with a similarity greater than a specified similarity is determined as the target contour of the target object.

[0074] The specified similarity in this application embodiment can be set according to the actual situation. This application embodiment does not limit the specific value of the specified similarity. The value range of the specified similarity in this application embodiment can be a real number greater than a certain set value.

[0075] In one embodiment, the similarity between the contour and the target object is obtained in the following way:

[0076] Based on the current feature vector of the contour and the standard contour feature vector of the target object, the chi-square distance between the current feature vector and the standard contour feature vector is obtained; the similarity between the current feature vector and the standard contour feature vector is determined according to the chi-square distance, and the similarity is negatively correlated with the chi-square distance. The chi-square distance between the current feature vector and the standard contour feature vector can be obtained using formula (1):

[0077] .....(1)

[0078] in, Let be the chi-square distance between the current feature vector i of the contour and the feature vector j of the standard contour. Let m be the m-th parameter of the Hu moment in the current feature vector i. Let be the m-th parameter of the Hu moment in the standard contour feature vector j. The mean of the R components in the current feature vector i. The mean of the R component in the standard contour feature vector j. The mean of the G components in the current feature vector i. The mean value of the G component in the standard contour feature j. The mean of the B components in the current feature vector i. The mean value of the B component in the standard contour feature j.

[0079] Step 305: Determine the three-dimensional position coordinates of the center point of the target object in the radar coordinate system based on the target outline of the target object;

[0080] like Figure 5The diagram shown illustrates a process for determining the three-dimensional position coordinates of the center point of a target object in a radar coordinate system, which may include the following steps:

[0081] Step 501: Obtain the 0th and 1st moments of the target contour by using the positions of each pixel in the target contour of the target object; wherein, the 0th and 1st moments of the target contour can be obtained by formula (2):

[0082] p,q=0,1,2,3……(2;

[0083] in, The maximum horizontal position coordinate of the target contour. The maximum vertical coordinate of the target contour. Let p be the pixel value of the pixel with x-coordinate and y-coordinate, and q be the 0th moment when p=0 and q=0. The value of the first moment when p=1 and q=0. The values ​​and when p=0, q=1 The value of .

[0084] Step 502: Based on the 0th and 1st moments of the target contour, obtain the two-dimensional position coordinates of the center point of the target contour in the RGB image; wherein, the abscissa of the two-dimensional position of the center point of the target contour in the image can be obtained by formula (3):

[0085] ... (3);

[0086] in, The x-coordinate of the center point of the target contour in the image is given by the following two-dimensional coordinates. The first moment of the target contour, The zeroth moment of the target profile is given.

[0087] And the two-dimensional ordinate of the center point of the target contour in the image can be obtained by formula (4):

[0088] ... (4);

[0089] in, The vertical coordinate of the center point of the target contour in the image is the two-dimensional position. This is another first-order moment of the target contour of the target object.

[0090] Step 503: Based on the two-dimensional position coordinates of the center point of the target contour in the RGB image and the camera intrinsic parameters, obtain the three-dimensional position coordinates of the center point of the target contour in the camera coordinate system;

[0091] In one embodiment, step 503 can be specifically implemented as follows: the product of the two-dimensional position coordinates of the center point of the target contour in the RGB image, the matrix corresponding to the camera intrinsic parameters, and the depth value of the center point of the target contour in the depth image containing the target object is determined as the three-dimensional position coordinates of the center point of the target contour in the camera coordinate system; wherein, the three-dimensional position coordinates of the center point of the target contour in the camera coordinate system can be obtained by formula (5):

[0092] ... (5);

[0093] in, Let x be the abscissa of the center point of the target contour in the camera coordinate system. Let be the ordinate of the three-dimensional position of the center point of the target contour in the camera coordinate system. Let be the vertical coordinate of the three-dimensional position of the center point of the target contour in the camera coordinate system. This is the matrix corresponding to the camera's intrinsic parameters. The x-coordinate of the center point of the target contour in the RGB image is the two-dimensional position. The vertical coordinate of the center point of the target contour in the RGB image. The depth value is the position of the center point of the target contour in the depth image containing the target object.

[0094] Step 504: Using the three-dimensional position coordinates of the center point of the target contour in the camera coordinate system and the height of the target object itself, obtain the three-dimensional position coordinates of the center point of the target object in the camera coordinate system;

[0095] In one embodiment, step 504 can be specifically implemented as follows: determining the three-dimensional abscissa of the center point of the target contour in the three-dimensional position coordinates of the camera coordinate system as the three-dimensional abscissa of the center point of the target object in the camera coordinate system; and determining the three-dimensional ordinate of the center point of the target contour in the three-dimensional position coordinates of the camera coordinate system as the three-dimensional ordinate of the center point of the target object in the camera coordinate system; and taking the vertical coordinate of the center point of the target contour in the three-dimensional position coordinates of the camera coordinate system and half the height of the target object to obtain the vertical coordinate of the center point of the target object in the camera coordinate system. The three-dimensional position coordinates of the center point of the target object in the camera coordinate system can be obtained through formula (6):

[0096] ... (6);

[0097] in, The x-coordinate of the center point of the target object in the camera coordinate system is given. The vertical coordinate of the center point of the target object in the camera coordinate system is given. The vertical coordinates of the center point of the target object in the camera coordinate system are given. The height is the height of the target object itself.

[0098] The following describes the method for determining the height of the target object in the embodiments of this application, such as... Figure 6 The flowchart shown illustrates the process of determining the height of a target object, which may include the following steps:

[0099] Step 601: Based on the crane's amplitude adjustment angle, the crane's boom length, and the crane's body height, obtain the first distance between the camera and the ground, wherein the amplitude adjustment angle is the angle between the crane's boom and the ground when the target object is not off the ground during the lifting process; wherein the first distance is obtained through formula (7):

[0100] ... (7);

[0101] in, For the first distance, The length of the boom is... The amplitude modulation angle is... The height of the crane body is given.

[0102] Step 602: Based on the depth value of the center point of the target contour in the depth image of the target object, obtain the second distance between the center point of the target contour and the camera;

[0103] In one embodiment, step 602 may be specifically implemented as: determining the depth value of the center point of the target contour in the depth image of the target object as the second distance.

[0104] Step 603: Obtain the height of the target object itself using the first distance and the second distance.

[0105] In one embodiment, step 603 may be specifically implemented as: determining the difference between the first distance and the second distance as the height of the target object itself.

[0106] It should be noted that the height of the target object in this embodiment only needs to be calculated once when the crane lifts the object and before it leaves the ground. The height value can be directly used in the subsequent calculation of the three-dimensional position of the center point of the target object in the camera coordinate system and the radius of the circumscribed sphere.

[0107] Step 505: Obtain the three-dimensional position coordinates of the center point of the target object in the radar coordinate system using the three-dimensional position coordinates of the center point of the target object in the camera coordinate system and the camera extrinsic parameters.

[0108] In one embodiment, step 505 can be specifically implemented as follows: multiplying the three-dimensional position coordinates of the center point of the target object in the camera coordinate system with the matrix corresponding to the camera extrinsic parameters to obtain the three-dimensional position coordinates of the center point of the target object in the radar coordinate system. The three-dimensional position coordinates of the center point of the target object in the radar coordinate system can be obtained through formula (8):

[0109] ... (8);

[0110] in, The x-coordinate of the center point of the target object in the radar coordinate system is given. The vertical coordinate of the center point of the target object in the radar coordinate system is given. Let be the vertical coordinate of the three-dimensional position of the center point of the target object in the radar coordinate system. This is the matrix corresponding to the camera's extrinsic parameters.

[0111] The following is a detailed description of the method for determining the depth image of the target object in the embodiments of this application, such as... Figure 7 The diagram shown illustrates a process for determining the depth image of a target object, which may include the following steps:

[0112] Step 701: Use the camera extrinsic parameters to perform coordinate transformation on the point cloud data containing the target object corresponding to the specified duration, and obtain the camera position coordinates of each discrete point in the point cloud data in the camera coordinate system; wherein, the camera position coordinates of any discrete point in the point cloud data in the camera coordinate system can be obtained by formula (9):

[0113] ... (9);

[0114] in, Let l be the abscissa of a discrete point l in the point cloud data containing the target object, in the radar coordinate system. The longitudinal coordinates of a discrete point l in the point cloud data containing the target object within the radar coordinate system. This refers to the vertical coordinates of a discrete point l in the point cloud data containing the target object, located in the radar coordinate system. This is the matrix corresponding to the camera's extrinsic parameters. Let l be the horizontal coordinate of a discrete point l in the point cloud data containing the target object, located in the camera coordinate system. Let l be the ordinate of a scatter point in the point cloud data containing the target object, located in the camera coordinate system. Let l be the vertical coordinate of a scatter point in the point cloud data containing the target object, located in the camera coordinate system. P represents the total number of discrete points in the point cloud data.

[0115] Step 702: Project the camera position coordinates of each discrete point in the point cloud data onto a preset depth image using the camera intrinsic parameters to obtain the two-dimensional position coordinates of each discrete point in the point cloud data in the depth image, wherein the size of the preset depth image is the same as the size of the RGB image; wherein the two-dimensional position coordinates of each discrete point in the point cloud data in the depth image can be obtained by formula (10):

[0116] ... (10);

[0117] in, Let l be the abscissa of the two-dimensional position of a discrete point l in the point cloud data in the depth image. Let be the ordinate of the two-dimensional position of a discrete point l in the point cloud data within the depth image. This is the matrix corresponding to the intrinsic parameters of the camera.

[0118] Step 703: For any discrete point, set the vertical coordinate value in the camera position coordinates corresponding to the discrete point to the depth value of the discrete point in the preset depth image;

[0119] Step 704: Obtain the depth image of the target object based on the depth values ​​of each discrete point in the preset depth image and the two-dimensional position coordinates of each discrete point in the depth image.

[0120] Step 306: Determine the circumscribed sphere radius of the target object based on the extreme values ​​of the horizontal and vertical coordinates in the target contour and the camera intrinsic parameters;

[0121] like Figure 8 The diagram shown illustrates the process for determining the circumscribed sphere radius of a target object, which may include the following steps:

[0122] Step 801: Obtain the length of the target contour based on the maximum and minimum values ​​among the extreme values ​​of the horizontal position coordinates of the target contour;

[0123] In one embodiment, step 801 can be specifically implemented as follows: the difference between the maximum and minimum values ​​of the horizontal position coordinates in the target contour is determined as the length of the target contour. The length of the target contour can be obtained using formula (11):

[0124] ... (11);

[0125] in, The length of the target contour, This represents the maximum value of the horizontal position coordinate in the target contour. It represents the minimum value of the horizontal position coordinate in the target contour.

[0126] Step 802: Obtain the width of the target contour based on the maximum and minimum values ​​among the extreme values ​​of the vertical position coordinates of the target contour;

[0127] In one embodiment, step 802 can be specifically implemented as follows: determining the width of the target contour as the difference between the maximum and minimum values ​​of the vertical position coordinates in the target contour. The width of the target contour can be obtained using formula (12):

[0128] …….(12)

[0129] in, The width of the target contour. The maximum value of the vertical position coordinate of the target contour. It represents the minimum value of the vertical position coordinate in the target contour.

[0130] Step 803: Based on the length of the target contour, the depth value of the center point of the target contour in the depth image of the target object, and the horizontal focal length in the camera intrinsics, the length of the target object is obtained; wherein, the length of the target object can be obtained by formula (13):

[0131] ... (13);

[0132] in, The length of the target object. The length of the target contour. The horizontal focal length is one of the camera's intrinsic parameters. The depth value of the center point of the target contour in the depth image of the target object.

[0133] Step 804: Based on the width of the target contour, the depth value of the center point of the target contour in the depth image of the target object, and the vertical focal length in the camera intrinsics, the width of the target object is obtained; wherein, the width of the target object can be obtained by formula (14):

[0134] ... (14);

[0135] in, The width of the target object. The width of the target contour. The vertical focal length is the intrinsic parameter of the camera.

[0136] Step 805: Determine half of the maximum value among the length, width, and height of the target object as the circumscribed sphere radius of the target object. The circumscribed sphere radius of the target object can be obtained using formula (15):

[0137] …….(15)

[0138] in, Let be the radius of the circumscribed sphere of the target object. The height is the height of the target object itself.

[0139] Step 307: Use the three-dimensional position coordinates of the center point of the target object in the radar coordinate system and the radius of the circumscribed sphere of the target object to segment the point cloud data containing the target object, and obtain the point cloud data corresponding to the target object.

[0140] In one embodiment, step 307 can be specifically implemented as follows: inputting the three-dimensional position coordinates of the center point of the target object in the radar coordinate system, the radius of the circumscribed sphere of the target object, and the point cloud data containing the target object into the minimum cut algorithm for point cloud segmentation to obtain point cloud data corresponding to the target object.

[0141] The minimum segmentation algorithm in this application embodiment is an algorithm in the prior art, and this application embodiment will not elaborate on the minimum segmentation algorithm here.

[0142] Since the target object is generally placed on the ground before lifting, the ground point cloud and the lifting point cloud are adhered together. To further improve the accuracy of point cloud segmentation of the target object, in one embodiment, the ground point cloud in the acquired point cloud data needs to be filtered before performing step 307. Because the radar faces the ground, the coordinate system needs to be rotated before filtering the ground point cloud so that the projection of the ground point cloud lies on the plane formed by the radar's XY axes. Figure 9 The diagram shown illustrates the process of filtering ground point cloud data, which may include the following steps:

[0143] Step 901: Using a pre-set first rotation matrix, rotate the point cloud data containing the target object clockwise by a specified angle around the Y-axis to obtain rotated point cloud data;

[0144] The specified angle in this embodiment is 90 degrees, but this does not limit the specified angle in this embodiment. The specified angle in this embodiment can be limited according to the actual situation.

[0145] In one embodiment, step 901 can be specifically implemented as follows: for any discrete point in the point cloud data containing the target object, the discrete point is rotated clockwise around the Y-axis by a specified angle using a pre-set first rotation matrix to obtain the rotated position coordinates of the discrete point, and the rotated position coordinates of each discrete point are determined as the rotated point cloud data.

[0146] The rotated position coordinates of any discrete point in the point cloud data can be obtained through formula (16):

[0147] ... (16);

[0148] in, for The x-coordinate of the discrete point l after rotation. Let l be the ordinate of the discrete point after rotation. Let l be the vertical coordinates of the discrete point after rotation. Let l be the x-coordinate of the discrete point before rotation. The ordinate of discrete point l before rotation Let l be the vertical coordinates of the discrete point before rotation. The first rotation matrix.

[0149] Step 902: Use a filtering algorithm to perform ground point cloud filtering on the rotating point cloud data to obtain ground-filtered rotating point cloud data;

[0150] The filtering algorithm in this embodiment is the CSF (Cloth Simulation Filter) algorithm. However, this embodiment does not limit the filtering algorithm. The filtering algorithm in this embodiment can be set according to the actual situation.

[0151] Step 903: Using a pre-set second rotation matrix, rotate the ground-filtered rotating point cloud data counterclockwise around the Y-axis by a specified angle to obtain the ground-filtered point cloud data in the initial coordinate system, and determine the ground-filtered point cloud data in the initial coordinate system as the point cloud data containing the target object.

[0152] In one embodiment, step 903 can be specifically implemented as follows: for any discrete point in the point cloud data containing the target object, the position coordinates of the discrete point are rotated counterclockwise around the Y-axis by a specified angle using a pre-set second rotation matrix to obtain the initial position coordinates of the discrete point, and the filtered point cloud data is obtained based on the initial position coordinates of each discrete point.

[0153] The filtered point cloud data can be obtained through formula (17):

[0154] ... (17);

[0155] in, Let l be the initial x-coordinate of the discrete point. The initial ordinate of the discrete point l. Let be the initial vertical coordinate of the discrete point l. Let be the second rotation matrix.

[0156] To further understand the technical solution of this disclosure, the following is in conjunction with... Figure 10 A detailed explanation may include the following steps:

[0157] Step 1001: During the crane hoisting process, acquire point cloud data and RGB images containing the target object corresponding to a specified time period, wherein the target object is the object that the crane needs to lift;

[0158] Step 1002: Extract contours from the RGB image to obtain each contour in the RGB image;

[0159] Step 1003: Determine the Hu moment of the contour based on the position of each pixel in the contour; and determine the mean value of the R component, the mean value of the G component, and the mean value of the B component based on the RGB values ​​of each pixel in the contour.

[0160] Step 1004: Based on the Hu moments of the contour, the mean of the R components, the mean of the G components, and the mean of the B components, obtain the current feature vector of the contour;

[0161] Step 1005: For any contour, based on the current feature vector of the contour and the standard contour feature vector of the target object, obtain the similarity between the contour and the target object;

[0162] Step 1006: Determine the contour with the largest similarity value among all contours with a similarity greater than a specified similarity value as the target contour of the target object;

[0163] Step 1007: Obtain the 0th and 1st moments of the target contour by using the positions of each pixel in the target contour of the target object;

[0164] Step 1008: Based on the 0th and 1st moments of the target contour, obtain the two-dimensional position coordinates of the center point of the target contour in the RGB image;

[0165] Step 1009: Based on the two-dimensional position coordinates of the center point of the target contour in the RGB image and the camera intrinsic parameters, obtain the three-dimensional position coordinates of the center point of the target contour in the camera coordinate system;

[0166] Step 1010: Using the three-dimensional position coordinates of the center point of the target contour in the camera coordinate system and the height of the target object itself, obtain the three-dimensional position coordinates of the center point of the target object in the camera coordinate system;

[0167] Step 1011: Obtain the three-dimensional position coordinates of the center point of the target object in the radar coordinate system using the three-dimensional position coordinates of the center point of the target object in the camera coordinate system and the camera extrinsic parameters;

[0168] Step 1012: Obtain the length of the target contour based on the maximum and minimum values ​​of the extreme values ​​of the horizontal position coordinates of the target contour; and obtain the width of the target contour based on the maximum and minimum values ​​of the extreme values ​​of the vertical position coordinates of the target contour.

[0169] Step 1013: Based on the length of the target contour, the depth value of the center point of the target contour in the depth image of the target object, and the horizontal focal length in the camera intrinsics, the length of the target object is obtained; and based on the width of the target contour, the depth value of the center point of the target contour in the depth image of the target object, and the vertical focal length in the camera intrinsics, the width of the target object is obtained.

[0170] Step 1014: Determine half of the maximum value among the length of the target object, the width of the target object, and the height of the target object itself as the radius of the circumscribed sphere of the target object;

[0171] Step 1015: Using a pre-set first rotation matrix, rotate the point cloud data containing the target object clockwise by a specified angle around the Y-axis to obtain rotated point cloud data;

[0172] Step 1016: Use a filtering algorithm to perform ground point cloud filtering on the rotating point cloud data to obtain ground-filtered rotating point cloud data;

[0173] Step 1017: Using a pre-set second rotation matrix, rotate the filtered rotating point cloud data counterclockwise around the Y-axis by a specified angle to obtain the filtered point cloud data of the ground in the initial coordinate system, and determine the filtered point cloud data as the point cloud data containing the target object;

[0174] Step 1018: Use the three-dimensional position coordinates of the center point of the target object in the radar coordinate system and the radius of the circumscribed sphere of the target object to segment the point cloud data containing the target object, and obtain the point cloud data corresponding to the target object.

[0175] Based on the same disclosed concept, the point cloud segmentation method for the target object described above can also be implemented by a point cloud segmentation device for the target object. The effect of this point cloud segmentation device is similar to that of the aforementioned method, and will not be described again here.

[0176] Figure 11 This is a schematic diagram of the structure of a point cloud segmentation apparatus for a target object according to an embodiment of the present disclosure.

[0177] like Figure 11 As shown, the point cloud segmentation device 1100 for the target object disclosed herein may include an acquisition module 1110, a contour extraction module 1120, a contour vector determination module 1130, a matching module 1140, a center point position determination module 1150, a circumscribed sphere radius determination module 1160, and a segmentation module 1170.

[0178] The acquisition module 1110 is used to acquire point cloud data and RGB images containing a target object corresponding to a specified time period during the crane hoisting process, wherein the target object is the object that the crane needs to lift;

[0179] The contour extraction module 1120 is used to extract contours from the RGB image to obtain each contour in the RGB image.

[0180] The contour vector determination module 1130 is used to obtain the current feature vector of any contour based on the position of each pixel in the contour and the RGB value of each pixel.

[0181] The matching module 1140 is used to determine the target contour of the target object among the contours by using the current feature vector of each contour and the standard contour feature vector of the target object.

[0182] The center point position determination module 1150 is used to determine the three-dimensional position coordinates of the center point of the target object in the radar coordinate system based on the target outline of the target object;

[0183] The circumscribed sphere radius determination module 1160 is used to determine the circumscribed sphere radius of the target object based on the extreme values ​​of the horizontal position coordinates and the extreme values ​​of the vertical position coordinates in the target contour and the camera intrinsic parameters.

[0184] The segmentation module 1170 is used to segment the point cloud data containing the target object using the three-dimensional position coordinates of the center point of the target object in the radar coordinate system and the radius of the circumscribed sphere of the target object, so as to obtain point cloud data corresponding to the target object.

[0185] In one embodiment, the contour vector determination module 1130 is specifically used for:

[0186] Based on the positions of each pixel in the contour, determine the Hu moment of the contour; and,

[0187] Based on the RGB values ​​of each pixel, determine the mean value of the R component, the mean value of the G component, and the mean value of the B component.

[0188] Based on the Hu moments of the contour, the mean of the R components, the mean of the G components, and the mean of the B components, the current feature vector of the contour is obtained.

[0189] In one embodiment, the matching module 1140 is specifically used for:

[0190] For any given contour, the similarity between the contour and the target object is obtained based on the current feature vector of the contour and the standard contour feature vector of the target object.

[0191] The contour with the highest similarity value among all contours with a similarity greater than a specified similarity is determined as the target contour of the target object.

[0192] In one embodiment, the matching module 1140 executes the current feature vector based on the contour and the standard contour feature vector of the target object to obtain the similarity between the contour and the target object, specifically for:

[0193] Based on the current feature vector of the contour and the standard contour feature vector of the target object, the chi-square distance between the current feature vector and the standard contour feature vector is obtained;

[0194] The similarity between the current feature vector and the standard contour feature vector is determined based on the chi-square distance, and the similarity is negatively correlated with the chi-square distance.

[0195] In one embodiment, the center point location determination module 1150 is specifically used for:

[0196] The 0th and 1st moments of the target contour are obtained by using the positions of each pixel in the target contour of the target object.

[0197] Based on the 0th and 1st moments of the target contour, the two-dimensional position coordinates of the center point of the target contour in the RGB image are obtained;

[0198] Based on the two-dimensional position coordinates of the center point of the target contour in the RGB image and the camera intrinsic parameters, the three-dimensional position coordinates of the center point of the target contour in the camera coordinate system are obtained.

[0199] Using the three-dimensional position coordinates of the center point of the target contour in the camera coordinate system and the height of the target object itself, the three-dimensional position coordinates of the center point of the target object in the camera coordinate system are obtained;

[0200] The three-dimensional position coordinates of the center point of the target object in the radar coordinate system are obtained by using the three-dimensional position coordinates of the center point of the target object in the camera coordinate system and the camera extrinsic parameters.

[0201] In one embodiment, the center point location determination module 1150 performs the step of obtaining the two-dimensional position coordinates of the center point of the target contour in the image based on the 0th and 1st moments of the target contour, specifically for:

[0202] The horizontal coordinate of the center point of the target contour in the image is obtained by the following formula:

[0203] ;

[0204] in, The x-coordinate of the center point of the target contour in the image is given by the following two-dimensional coordinates. The first moment of the target contour, The zeroth moment of the target contour;

[0205] The ordinate of the two-dimensional position of the center point of the target contour in the image can be obtained using the following formula:

[0206] ;

[0207] in, The vertical coordinate of the center point of the target contour in the image is the two-dimensional position. This is another first-order moment of the target contour of the target object.

[0208] In one embodiment, the center point location determination module 1150 executes the two-dimensional position coordinates of the center point of the target contour in the RGB image and camera intrinsic parameters to obtain the three-dimensional position coordinates of the center point of the target contour in the camera coordinate system, specifically for:

[0209] The product of the two-dimensional position coordinates of the center point of the target contour in the RGB image, the matrix corresponding to the camera intrinsic parameters, and the depth value of the center point of the target contour in the depth image containing the target object is determined as the three-dimensional position coordinates of the center point of the target contour in the camera coordinate system.

[0210] The center point position determination module 1150 executes the process of obtaining the three-dimensional position coordinates of the center point of the target object in the camera coordinate system using the three-dimensional position coordinates of the center point of the target contour in the camera coordinate system and the height of the target object itself, specifically for:

[0211] The abscissa of the three-dimensional position of the center point of the target contour in the camera coordinate system is determined as the abscissa of the three-dimensional position of the center point of the target object in the camera coordinate system; and...

[0212] The ordinate of the three-dimensional position of the center point of the target contour in the camera coordinate system is determined as the ordinate of the three-dimensional position of the center point of the target object in the camera coordinate system; and...

[0213] The vertical coordinate of the center point of the target contour in the camera coordinate system is obtained by taking half the height of the target object and then dividing the vertical coordinate of the center point of the target object in the camera coordinate system.

[0214] The center point location determination module 1150 executes the process of obtaining the three-dimensional position coordinates of the center point of the target object in the camera coordinate system and the camera extrinsic parameters to obtain the three-dimensional position coordinates of the center point of the target object in the radar coordinate system, specifically for:

[0215] Multiply the three-dimensional position coordinates of the center point of the target object in the camera coordinate system with the matrix corresponding to the camera extrinsic parameters to obtain the three-dimensional position coordinates of the center point of the target object in the radar coordinate system.

[0216] In one embodiment, the apparatus further includes:

[0217] The target object height determination module 1180 is used to determine the height of the target object itself in the following manner:

[0218] Based on the crane's amplitude adjustment angle, the crane's boom length, and the crane's height, a first distance between the camera and the ground is obtained, wherein the amplitude adjustment angle is the angle between the crane's boom and the ground when the target object is not off the ground during the lifting process; and,

[0219] Based on the depth value of the center point of the target contour in the depth image of the target object, a second distance between the center point of the target contour and the camera is obtained;

[0220] The height of the target object is obtained by using the first distance and the second distance.

[0221] In one embodiment, the circumscribed sphere radius determination module 1160 is specifically used for:

[0222] The length of the target contour is obtained based on the maximum and minimum values ​​of the extreme values ​​of the horizontal position coordinates of the target contour; and,

[0223] The width of the target contour is obtained from the maximum and minimum values ​​of the extreme values ​​of the vertical position coordinates of the target contour.

[0224] The length of the target object is obtained based on the length of the target contour, the depth value of the center point of the target contour in the depth image of the target object, and the horizontal focal length in the camera intrinsics; and the width of the target object is obtained based on the width of the target contour, the depth value of the center point of the target contour in the depth image of the target object, and the vertical focal length in the camera intrinsics.

[0225] The radius of the circumscribed sphere of the target object is determined by half of the maximum value among the length, width, and height of the target object.

[0226] In one embodiment, the apparatus further includes:

[0227] The depth image determination module 1190 is used to obtain the depth image of the target object in the following manner:

[0228] The camera extrinsic parameters are used to perform coordinate system transformation on the point cloud data containing the target object corresponding to a specified duration, so as to obtain the camera position coordinates of each discrete point in the point cloud data in the camera coordinate system.

[0229] The camera position coordinates of each discrete point in the point cloud data are projected onto a preset depth image using the camera intrinsic parameters to obtain the two-dimensional position coordinates of each discrete point in the point cloud data in the depth image. The size of the preset depth image is the same as the size of the RGB image.

[0230] For any discrete point, the vertical coordinate value in the camera position coordinates corresponding to the discrete point is set as the depth value of the discrete point in the preset depth image;

[0231] The depth image of the target object is obtained based on the depth value of each discrete point in the preset depth image and the two-dimensional position coordinates of each discrete point in the depth image.

[0232] In one embodiment, the segmentation module 1170 is specifically used for:

[0233] The three-dimensional position coordinates of the center point of the target object in the radar coordinate system, the circumscribed sphere radius of the target object, and the point cloud data containing the target object are input into the minimum segmentation algorithm to perform point cloud segmentation, thereby obtaining the point cloud data corresponding to the target object.

[0234] After introducing a point cloud segmentation method and apparatus for a target object according to an exemplary embodiment of the present disclosure, an electronic device according to another exemplary embodiment of the present disclosure will be introduced next.

[0235] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0236] In some possible implementations, the electronic device according to this disclosure may include at least one processor and at least one computer storage medium. The computer storage medium stores program code that, when executed by the processor, causes the processor to perform the steps in the point cloud segmentation method for a target object according to the various exemplary embodiments of this disclosure described above. For example, the processor may perform actions such as... Figure 3 Steps 301-307 are shown in the diagram.

[0237] The following reference Figure 12 To describe an electronic device 1200 according to such an embodiment of the present disclosure. Figure 12 The electronic device 1200 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0238] like Figure 12 As shown, the electronic device 1200 is manifested in the form of a general electronic device. The components of the electronic device 1200 may include, but are not limited to: at least one processor 1201, at least one computer storage medium 1202, and a bus 1203 connecting different system components (including the computer storage medium 1202 and the processor 1201).

[0239] Bus 1203 represents one or more of several bus structures, including a computer storage media bus or computer storage media controller, peripheral bus, processor, or local bus using any of the various bus structures.

[0240] Computer storage medium 1202 may include readable media in the form of volatile computer storage media, such as random access computer storage medium (RAM) 1221 and / or cache storage medium 1222, and may further include read-only computer storage medium (ROM) 1223.

[0241] The computer storage medium 1202 may also include a program / utility 1225 having a set (at least one) of program modules 1224, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0242] Electronic device 1200 can also communicate with one or more external devices 1204 (e.g., keyboard, pointing device, etc.), and with one or more devices that enable a user to interact with electronic device 1200, and / or with any device that enables electronic device 1200 to communicate with one or more other electronic devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1205. Furthermore, electronic device 1200 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1206. As shown, network adapter 1206 communicates with other modules used in electronic device 1200 via bus 1203. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1200, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0243] In some possible implementations, various aspects of the point cloud segmentation method for a target object provided in this disclosure can also be implemented in the form of a program product, which includes program code that, when the program product is run on a computer device, causes the computer device to perform the steps in the point cloud segmentation method for a target object according to the various exemplary embodiments of this disclosure described above.

[0244] The program product may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access computer storage media (RAM), read-only computer storage media (ROM), erasable programmable read-only computer storage media (EPROM or flash memory), optical fibers, portable compact disk read-only computer storage media (CD-ROM), optical computer storage media, magnetic computer storage media, or any suitable combination thereof.

[0245] The point cloud segmentation program product of the target object of the embodiments of this disclosure can be in the form of a portable compact disc read-only computer storage medium (CD-ROM) and include program code, and can run on an electronic device. However, the program product of this disclosure is not limited thereto. In this document, the readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0246] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0247] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0248] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's electronic device, partially on the user's device, as a standalone software package, partially on the user's electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In cases involving remote electronic devices, the remote electronic devices can be connected to the user's electronic device via any type of network, including a local area network (LAN) or a wide area network (WAN), or they can be connected to external electronic devices (e.g., via the Internet using an Internet service provider).

[0249] It should be noted that although several modules of the apparatus have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided and embodied by multiple modules.

[0250] Furthermore, although the operations of the methods disclosed herein are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0251] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk computer storage media, CD-ROMs, optical computer storage media, etc.) containing computer-usable program code.

[0252] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0253] These computer program instructions may also be stored in a computer-readable computer storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable computer storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0254] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0255] Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from its spirit and scope. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. A point cloud segmentation method for a target object, characterized in that, The method includes: During the crane hoisting process, point cloud data and RGB images containing the target object are acquired for a specified duration, wherein the target object is the object that the crane needs to lift; Contour extraction is performed on the RGB image to obtain each contour in the RGB image; For any given contour, the current feature vector of the contour is obtained based on the position of each pixel within the contour and the RGB value of each pixel. The target contour of the target object is determined by using the current feature vector of each contour and the standard contour feature vector of the target object. Based on the target outline of the target object, determine the three-dimensional position coordinates of the center point of the target object in the radar coordinate system; Based on the extreme values ​​of the horizontal and vertical coordinates in the target contour and the camera intrinsic parameters, the radius of the circumscribed sphere of the target object is determined. The point cloud data containing the target object is segmented using the three-dimensional position coordinates of the center point of the target object in the radar coordinate system and the radius of the circumscribed sphere of the target object, to obtain the point cloud data corresponding to the target object.

2. The method according to claim 1, characterized in that, The step of obtaining the current feature vector of the contour based on the position of each pixel within the contour and the RGB values ​​of each pixel includes: Based on the positions of each pixel in the contour, determine the Hu moment of the contour; and, Based on the RGB values ​​of each pixel, determine the mean value of the R component, the mean value of the G component, and the mean value of the B component. Based on the Hu moments of the contour, the mean of the R components, the mean of the G components, and the mean of the B components, the current feature vector of the contour is obtained.

3. The method according to claim 1, characterized in that, The step of determining the target contour of the target object among the contours by using the current feature vectors of each contour and the standard contour feature vector of the target object includes: For any given contour, the similarity between the contour and the target object is obtained based on the current feature vector of the contour and the standard contour feature vector of the target object. The contour with the highest similarity value among all contours with a similarity greater than a specified similarity is determined as the target contour of the target object.

4. The method according to claim 3, characterized in that, The similarity between the contour and the target object is obtained based on the current feature vector of the contour and the standard contour feature vector of the target object, including: Based on the current feature vector of the contour and the standard contour feature vector of the target object, the chi-square distance between the current feature vector and the standard contour feature vector is obtained; The similarity between the current feature vector and the standard contour feature vector is determined based on the chi-square distance, and the similarity is negatively correlated with the chi-square distance.

5. The method according to claim 1, characterized in that, Determining the three-dimensional position coordinates of the center point of the target object in the radar coordinate system based on the target object's outline includes: The 0th and 1st moments of the target contour are obtained by using the positions of each pixel in the target contour of the target object. Based on the 0th and 1st moments of the target contour, the two-dimensional position coordinates of the center point of the target contour in the RGB image are obtained; Based on the two-dimensional position coordinates of the center point of the target contour in the RGB image and the camera intrinsic parameters, the three-dimensional position coordinates of the center point of the target contour in the camera coordinate system are obtained. Using the three-dimensional position coordinates of the center point of the target contour in the camera coordinate system and the height of the target object itself, the three-dimensional position coordinates of the center point of the target object in the camera coordinate system are obtained; The three-dimensional position coordinates of the center point of the target object in the radar coordinate system are obtained by using the three-dimensional position coordinates of the center point of the target object in the camera coordinate system and the camera extrinsic parameters.

6. The method according to claim 5, characterized in that, The step of obtaining the two-dimensional position coordinates of the center point of the target contour in the image based on the 0th and 1st moments of the target contour includes: The horizontal coordinate of the center point of the target contour in the image is obtained by the following formula: ； in, The x-coordinate of the center point of the target contour in the image is given by the following two-dimensional coordinates. The first moment of the target contour, The zeroth moment of the target contour; The ordinate of the two-dimensional position of the center point of the target contour in the image can be obtained using the following formula: ； in, The vertical coordinate of the center point of the target contour in the image is the two-dimensional position. This is another first-order moment of the target contour of the target object.

7. The method according to claim 5, characterized in that, The process of obtaining the three-dimensional position coordinates of the center point of the target contour in the camera coordinate system based on the two-dimensional position coordinates of the center point of the target contour in the RGB image and the camera intrinsic parameters includes: The product of the two-dimensional position coordinates of the center point of the target contour in the RGB image, the matrix corresponding to the camera intrinsic parameters, and the depth value of the center point of the target contour in the depth image containing the target object is determined as the three-dimensional position coordinates of the center point of the target contour in the camera coordinate system. The step of obtaining the three-dimensional position coordinates of the center point of the target object in the camera coordinate system using the three-dimensional position coordinates of the center point of the target contour in the camera coordinate system and the height of the target object itself includes: The abscissa of the three-dimensional position of the center point of the target contour in the camera coordinate system is determined as the abscissa of the three-dimensional position of the center point of the target object in the camera coordinate system; and... The ordinate of the three-dimensional position of the center point of the target contour in the camera coordinate system is determined as the ordinate of the three-dimensional position of the center point of the target object in the camera coordinate system; and... The vertical coordinate of the center point of the target contour in the camera coordinate system is obtained by taking half the height of the target object and then dividing the vertical coordinate of the center point of the target object in the camera coordinate system. The process of obtaining the three-dimensional position coordinates of the target object's center point in the radar coordinate system using the three-dimensional position coordinates of the target object's center point in the camera coordinate system and the camera's extrinsic parameters includes: Multiply the three-dimensional position coordinates of the center point of the target object in the camera coordinate system with the matrix corresponding to the camera extrinsic parameters to obtain the three-dimensional position coordinates of the center point of the target object in the radar coordinate system.

8. The method according to claim 5 or 7, characterized in that, The height of the target object itself is determined in the following way: Based on the crane's amplitude adjustment angle, the crane's boom length, and the crane's height, a first distance between the camera and the ground is obtained, wherein the amplitude adjustment angle is the angle between the crane's boom and the ground when the target object is not off the ground during the lifting process; and, Based on the depth value of the center point of the target contour in the depth image of the target object, a second distance between the center point of the target contour and the camera is obtained; The height of the target object is obtained by using the first distance and the second distance.

9. The method according to claim 1, characterized in that, Based on the extreme values ​​of the horizontal and vertical coordinates of the target contour and the camera intrinsic parameters, the circumscribed sphere radius of the target object is determined, including: The length of the target contour is obtained based on the maximum and minimum values ​​of the extreme values ​​of the horizontal position coordinates of the target contour; and, The width of the target contour is obtained from the maximum and minimum values ​​of the extreme values ​​of the vertical position coordinates of the target contour. The length of the target object is obtained based on the length of the target contour, the depth value of the center point of the target contour in the depth image of the target object, and the horizontal focal length in the camera intrinsics; and the width of the target object is obtained based on the width of the target contour, the depth value of the center point of the target contour in the depth image of the target object, and the vertical focal length in the camera intrinsics. The radius of the circumscribed sphere of the target object is determined by half of the maximum value among the length, width, and height of the target object.

10. The method according to claim 7 or 9, characterized in that, The depth image of the target object is obtained using the following method: The camera extrinsic parameters are used to perform coordinate system transformation on the point cloud data containing the target object corresponding to a specified duration, so as to obtain the camera position coordinates of each discrete point in the point cloud data in the camera coordinate system. The camera position coordinates of each discrete point in the point cloud data are projected onto a preset depth image using the camera intrinsic parameters to obtain the two-dimensional position coordinates of each discrete point in the point cloud data in the depth image. The size of the preset depth image is the same as the size of the RGB image. For any discrete point, the vertical coordinate value in the camera position coordinates corresponding to the discrete point is set as the depth value of the discrete point in the preset depth image; The depth image of the target object is obtained based on the depth value of each discrete point in the preset depth image and the two-dimensional position coordinates of each discrete point in the depth image.

11. The method according to claim 1, characterized in that, The step of segmenting the point cloud data containing the target object using the three-dimensional position coordinates of the center point of the target object in the radar coordinate system and the radius of the circumscribed sphere of the target object to obtain point cloud data corresponding to the target object includes: The three-dimensional position coordinates of the center point of the target object in the radar coordinate system, the radius of the circumscribed sphere of the target object, and the point cloud data containing the target object are input into the minimum segmentation algorithm to perform point cloud segmentation, thereby obtaining the point cloud data corresponding to the target object.

12. A point cloud segmentation device for a target object, characterized in that, The device includes: The acquisition module is used to acquire point cloud data and RGB images containing a target object corresponding to a specified time period during the crane hoisting process, wherein the target object is the object that the crane needs to lift; The contour extraction module is used to extract contours from the RGB image to obtain each contour in the RGB image; The contour vector determination module is used to obtain the current feature vector of any contour based on the position of each pixel in the contour and the RGB value of each pixel. The matching module is used to determine the target contour of the target object among the contours by using the current feature vector of each contour and the standard contour feature vector of the target object. The center point position determination module is used to determine the three-dimensional position coordinates of the center point of the target object in the radar coordinate system based on the target outline of the target object; The circumscribed sphere radius determination module is used to determine the circumscribed sphere radius of the target object based on the extreme values ​​of the horizontal and vertical coordinates in the target contour and the camera intrinsic parameters. The segmentation module is used to segment the point cloud data containing the target object using the three-dimensional position coordinates of the center point of the target object in the radar coordinate system and the radius of the circumscribed sphere of the target object, so as to obtain the point cloud data corresponding to the target object.

13. An electronic device, characterized in that, The method includes at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that are executed by the at least one processor; the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-11.

14. A computer storage medium, characterized in that, The computer storage medium stores a computer program for performing the method according to any one of claims 1-11.

Images