Target object recognition method and apparatus, electronic device, and storage medium

Abstract

The application provides a target object recognition method and device, electronic equipment and storage medium, wherein the method comprises: acquiring a to-be-recognized depth map containing a target object; performing clustering segmentation on distortion points in a point cloud corresponding to the to-be-recognized depth map to obtain a point cloud of the target object; determining three-dimensional coordinates of each point in the point cloud of the target object in a geodetic coordinate system based on the point cloud of the target object and a coordinate conversion relationship between a camera coordinate system and the geodetic coordinate system corresponding to the to-be-recognized depth map; and determining spatial information of the target object based on the three-dimensional coordinates of each point in the geodetic coordinate system. The method, device, electronic equipment and storage medium provided by the application reduce the influence of environmental light conditions and do not need to consider the color or texture features of the target object, thereby improving the position recognition accuracy of the target object.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a target object recognition method, apparatus, electronic device, and storage medium. Background Technology

[0002] Excavators and other construction machinery are widely used in mining, building construction, road and bridge construction, and other construction sites. Among these, crushing is one of the most frequent operations performed by excavators. In order for excavators to perform crushing operations automatically, they need to be able to accurately identify the location of rocks, thereby smoothly controlling the breaker hammer to crush the rocks.

[0003] In existing technologies, cameras are installed on excavators to acquire images, and the acquired images are then identified to obtain information such as the location of the target object (rocks) to be crushed. However, because the image acquisition by the camera is easily affected by ambient lighting conditions, and the color variations and texture types of the target object are quite complex, the accuracy of the target object's location identification is poor, making it difficult to guide the excavator in automated crushing operations. Summary of the Invention

[0004] This invention provides a target object identification method, apparatus, electronic device, and storage medium to solve the technical problem that existing target object identification methods are easily affected by ambient lighting conditions and have poor identification accuracy.

[0005] This invention provides a target object identification method, comprising:

[0006] Obtain the depth map containing the target object;

[0007] Clustering and segmenting the distorted points in the point cloud corresponding to the depth map to be identified yields the point cloud of the target object.

[0008] Based on the point cloud of the target object and the coordinate transformation relationship between the camera coordinate system and the geodetic coordinate system corresponding to the depth map to be identified, the three-dimensional coordinates of each point in the point cloud of the target object in the geodetic coordinate system are determined.

[0009] Based on the three-dimensional coordinates of each point in the geodetic coordinate system, the spatial information of the target object is determined.

[0010] According to the target object recognition method provided by the present invention, the coordinate transformation relationship between the camera coordinate system and the geodetic coordinate system is determined based on the calibration depth map containing the calibration object and the three-dimensional coordinates of the calibration points on the calibration object in the geodetic coordinate system;

[0011] The calibration depth map and the depth map to be identified have the same camera coordinate system.

[0012] According to the target object recognition method provided by the present invention, the coordinate transformation relationship between the camera coordinate system and the geodetic coordinate system is determined based on the following steps:

[0013] Determine the pixel coordinates and depth values ​​of the calibration points on the calibration object in the calibration depth map;

[0014] Based on the pixel coordinates and depth value of the calibration point, the camera focal length of the depth camera, and the center pixel coordinates of the depth camera in the camera coordinate system, the three-dimensional coordinates of the calibration point in the camera coordinate system are determined.

[0015] Based on the three-dimensional coordinates of the calibration point in the camera coordinate system and the three-dimensional coordinates of the calibration point in the geodetic coordinate system, the coordinate transformation relationship between the camera coordinate system and the geodetic coordinate system is determined.

[0016] According to the target object identification method provided by the present invention, the calibrated object is a cylinder, and the calibrated point is the center of the upper surface of the cylinder.

[0017] According to the target object recognition method provided by the present invention, before the step of clustering and segmenting the distorted points in the point cloud corresponding to the depth map to be identified to obtain the point cloud of the target object, the method further includes:

[0018] Taking any point in the point cloud corresponding to the depth map to be identified as the target point, determine the neighboring points corresponding to the target point;

[0019] Based on the feature dimension values ​​of each neighboring point of the target point, the distortion point corresponding to the target point is determined;

[0020] The feature dimension value is determined by the three-dimensional coordinates of each point in the point cloud corresponding to the depth map to be identified in the camera coordinate system.

[0021] According to the target object recognition method provided by the present invention, determining the distortion point corresponding to the target point based on the feature dimension values ​​of each neighboring point of the target point includes:

[0022] Based on the distances between each neighboring point of the target point and the target point, determine the interior points corresponding to the target point;

[0023] Based on the feature dimension value change weights of each interior point of the target point and the preset threshold weights, the distortion point corresponding to the target point is determined.

[0024] The feature dimension value change weight of each interior point is determined based on the feature dimension value of each interior point and the feature dimension value of the target point.

[0025] According to the target object identification method provided by the present invention, the step of obtaining a depth map containing the target object includes:

[0026] Determine the collection area covering the target object;

[0027] Based on the size of the acquisition area, the acquisition tilt angle of the depth camera is determined;

[0028] The acquisition height of the depth camera is determined based on the acquisition tilt angle of the depth camera, the size of the acquisition area, the field of view angle and pixel accuracy of the depth camera;

[0029] The depth map to be identified is obtained based on the acquisition tilt angle and acquisition height of the depth camera.

[0030] This invention provides a target object identification device, comprising:

[0031] The acquisition unit is used to acquire a depth map containing the target object to be identified.

[0032] The identification unit is used to cluster and segment the distorted points in the point cloud corresponding to the depth map to be identified, so as to obtain the point cloud of the target object.

[0033] The transformation unit is used to determine the three-dimensional coordinates of each point in the point cloud of the target object in the geodetic coordinate system based on the coordinate transformation relationship between the camera coordinate system and the geodetic coordinate system corresponding to the depth map to be identified.

[0034] The determining unit is used to determine the spatial information of the target object based on the three-dimensional coordinates of each point in the geodetic coordinate system.

[0035] The present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the target object recognition method.

[0036] The present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the target object identification method.

[0037] The target object identification method, device, electronic device, and storage medium provided by this invention obtain the target object's point cloud by clustering and segmenting distorted points in the point cloud corresponding to the depth map to be identified, and determine the three-dimensional coordinates of each point in the target object's point cloud in the geodetic coordinate system based on the coordinate transformation relationship between the camera coordinate system and the geodetic coordinate system corresponding to the depth map to be identified, thereby determining the spatial information of the target object. Since the point cloud clustering and segmentation method is used to identify the target object from the environment, and the three-dimensional coordinates of the target object are transformed from the camera coordinate system to the geodetic coordinate system, the influence of ambient lighting conditions is reduced, and there is no need to consider the target object's color or texture features, which improves the accuracy of target object position identification, so that the spatial information of the target object can be directly used to guide the construction machinery. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0039] Figure 1 A schematic flowchart illustrating the target object identification method provided by the present invention;

[0040] Figure 2 A schematic diagram of the calibration method provided by the present invention;

[0041] Figure 3 A schematic diagram of the field of view of the depth camera provided by the present invention;

[0042] Figure 4 One of the schematic diagrams for data acquisition by the depth camera provided by the present invention;

[0043] Figure 5 This is the second schematic diagram of the depth camera acquisition provided by the present invention;

[0044] Figure 6 A schematic diagram of the target object identification device provided by the present invention;

[0045] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0047] Figure 1 This is a flowchart illustrating the target object identification method provided by the present invention. Figure 1 As shown, the method includes:

[0048] Step 110: Obtain the depth map containing the target object to be identified.

[0049] Specifically, the target object is the object that needs to be identified spatially. The depth map to be identified is the depth map obtained after collecting depth information from the scene containing the target object. The pixel values ​​in the depth map to be identified represent the distance (depth) from the image acquisition device to each point in the scene, containing the depth information of the target object. After identifying the depth information of the target object, the spatial information of the target object can be obtained. The spatial information includes the target object's position, shape, and size.

[0050] The depth map to be identified can be acquired using a depth camera. Depth cameras include Time-of-Flight (TOF) cameras, structured light cameras, and binocular vision cameras. Preferably, a TOF camera can be used to acquire the depth map. TOF cameras determine the precise distance to the target object by measuring the round-trip time of a light pulse. They have advantages such as long detection range and minimal susceptibility to ambient light interference, making them suitable for use in harsh environments like construction sites.

[0051] Step 120: Cluster and segment the distorted points in the point cloud corresponding to the depth map to be identified to obtain the point cloud of the target object.

[0052] Specifically, the depth map to be identified is transformed into a corresponding point cloud. Each pixel in the depth map represents the distance from an object at a specific coordinate within the depth camera's field of view to the camera's view plane. The depth camera's internal calibration parameters include the camera focal length and the center pixel coordinates. The camera focal length includes the lateral focal length f. x and longitudinal focal length f y The center pixel coordinates are the center coordinates of the depth camera sensor center on the imaging plane (c...). x c yIf the coordinates of any pixel in the depth map to be identified are (u, v) and the depth value corresponding to this pixel is dep, then the three-dimensional coordinates (x, y, z) of the point corresponding to this pixel in the camera coordinate system of the depth camera can be resolved as follows:

[0053]

[0054] By transforming each pixel in the depth map to be identified, the point cloud corresponding to the depth map can be obtained.

[0055] In real-world scenarios, the positional relationships of points on the same object's surface are usually continuous, meaning there are no abrupt changes in the positional relationships between points. Only at the object's edges will there be changes in spatial distance from the object's surrounding environment. Correspondingly, distortion points will appear at the edge locations in the object's point cloud. Distortion points in the point cloud are points of abrupt positional changes; that is, the positional relationship between a distortion point and other points in its neighborhood is discontinuous.

[0056] The point cloud containing the target object can be obtained by calculating the normal vector and / or curvature of each point in the point cloud corresponding to the depth map to be identified, and by determining the changes in the normal vector and / or curvature. After clustering and segmenting these distorted points, the point cloud of the target object can be obtained.

[0057] Clustering segmentation algorithms can include RanSaC (random sampling consensus) algorithm, point cloud segmentation algorithm based on proximity information, Euclidean algorithm, region growing algorithm, and super-volume clustering segmentation algorithm, etc., and the embodiments of the present invention do not specifically limit them.

[0058] Step 130: Based on the point cloud of the target object and the coordinate transformation relationship between the camera coordinate system and the geodetic coordinate system corresponding to the depth map to be identified, determine the three-dimensional coordinates of each point in the point cloud of the target object in the geodetic coordinate system.

[0059] Specifically, the camera coordinate system corresponding to the depth map to be identified is a three-dimensional coordinate system established with the optical center of the depth camera that acquired the depth map as its origin, and its central axis pointing directly in front of the camera. The geodetic coordinate system is a three-dimensional spatial coordinate system established with the Earth's center as its reference point.

[0060] The coordinate transformation relationship between two coordinate systems can be represented by a rotation matrix and a translation vector. The rotation matrix describes the rotation relationship between the coordinate axes in the two coordinate systems, and the translation vector describes the translation relationship between the origins of the two coordinate systems.

[0061] Based on the coordinate transformation relationship between the camera coordinate system and the geodetic coordinate system corresponding to the depth map to be identified, each point in the point cloud of the target object can be transformed to the geodetic coordinate system to obtain the three-dimensional coordinates of each point.

[0062] Step 140: Determine the spatial information of the target object based on the three-dimensional coordinates of each point in the geodetic coordinate system.

[0063] Based on the three-dimensional coordinates of each point, the spatial information of the target object in the geodetic coordinate system is determined. The positional information of the target object can be determined based on the three-dimensional coordinates of each point; for example, the average value of the three-dimensional coordinates of each point can be calculated, and the point corresponding to the average value can be taken as the center point of the target object, thus determining its position. The shape information of the target object can also be determined based on the three-dimensional coordinates of each point; for example, curve fitting can be performed on the three-dimensional coordinates of each point, and the shape of the target object can be determined based on the fitting result. Furthermore, the size information of the target object can be determined based on the three-dimensional coordinates of each point; for example, the length, width, and height of the target object can be obtained by calculating the difference between the three-dimensional coordinates of the boundary points of the target object.

[0064] The following example illustrates the automatic ore identification and crushing operation using an excavator equipped with a hydraulic breaker. Before automatic crushing, the ore needs to be identified. First, a Time-of-Flight (TOF) camera can collect depth information of the work scene containing the ore, obtaining a depth map to be identified. Based on the TOF camera's internal calibration parameters, the depth map is converted into a point cloud. Second, by using the normal vector or curvature of each point in the point cloud, distortion points are identified, and then clustered to obtain the ore's point cloud. Third, based on the coordinate transformation relationship between the TOF camera's coordinate system and the geodetic coordinate system, each point in the ore's point cloud is transformed to the geodetic coordinate system, obtaining the three-dimensional coordinates of each point in the geodetic coordinate system. Finally, based on the three-dimensional coordinates of each point in the ore in the geodetic coordinate system, the center position and size of the ore are calculated, thereby controlling the excavator to perform the crushing operation.

[0065] The target object recognition method provided in this invention obtains the target object's point cloud by clustering and segmenting distorted points in the point cloud corresponding to the depth map containing the target object. Based on the coordinate transformation relationship between the camera coordinate system and the geodetic coordinate system corresponding to the depth map, the three-dimensional coordinates of each point in the target object's point cloud in the geodetic coordinate system are determined, thereby determining the spatial information of the target object. Since the point cloud clustering and segmentation method is used to identify the target object from the environment, and the three-dimensional coordinates of the target object are transformed from the camera coordinate system to the geodetic coordinate system, the influence of ambient lighting conditions is reduced, and there is no need to consider the target object's color or texture features, which improves the accuracy of target object location recognition. This allows the spatial information of the target object to be directly used to guide the construction machinery.

[0066] Based on the above embodiments, the coordinate transformation relationship between the camera coordinate system and the geodetic coordinate system is determined based on the calibration depth map containing the calibration object and the three-dimensional coordinates of the calibration points on the calibration object in the geodetic coordinate system;

[0067] The calibration depth map and the depth map to be identified have the same camera coordinate system.

[0068] Specifically, the coordinate transformation relationship between the camera coordinate system and the geodetic coordinate system can be obtained by setting up calibration objects and performing calculations. A calibration depth map is used to calibrate the coordinate transformation relationship between the camera coordinate system and the geodetic coordinate system.

[0069] The camera coordinate systems corresponding to the calibration depth map where the calibration object is located and the depth map to be identified where the target object is located should be the same. In other words, the internal calibration parameters of the depth camera that acquires the calibration depth map should be the same as the internal calibration parameters of the depth camera that acquires the depth map to be identified; the field of view of the depth camera that acquires the calibration depth map should be the same as the field of view of the depth camera that acquires the depth map to be identified.

[0070] During calibration, multiple calibration points can be set up in the scene where the target object is located. By using the 3D coordinates of the calibration points on the calibration points in the camera coordinate system and the 3D coordinates of the calibration points on the calibration points in the geodetic coordinate system, the coordinate transformation relationship between the camera coordinate system and the geodetic coordinate system can be obtained, namely the rotation matrix and translation vector between the two coordinate systems. The 3D coordinates of the calibration points on the calibration points in the camera coordinate system can be obtained by acquiring a calibration depth map.

[0071] Based on any of the above embodiments, the coordinate transformation relationship between the camera coordinate system and the geodetic coordinate system is determined based on the following steps:

[0072] Determine the pixel coordinates and depth values ​​of the calibration points on the calibration object in the calibration depth map;

[0073] Based on the pixel coordinates and depth value of the calibration point, the camera focal length of the depth camera, and the center pixel coordinates of the depth camera in the camera coordinate system, determine the three-dimensional coordinates of the calibration point in the camera coordinate system.

[0074] Based on the three-dimensional coordinates of the calibration point in the camera coordinate system and the three-dimensional coordinates of the calibration point in the geodetic coordinate system, the coordinate transformation relationship between the camera coordinate system and the geodetic coordinate system is determined.

[0075] Specifically, the calibration object can be an object with a specific structure, and the calibration point can be a specific point on the calibration object. For example, the calibration object can be a cylinder, and the calibration point can be the center of the upper surface of the cylinder; the calibration object can be a cube, and the calibration point can be a vertex of the cube, etc.

[0076] First, multiple calibration objects are set up in the field of view of the depth camera, and the pixel coordinates and depth values ​​of the calibration points on the calibration objects in the calibration depth map are determined. Following the method in the above embodiment, the three-dimensional coordinates P of the calibration points in the camera coordinate system are calculated based on the pixel coordinates and depth values ​​of the calibration points, the camera focal length of the depth camera, and the center pixel coordinates of the depth camera in the camera coordinate system. cam (x cam y cam , z cam ).

[0077] Secondly, GPS (Global Positioning System) equipment or RTK (Real-time kinematic) equipment can be used to measure the three-dimensional coordinates P of the calibration point in the geodetic coordinate system. gps (x gps y gps , z gps ).

[0078] Furthermore, since the coordinate transformation relationship between the camera coordinate system and the geodetic coordinate system satisfies a rigid body transformation, the equation for the coordinate transformation relationship can be established:

[0079]

[0080] Where R is the rotation matrix between the camera coordinate system and the geodetic coordinate system, and T is the translation vector between the camera coordinate system and the geodetic coordinate system.

[0081] Figure 2 A schematic diagram of the calibration method provided by the present invention is shown below. Figure 2 As shown, the above equations can be solved by using the three-dimensional coordinates of multiple calibration points in the camera coordinate system O_cam and the three-dimensional coordinates in the geodetic coordinate system O_gps, to obtain the rotation matrix R and the translation vector T, that is, to obtain the coordinate transformation relationship between the camera coordinate system and the geodetic coordinate system.

[0082] Based on any of the above embodiments, the calibration object is a cylinder, and the calibration point is the center of the upper surface of the cylinder.

[0083] Specifically, based on the geometric properties of a cylinder, the distance from a point on the cylinder's surface to the cylinder's central axis is always equal to the cylinder's cross-sectional radius r0. Assuming P(x, y, z) is any point on the cylinder's surface, P0(x0, y0, z0) is a point on the cylinder's central axis, and (a, b, c) is the unit vector of the cylinder's central axis, then the cylinder can be expressed as:

[0084]

[0085] As can be seen from the above formula, only six parameters {x0, y0, z0, a, b, c} are needed to fit a cylinder.

[0086] Since the center of the upper surface of the cylinder can be easily determined from different viewpoints, its cross-sectional radius can be determined in advance. Based on a set of point cloud values ​​P0(x0, y0, z0), (a, b, c) can be fitted.

[0087] Therefore, in the above embodiments, considering the imaging error of the depth map, it is necessary to ensure that the calibration object can still be accurately located at the calibration point under certain changes in viewing angle. The calibration object can be selected as a cylinder, and the calibration point can be selected as the center of the upper surface of the cylinder.

[0088] Based on any of the above embodiments, the steps preceding step 120 include:

[0089] Take any point in the point cloud corresponding to the depth map to be identified as the target point and determine the neighboring points corresponding to the target point;

[0090] Based on the feature dimension values ​​of each neighboring point of the target point, determine the distortion point corresponding to the target point;

[0091] The feature dimension value is determined by the three-dimensional coordinates of each point in the point cloud corresponding to the depth map to be identified in the camera coordinate system.

[0092] Specifically, distortion points can be determined using a nearest neighbor search method. By comparing the positional change characteristics of a point with those of its neighbors, distortion points can be identified.

[0093] You can take any point in the point cloud corresponding to the depth map to be identified as the target point and determine the neighboring points corresponding to the target point. The selection of neighboring points can be based on the distance between the points.

[0094] Based on the three-dimensional coordinates and normal vectors of each neighboring point in the camera coordinate system, the feature dimension values ​​of each neighboring point of the target point are determined. These feature dimension values ​​are used to examine the positional changes of each neighboring point.

[0095] The method for determining feature dimension values ​​is as follows:

[0096] Step 1: In the established K-D tree, which contains all points in the point cloud corresponding to the depth map to be identified, determine any point P. a (x a y a , z a ), and determine point P. a The K most recent points are set as P;

[0097] Step 2, Order COV = Q TQ. Where Q is point P. a The nearest K points and point P a The three-dimensional coordinate difference matrix between them, COV is the covariance matrix, and after performing singular value decomposition on the covariance matrix, the eigenvector corresponding to the smallest eigenvalue is obtained [u a v a w a ], take this vector as P a The normal vector;

[0098] Step 3: Obtain point P a The feature dimension value can be (x a y a , z a u a v a w a ).

[0099] For a target point, the changes in the feature dimension values ​​of its neighboring points are examined to determine the distortion point corresponding to the target point.

[0100] Based on any of the above embodiments, determining the distortion point corresponding to the target point based on the feature dimension values ​​of each neighboring point of the target point includes:

[0101] Based on the distances between the target point and each of its neighboring points, determine the interior points corresponding to the target point;

[0102] Based on the feature dimension value change weights of each interior point of the target point and the preset threshold weights, the distortion points corresponding to the target point are determined.

[0103] The weights for the changes in the feature dimension values ​​of each inlier are determined based on the feature dimension values ​​of each inlier and the feature dimension values ​​of the target point.

[0104] Specifically, for each neighboring point of the target point, the interior points corresponding to the target point are determined. For example, if the target point is point P... a For any given K-neighborhood (the K nearest neighbors), calculate the distance between each neighboring point and the target point, and take the average as mean. For any neighboring point, if its distance to the target point is less than or equal to the average, then the neighboring point is an interior point of the target point; if its distance to the target point is greater than the average, then the neighboring point is not an interior point of the target point.

[0105] The feature dimension value change weights of each inlier are determined based on the distance between each inlier and the target point, as well as the normal vector deflection angle between each inlier and the target point.

[0106] For example, for interior point P b The feature dimension value of this point is (xb y b , z b u b v b w b According to the target point P a Feature dimension value (x) a y a , z a u a v a w a This allows us to calculate the distance between the point and the target point P. a The distance between them is:

[0107]

[0108] The distance between this point and the target point P can be calculated. a The deflection angle of the normal vector between them is:

[0109] |u a u b +v a v b +w a w b -1|

[0110] Then interior point P b The weight val for the change in feature dimension values ​​can be expressed by the formula:

[0111]

[0112] If the weight of the change in the feature dimension value of any inlier is greater than the preset threshold weight, then the inlier can be determined as the distortion point corresponding to the target point.

[0113] Based on any of the above embodiments, step 110 includes:

[0114] Determine the collection area covering the target object;

[0115] The acquisition tilt angle of the depth camera is determined based on the size of the acquisition area;

[0116] The acquisition height of the depth camera is determined based on the acquisition tilt angle, the size of the acquisition area, the field of view angle and pixel accuracy of the depth camera.

[0117] The depth map to be identified is obtained based on the acquisition tilt angle and acquisition height of the depth camera.

[0118] Specifically, when using a depth camera to acquire depth maps of ore, the depth camera is typically mounted on the operating machinery. However, the machinery generates strong vibrations during operation, affecting the lifespan and sensing accuracy of the depth camera. Therefore, the depth camera in this embodiment can be independently installed at the construction site of the operating machinery.

[0119] The field of view of a depth camera can be determined by the acquisition tilt angle and acquisition height. The acquisition tilt angle is the angle between the central axis of the depth camera and the vertical direction; the acquisition height is the installation height of the depth camera relative to the horizontal ground.

[0120] Figure 3 A schematic diagram of the field of view of the depth camera provided by the present invention, as shown below. Figure 3 As shown, for the depth camera O_camera, its field of view is a rectangular region. The four sides of the rectangular region are fx_a, fx_b, fy_a, and fy_b. The size of the field of view can be represented by x and y. x is the distance between fx_a and fx_b, and y is the distance between fy_a and fy_b.

[0121] The central axis can be represented by the field of view center line O_camera-f. The field of view angle of the depth camera can be represented by the angle fx_a-O_camera-fx_b (denoted as α) and the angle fy_a-O_camera-fy_b (denoted as β).

[0122] Figure 4 This is one of the schematic diagrams of depth camera acquisition provided by the present invention. Figure 5 This is the second schematic diagram of the depth camera acquisition provided by the present invention, as shown below. Figure 4 and Figure 5 As shown, the depth camera O_camera is mounted on a separate mounting rod P, with a acquisition height of len and an acquisition tilt angle of θ.

[0123] Before determining the acquisition tilt angle and acquisition height of the depth camera, the acquisition area covering the target object should first be determined. The acquisition tilt angle of the depth camera should ensure that the size of the acquisition field of view is greater than or equal to the size of the acquisition area.

[0124] Based on the depth camera's acquisition tilt angle, the size of the acquisition area, and the depth camera's field of view and pixel accuracy, the acquisition height of the depth camera is determined, expressed by the formula:

[0125]

[0126] Among them, pre level This represents the pixel precision level of the depth camera; pre0 sets the pixel precision.

[0127] To ensure the depth camera achieves the highest possible accuracy, it is necessary to solve the above four inequalities as constraints. The calculated len is the optimal value for the acquisition height.

[0128] Based on any of the above embodiments Figure 6 This is a schematic diagram of the target object identification device provided by the present invention, as shown below. Figure 6 As shown, the device includes:

[0129] The acquisition unit 610 is used to acquire a depth map containing the target object to be identified.

[0130] The recognition unit 620 is used to cluster and segment the distorted points in the point cloud corresponding to the depth map to be recognized, so as to obtain the point cloud of the target object.

[0131] The transformation unit 630 is used to determine the three-dimensional coordinates of each point in the point cloud of the target object in the geodetic coordinate system based on the coordinate transformation relationship between the camera coordinate system and the geodetic coordinate system corresponding to the depth map to be identified.

[0132] The determination unit 640 is used to determine the spatial information of the target object based on the three-dimensional coordinates of each point in the geodetic coordinate system.

[0133] The target object recognition device provided in this embodiment of the invention obtains the target object's point cloud by clustering and segmenting distorted points in the point cloud corresponding to the depth map to be recognized, and determines the three-dimensional coordinates of each point in the target object's point cloud in the geodetic coordinate system based on the coordinate transformation relationship between the camera coordinate system and the geodetic coordinate system corresponding to the depth map to be recognized, thereby determining the spatial information of the target object. Since the point cloud clustering and segmentation method is used to identify the target object from the environment, and the three-dimensional coordinates of the target object are transformed from the camera coordinate system to the geodetic coordinate system, the influence of ambient lighting conditions is reduced, and there is no need to consider the target object's color or texture features, which improves the accuracy of the target object's position recognition, so that the spatial information of the target object can be directly used to guide the construction machinery.

[0134] Based on any of the above embodiments, the coordinate transformation relationship between the camera coordinate system and the geodetic coordinate system is determined based on the calibration depth map containing the calibration object and the three-dimensional coordinates of the calibration points on the calibration object in the geodetic coordinate system;

[0135] The calibration depth map and the depth map to be identified have the same camera coordinate system.

[0136] Based on any of the above embodiments, it further includes:

[0137] The conversion unit is used to determine the pixel coordinates and depth values ​​of the calibration points on the calibration object in the calibration depth map;

[0138] Based on the pixel coordinates and depth value of the calibration point, the camera focal length of the depth camera, and the center pixel coordinates of the depth camera in the camera coordinate system, determine the three-dimensional coordinates of the calibration point in the camera coordinate system.

[0139] Based on the three-dimensional coordinates of the calibration point in the camera coordinate system and the three-dimensional coordinates of the calibration point in the geodetic coordinate system, the coordinate transformation relationship between the camera coordinate system and the geodetic coordinate system is determined.

[0140] Based on any of the above embodiments, the calibration object is a cylinder, and the calibration point is the center of the upper surface of the cylinder.

[0141] Based on any of the above embodiments, it further includes:

[0142] The neighboring point determination unit is used to determine the neighboring points corresponding to any point in the point cloud corresponding to the depth map to be identified as the target point.

[0143] The distortion point determination unit is used to determine the distortion point corresponding to the target point based on the feature dimension values ​​of each neighboring point of the target point;

[0144] The feature dimension value is determined by the three-dimensional coordinates of each point in the point cloud corresponding to the depth map to be identified in the camera coordinate system.

[0145] Based on any of the above embodiments, the distortion point determination unit is specifically used for:

[0146] Based on the distances between the target point and each of its neighboring points, determine the interior points corresponding to the target point;

[0147] Based on the feature dimension value change weights of each interior point of the target point and the preset threshold weights, the distortion points corresponding to the target point are determined.

[0148] The weights for the changes in the feature dimension values ​​of each inlier are determined based on the feature dimension values ​​of each inlier and the feature dimension values ​​of the target point.

[0149] Based on any of the above embodiments, the acquisition unit is specifically used for:

[0150] Determine the collection area covering the target object;

[0151] The acquisition tilt angle of the depth camera is determined based on the size of the acquisition area;

[0152] The acquisition height of the depth camera is determined based on the acquisition tilt angle, the size of the acquisition area, the field of view angle and pixel accuracy of the depth camera.

[0153] The depth map to be identified is obtained based on the acquisition tilt angle and acquisition height of the depth camera.

[0154] Based on any of the above embodiments Figure 7This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 7 As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communications bus 740, wherein the processor 710, communications interface 720, and memory 730 communicate with each other via the communications bus 740. The processor 710 can call logical commands stored in the memory 730 to execute the following methods:

[0155] Obtain a depth map containing the target object; cluster and segment the distorted points in the point cloud corresponding to the depth map to obtain the point cloud of the target object; based on the point cloud of the target object and the coordinate transformation relationship between the camera coordinate system and the geodetic coordinate system corresponding to the depth map to be identified, determine the three-dimensional coordinates of each point in the point cloud of the target object in the geodetic coordinate system; based on the three-dimensional coordinates of each point in the geodetic coordinate system, determine the spatial information of the target object.

[0156] Furthermore, the logical commands in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several commands to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0157] The processor in the electronic device provided in this embodiment of the invention can call logical instructions in the memory to implement the above method. Its specific implementation method is the same as the aforementioned method implementation method and can achieve the same beneficial effects, which will not be repeated here.

[0158] This invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the methods provided in the above embodiments, including, for example:

[0159] Obtain a depth map containing the target object; cluster and segment the distorted points in the point cloud corresponding to the depth map to obtain the point cloud of the target object; based on the point cloud of the target object and the coordinate transformation relationship between the camera coordinate system and the geodetic coordinate system corresponding to the depth map to be identified, determine the three-dimensional coordinates of each point in the point cloud of the target object in the geodetic coordinate system; based on the three-dimensional coordinates of each point in the geodetic coordinate system, determine the spatial information of the target object.

[0160] When the computer program stored on the non-transitory computer-readable storage medium provided in this embodiment of the invention is executed, it implements the above method. Its specific implementation method is the same as the aforementioned method implementation method and can achieve the same beneficial effects, which will not be repeated here.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A target object recognition method, characterized in that, include: Obtain the depth map containing the target object; Clustering and segmenting the distorted points in the point cloud corresponding to the depth map to be identified yields the point cloud of the target object. Based on the point cloud of the target object and the coordinate transformation relationship between the camera coordinate system and the geodetic coordinate system corresponding to the depth map to be identified, the three-dimensional coordinates of each point in the point cloud of the target object in the geodetic coordinate system are determined. Based on the three-dimensional coordinates of each point in the geodetic coordinate system, the spatial information of the target object is determined; Before the step of clustering and segmenting the distorted points in the point cloud corresponding to the depth map to be identified to obtain the point cloud of the target object, the method further includes: Taking any point in the point cloud corresponding to the depth map to be identified as the target point, determine the neighboring points corresponding to the target point; Based on the feature dimension values ​​of each neighboring point of the target point, the distortion point corresponding to the target point is determined; The feature dimension values ​​of each neighboring point include the three-dimensional coordinates and normal vectors of each neighboring point in the camera coordinate system; The step of determining the distortion point corresponding to the target point based on the feature dimension values ​​of each neighboring point of the target point includes: Based on the distances between each neighboring point of the target point and the target point, determine the interior points corresponding to the target point; Based on the feature dimension value change weights of each interior point of the target point and the preset threshold weights, the distortion point corresponding to the target point is determined. The feature dimension value change weight of each interior point is determined based on the feature dimension value of each interior point and the feature dimension value of the target point. The feature dimension value change weights for each interior point are determined based on the following steps: Based on the feature dimension values ​​of each inlier point and the feature dimension value of the target point, determine the distance and normal vector deflection angle between each inlier point and the target point. Based on the distance and normal vector angle between each inlier point and the target point, and the average distance between each neighboring point of the target point and the target point, the feature dimension value change weight of each inlier point is determined.

2. The target object identification method according to claim 1, characterized in that, The coordinate transformation relationship between the camera coordinate system and the geodetic coordinate system is determined based on the calibration depth map containing the calibration object and the three-dimensional coordinates of the calibration points on the calibration object in the geodetic coordinate system; The calibration depth map and the depth map to be identified have the same camera coordinate system.

3. The target object identification method according to claim 2, characterized in that, The coordinate transformation relationship between the camera coordinate system and the geodetic coordinate system is determined based on the following steps: Determine the pixel coordinates and depth values ​​of the calibration points on the calibration object in the calibration depth map; Based on the pixel coordinates and depth value of the calibration point, the camera focal length of the depth camera, and the center pixel coordinates of the depth camera in the camera coordinate system, the three-dimensional coordinates of the calibration point in the camera coordinate system are determined. Based on the three-dimensional coordinates of the calibration point in the camera coordinate system and the three-dimensional coordinates of the calibration point in the geodetic coordinate system, the coordinate transformation relationship between the camera coordinate system and the geodetic coordinate system is determined.

4. The target object identification method according to claim 3, characterized in that, The calibration object is a cylinder, and the calibration point is the center of the upper surface of the cylinder.

5. The target object identification method according to any one of claims 1 to 4, characterized in that, The process of obtaining the depth map containing the target object includes: Determine the collection area covering the target object; Based on the size of the acquisition area, the acquisition tilt angle of the depth camera is determined; The acquisition height of the depth camera is determined based on the acquisition tilt angle of the depth camera, the size of the acquisition area, the field of view angle and pixel accuracy of the depth camera; The depth map to be identified is obtained based on the acquisition tilt angle and acquisition height of the depth camera.

6. A target object identification device, characterized in that, include: The acquisition unit is used to acquire a depth map containing the target object to be identified. The identification unit is used to cluster and segment the distorted points in the point cloud corresponding to the depth map to be identified, so as to obtain the point cloud of the target object. The transformation unit is used to determine the three-dimensional coordinates of each point in the point cloud of the target object in the geodetic coordinate system based on the coordinate transformation relationship between the camera coordinate system and the geodetic coordinate system corresponding to the point cloud of the target object and the depth map to be identified. The determining unit is used to determine the spatial information of the target object based on the three-dimensional coordinates of each point in the geodetic coordinate system; Also includes: The neighboring point determination unit is used to determine the neighboring points corresponding to the target point, taking any point in the point cloud corresponding to the depth map to be identified as the target point; The distortion point determination unit is used to determine the distortion point corresponding to the target point based on the feature dimension values ​​of each neighboring point of the target point; The feature dimension values ​​of each neighboring point include the three-dimensional coordinates and normal vectors of each neighboring point in the camera coordinate system; The step of determining the distortion point corresponding to the target point based on the feature dimension values ​​of each neighboring point of the target point includes: Based on the distances between each neighboring point of the target point and the target point, determine the interior points corresponding to the target point; Based on the feature dimension value change weights of each interior point of the target point and the preset threshold weights, the distortion point corresponding to the target point is determined. The feature dimension value change weight of each interior point is determined based on the feature dimension value of each interior point and the feature dimension value of the target point. The feature dimension value change weights for each interior point are determined based on the following steps: Based on the feature dimension values ​​of each inlier point and the feature dimension value of the target point, determine the distance and normal vector deflection angle between each inlier point and the target point. Based on the distance and normal vector angle between each inlier point and the target point, and the average distance between each neighboring point of the target point and the target point, the feature dimension value change weight of each inlier point is determined.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the target object identification method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the target object identification method as described in any one of claims 1 to 5.

Images