Laser radar data processing method and device

Abstract

The present application discloses a laser radar data processing method and device, which belongs to the field of computer vision technology. The method includes: obtaining a target point cloud of a reference object through a laser radar included in a target sensor, the reference object is used to calibrate the target sensor, and the target point cloud includes a first feature point located in the reference object and a second feature point located at the boundary of the reference object; determining a first error function based on the first feature point in the target point cloud, and determining a second error function based on the second feature point in the target point cloud, the first error function and the second error function are used to determine the relative transformation relationship between the coordinate system where the laser radar is located and the coordinate system where the reference object is located; determining the first position data of the target corner point of the reference object in the coordinate system where the laser radar is located based on the first error function and the second error function. The data used to determine the error function is more comprehensive, and the determined error function has high stability, thereby improving the accuracy of the determined corner point position.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer vision technology, and in particular to a laser radar data processing method and device. Background Art

[0002] With the development of computer vision technology, a target sensor including a camera and a laser radar, namely a lidar-camera, has emerged. The use of this target sensor can combine the advantages of cameras and laser radars in acquiring image data. Before using the target sensor, the target sensor needs to be calibrated with external parameters. During the external parameter calibration process, it is necessary to determine the first position data of the corner point in the coordinate system where the laser radar is located, that is, to perform laser radar data processing. Among them, the corner point refers to the boundary intersection of two different areas on the reference object, and the reference object is, for example, a calibration plate. Summary of the invention

[0003] The embodiment of the present application provides a laser radar data processing method and device, which can be used to achieve the effect of laser radar data processing during external parameter calibration. The technical solution is as follows:

[0004] On the one hand, an embodiment of the present application provides a laser radar data processing method, the method comprising:

[0005] Acquire a target point cloud of a reference object by a laser radar included in the target sensor, wherein the reference object is used to calibrate the target sensor, and the target point cloud includes a first feature point located within the reference object and a second feature point located at a boundary of the reference object;

[0006] Determine a first error function based on a first feature point in the target point cloud, and determine a second error function based on a second feature point in the target point cloud, wherein the first error function and the second error function are used to determine a relative transformation relationship between a coordinate system where the laser radar is located and a coordinate system where the reference object is located;

[0007] Based on the first error function and the second error function, first position data of the target corner point of the reference object in the coordinate system where the laser radar is located is determined.

[0008] In a possible implementation, the step of acquiring a target point cloud of a reference object by using a laser radar included in the target sensor includes:

[0009] Acquire an initial transformation matrix of the target sensor, where the initial transformation matrix is ​​used to represent an initial posture relationship between a laser radar included in the target sensor and the reference object;

[0010] Acquire an initial point cloud based on the laser radar scanning;

[0011] The initial point cloud is segmented based on the initial transformation matrix to obtain the target point cloud.

[0012] In a possible implementation manner, determining a first error function based on a first feature point in the target point cloud, and determining a second error function based on a second feature point in the target point cloud includes:

[0013] determining the first error function based on the first feature point, the initial transformation matrix, and the width of the reference object;

[0014] The second error function is determined based on the second feature points, the initial transformation matrix, and the width of the reference object.

[0015] In a possible implementation manner, the first error function is:

[0016] Among them, the

[0017] Said

[0018] Said

[0019] The x i , the y i With the z i The calculation formula is:

[0020] The loss1 is the first error function, N is the number of the first feature points, i is the identifier of the first feature point, d is the width of the reference object, H is the initial transformation matrix, and pi is the three-dimensional coordinate of the first feature point.

[0021] In a possible implementation, the second error function is:

[0022] Among them, the

[0023] The x k With the y k The calculation formula is:

[0024] The loss2 is the second error function, the M is the number of the second feature points, the k is the identifier of the second feature point, the d is the width of the reference object, the H is the initial transformation matrix, and the Qk is the three-dimensional coordinate of the second feature point,

[0025] In a possible implementation manner, obtaining an initial transformation matrix of the target sensor includes:

[0026] Acquire second position data of an initial corner point of the reference object in a coordinate system where a camera included in the target sensor is located;

[0027] Determine first extrinsic parameter data of the target sensor based on the second position data of the initial corner point and the position information of each marking point included in the reference object, wherein the first extrinsic parameter data is used to represent the target posture relationship between the camera included in the target sensor and the reference object;

[0028] Acquire second extrinsic parameter data of the target sensor, where the second extrinsic parameter data is used to represent an initial posture relationship between a camera and a laser radar included in the target sensor;

[0029] The initial transformation matrix is ​​calculated based on the first extrinsic parameter data and the second extrinsic parameter data.

[0030] In a possible implementation manner, determining first position data of a target corner point of the reference object in a coordinate system where the laser radar is located based on the first error function and the second error function includes:

[0031] Iteratively optimizing an initial transformation matrix of the target sensor based on the first error function and the second error function to obtain a target transformation matrix, wherein the target transformation matrix is ​​used to represent a relative transformation relationship between a coordinate system where the laser radar is located and a coordinate system where the reference object is located;

[0032] The first position data of the target corner point is calculated based on the target transformation matrix and the size information of the reference object.

[0033] In a possible implementation manner, segmenting the initial point cloud based on the initial transformation matrix to obtain the target point cloud includes:

[0034] Selecting, from the initial point cloud, a feature point whose distance measurement meets the distance requirement and whose data type is single echo data as the first feature point, wherein the distance requirement is determined according to the initial transformation matrix;

[0035] The second feature point is determined from the initial point cloud based on the first feature point to obtain a target point cloud including the first feature point and the second feature point.

[0036] In a possible implementation manner, determining the second feature point from the initial point cloud based on the first feature point includes:

[0037] A feature point whose distance from the first feature point is not greater than a distance threshold and whose data type is dual echo data is selected from the initial point cloud as the second feature point.

[0038] On the other hand, a laser radar data processing device is provided, the device comprising:

[0039] an acquisition module, configured to acquire a target point cloud of a reference object through a laser radar included in the target sensor, wherein the reference object is used to calibrate the target sensor, and the target point cloud includes a first feature point located within the reference object and a second feature point located at a boundary of the reference object;

[0040] A determination module, configured to determine a first error function based on a first feature point in the target point cloud, and to determine a second error function based on a second feature point in the target point cloud, wherein the first error function and the second error function are used to determine a relative transformation relationship between a coordinate system where the laser radar is located and a coordinate system where the reference object is located;

[0041] The determination module is further used to determine first position data of the target corner point of the reference object in the coordinate system where the laser radar is located based on the first error function and the second error function.

[0042] In one possible implementation, the acquisition module is used to acquire an initial transformation matrix of the target sensor, where the initial transformation matrix is ​​used to represent an initial posture relationship between a laser radar included in the target sensor and the reference object; to acquire an initial point cloud obtained based on scanning by the laser radar; and to segment the initial point cloud based on the initial transformation matrix to obtain the target point cloud.

[0043] In a possible implementation, the determination module is used to determine the first error function based on the first feature point, the initial transformation matrix and the width of the reference object; and determine the second error function based on the second feature point, the initial transformation matrix and the width of the reference object.

[0044] In a possible implementation manner, the first error function is:

[0045] Among them, the

[0046] Said

[0047] Said

[0048] The x i , the y i With the z i The calculation formula is:

[0049] The loss1 is the first error function, N is the number of the first feature points, i is the identifier of the first feature point, d is the width of the reference object, H is the initial transformation matrix, and pi is the three-dimensional coordinate of the first feature point.

[0050] In a possible implementation, the second error function is: Among them, the The x k With the y k The calculation formula is: The loss2 is the second error function, the M is the number of the second feature points, the k is the identifier of the second feature point, the d is the width of the reference object, the H is the initial transformation matrix, the Qk is the three-dimensional coordinate of the second feature point,

[0051] In a possible implementation, the acquisition module is used to obtain the second position data of the initial corner point of the reference object in the coordinate system where the camera included in the target sensor is located; determine the first external parameter data of the target sensor based on the second position data of the initial corner point and the position information of each mark point included in the reference object, and the first external parameter data is used to represent the target pose relationship between the camera included in the target sensor and the reference object; obtain the second external parameter data of the target sensor, and the second external parameter data is used to represent the initial pose relationship between the camera included in the target sensor and the laser radar; calculate the initial transformation matrix based on the first external parameter data and the second external parameter data.

[0052] In one possible implementation, the determination module is used to iteratively optimize the initial transformation matrix of the target sensor based on the first error function and the second error function to obtain a target transformation matrix, wherein the target transformation matrix is ​​used to represent the relative transformation relationship between the coordinate system where the laser radar is located and the coordinate system where the reference object is located; and calculate the first position data of the target corner point based on the target transformation matrix and the size information of the reference object.

[0053] In a possible implementation, the acquisition module is used to select a feature point whose ranging distance meets the distance requirement and whose data type is single echo data from the initial point cloud as the first feature point, and the distance requirement is determined according to the initial transformation matrix; based on the first feature point, the second feature point is determined from the initial point cloud to obtain a target point cloud including the first feature point and the second feature point.

[0054] In a possible implementation, the acquisition module is used to select, from the initial point cloud, a feature point whose distance from the first feature point is not greater than a distance threshold and whose data type is dual echo data, as the second feature point.

[0055] On the other hand, a computer device is provided, which includes a processor and a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the computer device implements any of the above-mentioned lidar data processing methods.

[0056] On the other hand, a computer-readable storage medium is also provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor so that the computer implements any of the above-mentioned lidar data processing methods.

[0057] On the other hand, a computer program product or a computer program is also provided, wherein the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs any of the above-mentioned laser radar data processing methods.

[0058] The technical solution provided by the embodiments of the present application brings at least the following beneficial effects:

[0059] In addition to determining the first error function based on the first feature point located within the reference object, the second error function is also determined based on the second feature point located at the boundary of the reference object. The reference data is more comprehensive, and the error functions (including the first error function and the second error function) determined based on the more comprehensive data have high stability and high accuracy. The first position data of the target corner point determined using the error function with higher stability and accuracy is less affected by errors and has higher accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0061] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0062] Figure 2 is a schematic diagram of another implementation environment provided by an embodiment of the present application;

[0063] Figure 3 is a flow chart of a laser radar data processing method provided in an embodiment of the present application;

[0064] Figure 4 is a schematic diagram of a reference object provided in an embodiment of the present application;

[0065] Figure 5 is a flowchart of another laser radar data processing method provided in an embodiment of the present application;

[0066] Figure 6 It is a structural schematic diagram of a laser radar data processing device provided in an embodiment of the present application;

[0067] Figure 7 It is a structural diagram of a server provided in an embodiment of the present application;

[0068] Figure 8 It is a structural schematic diagram of a laser radar data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0070] This application embodiment provides a laser radar data processing method, please refer to Figure 1 , which shows a schematic diagram of an implementation environment of the method provided in an embodiment of the present application. The implementation environment may include: an electronic device 11, a target sensor 12 and a reference object 13. Among them, the target sensor 12 includes a camera and a laser radar.

[0071] Among them, the electronic device 11 and the target sensor 12 are communicatively connected in a wired or wireless manner. Based on the communication connection with the target sensor 12, the electronic device 11 obtains a target point cloud obtained by scanning a reference object 13 by a laser radar included in the target sensor 12. The electronic device 11 applies the method provided in an embodiment of the present application to determine, based on the target point cloud, the first position data of the target corner point of the reference object 13 in the coordinate system where the laser radar in the target sensor 12 is located.

[0072] For example, Figure 2 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, see Figure 2 , the electronic device 11 includes a terminal 111 and a server 112. When the electronic device 11 executes the method provided in the embodiment of the present application, the method of the embodiment of the present application can be executed independently through the terminal 111 or the server 112, or the method of the embodiment of the present application can be executed interactively through the terminal 111 and the server 112.

[0073] Optionally, the terminal 111 is installed with an application capable of acquiring a target point cloud. After acquiring the target point cloud through the application, the acquired target point cloud is sent to the server 112. The server 112 determines the first error function and the second error function based on the method provided in the embodiment of the present application, and the server 112 determines the first position data of the target corner point based on the first error function and the second error function. Optionally, the server 112 sends the first error function and the second error function to the terminal 111, and the terminal 111 determines the first position data of the target corner point based on the first error function and the second error function.

[0074] Optionally, the terminal 111 can be any electronic product that can interact with the user through one or more methods such as a keyboard, a touch pad, a touch screen, a remote control, voice interaction or a handwriting device, such as a PC (Personal Computer), a mobile phone, a smart phone, a PDA (Personal Digital Assistant), a wearable device, a PPC (Pocket PC), a tablet computer, a smart car machine, a smart TV, a smart speaker, etc. The server 112 can be a server, a server cluster consisting of multiple servers, or a cloud computing service center. The terminal 111 establishes a communication connection with the server 112 through a wired or wireless network.

[0075] Those skilled in the art should understand that the above-mentioned terminal 111 and server 112 are only examples, and other existing or future terminals or servers that are applicable to the present application should also be included in the protection scope of the present application and are included here by reference.

[0076] The present application embodiment provides a laser radar data processing method such as Figure 3 As shown, the laser radar data processing method is based on the above Figure 1 and Figure 2 In the implementation environment shown, the laser radar data processing method can be executed by a terminal or a server, or can be implemented interactively by a terminal or a server. Taking the method applied to a server as an example, the method includes steps 301 to 303.

[0077] In step 301, a target point cloud of a reference object is acquired by a laser radar included in a target sensor. The reference object is used to calibrate the target sensor. The target point cloud includes a first feature point located within the reference object and a second feature point located at a boundary of the reference object.

[0078] Exemplarily, the target sensor is a sensor including a laser radar and a camera, which combines the advantages of the laser radar and the camera and makes up for the disadvantages of the laser radar and the camera. Among them, the advantage of the laser radar is that it has a long detection distance and can obtain three-dimensional information of the object, while the disadvantage of the laser radar is that it is expensive. The advantage of the camera is that it is low-cost and the technology is relatively mature, while the disadvantage of the camera is that it is difficult to obtain three-dimensional information of the object.

[0079] Optionally, the camera included in the target sensor can be an image acquisition device for acquiring static images, that is, a camera. The camera can also be an image acquisition device for acquiring dynamic images, that is, a video camera. The embodiment of the present application does not limit the parameters and quantity of the camera and laser radar included in the target sensor. The laser radar and the camera can be any parameters, for example, the field of view angle of the camera is 120 degrees, the horizontal field of view angle of the laser radar is 360 degrees, and the vertical field of view angle is 40 degrees. Optionally, a target sensor can include a camera and a laser radar, or it can include multiple cameras and multiple laser radars. The types of multiple cameras can be the same or different, and the types of multiple laser radars can be the same or different.

[0080] In a possible implementation, since the position data in the coordinate system where the laser radar is located is different from the position data in the coordinate system where the camera is located, for example, the position data in the coordinate system where the laser radar is located belongs to three-dimensional coordinates, while the position data in the coordinate system where the camera is located belongs to two-dimensional coordinates. Therefore, before using the target sensor, it is necessary to calibrate the target sensor with external parameters, that is, to obtain the relative transformation relationship between the coordinate system where the laser radar in the target sensor is located and the coordinate system where the camera is located, so as to realize the mutual mapping of three-dimensional coordinates and two-dimensional coordinates.

[0081] When performing extrinsic calibration on the target sensor, it can be achieved by scanning the same reference object as the camera in the target sensor and the lidar. Optionally, the reference object used for extrinsic calibration can be a calibration plate with a fixed image array. The image array on the calibration plate can be any image, including but not limited to a checkerboard or circular grid from apriltag (a visual benchmark library). Optionally, Figure 4 A schematic diagram of a reference object provided in an embodiment of the present application, Figure 4 The reference object shown is a calibration plate of a checkerboard type. Of course, the reference object can also be other planar objects with known size information, and is not limited to the calibration plate. In addition, the reference object used for external parameter calibration can be one or more. Determining the first position data of the target corner point based on multiple reference objects is to repeatedly perform the operation of determining the first position data of the target corner point based on one reference object, that is, to repeatedly perform the operation of steps 301-303. The embodiment of the present application takes one reference object as an example to describe the process of determining the first position data of the target corner point.

[0082] Regardless of the situation of the target sensor and the reference object, the target point cloud of the reference object can be obtained by the laser radar included in the target sensor. In a possible implementation, the process of obtaining the target point cloud of the reference object includes: obtaining an initial transformation matrix of the target sensor, the initial transformation matrix is ​​used to represent the initial posture relationship between the laser radar included in the target sensor and the reference object; obtaining an initial point cloud obtained based on laser radar scanning; and segmenting the initial point cloud based on the initial transformation matrix to obtain a target point cloud.

[0083] Optionally, the process of obtaining the initial transformation matrix includes: obtaining second position data of the initial corner point of the reference object in the coordinate system where the camera included in the target sensor is located; determining first extrinsic parameter data of the target sensor based on the second position data of the initial corner point and the position information of each marking point included in the reference object, the first extrinsic parameter data being used to represent the target posture relationship between the camera included in the target sensor and the reference object; obtaining second extrinsic parameter data of the target sensor, the second extrinsic parameter data being used to represent the initial posture relationship between the camera included in the target sensor and the laser radar; and calculating the initial transformation matrix based on the first extrinsic parameter data and the second extrinsic parameter data.

[0084] Exemplarily, the second position data of the initial corner point is obtained based on the camera included in the target sensor. For example, based on the image of the reference object captured by the camera included in the target sensor, the two-dimensional coordinates of the initial corner point of the reference object in the coordinate system of the camera are determined from the image by a corner point extraction algorithm to obtain the second position data. The embodiment of the present application does not limit the corner point extraction algorithm used in determining the second position data of the initial corner point, such as the Thomas algorithm, the FAST (Features from Accelerated Segment Test, fast feature detection) algorithm and the ORB (Oriented Fast and Rotated Brief, scale-invariant feature transform) and the like.

[0085] In one possible implementation, the number of initial corner points determined based on the corner point extraction algorithm is limited. For example, the accuracy of the corner point extraction algorithm is low. Based on the low-accuracy corner point extraction algorithm, only the second position data of some marking points can be determined. As a result, the initial corner points may be part of the marking points of the reference object, that is, not all marking points on the reference object can be used as initial corner points. The marking points on the reference object refer to the feature points used for external parameter calibration. The marking points include but are not limited to the boundary intersection of two different areas on the reference object, points with local maximum or minimum grayscale, etc. Figure 4 Taking the reference object shown in FIG. 1 as an example, the marking point of the reference object is also the intersection point of the boundaries of different colors (the intersection point of the boundaries of different regions), for example Figure 4 Point A, point B and point C in the diagram.

[0086] It should be noted that no matter whether the initial corner points extracted by the corner point extraction algorithm are all the marking points or some of the marking points, the corresponding relationship between the initial corner points and the marking points can be calculated based on the second position data of the initial corner points and the position information of each marking point included in the reference object, and the target pose relationship between the camera in the target sensor and the reference object can be determined, thereby determining the relative transformation relationship between the coordinate system where the camera is located and the coordinate system where the reference object is located according to the target pose relationship between the camera and the reference object. Among them, since the reference object is an object selected in advance for external parameter calibration, the prior information of the reference object will be obtained during the selection process to realize the external parameter calibration of the target sensor based on the prior information of the reference object.

[0087] Exemplarily, the prior information of the reference object includes data such as the positional relationship of the marking points on the reference object and the size information of the reference object. The prior information of the reference object can be obtained by measurement or based on the provider of the reference object. For example, if the reference object is a calibration plate and the provider of the reference object is the manufacturer of the calibration plate, in this case, the prior information of the calibration plate is determined based on the provider of the reference object, that is, when the calibration plate leaves the factory.

[0088] Since the relative positional relationship of each marking point included in the reference object is known, once the coordinate system where the reference object is located is determined, the position information of each marking point can be determined. The coordinate system where the reference object is located can be set based on experience. Figure 4 As point A in the figure, establish the coordinate system where the reference object is located. For the X-axis direction, Y-axis direction and Z-axis direction in the coordinate system where the reference object is located, refer to Figure 4 The coordinate system shown, in this case, the position information of point A is (0, 0, 0), the position information of point B is (0, 1, 0), and the position information of point C is (0, 2, 0). The X-axis direction, Y-axis direction and Z-axis direction of the coordinate system where the reference object is located can also be other directions. The corresponding relationship between the initial corner point and the marking point is determined by the position information of each marking point and the second position data of the initial corner point, and then the target pose relationship between the camera and the reference object is obtained, and the target pose relationship is also called the first external parameter data. Optionally, based on the corresponding relationship between the initial corner point and the marking point, the process of determining the target pose relationship between the camera and the reference object is similar to the process of determining the target pose relationship between the laser radar and the camera based on the second position data of the initial corner point and the first position data of the target corner point in step 303, which will not be described in detail here.

[0089] Of course, the origin on the reference object can be any mark point selected from multiple mark points as the origin as shown in the above embodiment. The origin on the reference object can also be determined by other methods, for example, the geometric center of the reference object is determined as the origin. However, no matter which method is used to determine the origin of the reference object, the position information of each mark point can be determined based on the position of the mark point on the reference object at the reference object.

[0090] In one possible implementation, after determining the first extrinsic parameter data, the server also needs to obtain the second extrinsic parameter data, so as to determine the initial transformation matrix based on the first extrinsic parameter data and the second extrinsic parameter data. Optionally, the second extrinsic parameter data can be a rough value obtained by measurement, and a rough value refers to a numerical value that needs to be optimized accurately. For example, the initial posture relationship between the camera and the lidar includes the initial orientation relationship between the lidar and the camera. However, since the orientation relationship between the lidar and the camera cannot be measured, the initial orientation relationship can be set based on experience. The initial posture relationship determined by the initial orientation relationship is the second extrinsic parameter data, and the initial orientation relationship is set empirically and needs to be optimized accurately. Therefore, the second extrinsic parameter data determined based on the initial orientation relationship that needs to be optimized accurately belongs to a rough value.

[0091] Since the first extrinsic parameter data is used to represent the target pose relationship between the camera included in the target sensor and the reference object, and the second extrinsic parameter data is used to represent the initial pose relationship between the camera included in the target sensor and the laser radar, the initial transformation matrix of the laser radar included in the target sensor and the reference object can be obtained based on the first extrinsic parameter data and the second extrinsic parameter data. The process of obtaining the initial transformation matrix based on the first extrinsic parameter data and the second extrinsic parameter data is, for example, multiplying the first extrinsic parameter data and the second extrinsic parameter data, and the product obtained by the multiplication is the initial transformation matrix.

[0092] Optionally, after obtaining the initial transformation matrix, the initial point cloud obtained by the laser radar scanning can be segmented based on the initial transformation matrix to obtain the target point cloud. Among them, the scanned reference object is an object placed in the scanning environment, and the scanning environment will include other backgrounds in addition to the reference object. Taking the scanning environment as a study and the reference object as calibration plate A as an example, calibration plate A is hung in the middle of the bookcase in the study. During the scanning process, the laser radar has a horizontal field of view of 360°. Therefore, in addition to scanning calibration plate A, the laser radar will also scan the bookcase. The initial point cloud obtained in this case includes the first feature point falling within the calibration plate A, the second feature point falling on the boundary of the calibration plate A, and the feature point falling on the bookcase. Since the target corner point to be extracted in this application is located on the calibration plate A, it is necessary to segment the initial point cloud to further screen out the first feature point and the second feature point.

[0093] It should be noted that the reference object in the above embodiment refers to an object used for external parameter calibration. When there are other objects in the scanning environment that meet the reference object requirements but are not used for external parameter calibration, the feature points located on the other objects do not belong to the first feature points. Still taking the above example of the scanning environment being a study, and the reference object being calibration plate A as an example, there is also a calibration plate B hanging on the left side of the bookcase, but calibration plate B is not used for external parameter calibration of this target sensor. Therefore, the feature points included in the initial point cloud that fall on calibration plate B do not belong to the first feature points, that is, the target point cloud obtained by segmenting the initial point cloud does not include feature points that fall on calibration plate B.

[0094] In a possible implementation, segmenting the initial point cloud based on the initial transformation matrix includes: selecting a feature point whose ranging distance meets the distance requirement and whose data type is single echo data from the initial point cloud as a first feature point, and the distance requirement is determined according to the initial transformation matrix; determining a second feature point from the initial point cloud based on the first feature point, and obtaining a target point cloud including the first feature point and the second feature point.

[0095] Exemplarily, the initial transformation matrix is ​​used to indicate the initial conversion relationship between the coordinate system where the reference object is located and the coordinate system where the laser radar is located. The initial transformation matrix includes initial rotation parameters and initial translation parameters for representing the distance between the origin of the coordinate system where the reference object is located and the origin of the coordinate system where the laser radar is located. The distance between the origin of the coordinate system where the reference object is located and the origin of the coordinate system where the laser radar is located is determined as the distance between the reference object and the laser radar, the distance requirement is determined based on the distance between the reference object and the laser radar, and feature points whose ranging distances meet the distance requirements are selected from the initial point cloud.

[0096] Among them, the feature points in the initial point cloud are three-dimensional coordinate points in the coordinate system where the laser radar is located. The ranging distance of each feature point can be calculated through the three-dimensional coordinate points, and then the feature points that meet the distance requirements are segmented from the feature points through the ranging distance of each feature point. For example, the laser radar and the reference object are placed in parallel, and the reference object is placed directly in front of the laser radar. The distance between the two is 5 meters. The distance requirement is determined to be between 4.99 meters and 5.01 meters. The feature points in the initial point cloud scanned by the laser radar and the plane formed by the x-axis and z-axis in the coordinate system where the laser radar is located are determined as feature points that meet the distance requirements.

[0097] Optionally, the initial point cloud is segmented based on the RANSAC (Random Sample Consensus) algorithm to obtain feature points that meet the distance requirement. For example, based on the distance between the reference object and the laser radar, the data distribution conditions of the feature points that meet the distance requirement are determined, and the feature points that meet the data distribution conditions are segmented from the initial point cloud based on the RANSAC algorithm as the feature points that meet the distance requirement. The data distribution conditions are, for example, the feature points whose vertical distance between the plane formed by the x-axis and the z-axis in the coordinate system where the laser radar is located is between 4.99 meters and 5.01 meters.

[0098] After segmenting the feature points that meet the distance requirements, it is necessary to further select feature points whose data type is single echo data as the first feature point. In one possible implementation, when the laser radar included in the target sensor operates in dual echo mode, the data type also differs between single echo data and dual echo data according to the number of reflections of the emitted laser point. After the laser radar emits a laser point, when the laser point is directly reflected back to the laser radar, that is, the laser point is reflected once, the laser radar receives one echo data based on one laser point, and the data type is single echo data. When the laser point emitted by the laser radar falls on the boundary of any object including the reference object, it will first be reflected to other places, and then reflected back to the laser radar from other places, that is, the laser point falling on the boundary of any object will be reflected twice. In this case, the laser radar receives two echo data based on one laser point, and the data type is dual echo data.

[0099] As for the process of determining the data type of the feature point, illustratively, when the server receives the initial point cloud sent by the laser radar, each feature point included in the initial point cloud has a type identifier for distinguishing whether the data type of the feature point belongs to dual echo data or single echo data. The server can determine the data type of the feature point based on the type identifier, and select the feature point whose data type is single echo data as the first feature point.

[0100] For feature points whose data type is dual echo data included in the feature points that meet the distance requirements, the server can choose to discard them, or, based on the conditions for determining the second feature points, determine whether the feature points whose data type is dual echo data included in the feature points that meet the distance requirements belong to the second feature points, and if they belong to the second feature points, classify the feature points whose data type is dual echo data that meet the distance requirements into the second feature points.

[0101] Exemplarily, the process of determining the second feature point includes: selecting a feature point whose distance from the first feature point is not greater than a distance threshold from the initial point cloud and whose data type is dual echo data as the second feature point. The distance threshold can be set based on experience and size information of a reference object.

[0102] Optionally, in addition to determining the feature point whose data type is dual echo data as the second feature point based on the first feature point as shown in the above embodiment, the second feature point can also be determined from the initial point cloud based on the first feature point and the distance difference. For example, if the distance interval between the feature point B in the initial point cloud and the first feature point A is less than the distance difference, the feature point B is determined to be the second feature point. The distance difference is a value set based on experience and the size information of the reference object.

[0103] In step 302, a first error function is determined based on the first feature point in the target point cloud, and a second error function is determined based on the second feature point in the target point cloud. The first error function and the second error function are used to determine the relative transformation relationship between the coordinate system where the laser radar is located and the coordinate system where the reference object is located.

[0104] In a possible implementation, the first feature point used in the present application to determine the first error function needs to meet the requirement that the data type belongs to single echo data, and the second feature point used to determine the second error function needs to meet the requirement that the data type belongs to dual echo data. With respect to the case where a feature point with a data type of dual echo data is selected from the initial point cloud as the second feature point as shown in the above embodiment, the second feature point obtained by screening at this time has met the requirement for the second feature point when determining the second error function, and the second error function can be determined based on the second feature point. With respect to the method of determining the second feature point based on the first feature point and the distance difference as shown in the above embodiment, in this case, the second feature point needs to be preprocessed, that is, the second feature point with a data type of dual echo data is screened out from each second feature point. The screening process is similar to the process of screening single echo data shown in the above embodiment, and will not be repeated here.

[0105] In a possible implementation, a first error function may be determined based on a first feature point, an initial transformation matrix, and a width of a reference object; and a second error function may be determined based on a second feature point, an initial transformation matrix, and a width of a reference object. Exemplarily, the process of determining the first error function includes: calculating a first variable of the first error function based on the first feature point and the initial transformation matrix; determining a calculation error of the first variable based on the width of the reference object, and determining the first error function based on the calculation error of the first variable. Optionally, the first error function (loss1) determined based on the first feature point is as shown in Formula 1.

[0106]

[0107] in,

[0108]

[0109] N is the number of first feature points, i is the identifier of the first feature point, and d is the width of the reference object. Used to indicate taking and The minimum value between , the calculation principle of substituting other values ​​into min is similar, and will not be repeated here. The first variable of the first error function H is the initial transformation matrix, and pi is the three-dimensional coordinate of the first feature point. Based on the first variable and the width of the reference object, the calculation error of the first variable in the three directions of x-axis, y-axis and z-axis is determined, and the calculation errors of the first variable in the three directions of x-axis, y-axis and z-axis are added to obtain the first error function. Among them, the width of the reference object is determined based on the size information of the reference object. Taking the reference object as a calibration plate as an example, Figure 4 As shown, the width of the reference object refers to the side length in the size information of the reference plate.

[0110] In a possible implementation, the three-dimensional coordinates of the first feature point are multiplied by the initial transformation matrix, and the first variable obtained is used to represent the position information of the first feature point in the coordinate system where the reference object is located. Since the first feature point is a feature point located inside the reference object, the calculation error of the first variable is related to whether the first feature point is located inside the reference object. and l1(z i , 0) is used to determine whether the first feature point is located within the reference object based on the first variable and the width of the reference object. When the first feature point falls within the reference object, the first variable is smaller than the boundary of the reference object. Taking the coordinate origin of the coordinate system where the reference object is located as the geometric origin of the reference object as an example, the first variable is smaller than the boundary value of the reference object, which is reflected in the x-axis and y-axis directions as the x-axis coordinate value and the y-axis coordinate value being smaller than the boundary value d / 2 of the reference object, and being equal to 0 in the z-axis direction. By calculating the calculation error of the first variable, when the calculation error of the first variable is 0, it indicates that the first feature point falls within the reference object. When the calculation error of the first variable is greater than 0, it indicates that the first feature point falls outside the reference object.

[0111] Optionally, since the three-dimensional coordinates of the first feature point and the width of the reference object included in the first error function are both measured accurate values, they are known numbers. Therefore, the unknown parameters in the first variable in the first error function refer to the target rotation parameters and target translation parameters that need to be obtained by optimizing the initial rotation parameters and initial translation parameters in the initial transformation matrix. By setting the initial rotation parameters and initial translation parameters in the initial transformation matrix as unknown numbers, multiplying the initial transformation matrix with the three-dimensional coordinates of the first feature point, a first variable including unknown parameters is obtained, and the first variable and the width of the reference object are calculated according to the x, y, and z coordinates to obtain the calculation error of the first variable. The calculation error of the first variable in any coordinate axis direction is taken as a monomial, and a polynomial is obtained by superimposing multiple monomials, which is the first error function. The value of the first error function can be calculated later, thereby optimizing the initial transformation matrix. Regarding the process of calculating the value of the first error function, refer to the process of optimizing the initial transformation matrix based on formula 3 in the following embodiment, which will not be repeated here.

[0112] Exemplarily, the process of determining the second error function based on the second feature point includes calculating a second variable of the second error function based on the second feature point and the initial transformation matrix; determining a calculation error of the second variable based on the width of the reference object, and determining the second error function based on the calculation error of the second variable. Optionally, the determined second error function (loss2) is as shown in Formula 2.

[0113]

[0114] Where M is the number of second feature points, k is the identifier of the second feature point, and d is the width of the reference object. The second variable of the second error function H is the initial transformation matrix, Qk is the three-dimensional coordinate of the second feature point, also,

[0115] The above otherwise refers to the occurrence of and / or In case of The value is 0. The meaning of the first error function is related to The meaning of is similar, so we will not elaborate on it here. By substituting the second variable and the width of the reference object into Determine the calculation error of the second variable in the x-axis or y-axis direction, add the calculation errors of the second variable in the x-axis or y-axis direction, and obtain a second error function.

[0116] The description of the second variable in the second error function is similar to the description of the first variable in the first error function, and will not be repeated here. Since the second feature point is a feature point that falls on the boundary of the reference object, the calculation error of the second variable is related to whether the second feature point is not located in the reference object. It is used to determine whether the second feature point is not located in the reference object based on the second variable and the width of the reference object. Principles and procedures for making judgments and l1(z i , 0) is similar to the principle of judgment. When the second feature point does not fall within the reference object, The calculated error of the second variable is 0. When the second feature point falls within the reference object, The calculated error of the calculated second variable is greater than 0.

[0117] In step 303, first position data of a target corner point of the reference object in the coordinate system where the laser radar is located is determined based on the first error function and the second error function.

[0118] Exemplarily, the process of determining the first position data of the target corner point includes: iteratively optimizing the initial transformation matrix of the target sensor based on the first error function and the second error function to obtain the target transformation matrix, the target transformation matrix is ​​used to represent the relative transformation relationship between the coordinate system where the laser radar is located and the coordinate system where the reference object is located; calculating the first position data of the target corner point based on the target transformation matrix and the size information of the reference object.

[0119] In a possible implementation, Formula 3 may be used to determine the target transformation matrix based on the first error function and the second error function.

[0120]

[0121] Among them, H m Used to represent the target transformation matrix, It is used to indicate that when loss1+loss2 reaches the minimum value, the value of H in loss1 and loss2 is determined to be H m Since both the first error function and the second error function include the initial transformation matrix, the sum of the first error function and the second error function can be reduced by adjusting the initial transformation matrix. By continuously reducing the sum of the first error function and the second error function, the minimum value of the sum of the first error function and the second error function is determined, and H when the sum of the first error function and the second error function is the minimum value is determined as the target transformation matrix.

[0122] Regarding the iterative optimization process, illustratively, the initial rotation parameters and the initial translation parameters in the initial transformation matrix are substituted into the first error function as initial values. At this time, the values ​​of the unknown parameters included in the first variable and the second variable are known, that is, the initial rotation parameters and the initial translation parameters. The calculation relationship of the error of the first variable based on the width of the reference object is already based on and l1(z i , 0) determine, based on the width of the reference object to determine the calculation error of the second variable operation relationship has been based on Determine. Substituting the known number into the known operation relationship, the initial value of the calculation error of the first variable and the initial value of the calculation error of the second variable can be obtained, and then the initial value of the first error function and the initial value of the second error function can be determined. By adjusting the size of the first variable and the second variable, the first error calculated by the first error function and the second error calculated by the second error function can be adjusted, and then the sum of the first error and the second error can be adjusted.

[0123] Repeat the above steps to determine the minimum value of the sum of the first error and the second error, determine the value of the first variable when the sum of the first error and the second error reaches the minimum value as the target translation parameter and the target rotation parameter, and obtain the target transformation matrix. For example, when H in Formula 3 takes the value H1, since the sum of the first error determined based on the translation parameter and the rotation parameter in H1 and the second error determined based on the translation parameter and the rotation parameter in H1 reaches the minimum value, H1 is determined to be the target transformation matrix.

[0124] In addition, in the process of determining the iterative optimization direction of the initial transformation matrix according to the size of the sum of the first error function and the second error function, the initial transformation matrix in the first error function is used in conjunction with the initial transformation matrix in the second error function. In the iterative optimization process, they are calibrated with each other to ensure that when the sum of the first error function and the second error function reaches a minimum value, the target transformation matrix obtained has high accuracy. Since the first error function is determined based on the first feature point located in the reference object, and the second error function is determined based on the second feature point located at the boundary of the reference object. In the iterative optimization of the initial transformation matrix through the first error function and the second error function, the correspondence between the first feature point in the reference object and the second feature point at the boundary of the reference object is also used as a constraint, thereby improving the stability of the iterative optimization process.

[0125] Optionally, the first position data of the target corner point is determined through the target pose relationship between the laser radar and the reference object, and the size information of the reference object. The target pose relationship between the laser radar and the reference object is also the relative transformation relationship between the coordinate system where the laser radar is located and the coordinate system where the reference object is located. Based on the relative transformation relationship between the coordinate system where the laser radar is located and the coordinate system where the reference object is located and the position information of each marking point included in the size information of the reference object, the first position data of the target corner point in the coordinate system where the laser radar is located is calculated. In addition, the target corner point is selected from the marking point according to the implementation environment, and the implementation environment is, for example, the boundary point of the reference object. The boundary point is such as Figure 4 The determined target corner points may be part of the marker points, and not all marker points on the reference object are target corner points.

[0126] The first position data of the target corner points on the reference object are calculated by using the target transformation matrix and the size information of the reference object. Even when the initial point cloud obtained by the laser radar scanning is sparse, the first position data of the target corner points can be obtained by calculation, which has strong versatility. The sparse initial point cloud may be, for example, that some marking points of the reference object are missing. For example, the laser radar only scans and obtains the feature points of the lower half of the reference object, while the feature points of the upper half are missing. In this case, the initial point cloud obtained based on the laser radar scanning can only determine the first position data of the target corner points in the lower half of the reference object, and the determined first position data of the target corner points is incomplete and has low accuracy. The laser radar data processing method provided in the embodiment of the present application obtains the first position data of the target corner points based on the target transformation matrix and the size information of the reference object, rather than directly extracting it from the initial point cloud, so the determined first position data of the target corner points is more comprehensive and has high accuracy.

[0127] In a possible implementation, after determining the first position data of the target corner point, the server may also perform external parameter calibration on the target sensor based on the first position data of the target corner point and the second position data of the initial corner point. The method of external parameter calibration includes but is not limited to 3D-2D (3-dimension-2-dimension) matching of the first position data of the target corner point with the second position data of the initial corner point, obtaining the relative transformation relationship between the coordinate system where the laser radar is located and the coordinate system where the camera is located, and then obtaining the target posture relationship between the laser radar included in the target sensor and the camera. Optionally, the 2D-3D matching is implemented based on the PnP (Perspective-n-Point, perspective N-point projection) RANSAC algorithm. Exemplarily, the server may also optimize the second external parameter data in step 301 based on the second position data of the initial corner point and the first position data of the target corner point, obtain the target posture relationship between the laser radar included in the target sensor and the camera, and complete the external parameter calibration of the target sensor through optimization, and the optimization can be achieved by minimizing the error.

[0128] In one possible case, the initial corner point and the target corner point may be the same or different. However, no matter what relative relationship the initial corner point and the target corner point have, since the relative position relationship of each marking point on the reference object is known, that is, the relative position relationship between the initial corner point and the target corner point is also known. Therefore, even if the initial corner point and the target corner point are different, the external parameter calibration of the target sensor can be completed based on the second position data of the initial corner point and the first position data of the target corner point.

[0129] It should be noted that the above embodiment is intended to illustrate the process of determining the first position data of the target corner point of the reference object, rather than to limit the reference object. The reference object used for the external parameter calibration of the target sensor can be one or more of the examples in the above embodiment. The use of multiple reference objects can be that the target sensor scans multiple reference objects at a time, or that the target sensor scans one reference object at a time, and the target sensor scans multiple times. The process of determining the first position data of the target corner points of multiple reference objects is similar to the process of determining the first position data of the target corner point of one reference object, and will not be repeated here.

[0130] In summary, the laser radar data processing method provided in the embodiment of the present application, in addition to determining the first error function based on the first feature point located within the reference object, also determines the second error function based on the second feature point located at the boundary of the reference object. The reference feature points are more comprehensive, and the error function (including the first error function and the second error function) determined based on the more comprehensive feature points has high stability and high precision. The first position data of the target corner point determined by the error function with higher stability and precision is less affected by the error and has higher accuracy. In addition, by calculating the first position data of the target corner point through the first error function and the second error function, even if the initial point cloud obtained by the laser radar scanning is sparse, the first position data of the missing target corner point can be determined, and the determined target corner point is more comprehensive and more versatile.

[0131] In one possible implementation, Figure 5 A flowchart of another lidar data processing method provided in an embodiment of the present application, taking the method applied to a terminal as an example, the method includes the following steps.

[0132] Exemplarily, an image is captured by a camera included in the target sensor, the second position data of the initial corner point is obtained based on the image, and the first external parameter data is determined based on the second position data of the initial corner point and the position information of the marking point. Optionally, the process of determining the first external parameter data is the same as the above Figure 3 The process of determining the first external parameter data in step 301 of the illustrated embodiment is similar and will not be described again here.

[0133] Exemplarily, the terminal determines the initial transformation matrix based on the first external parameter data and the second external parameter data. That is, based on the target posture relationship between the camera and the reference object, and the initial posture relationship between the laser radar and the camera, the initial posture relationship between the laser radar and the reference object is determined. Optionally, the process of the terminal determining the initial transformation matrix based on the first external parameter data and the second external parameter data is the same as the above Figure 3 The process of determining the initial transformation matrix based on the first extrinsic parameter data and the second extrinsic parameter data in step 301 of the illustrated embodiment is similar and will not be described in detail here.

[0134] Exemplarily, the terminal segments the initial point cloud based on the initial transformation matrix to obtain the first feature point. The second feature point is determined from the initial point cloud based on the first feature point. Optionally, the process of segmenting the initial point cloud by the terminal is the same as the above Figure 3 The process of segmenting the initial point cloud in step 301 of the illustrated embodiment is similar to that of the above-mentioned embodiment. The process of determining the second feature point from the initial point cloud based on the first feature point by the terminal is similar to that of the above-mentioned embodiment. Figure 3 The process of the server determining the second feature point from the initial point cloud based on the first feature point in step 301 of the illustrated embodiment is similar and will not be described in detail here.

[0135] Exemplarily, a first error function is constructed based on the first feature point, and a second error function is constructed based on the second feature point. Optionally, the process of the terminal constructing the first error function based on the first feature point and constructing the second error function based on the second feature point is the same as the above Figure 3 The process of determining the first error function and the second error function in step 302 of the illustrated embodiment is similar and will not be described in detail here.

[0136] Exemplarily, the terminal determines the target transformation matrix based on the first error function and the second error function, and calculates the first position data of the target corner point based on the target transformation matrix. Optionally, the process of the terminal determining the target transformation matrix based on the first error function and the second error function is the same as the above Figure 3 The process of determining the target transformation matrix based on the first error function and the second error function in step 303 of the embodiment shown is similar to the process of calculating the first position data of the target corner point based on the target transformation matrix. Figure 3 The process of calculating the first position data of the target corner point based on the target transformation matrix in step 303 of the illustrated embodiment is similar and will not be described in detail here.

[0137] Exemplarily, the terminal calibrates the target sensor based on the first position data of the target corner point and the second position data of the initial corner point. Optionally, the process of calibrating the target sensor based on the first position data of the target corner point and the second position data of the initial corner point is the same as the above process. Figure 3 The process of calibrating the target sensor based on the first position data of the target corner point and the second position data of the initial corner point in step 303 of the illustrated embodiment is similar and will not be described in detail here.

[0138] In a possible implementation, based on the above process, the terminal has conducted multiple tests on the laser radar data processing method. The translation difference of the test consistency under multiple sets of target external parameter data (used to represent the target posture relationship between the laser radar and the camera) is less than 2 cm (basically millimeter level), and the rotation difference is less than 0.1 degree, and it is very close to the indoor calibration result (the translation difference is less than 3 cm, and the maximum rotation difference is about 0.1 degree). It can be seen that the first position data of the target corner point determined by the laser radar data processing method provided in the embodiment of the present application improves the calibration accuracy and consistency of the target sensor during the external parameter calibration process.

[0139] See also Figure 6 , an embodiment of the present application provides a laser radar data processing device, which includes: an acquisition module 601 and a determination module 602.

[0140] An acquisition module 601 is used to acquire a target point cloud of a reference object through a laser radar included in the target sensor, where the reference object is used to calibrate the target sensor, and the target point cloud includes a first feature point located in the reference object and a second feature point located at a boundary of the reference object;

[0141] A determination module 602 is used to determine a first error function based on a first feature point in the target point cloud, and to determine a second error function based on a second feature point in the target point cloud, wherein the first error function and the second error function are used to determine a relative transformation relationship between a coordinate system where the laser radar is located and a coordinate system where the reference object is located;

[0142] The determination module 602 is further used to determine the first position data of the target corner point of the reference object in the coordinate system where the laser radar is located based on the first error function and the second error function.

[0143] Optionally, acquisition module 601 is used to obtain an initial transformation matrix of the target sensor, where the initial transformation matrix is ​​used to represent the initial posture relationship between the laser radar included in the target sensor and the reference object; obtain an initial point cloud obtained based on the laser radar scanning; and segment the initial point cloud based on the initial transformation matrix to obtain a target point cloud.

[0144] Optionally, the determination module 602 is used to determine a first error function based on the first feature point, the initial transformation matrix and the width of the reference object; and determine a second error function based on the second feature point, the initial transformation matrix and the width of the reference object.

[0145] Optionally, the first error function is:

[0146] in,

[0147]

[0148]

[0149] x i ,y i With z i The calculation formula is: loss1 is the first error function, N is the number of first feature points, i is the identifier of the first feature point, d is the width of the reference object, H is the initial transformation matrix, and pi is the three-dimensional coordinate of the first feature point.

[0150] Optionally, the second error function is: in, x k With y k The calculation formula is: loss2 is the second error function, M is the number of second feature points, k is the identifier of the second feature point, d is the width of the reference object, H is the initial transformation matrix, Q k is the three-dimensional coordinate of the second feature point,

[0151] Optionally, an acquisition module 601 is used to obtain second position data of an initial corner point of the reference object in a coordinate system where a camera included in the target sensor is located; determine first extrinsic parameter data of the target sensor based on the second position data of the initial corner point and position information of each mark point included in the reference object, the first extrinsic parameter data being used to represent a target pose relationship between the camera included in the target sensor and the reference object; obtain second extrinsic parameter data of the target sensor, the second extrinsic parameter data being used to represent an initial pose relationship between the camera included in the target sensor and a laser radar; and calculate an initial transformation matrix based on the first extrinsic parameter data and the second extrinsic parameter data.

[0152] Optionally, the determination module 602 is used to iteratively optimize the initial transformation matrix of the target sensor based on the first error function and the second error function to obtain a target transformation matrix, where the target transformation matrix is ​​used to represent the relative transformation relationship between the coordinate system where the laser radar is located and the coordinate system where the reference object is located; and calculate the first position data of the target corner point based on the target transformation matrix and the size information of the reference object.

[0153] Optionally, the acquisition module 601 is used to select a feature point whose ranging distance meets the distance requirement and whose data type is single echo data from the initial point cloud as the first feature point, and the distance requirement is determined according to the initial transformation matrix; based on the first feature point, the second feature point is determined from the initial point cloud to obtain a target point cloud including the first feature point and the second feature point.

[0154] Optionally, the acquisition module 601 is used to select, from the initial point cloud, a feature point whose distance from the first feature point is not greater than a distance threshold and whose data type is dual echo data, as the second feature point.

[0155] When determining the error function (including the first error function and the second error function), the above-mentioned device not only determines the first error function based on the first feature point located in the reference object, but also determines the second error function based on the second feature point located at the boundary of the reference object. The reference data is more comprehensive, and the error function determined based on the more comprehensive data has high stability and high accuracy. The first position data of the target corner point determined using the error function with higher stability and accuracy is less affected by errors and has higher accuracy.

[0156] It should be noted that the device provided in the above embodiment only uses the division of the above functional modules as an example to implement its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and their specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0157] Figure 7 It is a structural diagram of a server provided in an embodiment of the present application. The server may have relatively large differences due to different configurations or performances, and may include one or more processors 701 and one or more memories 702, wherein the one or more memories 702 store at least one computer program, and the at least one computer program is loaded and executed by the one or more processors 701, so that the server implements the laser radar data processing method provided by the above-mentioned various method embodiments. The processor is, for example, a central processing unit (CPU). Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server may also include other components for implementing device functions, which will not be repeated here.

[0158] Figure 81 is a schematic diagram of the structure of a laser radar data processing device provided in an embodiment of the present application. The device may be a terminal, such as a smart phone, a tablet computer, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer or a desktop computer. The terminal may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0159] Typically, the terminal includes: a processor 801 and a memory 802 .

[0160] The processor 801 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 801 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 801 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 801 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 801 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0161] The memory 802 may include one or more computer-readable storage media, which may be non-transitory. The memory 802 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 802 is used to store at least one instruction, which is used to be executed by the processor 801 so that the terminal implements the laser radar data processing method provided in the method embodiment of the present application.

[0162] In some embodiments, the terminal may further optionally include: a peripheral device interface 803 and at least one peripheral device. The processor 801, the memory 802 and the peripheral device interface 803 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 803 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 804, a display screen 805, a camera assembly 806, an audio circuit 807, a positioning assembly 808 and a power supply 809.

[0163] The peripheral device interface 803 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 801 and the memory 802. In some embodiments, the processor 801, the memory 802, and the peripheral device interface 803 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 801, the memory 802, and the peripheral device interface 803 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0164] The radio frequency circuit 804 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 804 communicates with communication networks and other communication devices through electromagnetic signals. The radio frequency circuit 804 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 804 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 804 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 804 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0165] The display screen 805 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos and any combination thereof. When the display screen 805 is a touch display screen, the display screen 805 also has the ability to collect touch signals on the surface or above the surface of the display screen 805. The touch signal can be input to the processor 801 as a control signal for processing. At this time, the display screen 805 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 805 can be one, set on the front panel of the terminal; in other embodiments, the display screen 805 can be at least two, respectively set on different surfaces of the terminal or in a folding design; in other embodiments, the display screen 805 can be a flexible display screen, set on a curved surface or a folding surface of the terminal. Even, the display screen 805 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 805 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode) and the like.

[0166] The camera assembly 806 is used to capture images or videos. Optionally, the camera assembly 806 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 806 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0167] The audio circuit 807 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 801 for processing, or input them into the radio frequency circuit 804 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 801 or the radio frequency circuit 804 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 807 may also include a headphone jack.

[0168] The positioning component 808 is used to locate the current geographical location of the terminal to implement navigation or LBS (Location Based Service). The positioning component 808 can be a positioning component based on the US GPS (Global Positioning System), China's Beidou system, Russia's Grenas system or the European Union's Galileo system.

[0169] The power supply 809 is used to power various components in the terminal. The power supply 809 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 809 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0170] In some embodiments, the terminal further includes one or more sensors 810 , including but not limited to: an acceleration sensor 811 , a gyroscope sensor 812 , a pressure sensor 813 , a fingerprint sensor 814 , an optical sensor 815 , and a proximity sensor 816 .

[0171] The acceleration sensor 811 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal. For example, the acceleration sensor 811 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 801 can control the display screen 805 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 811. The acceleration sensor 811 can also be used to collect game or user motion data.

[0172] The gyroscope sensor 812 can detect the body direction and rotation angle of the terminal, and the gyroscope sensor 812 can cooperate with the acceleration sensor 811 to collect the user's 3D actions on the terminal. The processor 801 can implement the following functions based on the data collected by the gyroscope sensor 812: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0173] The pressure sensor 813 can be set in the side frame of the terminal and / or the lower layer of the display screen 805. When the pressure sensor 813 is set in the side frame of the terminal, the user's holding signal of the terminal can be detected, and the processor 801 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 813. When the pressure sensor 813 is set in the lower layer of the display screen 805, the processor 801 controls the operability controls on the UI interface according to the user's pressure operation on the display screen 805. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0174] The fingerprint sensor 814 is used to collect the user's fingerprint, and the processor 801 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 814, or the fingerprint sensor 814 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 801 authorizes the user to perform relevant sensitive operations, which include unlocking the screen, viewing encrypted information, downloading software, paying, and changing settings. The fingerprint sensor 814 can be set on the front, back or side of the terminal. When a physical button or a manufacturer logo is set on the terminal, the fingerprint sensor 814 can be integrated with the physical button or the manufacturer logo.

[0175] The optical sensor 815 is used to collect the ambient light intensity. In one embodiment, the processor 801 can control the display brightness of the display screen 805 according to the ambient light intensity collected by the optical sensor 815. Specifically, when the ambient light intensity is high, the display brightness of the display screen 805 is increased; when the ambient light intensity is low, the display brightness of the display screen 805 is reduced. In another embodiment, the processor 801 can also dynamically adjust the shooting parameters of the camera assembly 806 according to the ambient light intensity collected by the optical sensor 815.

[0176] The proximity sensor 816, also known as a distance sensor, is usually arranged on the front panel of the terminal. The proximity sensor 816 is used to collect the distance between the user and the front of the terminal. In one embodiment, when the proximity sensor 816 detects that the distance between the user and the front of the terminal is gradually decreasing, the processor 801 controls the display screen 805 to switch from the screen-on state to the screen-off state; when the proximity sensor 816 detects that the distance between the user and the front of the terminal is gradually increasing, the processor 801 controls the display screen 805 to switch from the screen-off state to the screen-on state.

[0177] Those skilled in the art will understand that Figure 8 The structure shown in the figure does not constitute a limitation on the laser radar data processing device, and may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0178] In an exemplary embodiment, a computer device is also provided, the computer device comprising a processor and a memory, wherein at least one computer program is stored in the memory. The at least one computer program is loaded and executed by one or more processors, so that the computer device implements any of the above-mentioned laser radar data processing methods.

[0179] In an exemplary embodiment, a computer-readable storage medium is also provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor of a computer device so that the computer implements any of the above-mentioned lidar data processing methods.

[0180] In one possible implementation, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0181] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs any of the above-mentioned laser radar data processing methods.

[0182] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions. For example, the target point clouds involved in this application are all obtained with full authorization.

[0183] It should be understood that the "plurality" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0184] The above description is only an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A laser radar data processing method, characterized in that: The method comprises: Acquire a target point cloud of a reference object through a laser radar included in the target sensor, the reference object is used to calibrate the target sensor, the target point cloud includes a first feature point located in the reference object and a second feature point located at a boundary of the reference object, the data type of the first feature point is single echo data, and the data type of the second feature point is double echo data; Determine a first error function based on a first feature point in the target point cloud, and determine a second error function based on a second feature point in the target point cloud, wherein the first error function and the second error function are used to determine a relative transformation relationship between a coordinate system where the laser radar is located and a coordinate system where the reference object is located; Iteratively optimizing an initial transformation matrix of the target sensor based on the first error function and the second error function to obtain a target transformation matrix, wherein the target transformation matrix is ​​used to represent a relative transformation relationship between a coordinate system where the laser radar is located and a coordinate system where the reference object is located; The first position data of the target corner point of the reference object in the coordinate system where the laser radar is located is calculated based on the target transformation matrix and the size information of the reference object.

2. The method according to claim 1, characterized in that The step of acquiring a target point cloud of a reference object by using a laser radar included in the target sensor includes: Acquire an initial transformation matrix of the target sensor, where the initial transformation matrix is ​​used to represent an initial posture relationship between a laser radar included in the target sensor and the reference object; Acquire an initial point cloud based on the laser radar scanning; The initial point cloud is segmented based on the initial transformation matrix to obtain the target point cloud.

3. The method according to claim 2, characterized in that The step of determining a first error function based on a first feature point in the target point cloud and determining a second error function based on a second feature point in the target point cloud comprises: determining the first error function based on the first feature point, the initial transformation matrix, and the width of the reference object; The second error function is determined based on the second feature points, the initial transformation matrix, and the width of the reference object.

4. The method according to claim 3, characterized in that The first error function is: Among them, the Said Said The x i , the y i With the z i The calculation formula is: The loss1 is the first error function, the N is the number of the first feature points, the i is the identifier of the first feature point, the d is the width of the reference object, the H is the initial transformation matrix, and the p i is the three-dimensional coordinate of the first feature point; is the first variable of the first error function, and l1 is the calculated error of the first variable.

5. The method according to claim 3, characterized in that: The second error function is: Among them, the The x k With the y k The calculation formula is: The loss2 is the second error function, the M is the number of the second feature points, the k is the identifier of the second feature point, the d is the width of the reference object, the H is the initial transformation matrix, and the Q k is the three-dimensional coordinate of the second feature point, Said is the second variable of the second error function, and l2 is the calculated error of the second variable.

6. The method according to any one of claims 2 to 5, characterized in that: The obtaining of the initial transformation matrix of the target sensor comprises: Acquire second position data of an initial corner point of the reference object in a coordinate system where a camera included in the target sensor is located; Determine first extrinsic parameter data of the target sensor based on the second position data of the initial corner point and the position information of each marking point included in the reference object, wherein the first extrinsic parameter data is used to represent the target posture relationship between the camera included in the target sensor and the reference object; Acquire second extrinsic parameter data of the target sensor, where the second extrinsic parameter data is used to represent an initial posture relationship between a camera and a laser radar included in the target sensor; The initial transformation matrix is ​​calculated based on the first extrinsic parameter data and the second extrinsic parameter data.

7. The method according to any one of claims 2 to 5, characterized in that: The step of segmenting the initial point cloud based on the initial transformation matrix to obtain the target point cloud includes: Selecting, from the initial point cloud, a feature point whose distance measurement meets the distance requirement and whose data type is single echo data as the first feature point, wherein the distance requirement is determined according to the initial transformation matrix; The second feature point is determined from the initial point cloud based on the first feature point to obtain a target point cloud including the first feature point and the second feature point.

8. The method according to claim 7, characterized in that The determining the second feature point from the initial point cloud based on the first feature point includes: A feature point whose distance from the first feature point is not greater than a distance threshold and whose data type is dual echo data is selected from the initial point cloud as the second feature point.

9. A laser radar data processing device, characterized in that: The device comprises: an acquisition module, configured to acquire a target point cloud of a reference object through a laser radar included in the target sensor, wherein the reference object is used to calibrate the target sensor, the target point cloud includes a first feature point located within the reference object and a second feature point located at a boundary of the reference object, the data type of the first feature point is single echo data, and the data type of the second feature point is double echo data; A determination module, configured to determine a first error function based on a first feature point in the target point cloud, and to determine a second error function based on a second feature point in the target point cloud, wherein the first error function and the second error function are used to determine a relative transformation relationship between a coordinate system where the laser radar is located and a coordinate system where the reference object is located; The determination module is also used to iteratively optimize the initial transformation matrix of the target sensor based on the first error function and the second error function to obtain a target transformation matrix, where the target transformation matrix is ​​used to represent the relative transformation relationship between the coordinate system where the laser radar is located and the coordinate system where the reference object is located; and calculate the first position data of the target corner point of the reference object in the coordinate system where the laser radar is located based on the target transformation matrix and the size information of the reference object.

Images