Laser radar point cloud marking method, device, equipment and medium

Abstract

The invention discloses a laser radar point cloud marking method and device, equipment and a medium, and relates to the technical field of image processing, and the method comprises the steps: constructing a projection mapping relation between an omnidirectional visual image and a laser radar point cloud based on a target space position relation; performing image recognition on the omnidirectional visual image by using a preset image recognition model to obtain a plurality of first target detection frames including object category labels; and determining a target area of the first target detection frame in the laser radar point cloud according to the projection mapping relation, performing point cloud clustering processing in the target area to obtain a target point cloud cluster, and fitting the target point cloud cluster to obtain a second target detection frame to complete marking operation of the laser radar point cloud. The detection frame with the category label is obtained through the image recognition model and mapped to the point cloud for clustering, and the technical problem that the point cloud processing delay is large is solved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus, device, and medium for marking point clouds using lidar. Background Technology

[0002] Point cloud recognition models can only segment and label all point cloud clusters that approximate tree outlines as "trees." Image-based recognition models, however, can identify specific subcategories within the "tree" category, such as "camphor tree," "maple tree," and "birch tree," based on visual information, which better aligns with human visual semantic observation habits.

[0003] To obtain 3D point clouds with rich visual color and texture information, allowing users to intuitively distinguish "which point cloud represents which specific object", the current common practice is to use a real-time acquisition + offline rendering method to register and fuse the geometric point cloud acquired by the LiDAR with the image captured by the camera. However, this method has the problem of not being able to operate and process the coordinate information acquired by the LiDAR in real time. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a method, apparatus, device, and medium for marking point clouds in LiDAR, which can obtain detection boxes with category labels through an image recognition model and map them to the point cloud for clustering, thus solving the technical problem of high latency in point cloud processing. The specific solution is as follows:

[0005] Firstly, this application provides a method for marking point clouds using lidar, including:

[0006] Deploy a preset omnidirectional camera array and a preset lidar according to the target's spatial location, and establish a time synchronization relationship between the preset omnidirectional camera array and the preset lidar when collecting data;

[0007] Based on the time synchronization relationship, the preset omnidirectional camera array and the preset lidar are controlled to collect data to obtain omnidirectional visual images and corresponding lidar point clouds, and a projection mapping relationship between the omnidirectional visual images and lidar point clouds is constructed based on the target spatial position relationship.

[0008] The omnidirectional visual image is subjected to image recognition using a preset image recognition model to obtain several first target detection boxes including object category labels;

[0009] The target region of the first target detection box in the lidar point cloud is determined according to the projection mapping relationship. Point cloud clustering is performed in the target region to obtain the target point cloud cluster corresponding to the first target detection box. The target point cloud cluster is then fitted using a preset 3D point cloud processing tool to obtain the second target detection box corresponding to the target point cloud cluster, thereby completing the marking operation of the lidar point cloud.

[0010] Optionally, the preset omnidirectional camera array includes several cameras with identical parameters deployed around a central point; wherein the field of view of each camera is adjacent to the other.

[0011] Optionally, the step of deploying a preset omnidirectional camera array and a preset lidar according to the target's spatial location includes:

[0012] The mounting plane of the preset omnidirectional camera array is controlled to be parallel to the horizontal plane in the coordinate system of the preset lidar, and the central axis of the preset omnidirectional camera array is controlled to be coaxial with the central axis of the preset lidar.

[0013] Optionally, the step of constructing the projection mapping relationship between the omnidirectional visual image and the lidar point cloud based on the target spatial position relationship includes:

[0014] The cameras in the preset omnidirectional camera array are calibrated to obtain the camera's intrinsic parameters, and the external parameters of the cameras relative to the preset lidar are determined according to the target spatial position relationship; wherein, the intrinsic parameters include the camera intrinsic parameter matrix and distortion coefficients, and the external parameters include rotation matrix and translation matrix, wherein the translation matrix is ​​a matrix representing the three-dimensional coordinates of the camera's optical center in the lidar coordinate system, and the rotation matrix is ​​a matrix representing the angle of the camera coordinate system relative to the lidar coordinate system;

[0015] The point cloud corresponding to the pixels in the omnidirectional visual image is determined based on the internal parameters and the external parameters to obtain the projection mapping relationship.

[0016] Optionally, determining the point cloud corresponding to the pixels in the omnidirectional visual image based on the internal parameters and the external parameters includes:

[0017] The intrinsic parameter matrix and the distortion coefficient are used to perform distortion correction on the pixels in the omnidirectional visual image to obtain the corrected pixel coordinates, and the intrinsic parameter matrix and the corrected pixel coordinates are used to determine the first coordinate of the pixel in the camera coordinate system;

[0018] The second coordinates of the reference point of the pixel in the lidar coordinate system are calculated using the translation matrix, the rotation matrix, and the first coordinates;

[0019] The cylindrical coordinates of the reference point are determined based on the second coordinates. The horizontal azimuth and elevation azimuth of the reference point are determined based on the cylindrical coordinates. The point cloud corresponding to the pixel in the lidar point cloud is determined using the horizontal azimuth and elevation azimuth.

[0020] Optionally, the point cloud clustering processing within the target area includes:

[0021] The point within the target area that is closest to the preset lidar is used as the initial cluster center. Neighboring points are clustered based on a preset clustering distance threshold to obtain the target point cloud cluster. The preset clustering distance threshold is a threshold that is dynamically set according to the point cloud density.

[0022] Optionally, the second target detection box includes the geometric center coordinates of the object being measured, the detection box size, and the object category label of the object being measured.

[0023] Secondly, this application provides a lidar point cloud marking device, comprising:

[0024] The radar deployment module is used to deploy a preset omnidirectional camera array and a preset lidar according to the spatial position relationship of the target, and to establish a time synchronization relationship between the preset omnidirectional camera array and the preset lidar when collecting data.

[0025] The mapping relationship construction module is used to control the preset omnidirectional camera array and the preset lidar to collect data based on the time synchronization relationship, so as to obtain omnidirectional visual images and corresponding lidar point clouds, and to construct the projection mapping relationship between the omnidirectional visual images and lidar point clouds based on the target spatial position relationship.

[0026] The detection box acquisition module is used to perform image recognition on the omnidirectional visual image using a preset image recognition model to obtain a number of first target detection boxes including object category labels.

[0027] The point cloud marking module is used to determine the target region of the first target detection box in the lidar point cloud according to the projection mapping relationship, perform point cloud clustering processing in the target region to obtain the target point cloud cluster corresponding to the first target detection box, and use a preset three-dimensional point cloud processing tool to fit the target point cloud cluster to obtain the second target detection box corresponding to the target point cloud cluster, so as to complete the marking operation of the lidar point cloud.

[0028] Thirdly, this application provides an electronic device, comprising:

[0029] Memory, used to store computer programs;

[0030] A processor is used to execute the computer program to implement the aforementioned lidar point cloud marking method.

[0031] Fourthly, this application provides a computer-readable storage medium for storing a computer program that, when executed by a processor, implements the aforementioned lidar point cloud marking method.

[0032] This application first deploys a preset omnidirectional camera array and a preset LiDAR based on the target spatial location relationship, and establishes a time synchronization relationship between the preset omnidirectional camera array and the preset LiDAR when acquiring data. Then, based on the time synchronization relationship, it controls the preset omnidirectional camera array and the preset LiDAR to acquire data to obtain omnidirectional visual images and corresponding LiDAR point clouds. Based on the target spatial location relationship, it constructs a projection mapping relationship between the omnidirectional visual images and the LiDAR point clouds. Then, it uses a preset image recognition model to perform image recognition on the omnidirectional visual images to obtain several first target detection boxes including object category labels. Finally, based on the projection mapping relationship, it determines the target region of the first target detection box in the LiDAR point cloud, performs point cloud clustering processing in the target region to obtain the target point cloud cluster corresponding to the first target detection box, and uses a preset 3D point cloud processing tool to fit the target point cloud cluster to obtain the second target detection box corresponding to the target point cloud cluster, thereby completing the labeling operation of the LiDAR point cloud. Therefore, this application achieves real-time aligned acquisition of visual texture information and point cloud geometric information by deploying an omnidirectional camera array and LiDAR and establishing a time synchronization relationship; it achieves real-time and accurate mapping of visual detection results to point cloud space by constructing a projection mapping relationship between images and point clouds, replacing the time-consuming offline registration process; and it achieves fine target segmentation and 3D labeling based on visual semantic guidance by obtaining detection boxes with category labels through an image recognition model and mapping them to the point cloud for clustering and fitting. Thus, it simultaneously solves the technical problems of coarse point cloud segmentation results and large processing latency without the need for offline rendering. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0034] Figure 1 This is a schematic diagram of a lidar point cloud marking method disclosed in this application;

[0035] Figure 2 This is a schematic diagram of the deployment location of a camera radar disclosed in this application;

[0036] Figure 3 This is a schematic diagram of a point cloud marker disclosed in this application;

[0037] Figure 4 This is a schematic diagram of the structure of a lidar point cloud marking device disclosed in this application;

[0038] Figure 5 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] Current point cloud marking methods have the problem of being unable to operate and process coordinate information acquired in real time by lidar.

[0041] See Figure 1 As shown, this embodiment of the invention discloses a lidar point cloud marking method, including:

[0042] Step S11: Deploy a preset omnidirectional camera array and a preset lidar according to the target spatial location relationship, and establish a time synchronization relationship between the preset omnidirectional camera array and the preset lidar when collecting data.

[0043] In this embodiment, it is first necessary to deploy a preset omnidirectional camera array and a preset lidar according to the target spatial position relationship. The process may specifically include: controlling the installation plane of the preset omnidirectional camera array to be parallel to the horizontal plane in the lidar coordinate system corresponding to the preset lidar, and controlling the central axis of the preset omnidirectional camera array to be coaxial with the central axis of the preset lidar.

[0044] It should be noted that the preset omnidirectional camera array in this embodiment includes several cameras with identical parameters deployed around a central point; wherein the field of view of each camera is adjacent to that of the others. Specifically, the relative positional relationship between the preset omnidirectional camera array and the preset lidar is as follows: Figure 2 As shown.

[0045] Specifically, this embodiment designs a combined system of an omnidirectional camera array and a lidar, with the omnidirectional camera array mounted above the lidar. The camera array contains four identical camera units, denoted as follows: (n=1, 2, 3, 4), each camera has a resolution of U×V, ensuring that the horizontal field of view is greater than 90 degrees, and the camera array can cover a 360-degree horizontal field of view.

[0046] This integrated system uses the coordinate system of the lidar device. The reference coordinate system is the plane of the camera array. The mounting plane of the camera array is parallel to the coordinate system of the lidar. The planes are parallel, at a distance of H, and the center of the installation plane is perpendicular to the coordinate system of the lidar. The axes are coaxial.

[0047] On the mounting plane of the camera array, determine the scanning direction of the LiDAR transmitter / receiver during operation, and arrange the camera array sequentially. For example, if the LiDAR scanning direction is clockwise,

[0048] camera The reference optical axis and the lidar coordinate system The coordinate axes are parallel and in the same direction, and are located at... In the plane.

[0049] camera The reference optical axis lidar coordinate system The coordinate axes are parallel and in the same direction, and are located at... In the plane.

[0050] camera The reference optical axis and the lidar coordinate system The coordinate axes are parallel and opposite, and are in the same position. In the plane.

[0051] camera The reference optical axis lidar coordinate system The coordinate axes are parallel and opposite, and are in the same position. In the plane.

[0052] Step S12: Based on the time synchronization relationship, control the preset omnidirectional camera array and the preset lidar to collect data to obtain omnidirectional visual images and corresponding lidar point clouds, and construct the projection mapping relationship between the omnidirectional visual images and the lidar point clouds based on the target spatial position relationship.

[0053] In this embodiment, the process of constructing a projection mapping relationship between an omnidirectional visual image and a LiDAR point cloud based on the target spatial position relationship includes: calibrating the cameras in a preset omnidirectional camera array to obtain the camera's intrinsic parameters, and determining the camera's extrinsic parameters relative to a preset LiDAR based on the target spatial position relationship; wherein, the intrinsic parameters include the camera's intrinsic parameter matrix and distortion coefficients, and the extrinsic parameters include a rotation matrix and a translation matrix, the translation matrix being a matrix representing the three-dimensional coordinates of the camera's optical center in the LiDAR coordinate system, and the rotation matrix being a matrix representing the angle of the camera coordinate system relative to the LiDAR coordinate system; and determining the point cloud corresponding to the pixels in the omnidirectional visual image based on the intrinsic and extrinsic parameters to obtain the projection mapping relationship.

[0054] The above-mentioned determination of the point cloud corresponding to a pixel in an omnidirectional visual image based on internal and external parameters may specifically include: performing distortion correction on the pixels in the omnidirectional visual image using an intrinsic parameter matrix and distortion coefficients to obtain the corrected pixel coordinates, and determining the first coordinate of the pixel in the camera coordinate system using the intrinsic parameter matrix and the corrected pixel coordinates; calculating the second coordinate of the reference point of the pixel in the lidar coordinate system using a translation matrix, a rotation matrix, and the first coordinate; determining the cylindrical coordinate of the reference point based on the second coordinate; determining the horizontal azimuth and pitch azimuth of the reference point based on the cylindrical coordinates; and determining the point cloud corresponding to the pixel in the lidar point cloud using the horizontal azimuth and pitch azimuth.

[0055] Specifically, the process of establishing the mutual projection framework between omnidirectional image pixels and LiDAR point clouds is as follows:

[0056] The LiDAR scanning frequency is set to f = 1 / T, where T is the time required for the LiDAR transmitter and receiver to complete one rotation and acquire the real-time point cloud. When the LiDAR begins acquiring the point cloud from all directions, the system time is recorded as t = 0, and the camera array is synchronously triggered to capture images. Settings: During the shooting process, record the system time as t=0; During the shooting, the system time was recorded as t=0.25T; During shooting, the system time is recorded as t=0.5T; During shooting, the system time is recorded as t=0.75T; this ensures the synchronization of data acquired by the lidar and camera array.

[0057] Each camera unit is calibrated to obtain the camera unit... The internal parameters include the intrinsic parameter matrix (i.e., the camera intrinsic parameter matrix). Distortion coefficient External parameters include the rotation matrix relative to the lidar coordinate system. Translation matrix For the camera unit Capture any pixel in the image The projection process is performed in the following order:

[0058] 1. Using the intrinsic parameter matrix and distortion coefficient Calculate the distortion-free pixel coordinates (i.e., the corrected pixel coordinates). ;

[0059] 2. Using camera projection model and intrinsic parameter matrix Calculate its position in the camera Normalized coordinates in coordinate system ;

[0060] 3. Using extrinsic rotation matrices Translation matrix Combined with back projection method Calculate the normalized reference point coordinates (i.e., the second coordinates) of the pixel in the LiDAR point cloud space. ;

[0061] 4. Calculate the normalized cylindrical coordinates of the reference point to obtain its horizontal and vertical azimuth angles, and then select the lidar point cloud with the best azimuth match. .

[0062] Through the above process, a mutual projection framework is established that corresponds one-to-one between each pixel of the omnidirectional photography system and the LiDAR point cloud.

[0063] Step S13: Use a preset image recognition model to perform image recognition on the omnidirectional visual image to obtain several first target detection boxes including object category labels.

[0064] In this embodiment, for a set of real-time point cloud and omnidirectional photographic image data, the system calls an artificial intelligence image processing model to perform recognition processing on the images acquired by the camera unit, obtaining several visual recognition results. Each visual recognition result includes a detection box composed of two pixel coordinates (i.e., the first target detection box), a classification label for the recognition result, etc.

[0065] Step S14: Determine the target region of the first target detection box in the lidar point cloud according to the projection mapping relationship, perform point cloud clustering processing in the target region to obtain the target point cloud cluster corresponding to the first target detection box, and use a preset three-dimensional point cloud processing tool to fit the target point cloud cluster to obtain the second target detection box corresponding to the target point cloud cluster, so as to complete the marking operation of the lidar point cloud.

[0066] In this embodiment, the process of performing point cloud clustering within the target area may specifically include: taking the point within the target area that is closest to the preset lidar as the initial cluster center, and performing neighboring point clustering based on a preset clustering distance threshold to obtain the target point cloud cluster; wherein, the preset clustering distance threshold is a threshold dynamically set according to the point cloud density.

[0067] Specifically, for a visual recognition detection box, the system uses the projection framework proposed in S2 to delineate an initial screening range in the real-time point cloud of the LiDAR based on the pixel coordinates. And perform the following operations:

[0068] 1. In this part of the point cloud, obtain the point cloud ranging information, filter out invalid points with a distance of 0, and select the point with the shortest laser ranging distance as the reference point.

[0069] 2. Use the reference point as the cluster center (i.e., the initial cluster center). ,calculate any point in the middle arrive Distance from the reference point Here, a cluster radius of 0.01 meters is used (i.e., a preset cluster distance threshold). That is, if d < 0.01, then P is considered to be in the cluster; otherwise, it is outside the cluster. Perform the "Calculate distance, cluster determination" operation on all points until... All points are traversed, and the resulting cluster is recorded as a point cloud cluster. ;

[0070] 3. With point cloud clusters Each point in the cluster becomes a new cluster center. Following step 2, a second round of clustering is performed to obtain point cloud clusters. Repeat this step until the point cloud cluster S is reached, such that... There is no point in the middle The distance to a point in point cloud cluster S is less than the cluster radius d. In this case, point cloud cluster S (i.e., the target point cloud cluster) represents the set of points of the visually detected object in the LiDAR point cloud space.

[0071] 4. Use a 3D point cloud processing tool to fit a 3D detection box (i.e., the second target detection box) to the point cloud cluster S. This detection box includes the geometric center coordinates of the visually recognized detection object and the length, width, and height dimensions of the 3D detection box. Visual recognition information, such as classification labels for the detection results, is also added.

[0072] That is, the second target detection box includes the geometric center coordinates of the object being measured, the size of the detection box, and the object category label of the object being measured.

[0073] After acquiring visual recognition and detection information, the omnidirectional photography system can achieve fully automatic real-time segmentation and labeling of LiDAR point clouds through the above processing, resulting in richer and faster processing.

[0074] It should be noted that, as Figure 3 As shown, current single-LiDAR labeling can only segment and label point clouds according to relatively coarse classifications, such as people, vehicles, and obstacles, making it difficult to correspond one-to-one with the specific classification of the object of interest (for example, because LiDAR point clouds lack visual information such as color and grayscale variations, point cloud recognition models can only segment and label all point cloud clusters that are close to the outline of a tree as "trees." In contrast, image-based recognition models can label specific subcategories such as "camphor trees," "maple trees," and "birch trees" under the "tree" category in an image based on visual information, which is more in line with human visual semantic observation habits). This application, however, achieves fine classification of objects in point clouds by registering the recognition results of an artificial intelligence recognition model with point cloud data.

[0075] Therefore, this application achieves real-time aligned acquisition of visual texture information and point cloud geometric information by deploying an omnidirectional camera array and LiDAR and establishing a time synchronization relationship; it achieves real-time and accurate mapping of visual detection results to point cloud space by constructing a projection mapping relationship between images and point clouds, replacing the time-consuming offline registration process; and it achieves fine target segmentation and 3D labeling based on visual semantic guidance by obtaining detection boxes with category labels through an image recognition model and mapping them to the point cloud for clustering and fitting. Thus, it simultaneously solves the technical problems of coarse point cloud segmentation results and large processing latency without the need for offline rendering.

[0076] See Figure 4 As shown, an embodiment of the present invention discloses a lidar point cloud marking device, comprising:

[0077] The radar deployment module 11 is used to deploy a preset omnidirectional camera array and a preset lidar according to the spatial position relationship of the target, and to establish a time synchronization relationship between the preset omnidirectional camera array and the preset lidar when collecting data.

[0078] The mapping relationship construction module 12 is used to control the preset omnidirectional camera array and the preset lidar to collect data based on the time synchronization relationship, so as to obtain omnidirectional visual images and corresponding lidar point clouds, and to construct the projection mapping relationship between the omnidirectional visual images and lidar point clouds based on the target spatial position relationship.

[0079] The detection box acquisition module 13 is used to perform image recognition on the omnidirectional visual image using a preset image recognition model to obtain a number of first target detection boxes including object category labels.

[0080] The point cloud marking module 14 is used to determine the target region of the first target detection box in the lidar point cloud according to the projection mapping relationship, perform point cloud clustering processing in the target region to obtain the target point cloud cluster corresponding to the first target detection box, and use a preset three-dimensional point cloud processing tool to fit the target point cloud cluster to obtain the second target detection box corresponding to the target point cloud cluster, so as to complete the marking operation of the lidar point cloud.

[0081] In some specific embodiments, the radar deployment module 11 may specifically include:

[0082] The planar control unit is used to control the mounting plane of the preset omnidirectional camera array to be parallel to the horizontal plane in the coordinate system of the preset lidar, and to control the central axis of the preset omnidirectional camera array to be coaxial with the central axis of the preset lidar.

[0083] In some specific embodiments, the mapping relationship construction module 12 may specifically include:

[0084] A camera calibration unit is used to calibrate the cameras in the preset omnidirectional camera array to obtain the camera's intrinsic parameters and determine the camera's extrinsic parameters relative to the preset lidar based on the target spatial position relationship. The intrinsic parameters include the camera's intrinsic parameter matrix and distortion coefficients, and the extrinsic parameters include a rotation matrix and a translation matrix. The translation matrix is ​​a matrix representing the three-dimensional coordinates of the camera's optical center in the lidar coordinate system, and the rotation matrix is ​​a matrix representing the angle of the camera coordinate system relative to the lidar coordinate system.

[0085] The point cloud determination submodule is used to determine the point cloud corresponding to the pixels in the omnidirectional visual image based on the internal parameters and the external parameters, so as to obtain the projection mapping relationship.

[0086] In some specific embodiments, the point cloud determination submodule may specifically include:

[0087] The distortion correction unit is used to perform distortion correction on the pixels in the omnidirectional visual image using the intrinsic parameter matrix and the distortion coefficients to obtain the corrected pixel coordinates, and to determine the first coordinates of the pixel in the camera coordinate system using the intrinsic parameter matrix and the corrected pixel coordinates.

[0088] A coordinate calculation unit is used to calculate the second coordinates of a reference point of a pixel in the lidar coordinate system using the translation matrix, the rotation matrix, and the first coordinates.

[0089] The point cloud determination unit is used to determine the cylindrical coordinates of the reference point based on the second coordinates, determine the horizontal azimuth and elevation azimuth of the reference point based on the cylindrical coordinates, and use the horizontal azimuth and elevation azimuth to determine the point cloud corresponding to the pixel in the lidar point cloud.

[0090] In some specific embodiments, the point cloud marking module 14 may specifically include:

[0091] The point cloud cluster acquisition unit is used to take the point with the shortest distance to the preset lidar in the target area as the initial cluster center, and perform neighbor point clustering based on a preset clustering distance threshold to obtain the target point cloud cluster; wherein, the preset clustering distance threshold is a threshold dynamically set according to the point cloud density.

[0092] Furthermore, embodiments of this application also disclose an electronic device, Figure 5 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0093] Figure 5 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the lidar point cloud marking method disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0094] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0095] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0096] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the lidar point cloud marking method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0097] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed lidar point cloud marking method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0098] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0099] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0100] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0101] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0102] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for marking point clouds using lidar, characterized in that, include: Deploy a preset omnidirectional camera array and a preset lidar according to the target's spatial location, and establish a time synchronization relationship between the preset omnidirectional camera array and the preset lidar when collecting data; Based on the time synchronization relationship, the preset omnidirectional camera array and the preset lidar are controlled to collect data to obtain omnidirectional visual images and corresponding lidar point clouds, and a projection mapping relationship between the omnidirectional visual images and lidar point clouds is constructed based on the target spatial position relationship. The omnidirectional visual image is subjected to image recognition using a preset image recognition model to obtain several first target detection boxes including object category labels; The target region of the first target detection box in the lidar point cloud is determined according to the projection mapping relationship. Point cloud clustering is performed in the target region to obtain the target point cloud cluster corresponding to the first target detection box. The target point cloud cluster is then fitted using a preset 3D point cloud processing tool to obtain the second target detection box corresponding to the target point cloud cluster, thereby completing the marking operation of the lidar point cloud.

2. The lidar point cloud marking method according to claim 1, characterized in that, The preset omnidirectional camera array includes several cameras with identical parameters deployed around a central point; wherein the field of view of each camera is adjacent to that of the others.

3. The lidar point cloud marking method according to claim 1, characterized in that, The deployment of a preset omnidirectional camera array and a preset lidar based on the target's spatial location includes: The mounting plane of the preset omnidirectional camera array is controlled to be parallel to the horizontal plane in the coordinate system of the preset lidar, and the central axis of the preset omnidirectional camera array is controlled to be coaxial with the central axis of the preset lidar.

4. The lidar point cloud marking method according to claim 3, characterized in that, The construction of the projection mapping relationship between the omnidirectional visual image and the lidar point cloud based on the target spatial position relationship includes: The cameras in the preset omnidirectional camera array are calibrated to obtain the camera's intrinsic parameters, and the external parameters of the cameras relative to the preset lidar are determined according to the target spatial position relationship; wherein, the intrinsic parameters include the camera intrinsic parameter matrix and distortion coefficients, and the external parameters include rotation matrix and translation matrix, wherein the translation matrix is ​​a matrix representing the three-dimensional coordinates of the camera's optical center in the lidar coordinate system, and the rotation matrix is ​​a matrix representing the angle of the camera coordinate system relative to the lidar coordinate system; The point cloud corresponding to the pixels in the omnidirectional visual image is determined based on the internal parameters and the external parameters to obtain the projection mapping relationship.

5. The lidar point cloud marking method according to claim 4, characterized in that, Determining the point cloud corresponding to pixels in the omnidirectional visual image based on the internal parameters and the external parameters includes: The intrinsic parameter matrix and the distortion coefficient are used to perform distortion correction on the pixels in the omnidirectional visual image to obtain the corrected pixel coordinates, and the intrinsic parameter matrix and the corrected pixel coordinates are used to determine the first coordinate of the pixel in the camera coordinate system; The second coordinates of the reference point of the pixel in the lidar coordinate system are calculated using the translation matrix, the rotation matrix, and the first coordinates; The cylindrical coordinates of the reference point are determined based on the second coordinates. The horizontal azimuth and elevation azimuth of the reference point are determined based on the cylindrical coordinates. The point cloud corresponding to the pixel in the lidar point cloud is determined using the horizontal azimuth and elevation azimuth.

6. The lidar point cloud marking method according to claim 1, characterized in that, The point cloud clustering process within the target area includes: The point within the target area that is closest to the preset lidar is used as the initial cluster center. Neighboring points are clustered based on a preset clustering distance threshold to obtain the target point cloud cluster. The preset clustering distance threshold is a threshold that is dynamically set according to the point cloud density.

7. The lidar point cloud marking method according to any one of claims 1 to 6, characterized in that, The second target detection box includes the geometric center coordinates of the object being measured, the size of the detection box, and the object category label of the object being measured.

8. A laser radar point cloud marking device, characterized in that, include: The radar deployment module is used to deploy a preset omnidirectional camera array and a preset lidar according to the spatial position relationship of the target, and to establish a time synchronization relationship between the preset omnidirectional camera array and the preset lidar when collecting data. The mapping relationship construction module is used to control the preset omnidirectional camera array and the preset lidar to collect data based on the time synchronization relationship, so as to obtain omnidirectional visual images and corresponding lidar point clouds, and to construct the projection mapping relationship between the omnidirectional visual images and lidar point clouds based on the target spatial position relationship. The detection box acquisition module is used to perform image recognition on the omnidirectional visual image using a preset image recognition model to obtain a number of first target detection boxes including object category labels. The point cloud marking module is used to determine the target region of the first target detection box in the lidar point cloud according to the projection mapping relationship, perform point cloud clustering processing in the target region to obtain the target point cloud cluster corresponding to the first target detection box, and use a preset three-dimensional point cloud processing tool to fit the target point cloud cluster to obtain the second target detection box corresponding to the target point cloud cluster, so as to complete the marking operation of the lidar point cloud.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the lidar point cloud marking method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the lidar point cloud marking method as described in any one of claims 1 to 7.

Images