A target automatic labeling method, device and vehicle

Abstract

A target automatic labeling method, acquires three-dimensional point cloud of at least one laser radar and image of at least one camera, and inputs three-dimensional point cloud of at least one laser radar and image of at least one camera into multi-modal MVF model, obtains detection frame of each target of three-dimensional point cloud.In the application, the multi-modal MVF algorithm utilizes the difference and complementarity of different types of views, fuses three-dimensional point cloud and ordinary image, and the attribute information of the detection frame of each target of three-dimensional point cloud obtained is more accurate.The multi-modal MVF algorithm superimposes the features of the ordinary image at the same time on the target features of the three-dimensional point cloud, makes the features of each target of the three-dimensional point cloud more prominent, and the target features of the three-dimensional point cloud obtained are more accurate.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a method, apparatus and vehicle for automatic target labeling. Background Technology

[0002] With the development of artificial intelligence, autonomous driving technology has become a hot development area. Laser-based 3D target detection modules are a core component of autonomous driving technology. These modules input 3D point clouds captured by LiDAR into a neural network (NN) model, which then annotates the targets within the 3D point cloud to detect them. The accuracy of the NN model's annotation of the 3D point cloud directly impacts the safety of autonomous driving technology. Therefore, how to efficiently and cost-effectively improve the accuracy of NN model annotation of 3D point clouds is a pressing issue that needs to be addressed. Summary of the Invention

[0003] To address the aforementioned problems, embodiments of this application provide an automatic target annotation method. This method utilizes a multimodal MVF algorithm to fuse 3D point clouds and ordinary images, resulting in more accurate target detection bounding boxes in the 3D point cloud. After obtaining the attribute information of each target detection bounding box in the 3D point cloud, the attribute information is then corrected to further improve the accuracy of the target detection bounding boxes. Furthermore, this application also provides an apparatus and vehicle corresponding to this rule-based reorganization method.

[0004] Therefore, the following technical solutions are adopted in the embodiments of this application:

[0005] In a first aspect, this application provides an automatic target annotation method, comprising: acquiring a 3D point cloud of at least one lidar and an image of at least one camera; inputting the 3D point cloud of the at least one lidar and the image of the at least one camera into a multimodal multiview fusion model to obtain detection boxes for each target in the 3D point cloud; the multimodal multiview fusion model is used to superimpose the features of the images at the same time onto the target features of the 3D point cloud, and to annotate the superimposed 3D point cloud to obtain the target detection boxes; the detection boxes refer to the boxes that annotate the targets in the 3D point cloud.

[0006] In this implementation, the multimodal MVF model fuses features from ordinary images into a 3D point cloud, compensating for the lack of texture and color in the 3D point cloud. This makes the features of each target in the 3D point cloud more prominent and improves the accuracy of the detection bounding boxes for each target. Furthermore, compared to the standard MVF model, the multimodal MVF model has a larger number of network parameters and acquires more accurate target features from the 3D point cloud.

[0007] In one embodiment, the step of inputting the 3D point cloud of the at least one lidar and the image of the at least one camera into a multimodal multi-view fusion model to obtain the detection boxes of each target in the 3D point cloud specifically includes: performing rotation and / or translation operations on the viewing angles of the 3D point cloud of the at least one lidar and the image of the at least one camera.

[0008] In this implementation, the multimodal MVF model performs rotation and translation operations on the perspectives of the LiDAR-acquired 3D point cloud and the camera-acquired ordinary image, ensuring that the perspectives of the multimodal MVF model on the LiDAR-acquired 3D point cloud and the camera-acquired ordinary image are at the same position. The multimodal MVF model superimposes features from the ordinary image at the same perspective position onto the 3D point cloud, avoiding misalignment between the features of the ordinary image and the 3D point cloud, which would otherwise cause confusion among the various targets in the 3D point cloud.

[0009] In one embodiment, the step of inputting the 3D point cloud of the at least one lidar and the image of the at least one camera into a multimodal multi-view fusion model to obtain the detection boxes of each target in the 3D point cloud specifically includes: dividing the 3D point cloud of the at least one lidar into multiple sub-3D point clouds; superimposing the features of the images at the same position at the same time onto the target features of the sub-3D point clouds; and labeling the superimposed sub-3D point clouds to obtain the detection boxes of each target in the 3D point cloud.

[0010] In this implementation, the multimodal MVF model can divide a 3D point cloud into multiple sub-3D point clouds, and then assign different sub-3D point clouds to different detectors, allowing each detector to detect only specific regions, thus improving the specificity of individual region detectors. Partition detection can be used to create larger networks, reducing voxel size, increasing feature maps, and increasing network parameters, thereby improving the accuracy of bounding boxes.

[0011] In one implementation, the attribute information of the target detection box includes GIOU, where GIOU represents the degree of overlap between the detection box and the ground view box. The method further includes: detecting whether the number of detection boxes of the target at the same location is greater than 1; in response to the number of detection boxes of the target at the same location being greater than 1, using the detection box of the target with the largest GIOU value as the detection box of the target at the same location.

[0012] In this implementation, when the sub-3D point clouds segmented by the multimodal MVF model overlap in their positions, multiple targets may be output at the same location, resulting in overlap. The attribute information of the detection boxes output by the multimodal MVF model also includes GIOU. Based on the GIOU of multiple detection boxes at the same location, the multimodal MVF model selects the 3D detection box with the larger GIOU as the true detection box and outputs that box, thus removing duplicate targets.

[0013] In one implementation, the attribute information of the target detection box includes the size of the target, and the method further includes increasing the length, width, and height of the detection box of each target in the 3D point cloud.

[0014] In this implementation, the multimodal MVF model increases the length, width, and height of the bounding boxes for each target in the 3D point cloud to prevent the bounding boxes from not fully encompassing the entire 3D point cloud of the target. If the multimodal MVF model extracts the 3D point cloud of the target from the bounding box, the target's 3D point cloud will be missing, leading to a decrease in the accuracy of automatic target labeling by the vehicle.

[0015] In one embodiment, the attribute information of the target detection box includes the target orientation, and the method further includes: stitching together the detection boxes of each target in the three-dimensional point cloud of multiple frames to obtain the motion trajectory of each target; and using an orientation optimization network to correct the orientation of the target in different frames of the motion trajectory of each target.

[0016] In this implementation, the multimodal MVF model stitches together the motion trajectory of the detection boxes of the same target from multiple frames of 3D point clouds, and then smooths the orientation of the detection boxes of each multimodal MVF model, correcting the orientation of detection boxes with large changes, so that the motion trajectory of the stitched detection boxes of the same target is smoother.

[0017] In one embodiment, the method further includes: stitching together the detection boxes of each target in a multi-frame 3D point cloud to obtain the motion trajectory of each target; and setting the detection boxes of different frames of the motion trajectory of each target to the same size.

[0018] In this implementation, the multimodal MVF model stitches together the detection boxes of the same target from the 3D point clouds of multiple frames to form a motion trajectory, and then sets the size of the multiple detection boxes to the same size, so that the same target has the same size in all frames, making the motion trajectory of the detection boxes of the same target smoother.

[0019] In one embodiment, setting the detection frames of different frames of the motion trajectories of the various targets to the same size includes: acquiring lane line information, the lane line information indicating the lane lines of the road where the vehicle is located; detecting whether the target is in a predetermined number of adjacent lanes of the vehicle; in response to the target being in a predetermined number of adjacent lanes of the vehicle, using the maximum size of the detection frames of different frames of the target's motion trajectory as the size of the detection frames of different frames of the target's motion trajectory; in response to the target not being in a predetermined number of adjacent lanes of the vehicle, using the maximum length of the projection of the detection frames of different frames of the target's motion trajectory onto the target's motion trajectory as the size of the detection frames of different frames of the target's motion trajectory.

[0020] In one embodiment, the method further includes: in response to the fact that the size of the detection boxes for different frames of the target's motion trajectory is greater than a set size, setting the size of the detection boxes for different frames of the target's motion trajectory to the set size.

[0021] In this implementation, the size of the detection boxes obtained by the multimodal MVF model may be too large, resulting in a large 3D point cloud for each detection box of the target's motion trajectory, which increases the workload of the processor. The multimodal MVF model can pre-set a preset size for each detection box of the target's motion trajectory, limiting the size of the detection boxes to within the preset size and avoiding excessive consumption of processing computing resources.

[0022] In one embodiment, the attribute information of the target detection box includes the center position of the target. The method further includes: stitching together the detection boxes of each target in the three-dimensional point cloud of multiple frames to obtain the motion trajectory of each target; and using a center point optimization network to correct the center position of the target in different frames of the motion trajectory of each target.

[0023] In this implementation, the multimodal MVF model stitches together the motion trajectory of the detection boxes of the same target from multiple frames of 3D point clouds, and then smooths the center position of each multimodal MVF model detection box, correcting the center position of detection boxes with large changes, so that the motion trajectory of the stitched detection boxes of the same target is smoother.

[0024] In one embodiment, the method further includes: inputting lane line information into a multimodal multi-view fusion model to obtain detection boxes of each target in a 3D point cloud based on lane lines; the attribute information of the target detection boxes includes confidence, the confidence being the probability that the detection box is a real target; in response to the confidence of the detection boxes of each target in the 3D point cloud based on lane lines being greater than a set threshold, outputting the detection boxes of the targets in the 3D point cloud with confidence greater than the set threshold.

[0025] In this implementation, the multimodal MVF model, based on 3D point cloud, ordinary image and lane line information, obtains higher confidence in the detection boxes of each target, allowing low-confidence targets to be identified as real targets and avoiding missed targets by vehicles.

[0026] Secondly, this application provides an automatic target labeling device, comprising: a transceiver unit for acquiring a 3D point cloud from at least one lidar and an image from at least one camera; a processing unit for inputting the 3D point cloud from the at least one lidar and the image from the at least one camera into a multimodal multiview fusion model to obtain detection boxes for each target in the 3D point cloud; the multimodal multiview fusion model is used to superimpose features from images at the same time onto target features in the 3D point cloud and label the superimposed 3D point cloud to obtain target detection boxes; the detection box refers to the box that labels the target in the 3D point cloud.

[0027] In one embodiment, the processing unit is further configured to perform rotation and / or translation operations on the viewing angles of the three-dimensional point cloud of the at least one lidar and the image of the at least one camera.

[0028] In one embodiment, the processing unit is further configured to divide the three-dimensional point cloud of the at least one lidar into multiple sub-three-dimensional point clouds; superimpose the features of images at the same location at the same time onto the target features of the sub-three-dimensional point clouds; and annotate each of the superimposed sub-three-dimensional point clouds to obtain the detection boxes of each target in the three-dimensional point cloud.

[0029] In one implementation, the attribute information of the target detection box includes GIOU, where GIOU represents the degree of overlap between the detection box and the ground view box. The processing unit is further configured to detect whether the number of detection boxes of the target at the same location is greater than 1; in response to the number of detection boxes of the target at the same location being greater than 1, the detection box of the target with the largest GIOU value is used as the detection box of the target at the same location.

[0030] In one implementation, the attribute information of the target detection box includes the size of the target, and the processing unit is further configured to increase the length, width, and height of the detection box of each target in the 3D point cloud.

[0031] In one implementation, the attribute information of the target detection box includes the target orientation. The processing unit is further configured to stitch together the detection boxes of each target in the three-dimensional point cloud of multiple frames to obtain the motion trajectory of each target; and to use an orientation optimization network to correct the orientation of the target in different frames of the motion trajectory of each target.

[0032] In one embodiment, the processing unit is further configured to stitch together the detection boxes of each target in the three-dimensional point cloud of multiple frames to obtain the motion trajectory of each target; and to set the detection boxes of different frames of the motion trajectory of each target to the same size.

[0033] In one embodiment, the processing unit is specifically configured to acquire lane line information, the lane line information indicating the lane lines of the road where the vehicle is located; detect whether a target is located in a predetermined number of adjacent lanes of the vehicle; in response to the target being located in a predetermined number of adjacent lanes of the vehicle, use the maximum size of the detection frame of different frames of the target's motion trajectory as the size of the detection frame of different frames of the target's motion trajectory; in response to the target not being located in a predetermined number of adjacent lanes of the vehicle, use the maximum length of the detection frame of different frames of the target's motion trajectory projected onto the target's motion trajectory as the size of the detection frame of different frames of the target's motion trajectory.

[0034] In one embodiment, the processing unit is further configured to set the size of the detection boxes of different frames of the target's motion trajectory to the set size in response to the fact that the size of the detection boxes of different frames of the target's motion trajectory is greater than the set size.

[0035] In one implementation, the attribute information of the target detection box includes the center position of the target. The processing unit is further configured to stitch together the detection boxes of each target in the three-dimensional point cloud of multiple frames to obtain the motion trajectory of each target; and to use a center point optimization network to correct the center position of the target in different frames of the motion trajectory of each target.

[0036] In one embodiment, the processing unit is further configured to input lane line information into a multimodal multi-view fusion model to obtain detection boxes of each target in a 3D point cloud based on lane lines; the attribute information of the target detection boxes includes confidence, the confidence being the probability that the detection box is a real target; in response to the confidence of the detection boxes of each target in the 3D point cloud based on lane lines being greater than a set threshold, the processing unit outputs the detection boxes of the targets in the 3D point cloud with confidence greater than the set threshold.

[0037] Thirdly, this application provides a vehicle, comprising: at least one lidar for acquiring a three-dimensional point cloud of the environment surrounding the vehicle; at least one camera for acquiring images of the environment surrounding the vehicle; and a processor connected to the at least one lidar and the at least one camera to perform various possible implementations of the methods described in the first aspect.

[0038] Fourthly, this application provides a computing device, comprising: at least one memory, at least one processor, the at least one processor being configured to execute instructions stored in the at least one memory to cause the computing device to perform various possible implementations of the methods described in the first aspect.

[0039] Fifthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform various possible implementations of the first aspect.

[0040] In a sixth aspect, this application provides a computer program product that stores instructions which, when executed by a computer, cause the computer to implement various possible implementations of the first aspect. Attached Figure Description

[0041] The accompanying drawings used in the description of the embodiments or prior art are briefly introduced below.

[0042] Figure 1 This is a schematic diagram showing the result of the laser 3D target detection module annotating the target.

[0043] Figure 2 This is a schematic diagram of the architecture of a vehicle provided in an embodiment of this application;

[0044] Figure 3 This is a schematic diagram illustrating the process of automatic target annotation using the multimodal MVF model provided in this application embodiment.

[0045] Figure 4 This is a schematic diagram showing the division of a 3D point cloud into multiple detection regions in the embodiments of this application.

[0046] Figure 5(a) is a schematic diagram of the orientation of multiple 3D detection boxes for the motion trajectory of the same target without optimization provided in the embodiments of this application;

[0047] Figure 5(b) is a schematic diagram of the orientation of multiple 3D detection boxes of the optimized motion trajectory of the same target provided in the embodiment of this application;

[0048] Figure 6 A schematic diagram illustrating the process of optimizing the size of 3D detection bounding boxes for multiple frames of the same target provided in this application embodiment.

[0049] Figure 7 This is a schematic diagram showing the center positions of multiple 3D detection boxes for the motion trajectory of the same target before and after optimization, as provided in the embodiments of this application.

[0050] Figure 8 This is a schematic diagram of the architecture of an automatic target labeling device provided in the embodiments of this application. Detailed Implementation

[0051] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0052] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0053] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0054] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0055] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0056] Autonomous driving technology is typically deployed in vehicles. These vehicles are equipped with data acquisition devices such as LiDAR and cameras. The vehicle's processor establishes a communication connection with these data acquisition devices, receiving information to acquire 3D point clouds, images, and other data. The vehicle's processor then runs a LiDAR 3D target detection module to detect obstacles such as vehicles and pedestrians around the vehicle.

[0057] Figure 1 This is a schematic diagram showing the result of the laser 3D target detection module annotating the target. For example... Figure 1As shown, the bounding boxes represent the results of the laser 3D target detection module annotating targets in the 3D point cloud. The laser 3D target detection module trains a neural network model using existing 3D annotation data, performs a series of processes, including target detection, on each frame of the 3D point cloud to be annotated, and obtains the 3D bounding boxes for the targets in each frame. The attribute information of the 3D bounding boxes includes the target's center position (x, y, z), target size (l, w, h), target orientation, target category (car, truck, cyclist, pedestrian), and the target's associated identity document (ID).

[0058] The data used to train the neural network (NN) model for the laser 3D target detection module is based on parameters such as a specific lidar and the lidar's fixed position on the vehicle. If the lidar model changes, or the lidar's fixed position on the vehicle changes, the laser 3D target detection module needs to retrain the NN model to label the 3D point cloud data collected by the changed lidar. The lidar's fixed position and orientation on the vehicle are prone to change, leading to a decrease in the accuracy of target detection by the laser 3D target detection module.

[0059] During the process of labeling targets in a 3D point cloud by the vehicle's processor, the neural network model first performs automatic labeling, and then the labeled 3D point cloud is displayed on the vehicle's screen. Users can correct the 3D bounding boxes displayed on the screen based on their observation of the vehicle's surrounding environment to improve the safety of the vehicle's autonomous driving technology. If the accuracy of the vehicle's processor's labeling is low, the number of targets that need correction by the user will be large, leading to a lack of trust in the vehicle's reliability.

[0060] To improve the accuracy of automatic labeling in existing autonomous driving technologies, embodiments of this application provide a method and apparatus for automatic labeling of vehicles and targets.

[0061] Figure 2 This is a schematic diagram of the architecture of a vehicle provided in an embodiment of this application. Figure 2 As shown, the vehicle 200 includes a data acquisition component 210, a memory 220, a processor 230, and a bus 240. The data acquisition component 210, the processor 220, and the processor 230 are connected via the bus 240 to transmit data.

[0062] The data acquisition component 210 can be a lidar, camera, ultrasonic radar, or other device, used to acquire environmental data and ground road data surrounding the vehicle 200. In this application, the data acquisition component 210 comprises a lidar and a camera. The lidar is used to acquire a 3D point cloud of the environment surrounding the vehicle 200. The camera is used to acquire general images of the environment surrounding the vehicle 200. The vehicle 200 includes multiple lidars and multiple cameras. The multiple lidars and multiple cameras are fixed at different positions on the vehicle 200.

[0063] Memory 220 may include volatile memory, such as random-access memory (RAM). Memory 220 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), solid-state drive (SSD), etc. Memory 201 may be a cloud server or a combination of the various types of memory mentioned above. In this application, memory 220 can serve as a cache database for storing data acquired by data acquisition component 210. Memory 220 is used to store applications running on processor 230, allowing processor 230 to execute automatic target annotation methods. Memory 220 is used to store data such as images, 3D point clouds, and attribute information of 3D detection boxes processed by processor 230. Memory 220 may also have other functions, which are not limited herein.

[0064] The processor 230 can be a central processing unit (CPU), a microcontroller unit (MCU), a domain controller (DC), or other devices with processing capabilities, and can also be a cloud server. The processor 230 is used to receive data from the data acquisition component 210 via the bus 240 and execute the automatic target labeling method.

[0065] In this application, the processor 230 can receive 3D point clouds from a LiDAR and ordinary images from a camera, and input the LiDAR 3D point cloud and the camera ordinary images into a multi-view fusion (MVF) model to obtain attribute information of the 3D detection boxes of the 3D point cloud. A detection box refers to a box that labels a target in a 3D point cloud. A 3D detection box refers to a box that labels the 3D structure of a target in a 3D point cloud.

[0066] In one embodiment, when there are multiple lidars, the processor 230 can stitch together the three-dimensional point clouds of the multiple lidars into a complete three-dimensional point cloud.

[0067] In one embodiment, when there are multiple cameras, the processor 230 can stitch together the ordinary images from the multiple cameras into a complete ordinary image.

[0068] The MVF algorithm fuses the differences and complementarities between different views to obtain a 3D point cloud of the vehicle's surrounding environment, and then annotates the targets in the 3D point cloud to obtain 3D bounding boxes for each target. The multimodal MVF algorithm overlays features from other types of images onto the target features of the 3D point cloud, then annotates the overlaid 3D point cloud to obtain 3D bounding boxes for the targets.

[0069] In this application, the processor 230 can receive 3D point clouds continuously transmitted by a LiDAR and ordinary images continuously transmitted by a camera, and input the 3D point clouds and ordinary images into a multimodal MVF model. A multimodal MVF model refers to a module capable of running the multimodal MVF algorithm. The multimodal MVF model can superimpose the features of an ordinary image frame at the same time onto the target features of a 3D point cloud frame, making the features of each target in the 3D point cloud more prominent.

[0070] The multimodal MVF model annotates targets in a superimposed 3D point cloud, obtaining 3D bounding boxes for each target and outputting the attribute information of each bounding box. A 3D bounding box refers to the 3D point cloud containing the target. The attribute information of the target's 3D bounding box output by the multimodal MVF model includes not only the target's center position (x, y, z), size (l, w, h), orientation, type (car, truck, cyclist, pedestrian), and ID, but also GIOU and a confidence score. GIOU represents the degree of overlap between the 3D bounding box and the ground truth (GT) bounding box, serving as a metric for the quality of the 3D bounding box. A higher GIOU value indicates a greater overlap between the 3D bounding box and the GT bounding box. The confidence score represents the probability that the bounding box is a real target, a measure of its existence.

[0071] In one embodiment, after receiving a normal image, the multimodal MVF model can use an image feature extraction algorithm to extract features from the normal image in order to obtain various targets in the normal image.

[0072] In one embodiment, the multimodal MVF model performs rotation and translation operations on the viewpoints of the 3D point cloud acquired by the LiDAR and the ordinary image acquired by the camera, ensuring that the viewpoints of the multimodal MVF model for the 3D point cloud acquired by the LiDAR and the ordinary image acquired by the camera are at the same position. The multimodal MVF model superimposes features from the ordinary image at the same viewpoint onto the 3D point cloud, avoiding misalignment between the features of the ordinary image and the 3D point cloud, which would otherwise cause confusion among the various targets in the 3D point cloud.

[0073] Figure 3 This is a schematic diagram illustrating the process of automatic target annotation using the multimodal MVF model provided in this application embodiment. Figure 3 As shown, processor 230 inputs 3D point clouds and ordinary images into the multimodal MVF model. The point-wise feature fetching unit of the multimodal MVF model extracts features from both the 3D point cloud and the ordinary image to obtain target features from the 3D point cloud and features from the ordinary image, respectively. The multi-view voxelization unit of the multimodal MVF model superimposes the target features of the 3D point cloud into a stereo feature according to a predetermined method, and also superimposes the features of the ordinary image into a stereo feature according to a predetermined method. The feature detector (encoder) further superimposes the target features of the 3D point cloud and the image features on a bird's-eye view (BEV) to form multimodal features. The 3D backbone of the multimodal MVF model further extracts features suitable for 3D detection based on the obtained multimodal stereo features. The region proposal network (RPN) of the multimodal MVF model outputs the target annotations, obtaining the 3D detection boxes for each target. The head unit, regression unit, and GIOU unit of the multimodal MVF model can extract various attribute information of the 3D bounding box.

[0074] Compared to the MVF model, the multimodal MVF model fuses features from ordinary images into a 3D point cloud, compensating for the lack of texture and color in the 3D point cloud. This makes the features of each target in the 3D point cloud more prominent and improves the confidence of each target. Furthermore, compared to the MVF model, the multimodal MVF model has a larger number of network parameters, resulting in more accurate acquisition of target features from the 3D point cloud.

[0075] As the number of network parameters in the multimodal MVF model increases, the core region of the 3D point cloud detected by the processor 230 may decrease. The core region of the 3D point cloud refers to the area within a set distance around the vehicle. In this application, the processor 230 can propose a partitioned multimodal MVF model based on the multimodal MVF model. The partitioned multimodal MVF model can divide the 3D point cloud into multiple detection regions, and then automatically label the targets in the sub-3D point clouds of each detection region. The partitioned multimodal MVF model divides the 3D point cloud into multiple detection regions without reducing the core region detected by the processor 230, and may even increase it.

[0076] In one embodiment, the processor 230 can input ordinary images and 3D point clouds into a partitioned multimodal MVF model. The partitioned multimodal MVF model detects the core regions of the 3D point clouds, dividing the key areas of the 3D point clouds into multiple detection regions according to predefined rules, resulting in multiple sub-3D point clouds. The partitioned multimodal MVF model assigns the multiple sub-3D point clouds to detectors in different partitions. The detectors detect the features of the sub-3D point clouds and, based on the sub-3D point clouds, obtain the features of the ordinary images at the same locations. The partitioned multimodal MVF model superimposes the features of the ordinary images at the same locations onto the features of the sub-3D point clouds, obtaining the multimodal features corresponding to the superimposed sub-3D point clouds. The partitioned multimodal MVF model annotates the multimodal feature targets of the superimposed sub-3D point clouds to obtain the 3D detection boxes of the superimposed sub-3D point clouds. The partitioned multimodal MVF model sequentially annotates the targets of other superimposed sub-3D point clouds, obtaining the 3D detection boxes of other superimposed sub-3D point clouds, and outputs the 3D detection boxes of all 3D point clouds.

[0077] In a partitioned multimodal MVF model, multiple sub-3D point clouds can partially overlap in the 3D point cloud. For example... Figure 4 As shown, the partitioned multimodal MVF model divides the key area of ​​the 3D point cloud into four detection regions, resulting in four sub-3D point clouds: sub-3D point cloud 1, sub-3D point cloud 2, sub-3D point cloud 3, and sub-3D point cloud 4. Parts of sub-3D point cloud 2 overlap with parts of sub-3D point cloud 1. Parts of sub-3D point cloud 2 overlap with parts of sub-3D point cloud 3. Parts of sub-3D point cloud 3 overlap with parts of sub-3D point cloud 4.

[0078] When the sub-3D point clouds segmented by the partitioned multimodal dynamic range (MVF) model overlap in their positions, multiple targets may be output at the same location, resulting in overlap. The 3D bounding boxes output by the partitioned MVF model also include GIOU (Geometric Target Identity). Based on the GIOU of multiple 3D bounding boxes at the same location, the partitioned MVF model selects the 3D bounding box with the larger GIOU as the true 3D bounding box and outputs it, thus removing duplicate targets.

[0079] Compared to multimodal multimodal (MVF) models, partitioned multimodal MVF models divide a 3D point cloud into multiple sub-3D point clouds, and then assign different sub-3D point clouds to different detectors. This allows each detector to perform detection only on a specific region, improving the specificity of individual region detectors. Partitioned detection can be used to create larger networks, reducing voxel size, increasing feature maps, and increasing network parameters, thereby improving the accuracy of 3D bounding boxes.

[0080] The processor 230 continuously receives 3D point clouds from the LiDAR and ordinary images from the camera, acquiring 3D bounding boxes corresponding to each target in each frame of the 3D point cloud. The processor 230 can use a baseline for 3D multi-object tracking (3D ABMOT) algorithm to track the 3D bounding boxes corresponding to each target in the multiple frames obtained, and obtain the motion trajectory of each target.

[0081] In this embodiment, the processor 230 can acquire 3D point clouds and ordinary images collected by the vehicle, and superimpose the target features of the 3D point cloud and the features of the ordinary image based on a multimodal MVF algorithm or a partitioned multimodal MVF algorithm. The multimodal MVF algorithm or the partitioned multimodal MVF algorithm marks the superimposed 3D point cloud, resulting in a more accurate 3D detection box corresponding to the target.

[0082] After the processor 230 obtains the 3D detection bounding boxes of the 3D point cloud, the attribute information of the 3D detection bounding boxes may contain errors. For example, the target size (l, w, h) may be inaccurate, the target orientation may be inaccurate, the size (l, w, h) of the same target in different frames may be different, the target center position (x, y, z) may be inaccurate, the target type (car, truck, cyclist, pedestrian) may be incorrectly identified, and there may be missed targets. The processor 230 can correct the various attribute information of the 3D detection bounding boxes based on various types of correction algorithms to improve the accuracy of the attribute information of the 3D detection bounding boxes.

[0083] The 3D detection bounding box is too small, its position is off-center from the target's position, and it cannot encompass the entire 3D point cloud of the target. When the processor 230 extracts the target's 3D point cloud from the 3D detection bounding box, the target's 3D point cloud will be missing, resulting in reduced accuracy of the vehicle 200's automatic target labeling.

[0084] In this application, the processor 230 can enlarge the size of the 3D detection box, allowing it to contain more 3D point clouds, ensuring that the 3D detection box can encompass all the 3D point clouds of the target. In one embodiment, when the processor 230 enlarges the size (l, w, h) of the 3D detection box, it multiplies the size (l, w, h) by a set coefficient k, resulting in a detection box size of (k×l, k×w, k×h). k is a positive number greater than 1. Preferably, k is equal to 1.5.

[0085] After enlarging the size of the 3D detection box, the processor 230 can identify the 3D point cloud of the target within the enlarged 3D detection box and extract the 3D point cloud of the target from the 3D detection box. In one embodiment, the processor 230 can utilize a foreground segmentation network to obtain foreground points within the enlarged 3D detection box and extract the foreground points from the 3D detection box. The foreground points are the 3D point clouds belonging to the target. Optionally, the foreground segmentation network can be a pointNet or other networks.

[0086] After acquiring the attribute information of the 3D detection boxes of multiple frames of 3D point clouds, the processor 230 can stitch together the 3D detection boxes of the same target from the multiple frames of 3D point clouds to obtain the motion trajectory of the 3D detection boxes of the same target. In one embodiment, the orientation of the 3D detection boxes of the same target in multiple frames of 3D point clouds varies greatly, and the motion trajectory after stitching together the 3D detection boxes of the same target in multiple frames of 3D point clouds may not be smooth. In this application, the processor 230 smooths the orientation of each 3D detection box after stitching together the motion trajectory of the 3D detection boxes of the same target in multiple frames of 3D point clouds, correcting the orientation of the 3D detection boxes with large changes, so that the motion trajectory of the stitched 3D detection boxes of the same target is smoother.

[0087] In one embodiment, the processor 230 can utilize an orientation optimization network to optimize the orientation of 3D detection boxes across multiple frames, constraining the orientation of 3D detection boxes with significant variations to the trend of target orientation changes. For example, the processor 230 inputs the orientations of 3D detection boxes from two frames of the same target into the orientation optimization network, and the orientation optimization network obtains the orientation optimization value LOSS for the two frames of 3D detection boxes, which is:

[0088]

[0089] Where α represents the angle classification loss weight, β represents the angle residual loss weight, γ represents the angle smoothing loss weight, T represents the current frame, S represents the maximum span of all selected frames, Lcls represents the classification loss, Lreg represents the residual loss, and Lfit represents the smoothing loss. The first term of formula (1) is the angle classification loss from the commonly used angle regression multi-bin loss, the second term is the residual loss from the regression orientation, and the third term is the smoothing constraint term. The orientation optimization loss of the third term is:

[0090]

[0091] Here, x and y represent a common corner point across all 3D bounding boxes. f() is a cubic fitting function for the corner point. A smoothing constraint term is used to measure the fitting error, assessing the smoothness of a common corner point between two 3D bounding boxes. The smaller the Loss value, the smoother the target's trajectory.

[0092] In this application, the processor 230 stitches together the 3D detection boxes of the same target from multiple frames to form the target's motion trajectory. As shown in Figure 5(a), the motion trajectory of the same target stitched together by the processor 230 is not smooth. As shown in Figure 5(b), after the processor 230 corrects the orientation of the 3D detection boxes of the target from multiple frames, the motion trajectory of the same target is relatively smooth. Therefore, it is evident that the processor 230 can correct the orientation of the 3D detection boxes of the target one by one, with a very significant effect.

[0093] When processor 230 stitches together 3D detection boxes from multiple frames of the same target, the sizes of the 3D detection boxes in the multiple frames may be different, resulting in inconsistent sizes of the 3D detection boxes for the same target across different frames. In actual target annotation, it is essential that the same target has the same size in different frames. In this application, processor 230 stitches together the 3D detection boxes of the same target from multiple frames of 3D point clouds to form a motion trajectory, and then sets the size of the multiple 3D detection boxes to the same size, ensuring that the same target has the same size in all frames.

[0094] Figure 6 This is a schematic diagram illustrating the process of optimizing the size of 3D detection bounding boxes for multiple frames of the same target, as provided in an embodiment of this application. Figure 6 As shown, the process by which the processor 230 optimizes the size of the 3D detection bounding boxes for multiple frames of the same target is as follows:

[0095] In step S601, the processor 230 acquires the target's motion trajectory and lane line information.

[0096] Lane line information refers to the lane lines on the road where vehicle 200 is located. In this application, the memory 220 of vehicle 200 stores a high-precision map. Vehicle 200 is equipped with a positioning module. During driving, vehicle 200 can obtain the lane lines within a set range of the current road based on its position and the high-precision map.

[0097] In one embodiment, after the processor 230 acquires the lane lines of the current road of the vehicle 200, it performs operations such as rotation and translation on the viewpoint of the lane lines so that the viewpoint of the multimodal MVF model for the 3D point cloud acquired by the LiDAR, the viewpoint of the camera for the ordinary image, and the viewpoint of the lane lines are in the same position.

[0098] In step S602, the processor 230 detects whether the target is in a predetermined number of lanes adjacent to the vehicle 200. If the processor 230 detects that the target is in a predetermined number of lanes adjacent to the vehicle 200, step S603 is executed. If the processor 230 detects that the target is not in a predetermined number of lanes adjacent to the vehicle 200, step S604 is executed.

[0099] When vehicle 200 is traveling on the road, obstacles such as vehicles and pedestrians close to vehicle 200 will affect its driving strategy. Obstacles farther away from vehicle 200 will not affect its driving strategy, or will have a minor impact. When vehicle 200 acquires a 3D point cloud using LiDAR, obstacles close to vehicle 200 are clearly displayed in the 3D point cloud, while obstacles farther away are not clearly displayed or are not fully displayed. Based on these two factors, vehicle 200 only needs to detect targets close to it.

[0100] In step S603, the processor 230 uses the maximum size of multiple 3D detection boxes in the target's motion trajectory as the standard size of the target's 3D detection box.

[0101] In step S604, the processor 230 determines the effective size of the target's 3D detection box based on the target's motion trajectory and the 3D point cloud of the target's 3D detection box.

[0102] The processor 230 can acquire the motion trajectories of targets located on a predetermined number of lanes on both sides of the vehicle 200 based on the motion trajectories of multiple targets in a 3D point cloud and the lane lines of the current road. Preferably, the predetermined number can be two lanes. The processor 230 can use the size of the largest 3D detection box of the target's motion trajectory as the standard size. The processor 230 can set the standard size for the dimensions of each 3D detection box of the target's motion trajectory.

[0103] The processor 230 acquires the motion trajectory of a target that is not located on a predetermined number of vehicles on either side of the vehicle 200. Based on the 3D point cloud of the target's 3D detection boxes in different frames, the processor projects the 3D point cloud of the 3D detection boxes in different frames onto a straight line along the side length of the corresponding detection box of the target's motion trajectory. The distance between the farthest projection points on the straight line along the side length of the corresponding detection box of the target's motion trajectory is used as the effective size. The processor 230 can set the effective size of each 3D detection box of the target's motion trajectory.

[0104] In step S605, the processor 230 determines whether the standard size or the effective size is greater than the set size. If the processor 230 determines that the standard size or the effective size is greater than the set size, step S606 is executed. If the processor 230 determines that the standard size or the effective size is not greater than the set size, step S607 is executed.

[0105] In step S606, the processor 230 sets the size of each 3D detection box of the target's motion trajectory to a set size.

[0106] In step S607, the processor 230 sets the size of each 3D detection box of the target's motion trajectory to the standard size or the effective size.

[0107] The standard and effective dimensions obtained by processor 230 may be too large, resulting in large 3D point clouds of the 3D detection boxes for the target's motion trajectory, increasing the workload of processor 230. Processor 230 can preset the dimensions of the 3D detection boxes for the target's motion trajectory. In one embodiment, if the standard dimension obtained by processor 230 is larger than the preset dimension, the dimensions of the 3D detection boxes for the motion trajectory of targets on a predetermined number of vehicles located on both sides of vehicle 200 can be set to the preset dimensions. In another embodiment, if the standard dimension obtained by processor 230 is not larger than the preset dimension, the dimensions of the 3D detection boxes for the motion trajectory of targets on a predetermined number of vehicles located on both sides of vehicle 200 can be set to the standard dimension.

[0108] In one embodiment, the effective size obtained by the processor 230 is greater than a preset size, and the size of each 3D detection box of the motion trajectory of a target not located on a predetermined number of vehicles on both sides of the vehicle 200 can be set to the preset size. In another embodiment, the effective size obtained by the processor 230 is not greater than the preset size, and the size of each 3D detection box of the motion trajectory of a target not located on a predetermined number of vehicles on both sides of the vehicle 200 can be set to the effective size.

[0109] When processor 230 automatically labels a target, the center position of the 3D detection box may be inaccurate. Processor 230 stitches together 3D detection boxes from multiple frames of the same target, resulting in a motion trajectory that may not be smooth. In this application, processor 230 stitches together the motion trajectory of the 3D detection boxes of the same target from multiple frames of 3D point clouds, and then smooths the center position of each 3D detection box, correcting for changes in the center position of 3D detection boxes, thus making the motion trajectory of the stitched 3D detection boxes of the same target smoother.

[0110] In one embodiment, the processor 230 can utilize a center point optimization network to optimize the center positions of 3D detection boxes across multiple frames, constraining the center positions of 3D detection boxes with significant variations within the trend of target center point changes. For example, the processor 230 inputs the center positions of 3D detection boxes from two frames of the same target into the center point optimization network, which obtains the center point optimization value (LOSS) for the two frames' 3D detection boxes, as follows:

[0111]

[0112] Where α represents the weight of the center point regression loss, β represents the weight of the center variance, T represents the current frame, S represents the maximum span of all selected frames, and c represents the center point. The first term of formula (3) is the conventional center point regression center point optimization value LOSS, and the second term is the improved value with added smoothing constraints. δ 2 The variance of the distance between the center points of consecutive frames is as follows:

[0113] δ 2 =var(c i -c i-1 (4)

[0114] Here, var represents the variance to be extracted.

[0115] In this application, the processor 230 stitches together the 3D detection bounding boxes of the same target from multiple frames to form the target's motion trajectory. For example... Figure 7 As shown in the dark box, the motion trajectory of the same target, which is normally stitched together by processor 230, is not smooth. For example... Figure 7 As shown in the light-colored box, after the processor 230 corrects the center position of the target's 3D detection box across multiple frames, the target's motion trajectory is relatively smooth. This demonstrates that the processor 230 can correct the center position of the target's 3D detection box one by one, with very noticeable results.

[0116] In this embodiment, after the processor 230 obtains the 3D detection boxes corresponding to the targets in the 3D point cloud based on the multimodal MVF algorithm or the partitioned multimodal MVF algorithm, it can perform operations such as reasonably increasing the size of the 3D detection boxes, smoothing the orientation of the 3D detection boxes in different frames of the running trajectory of each target, unifying the size of the 3D detection boxes in different frames of the running trajectory of each target, and smoothing the center position of the 3D detection boxes in different frames of the running trajectory of each target, based on the attribute information of the 3D detection boxes of each target, so as to further improve the accuracy of the vehicle 200 in automatically detecting the 3D detection boxes of the 3D point cloud.

[0117] If target detection errors or missed targets still occur after the processor 230 completes automatic target labeling, the processor 230 can eliminate the target detection errors and missed targets based on lane line information and two-dimensional (2D) detection results of ordinary images.

[0118] In one embodiment, the processor 230 can obtain targets located in a predetermined number of lanes on both sides of the vehicle 200 based on the 2D detection results of a normal image and the lane line information of the current road of the vehicle 200. The processor 230 compares the targets in the predetermined number of lanes on both sides of the vehicle 200 with the targets corresponding to the 3D detection boxes in the 3D point cloud. If the processor 230 determines that the targets in the predetermined number of lanes on both sides of the vehicle 200 are not the targets corresponding to the 3D detection boxes in the 3D point cloud, it can define the target as a real target and add a 3D detection box corresponding to the target to the 3D point cloud.

[0119] When the processor 230 performs automatic target annotation using a multimodal MVF model or a partitioned multimodal MVF model, it can simultaneously input 3D point clouds, ordinary images, and lane line information. The multimodal MVF model or partitioned multimodal MVF model additionally receives lane line information, which can improve the confidence level of the target's 3D detection bounding box.

[0120] Normally, the processor 230 can determine whether a target is a real target based on the confidence level of its 3D detection bounding box. When the confidence level of a target's 3D detection bounding box is lower than a set threshold, the processor 230 identifies the target as a false target and filters out its 3D detection bounding box. When the confidence level of a target's 3D detection bounding box is not lower than the set threshold, the processor 230 identifies the target as a real target. In this application, the multimodal MVF model or the partitioned multimodal MVF model increases the confidence level of each target's 3D detection bounding box, allowing targets with low confidence levels to be identified as real targets, thus preventing the vehicle 200 from missing targets.

[0121] Figure 8 This is a schematic diagram of the architecture of an automatic target labeling device provided in an embodiment of this application. Figure 8As shown, the automatic target labeling device 800 includes a transceiver unit 801 and a processing unit 802. The specific execution process of the automatic target labeling device 800 is as follows:

[0122] The transceiver unit 801 is used to acquire 3D point clouds from at least one lidar and images from at least one camera; the processing unit 802 is used to input the 3D point clouds from at least one lidar and the images from at least one camera into a multimodal multi-view fusion model to obtain detection boxes for each target in the 3D point cloud. The multimodal multi-view fusion model is used to superimpose the features of the images at the same time onto the target features of the 3D point cloud, and to annotate the superimposed 3D point cloud to obtain the target detection boxes. A detection box refers to a box that annotates a target in the 3D point cloud.

[0123] In one embodiment, the processing unit 802 is further configured to perform rotation and / or translation operations on the perspective of the three-dimensional point cloud of at least one lidar and the image of at least one camera.

[0124] In one embodiment, the processing unit 802 is further configured to divide the three-dimensional point cloud of at least one lidar into multiple sub-three-dimensional point clouds; superimpose the features of images at the same location at the same time onto the target features of the sub-three-dimensional point clouds; and annotate each sub-three-dimensional point cloud after superposition to obtain the detection box of each target in the three-dimensional point cloud.

[0125] In one implementation, the target detection bounding box attribute information includes GIOU. GIOU represents the degree of overlap between the detection bounding box and the ground view bounding box. The processing unit 802 is further configured to detect whether the number of detection bounding boxes for targets at the same location is greater than 1; in response to the number of detection bounding boxes for targets at the same location being greater than 1, the detection bounding box of the target with the largest GIOU value is selected as the detection bounding box for the target at the same location.

[0126] In one implementation, the attribute information of the target's detection bounding box includes the target's dimensions. The processing unit 802 is also used to increase the length, width, and height of the detection bounding boxes of each target in the 3D point cloud.

[0127] In one implementation, the attribute information of the target's detection bounding box includes the target's orientation. The processing unit 802 is further configured to stitch together the detection bounding boxes of each target in multiple frames of 3D point clouds to obtain the motion trajectory of each target; and to use an orientation optimization network to correct the orientation of the target in different frames of the motion trajectory of each target.

[0128] In one embodiment, the processing unit 802 is further configured to stitch together the detection boxes of each target in the three-dimensional point cloud of multiple frames to obtain the motion trajectory of each target; and set the detection boxes of different frames of the motion trajectory of each target to the same size.

[0129] In one embodiment, the processing unit 802 is specifically configured to acquire lane line information, which indicates the lane lines of the road where the vehicle is located; detect whether the target is in a predetermined number of adjacent lanes of the vehicle; in response to the target being in a predetermined number of adjacent lanes of the vehicle, use the maximum size of the detection frame of different frames of the target's motion trajectory as the size of the detection frame of different frames of the target's motion trajectory; in response to the target not being in a predetermined number of adjacent lanes of the vehicle, use the maximum length of the detection frame of different frames of the target's motion trajectory projected onto the target's motion trajectory as the size of the detection frame of different frames of the target's motion trajectory.

[0130] In one embodiment, the processing unit 802 is further configured to set the size of the detection frame of the different frames of the target's motion trajectory to the set size in response to the fact that the size of the detection frame of the different frames of the target's motion trajectory is greater than the set size.

[0131] In one implementation, the attribute information of the target's detection bounding box includes the target's center position. The processing unit 802 is further configured to stitch together the detection bounding boxes of each target in multiple frames of the 3D point cloud to obtain the motion trajectory of each target; and to use a center point optimization network to correct the center position of the target in different frames of the motion trajectory of each target.

[0132] In one embodiment, the processing unit 802 is further configured to input lane line information into a multimodal multi-view fusion model to obtain detection boxes of each target in a 3D point cloud based on lane lines; the attribute information of the target detection boxes includes confidence, which is the probability that the detection box is a real target; in response to the confidence of the detection boxes of each target in the 3D point cloud based on lane lines being greater than a set threshold, the processing unit 802 outputs the detection boxes of the targets in the 3D point cloud with confidence greater than the set threshold.

[0133] The present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform any of the methods described above.

[0134] The present invention provides a computing device, including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement any of the methods described above.

[0135] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0136] Furthermore, various aspects or features of the embodiments of this application can be implemented as methods, apparatus, or articles of manufacture using standard programming and / or engineering techniques. The term "article of manufacture" as used in this application encompasses a computer program accessible from any computer-readable device, carrier, or medium. For example, computer-readable media may include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks, or magnetic tapes), optical discs (e.g., compact discs (CDs), digital versatile discs (DVDs), etc.), smart cards, and flash memory devices (e.g., erasable programmable read-only memory (EPROMs), cards, sticks, or key drives, etc.). Additionally, the various storage media described herein may represent one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing, and / or carrying instructions and / or data.

[0137] In the above embodiments, the automatic target labeling device 800 can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented entirely or partially in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., SSD), etc.

[0138] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application.

[0139] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0140] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0141] The unit described as a separate component may or may not be physically separate. The component shown as a unit may or may not be a physical unit; that is, it may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0142] If this function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application embodiment, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or an access network device, etc.) to execute all or part of the steps of the method in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0143] The above are merely specific implementations of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application.

Claims

1. An automatic target annotation method, characterized in that, include: Acquire 3D point clouds from at least one lidar and images from at least one camera; The 3D point cloud of the at least one lidar and the image of the at least one camera are input into a multimodal multiview fusion model to obtain detection boxes for each target in the 3D point cloud; the multimodal multiview fusion model is used to superimpose the features of the images at the same time onto the target features of the 3D point cloud, and to annotate the superimposed 3D point cloud to obtain the target detection boxes; the detection box refers to the box that annotates the target in the 3D point cloud. The detection bounding boxes of each target in the 3D point cloud of multiple frames are stitched together to obtain the motion trajectory of each target. Obtain lane line information, which indicates the lane lines of the road in which the vehicle is located; Detect whether the target is in a predetermined number of lanes adjacent to the vehicle; In response to the target being located in a predetermined number of adjacent lanes of the vehicle, the maximum size of the detection box of different frames of the target's motion trajectory is used as the size of the detection box of different frames of the target's motion trajectory; In response to the target not being in a predetermined number of adjacent lanes of the vehicle, the maximum length of the detection box of different frames of the target's motion trajectory projected onto the target's motion trajectory is used as the size of the detection box of different frames of the target's motion trajectory; In response to the fact that the size of the detection box of different frames of the target's motion trajectory is larger than a set size, the size of the detection box of different frames of the target's motion trajectory is set to the set size.

2. The method according to claim 1, characterized in that, The step of inputting the 3D point cloud of the at least one lidar and the image of the at least one camera into a multimodal multi-view fusion model to obtain the detection bounding boxes of each target in the 3D point cloud specifically includes: Rotation and / or translation operations are performed on the viewing angles of the three-dimensional point cloud of the at least one lidar and the image of the at least one camera.

3. The method according to claim 1, characterized in that, The step of inputting the 3D point cloud of the at least one lidar and the image of the at least one camera into a multimodal multi-view fusion model to obtain the detection bounding boxes of each target in the 3D point cloud specifically includes: The three-dimensional point cloud of the at least one lidar is divided into multiple sub-three-dimensional point clouds; The features of images at the same location at the same time are superimposed on the target features of the sub-3D point cloud; The superimposed sub-3D point clouds are labeled to obtain the detection boxes of each target in the 3D point cloud.

4. The method according to claim 1, characterized in that, The target's bounding box attribute information includes GIOU, which represents the degree of overlap between the bounding box and the ground view bounding box. The method further includes: Check if the number of detection boxes for targets at the same location is greater than 1; If the number of detection boxes for a target at the same location is greater than 1, the detection box of the target with the largest GIOU value is taken as the detection box for the target at the same location.

5. The method according to claim 1, characterized in that, The properties of the target's detection bounding box include the target's size. The method further includes: The length, width, and height of the detection box for each target in the 3D point cloud are increased.

6. The method according to claim 1, characterized in that, The properties of the target's bounding box include the target's orientation. The method further includes: The detection bounding boxes of each target in the 3D point cloud of multiple frames are stitched together to obtain the motion trajectory of each target. An orientation optimization network is used to correct the orientation of the targets in different frames of the motion trajectory of each target.

7. The method according to claim 1, characterized in that, The properties of the target's bounding box include the target's center position. The method further includes: The detection bounding boxes of each target in the 3D point cloud of multiple frames are stitched together to obtain the motion trajectory of each target. The center point optimization network is used to correct the center position of the target in different frames of the motion trajectory of each target.

8. The method according to any one of claims 1-7, characterized in that, The method further includes: Lane line information is input into a multimodal multi-view fusion model to obtain detection boxes for each target in a 3D point cloud based on lane lines; the attribute information of the target detection box includes confidence, which is the probability that the detection box is a real target; In response to the fact that the confidence of the detection boxes of each target in the lane-line-based 3D point cloud is greater than a set threshold, the detection boxes of the targets in the 3D point cloud with a confidence of greater than the set threshold are output.

9. An automatic target labeling device, characterized in that, include: A transceiver unit is used to acquire 3D point clouds from at least one lidar and images from at least one camera; The processing unit is used to input the 3D point cloud of the at least one lidar and the image of the at least one camera into a multimodal multiview fusion model to obtain the detection boxes of each target in the 3D point cloud; the multimodal multiview fusion model is used to superimpose the features of the images at the same time onto the target features of the 3D point cloud, and to annotate the superimposed 3D point cloud to obtain the target detection boxes; the detection box refers to the box that annotates the target in the 3D point cloud. The detection bounding boxes of each target in the 3D point cloud of multiple frames are stitched together to obtain the motion trajectory of each target. Obtain lane line information, which indicates the lane lines of the road in which the vehicle is located; Detect whether the target is in a predetermined number of lanes adjacent to the vehicle; In response to the target being located in a predetermined number of adjacent lanes of the vehicle, the maximum size of the detection box of different frames of the target's motion trajectory is used as the size of the detection box of different frames of the target's motion trajectory; In response to the target not being in a predetermined number of adjacent lanes of the vehicle, the maximum length of the detection box of different frames of the target's motion trajectory projected onto the target's motion trajectory is used as the size of the detection box of different frames of the target's motion trajectory; In response to the fact that the size of the detection box of different frames of the target's motion trajectory is larger than a set size, the size of the detection box of different frames of the target's motion trajectory is set to the set size.

10. The apparatus according to claim 9, characterized in that, The processing unit is also used to perform rotation and / or translation operations on the viewing angles of the three-dimensional point cloud of the at least one lidar and the image of the at least one camera.

11. The apparatus according to claim 9, characterized in that, The processing unit is further configured to divide the three-dimensional point cloud of the at least one lidar into multiple sub-three-dimensional point clouds. The features of images at the same location at the same time are superimposed on the target features of the sub-3D point cloud; The superimposed sub-3D point clouds are labeled to obtain the detection boxes of each target in the 3D point cloud.

12. The apparatus according to claim 9, characterized in that, The target's bounding box attribute information includes GIOU, which represents the degree of overlap between the bounding box and the ground view bounding box. The processing unit is also used to detect whether the number of detection boxes for a target at the same location is greater than 1; If the number of detection boxes for a target at the same location is greater than 1, the detection box of the target with the largest GIOU value is taken as the detection box for the target at the same location.

13. The apparatus according to claim 9, characterized in that, The properties of the target's detection bounding box include the target's size. The processing unit is also used to increase the length, width, and height of the detection boxes of each target in the three-dimensional point cloud.

14. The apparatus according to claim 9, characterized in that, The properties of the target's bounding box include the target's orientation. The processing unit is also used to stitch together the detection boxes of each target in the three-dimensional point cloud of multiple frames to obtain the motion trajectory of each target. An orientation optimization network is used to correct the orientation of the targets in different frames of the motion trajectory of each target.

15. The apparatus according to claim 9, characterized in that, The properties of the target's bounding box include the target's center position. The processing unit is also used to stitch together the detection boxes of each target in the three-dimensional point cloud of multiple frames to obtain the motion trajectory of each target. The center point optimization network is used to correct the center position of the target in different frames of the motion trajectory of each target.

16. The apparatus according to any one of claims 9-15, characterized in that, The processing unit is further configured to input lane line information into a multimodal multi-view fusion model to obtain detection boxes of each target in a 3D point cloud based on lane lines; the attribute information of the target detection box includes confidence, which is the probability that the detection box is a real target; In response to the fact that the confidence of the detection boxes of each target in the lane-line-based 3D point cloud is greater than a set threshold, the detection boxes of the targets in the 3D point cloud with a confidence of greater than the set threshold are output.

17. A vehicle, characterized in that, include: At least one lidar is used to acquire a three-dimensional point cloud of the environment surrounding the vehicle; At least one camera is used to capture images of the environment surrounding the vehicle; A processor, connected to the at least one lidar and the at least one camera, performs the method as described in any one of claims 1-8.

18. A computing device, characterized in that, include: At least one memory, At least one processor, the at least one processor being configured to execute instructions stored in the at least one memory to cause the computing device to perform the method as described in any one of claims 1-8.

Images