Training methods for 3D object detection models, 3D object detection methods and devices

Abstract

This application provides a training method, a 3D object detection method, and an apparatus for a 3D object detection model, relating to the field of computer vision. The training method includes: acquiring training samples; training a 3D object detection model based on the training samples to obtain a trained 3D object detection model. The training samples include target point cloud frames and target image frames. The target point cloud frames and target image frames are obtained by modally synchronized resampling of initial image frames and initial point cloud frames, and by processing the resampling point cloud frames and image frames according to the occlusion relationships between objects. The scheme of this application embodiment can enhance the initial point cloud frames and initial image frames, improve the diversity of training samples, and is beneficial to improving the performance of the trained 3D object detection model.

Description

Technical Field

[0001] This application relates to the field of computer vision, and in particular to a training method for a 3D object detection model, a 3D object detection method, a data augmentation method, and an apparatus. Background Technology

[0002] Computer vision is an integral part of various intelligent / autonomous systems across diverse application fields, such as manufacturing, inspection, document analysis, medical diagnosis, and the military. It's the science of how to use cameras / video cameras and computers to acquire the data and information we need about the objects being photographed. Figuratively speaking, it's about equipping computers with eyes (cameras / video cameras) and a brain (algorithms) to replace human eyes in identifying, tracking, and measuring targets, thus enabling computers to perceive their environment. Because perception can be seen as extracting information from sensory signals, computer vision can also be viewed as the science of how to enable artificial systems to "perceive" from images or multidimensional data. In short, computer vision uses various imaging systems to replace visual organs in acquiring input information, and then the computer replaces the brain in processing and interpreting this input information. The ultimate research goal of computer vision is to enable computers to observe and understand the world through vision like humans, possessing the ability to autonomously adapt to their environment.

[0003] Three-dimensional (3D) object detection in computer vision provides various metrics such as the 3D size and rotation angle of objects in an image, attracting widespread attention across multiple application areas. Existing 3D object detection methods mainly include unimodal and multimodal approaches. Unimodal 3D object detection utilizes single-modal data, such as image data or laser data, to achieve object detection. Multimodal 3D object detection uses a combination of image and laser data, among other modalities, to achieve object detection, and this approach significantly outperforms unimodal 3D object detection. Training samples are crucial for the training effectiveness of 3D object detection models. Data augmentation methods can greatly increase the diversity of training samples and improve the robustness of the model.

[0004] Currently, a series of effective data augmentation methods have been developed for unimodal 3D object detection. For example, point clouds and 3D bounding boxes of objects from other scenes can be pasted into the current scene to increase the diversity of training samples and improve model accuracy. However, multimodal 3D object detection requires training samples from multiple modalities, and these data must be consistent across modalities. Therefore, data augmentation methods used in unimodal 3D object detection cannot be applied to multimodal 3D object detection training. Existing data augmentation methods used in multimodal 3D object detection training are limited to operations such as rotation, translation, and scaling, which cannot effectively increase the number of training samples and thus fail to improve model performance.

[0005] Therefore, improving the accuracy of 3D target detection has become an urgent problem to be solved. Summary of the Invention

[0006] This application provides a training method for a 3D object detection model, a 3D object detection method, a data augmentation method, and an apparatus, which can increase the diversity of training samples and improve the accuracy of the 3D object detection model.

[0007] Firstly, a training method for a 3D object detection model is provided, comprising: acquiring training samples, the training samples including: target point cloud frames and labels on the target point cloud frames, target image frames and labels on the target image frames, the labels on the target point cloud frames including 3D bounding boxes indicating the regions where the point cloud of an object is located in the target point cloud frame, and the labels on the target image frames including 2D bounding boxes indicating the regions where the image patches of an object are located in the target image frames; the target point cloud frames are obtained by filtering points in intermediate point cloud frames that are occluded by objects in intermediate point cloud frames, the target image frames are obtained by adjusting the coverage order of image patches of objects in intermediate image frames according to the occlusion relationship between objects in intermediate point cloud frames, the intermediate point cloud frames are obtained by adding the point cloud of at least one sample to an initial point cloud frame, the intermediate image frames are obtained by adding the image patches of at least one sample to the initial image frame, the initial image frames and the initial point cloud frames have a correspondence, and at least one sample is used to indicate at least one object; and training the 3D object detection model based on the training samples to obtain a trained 3D object detection model.

[0008] According to the scheme of this application embodiment, the training samples are obtained by enhancing the initial image frame and the initial point cloud frame, which increases the number of objects in the training samples. Compared with using the initial image frame and the initial point cloud frame as training samples, the scheme of this application can improve the diversity of training samples, thereby improving the performance of the trained model, that is, improving the detection accuracy and reducing the false detection rate. Moreover, based on the occlusion relationship between objects in the intermediate point cloud frame, occluded point clouds are filtered out, ensuring the rationality of the target point cloud frame. At the same time, based on the occlusion relationship between objects in the intermediate point cloud frame, the coverage order between the image patches of objects is adjusted, which further ensures the rationality of the target image frame and also ensures the consistency between the target point cloud frame and the target image frame.

[0009] Image frames and point cloud frames with a corresponding relationship can be understood as paired image frames and point cloud frames.

[0010] The point cloud of at least one sample and the image patch of at least one sample can be the point cloud and image patch of the labeled sample extracted from at least one point cloud frame and at least one image frame, respectively. Each image frame in the at least one image frame and each point cloud frame in the at least one point cloud frame are in one-to-one correspondence.

[0011] The point cloud and image patch of the at least one sample can be obtained by sampling in a sampling pool. The sampling pool includes point clouds and image patches of multiple candidate samples. The point clouds and image patches of the multiple candidate samples can be point clouds and image patches of labeled samples extracted from paired point cloud frames and image frames using the ground truth of the labeled samples.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, the intermediate point cloud frame is obtained by adding the point cloud of at least one sample to the initial point cloud frame, including: the intermediate point cloud frame is obtained by adding the point clouds of different samples from the at least one sample to different regions of the initial point cloud frame respectively.

[0013] In conjunction with the first aspect, in some implementations of the first aspect, there is no overlap between the point cloud region of at least one sample in the intermediate point cloud frame and the point cloud region of the object in the initial point cloud frame.

[0014] According to the scheme of the embodiment of this application, there is no overlapping area between the point cloud of the sample and the point cloud of the object in the initial point cloud frame. This can avoid the unreasonable situation of multiple objects colliding in three-dimensional space in the intermediate point cloud frame, thus ensuring the rationality of the intermediate point cloud frame. In turn, it ensures the rationality of the target point cloud frame obtained based on the intermediate point cloud frame. Using the target point cloud frame as a training sample can improve the performance of the trained 3D object detection model.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, there is no overlapping region between the point cloud region of at least one sample in the intermediate point cloud frame and the point cloud region of the object in the initial point cloud frame, including: on BEV, the 3D bounding box of the at least one sample and the 3D bounding box of the object region in the initial point cloud frame do not overlap in the intermediate point cloud frame.

[0016] In conjunction with the first aspect, in some implementations of the first aspect, the area in which the point cloud of at least one sample is located in the intermediate point cloud frame is the same as the area in which the point cloud of at least one sample is located in the point cloud frame of the original scene, and the point cloud frame of the original scene is a point cloud frame acquired based on the scene in which at least one object indicated by at least one sample is located.

[0017] According to the scheme of the embodiments of this application, the area where the point cloud of at least one sample is located in the intermediate point cloud frame is the same as the area where the point cloud of at least one sample is located in the point cloud frame of the original scene, which satisfies the sparsity of the point cloud in the intermediate point cloud frame, avoids the situation that does not match the actual situation, further improves the rationality of the intermediate point cloud frame, and thus ensures the rationality of the target point cloud frame obtained based on the intermediate point cloud frame. Using the target point cloud frame as a training sample can improve the performance of the trained 3D target detection model.

[0018] For example, the point cloud of the at least one sample is added to the area indicated by the coordinates of the 3D bounding box of the at least one sample in the initial point cloud frame.

[0019] In conjunction with the first aspect, in some implementations of the first aspect, the region in which the image patch of at least one sample is located in the intermediate image frame is obtained by projecting the point cloud of at least one sample in the region in the intermediate point cloud frame onto the initial image frame.

[0020] According to the scheme of the embodiments of this application, the region where the image block of the sample is located in the initial image frame is obtained based on the region where the point cloud of the sample is located in the initial point cloud frame. This can avoid the influence caused by camera extrinsic jitter, ensure the consistency between the position of the point cloud of the sample in the initial point cloud frame and the position of the image block of the sample in the initial image frame, thereby improving the accuracy of training samples and ensuring the training effect of the 3D object detection model.

[0021] In conjunction with the first aspect, in some implementations of the first aspect, points occluded by objects in the intermediate point cloud frame include: points located within the viewing range of objects in the intermediate point cloud frame, and whose distance from the observation point in the intermediate point cloud frame is greater than the distance between the object in the intermediate point cloud frame and the observation point.

[0022] In conjunction with the first aspect, in some implementations of the first aspect, the object in the intermediate point cloud frame includes the first object. In the case where the first object is an object in the initial point cloud frame, the point occluded by the first object includes the first 3D point in the intermediate point cloud frame. The first 3D point is a point in the point cloud of at least one sample, and the first 3D point is located within the view range of the first object, and the distance between the first 3D point and the origin is greater than the distance between the first object and the origin.

[0023] In conjunction with the first aspect, in some implementations of the first aspect, the object in the intermediate point cloud frame includes the first object. If the first object is also an object in the initial point cloud frame, the points occluded by the first object include the second and third 3D points in the intermediate point cloud frame. The second 3D point is a background point and is located within the viewpoint of the first object. The third 3D point is a foreground point and is also located within the viewpoint of the first object; the distance between the third 3D point and the origin is greater than the distance between the first object and the origin; and the third 3D point does not belong to the first object.

[0024] In conjunction with the first aspect, in some implementations of the first aspect, the occlusion relationship between objects in the intermediate point cloud frame is determined based on the distance between the objects in the intermediate point cloud frame and the observation point, and the target image frame is obtained by adjusting the coverage order of the image blocks of the objects in the intermediate image frame according to the occlusion relationship between the objects in the intermediate point cloud frame, including: the target image frame is obtained by sequentially covering the image blocks of the objects in the intermediate image frame in order from farthest to closest distance between the objects in the intermediate point cloud frame and the observation point.

[0025] In conjunction with the first aspect, in some implementations of the first aspect, the region in which the point cloud of at least one sample is located in the intermediate point cloud frame is the target region in the intermediate point cloud frame.

[0026] According to the solution of the embodiments of this application, the point cloud of the sample can be placed in the target area to increase the number of objects in the target area, so as to obtain a training sample that better meets the requirements.

[0027] In conjunction with the first aspect, in some implementations of the first aspect, the target area is the road surface area.

[0028] The model in this application embodiment can be applied to the field of autonomous driving. In the field of autonomous driving, more attention is usually paid to objects in the road surface area. Adding the point cloud of the sample to the road surface area can increase the number of objects in the road surface area, while ensuring the rationality of the intermediate point cloud frame, thereby ensuring the rationality of the target point cloud frame obtained based on the intermediate point cloud frame. Using the target point cloud frame as a training sample can improve the performance of the trained 3D object detection model.

[0029] Secondly, a 3D object detection method is provided, comprising: acquiring image frames to be processed and point cloud frames to be processed, wherein there is a correspondence between the image frames to be processed and the point cloud frames to be processed; using a trained 3D object detection model to perform 3D object detection on the image frames to be processed and the point cloud frames to be processed, and obtaining detection results, wherein the trained 3D object detection model is obtained by training the 3D object detection model based on training samples, the training samples including: target point cloud frames and labels on the target point cloud frames, target image frames and labels on the target image frames, the labels on the target point cloud frames including 3D tags indicating the region where the object is located in the target point cloud frame. The target image frame includes a 2D bounding box labeled with a label indicating the region where an image patch of an object is located in the target image frame. The target point cloud frame is obtained by filtering out points in the intermediate point cloud frame that are occluded by objects in the intermediate point cloud frame. The target image frame is obtained by adjusting the coverage order of the image patches of objects in the intermediate image frame according to the occlusion relationship between objects in the intermediate point cloud frame. The intermediate point cloud frame is obtained by adding the point cloud of at least one sample to the initial point cloud frame. The intermediate image frame is obtained by adding the image patch of at least one sample to the initial image frame. There is a correspondence between the initial image frame and the initial point cloud frame. At least one sample is used to indicate at least one object.

[0030] According to the scheme of this application embodiment, the trained 3D object detection model is trained by the method in the first aspect, wherein the training samples are obtained by enhancing the initial image frame and the initial point cloud frame. Compared with using the initial image frame and the initial point cloud frame as training samples, the scheme of this application can improve the diversity of training samples, thereby improving the performance of the trained model, that is, improving the detection accuracy and reducing the false detection rate. Moreover, based on the occlusion relationship between objects in the intermediate point cloud frame, occluded point clouds are filtered out, ensuring the rationality of the target point cloud frame. At the same time, based on the occlusion relationship between objects in the intermediate point cloud frame, the coverage order between the image blocks of the objects is adjusted, which further ensures the rationality of the target image frame and also ensures the consistency between the target point cloud frame and the target image frame.

[0031] In conjunction with the second aspect, in some implementations of the second aspect, there is no overlap between the point cloud region of at least one sample in the intermediate point cloud frame and the point cloud region of the object in the initial point cloud frame.

[0032] In conjunction with the second aspect, in some implementations of the second aspect, the area in which the point cloud of at least one sample is located in the intermediate point cloud frame is the same as the area in which the point cloud of at least one sample is located in the point cloud frame of the original scene, and the point cloud frame of the original scene is a point cloud frame collected based on the scene where at least one object indicated by at least one sample is located.

[0033] In conjunction with the second aspect, in some implementations of the second aspect, the region in which the image patch of at least one sample is located in the intermediate image frame is obtained by projecting the region in which the point cloud of at least one sample is located in the intermediate point cloud frame onto the initial image frame.

[0034] In conjunction with the second aspect, in some implementations of the second aspect, points occluded by objects in the intermediate point cloud frame include: points located within the viewing range of objects in the intermediate point cloud frame, and whose distance from the observation point in the intermediate point cloud frame is greater than the distance between the object in the intermediate point cloud frame and the observation point.

[0035] In conjunction with the second aspect, in some implementations of the second aspect, the occlusion relationship between objects in the intermediate point cloud frame is determined based on the distance between the objects in the intermediate point cloud frame and the observation point, and the target image frame is obtained by adjusting the coverage order of the image blocks of the objects in the intermediate image frame according to the occlusion relationship between the objects in the intermediate point cloud frame, including: the target image frame is obtained by sequentially covering the image blocks of the objects in the intermediate image frame in order from farthest to closest distance between the objects in the intermediate point cloud frame and the observation point.

[0036] In conjunction with the second aspect, in some implementations of the second aspect, the region in which the point cloud of at least one sample is located in the intermediate point cloud frame is the target region in the intermediate point cloud frame.

[0037] In conjunction with the second aspect, in some implementations of the second aspect, the target area is the road surface area.

[0038] Thirdly, a data augmentation method is provided, comprising: acquiring an initial point cloud frame and an initial image frame, wherein the initial point cloud frame and the initial image frame have a corresponding relationship; adding the point cloud of at least one sample to the initial point cloud frame to obtain an intermediate point cloud frame; adding the image patch of at least one sample to the initial image frame to obtain an intermediate image frame, wherein at least one sample is used to indicate at least one object; filtering points in the intermediate point cloud frame that are occluded by objects in the intermediate point cloud frame to obtain a target point cloud frame; and adjusting the coverage order of the image patches of objects in the intermediate image frame according to the occlusion relationship between objects in the intermediate point cloud frame to obtain the target image frame.

[0039] The target point cloud frame and target image frame obtained by the scheme in this application embodiment can be used as training samples. Compared with using the initial image frame and initial point cloud frame as training samples, the scheme of this application increases the diversity of training samples and improves the performance of the trained model, for example, improving the recognition rate and reducing the false detection rate. At the same time, based on the occlusion relationship between objects in the intermediate point cloud frame, occluded point clouds are filtered out, ensuring the rationality of the target point cloud frame. In addition, based on the occlusion relationship between objects in the intermediate point cloud frame, the coverage order between the image blocks of the objects is adjusted, which further ensures the rationality of the target image frame, and at the same time, it can ensure the consistency of the projection relationship between the target point cloud frame and the target image frame.

[0040] The point cloud of at least one sample and the image patch of at least one sample can be the point cloud and image patch of the labeled sample extracted from at least one point cloud frame and at least one image frame, respectively. Each image frame in the at least one image frame and each point cloud frame in the at least one point cloud frame are in one-to-one correspondence.

[0041] In conjunction with the third aspect, in some implementations of the third aspect, the point cloud and image patch of the at least one sample may be obtained by sampling in a sampling pool. The sampling pool includes point clouds and image patches of multiple candidate samples, which may be point clouds and image patches of labeled samples extracted from paired point cloud frames and image frames using the ground truth values ​​of labeled samples.

[0042] In conjunction with the third aspect, in some implementations of the third aspect, adding the point cloud of at least one sample to the initial point cloud frame includes: adding the point clouds of different samples from the at least one sample to different regions of the initial point cloud frame respectively.

[0043] In conjunction with the third aspect, in some implementations of the third aspect, there is no overlap between the point cloud region of at least one sample in the intermediate point cloud frame and the point cloud region of the object in the initial point cloud frame.

[0044] In conjunction with the third aspect, in some implementations of the third aspect, there is no overlapping region between the region where the point cloud of the at least one sample is located in the initial point cloud and the region where the point cloud of the object in the initial point cloud is located, including: from a bird's-eye view, there is no overlapping region between the 3D bounding box of the at least one sample and the 3D bounding box of the region where the object in the initial point cloud frame is located in the intermediate point cloud frame.

[0045] In conjunction with the third aspect, in some implementations of the third aspect, the area in which the point cloud of at least one sample is located in the intermediate point cloud frame is the same as the area in which the point cloud of at least one sample is located in the point cloud frame of the original scene, and the point cloud frame of the original scene is a point cloud frame acquired based on the scene in which the at least one object indicated by the at least one sample is located.

[0046] In conjunction with the third aspect, in some implementations of the third aspect, the region in which the image patch of the at least one sample is located in the intermediate image frame is obtained by projecting the point cloud of the at least one sample in the region in the intermediate point cloud frame onto the initial image frame.

[0047] In conjunction with the third aspect, in some implementations of the third aspect, points occluded by objects in the intermediate point cloud frame include: points located within the viewing range of objects in the intermediate point cloud frame, and whose distance from the observation point in the intermediate point cloud frame is greater than the distance between the object in the intermediate point cloud frame and the observation point.

[0048] In conjunction with the third aspect, in some implementations of the third aspect, the occlusion relationship between objects in the intermediate point cloud frame is determined based on the distance between the objects in the intermediate point cloud frame and the observation point, and the covering order of the image blocks of the objects in the intermediate image frame is adjusted according to the occlusion relationship between the objects in the intermediate point cloud frame to obtain the target image frame, including: sequentially covering the image blocks of the objects in the intermediate image frame in order from farthest to closest distance between the objects in the intermediate point cloud frame and the observation point to obtain the target image frame.

[0049] In conjunction with the third aspect, in some implementations of the third aspect, adding the point cloud of at least one sample to the initial point cloud frame includes: adding the point cloud of the at least one sample to the target region in the initial point cloud frame.

[0050] In conjunction with the third aspect, in some implementations of the third aspect, the target area is the road surface area.

[0051] Fourthly, a training apparatus for a 3D object detection model is provided, the apparatus including a module or unit for performing the methods described in the first aspect and any implementation thereof.

[0052] Fifthly, a 3D target detection apparatus is provided, the apparatus including a module or unit for performing the method in the second aspect and any implementation thereof.

[0053] In a sixth aspect, a data enhancement apparatus is provided, the apparatus including a module or unit for performing the methods of the third aspect and any implementation thereof.

[0054] It should be understood that the extensions, limitations, interpretations and descriptions of the relevant content in the first aspect above also apply to the same content in the second, third, fourth, fifth and sixth aspects.

[0055] In a seventh aspect, a training apparatus for a 3D object detection model is provided. The apparatus includes: a memory for storing a program; and a processor for executing the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method in the first aspect and any implementation thereof.

[0056] The processor mentioned in the seventh aspect above can be a central processing unit (CPU) or a combination of a CPU and a neural network processing processor. The neural network processing processor can include a graphics processing unit (GPU), a neural network processing unit (NPU), and a tensor processing unit (TPU), etc. The TPU is a dedicated integrated circuit developed by Google for a fully customized artificial intelligence accelerator for machine learning.

[0057] Eighthly, a 3D target detection device is provided, the device comprising: a memory for storing a program; and a processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the method of the second aspect and any implementation thereof.

[0058] The processor mentioned in the eighth aspect above can be a central processing unit (CPU) or a combination of a CPU and a neural network processing processor. The neural network processing processor can include a graphics processing unit (GPU), a neural network processor, and a tensor processor, among others. The TPU is a Google-designed application-specific integrated circuit (ASIC) for a fully customized AI accelerator designed for machine learning.

[0059] A ninth aspect provides a computer-readable medium storing program code for execution by a device, the program code including methods for performing any one of the first, second, or third aspects.

[0060] In a tenth aspect, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to perform the method in any one of the implementations of the first, second, or third aspects described above.

[0061] Eleventhly, a chip is provided, the chip including a processor and a data interface, the processor reading instructions stored in a memory through the data interface to execute the method in any one of the implementations of the first, second or third aspects described above.

[0062] Optionally, as one implementation, the chip may further include a memory storing instructions, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to perform the method in any one of the first, second, or third implementations.

[0063] The aforementioned chip can be a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).

[0064] In a twelfth aspect, a vehicle is provided that includes the means in any one of the implementations of the fourth, fifth, or sixth aspects described above. Attached Figure Description

[0065] Figure 1 This is a schematic diagram of the structure of an autonomous vehicle provided in an embodiment of this application;

[0066] Figure 2 This is a schematic flowchart illustrating the vehicle control process in an autonomous driving system provided in an embodiment of this application;

[0067] Figure 3 This is a schematic diagram of the structure of a data enhancement device provided in an embodiment of this application;

[0068] Figure 4 This is a schematic flowchart illustrating a data augmentation method provided in an embodiment of this application;

[0069] Figure 5 This is a schematic diagram of a road surface area extraction process and a sample screening process provided in an embodiment of this application;

[0070] Figure 6 This is an example diagram illustrating a point cloud filtering process and an image block pasting process provided in an embodiment of this application.

[0071] Figure 7 This is a schematic flowchart illustrating a point cloud filtering process provided in an embodiment of this application;

[0072] Figure 8 This is a schematic diagram of a point cloud filtering effect provided in an embodiment of this application;

[0073] Figure 9 This is a schematic flowchart of the training method for the 3D object detection model provided in the embodiments of this application;

[0074] Figure 10 This is a schematic flowchart of the 3D target detection method provided in the embodiments of this application;

[0075] Figure 11 A schematic block diagram of a training device for a 3D object detection model provided in an embodiment of this application;

[0076] Figure 12 A schematic block diagram of a 3D target detection device provided in an embodiment of this application;

[0077] Figure 13 A schematic block diagram of a training device for a 3D object detection model provided in an embodiment of this application;

[0078] Figure 14 A schematic block diagram of a 3D target detection device provided in an embodiment of this application;

[0079] Figure 15 This is a schematic diagram of a vehicle driving process provided in an embodiment of this application. Detailed Implementation

[0080] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0081] Object detection is one of the fundamental tasks in the field of computer vision. Object detection not only requires identifying objects in an image—that is, determining the category of the object—but also determining the object's location. Three-dimensional (3D) object detection can output the object's category and information such as its length, width, height, or rotation angle in three-dimensional space based on data such as two-dimensional images or laser point clouds.

[0082] The method of this application can be applied to fields that require 3D object detection, such as autonomous driving, robot navigation, drone navigation, and virtual / augmented reality. The following mainly describes the autonomous driving scenario.

[0083] Figure 1 This is a functional block diagram of a vehicle 100 provided in an embodiment of the present invention. In one embodiment, the vehicle 100 is configured in a fully or partially automated driving mode.

[0084] In one example, vehicle 100 can control a target vehicle while in autonomous driving mode, and can determine the current state of the vehicle and its surrounding environment through human intervention, determine the possible behaviors of at least one other vehicle in the surrounding environment, and determine the confidence level corresponding to the probability of other vehicles performing possible behaviors, and control vehicle 100 based on the determined information. When vehicle 100 is in autonomous driving mode, vehicle 100 can be set to operate without human interaction.

[0085] Vehicle 100 may include various subsystems, such as a mobility system 110, a sensing system 120, a control system 130, one or more peripheral devices 140, a power supply 160, a computer system 150, and a user interface 170. Optionally, vehicle 100 may include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and component of vehicle 100 may be interconnected via wired or wireless means.

[0086] Exemplarily, the mobility system 110 may include components for providing powered motion to the vehicle 100. In this embodiment, the mobility system can be used to drive the vehicle to perform corresponding motion behaviors, such as forward, backward, and steering, during obstacle avoidance. The mobility system 110 includes an engine 111, a transmission 112, an energy source 113, and wheels 114.

[0087] The sensing system 120 may include several sensors for sensing information about the environment surrounding the vehicle 100. In embodiments of this application, the sensing system can be used to acquire environmental information and road structure information, thereby performing subsequent control based on this acquired information.

[0088] For example, the sensing system 120 may include a positioning system 121 (e.g., a global positioning system, BeiDou system, or other positioning system), an inertial measurement unit (IMU) 122, a radar 123, a laser rangefinder 124, a camera 125, and a vehicle speed sensor 126. The sensing system 120 may also include sensors from the internal systems of the monitored vehicle 100 (e.g., an in-vehicle air quality monitor, fuel gauge, oil temperature gauge, etc.). Sensor data from one or more of these sensors can be used to detect objects and their corresponding characteristics (position, shape, orientation, speed, etc.). This detection and identification is a key function for the safe operation of the autonomous vehicle 100.

[0089] The positioning system 121 can be used to estimate the geographical location of the vehicle 100. The IMU 122 can be used to sense changes in the position and orientation of the vehicle 100 based on inertial acceleration. In one embodiment, the IMU 122 can be a combination of an accelerometer and a gyroscope.

[0090] For example, radar 123 can use radio signals to sense objects in the surrounding environment of vehicle 100. In some embodiments, in addition to sensing objects, radar 123 can also be used to sense the speed and / or direction of travel of objects.

[0091] For example, the laser rangefinder 124 can use a laser to sense objects in the environment in which the vehicle 100 is located. In some embodiments, the laser rangefinder 124 may include one or more laser sources, a laser scanner, and one or more detectors, as well as other system components.

[0092] For example, camera 125 can be used to capture multiple images of the surrounding environment of vehicle 100. For example, camera 125 can be a still camera or a video camera.

[0093] For example, the vehicle speed sensor 126 can be used to measure the speed of the vehicle 100. For instance, the vehicle speed can be measured in real time. The measured vehicle speed can be transmitted to the control system 130 to control the vehicle.

[0094] The control system 130 controls the operation of the vehicle 100 and its components. The control system 130 may include various components, such as a steering system 131, a throttle 132, a braking unit 133, a computer vision system 134, a route control system 135, and an obstacle avoidance system 136.

[0095] For example, the steering system 131 can be operated to adjust the forward direction of the vehicle 100. For example, in one embodiment, it can be a steering wheel system. The throttle 132 can be used to control the operating speed of the engine 111 and thus the speed of the vehicle 100.

[0096] For example, braking unit 133 can be used to control the deceleration of vehicle 100; braking unit 133 can use friction to slow down wheel 114. In other embodiments, braking unit 133 can convert the kinetic energy of wheel 114 into electric current. Braking unit 133 can also take other forms to slow down the rotational speed of wheel 114 to control the speed of vehicle 100.

[0097] like Figure 1 As shown, the computer vision system 134 is operable to process and analyze images captured by the camera 125 to identify objects and / or features in the environment surrounding the vehicle 100. These objects and / or features may include traffic signals, road boundaries, and obstacles. The computer vision system 134 may use object recognition algorithms, structure-from-motion (SFM) algorithms, video tracking, and other computer vision techniques. In some embodiments, the computer vision system 134 may be used to map the environment, track objects, estimate object velocities, and so on.

[0098] For example, the route control system 135 can be used to determine the driving route of the vehicle 100. The obstacle avoidance system 136 can be used to identify, assess, and avoid or otherwise traverse potential obstacles in the environment of the vehicle 100.

[0099] In one instance, the control system 130 may include additional or alternative components besides those shown and described. Alternatively, some of the components shown above may be reduced.

[0100] like Figure 1 As shown, vehicle 100 can interact with external sensors, other vehicles, other computer systems, or users through peripheral device 140.

[0101] In some embodiments, peripheral device 140 may provide a means for vehicle 100 to interact with user interface 170. Wireless communication system 141 may communicate wirelessly with one or more devices directly or via a communication network.

[0102] Power source 160 can provide power to various components of vehicle 100.

[0103] Some or all of the functions of vehicle 100 may be controlled by computer system 150, wherein computer system 150 may include at least one processor 151, which executes instructions 153 stored in a non-transitory computer-readable medium, such as memory 152. Computer system 150 may also be multiple computing devices that control individual components or subsystems of vehicle 100 in a distributed manner.

[0104] For example, processor 151 can be any conventional processor, such as a commercially available central processing unit (CPU).

[0105] Alternatively, the processor can be a dedicated device such as an application-specific integrated circuit (ASIC) or other hardware-based processor. Although Figure 1 The illustrations functionally depict a processor, memory, and other components of a computer within the same block; however, those skilled in the art will understand that the processor, computer, or memory may actually include multiple processors, computers, or memories that may or may not be stored in the same physical enclosure. For example, memory may be a hard disk drive or other storage media located in an enclosure different from that of the computer. Therefore, references to a processor or computer will be understood to include references to a collection of processors or computers or memories that may or may not operate in parallel. Unlike using a single processor to perform the steps described herein, some components, such as steering and deceleration assemblies, may each have their own processor that performs calculations only relevant to the component's specific function.

[0106] In the various aspects described herein, the processor may be located remotely from the vehicle and communicate wirelessly with the vehicle. In other aspects, some of the processes described herein are executed on a processor located within the vehicle, while others are executed by a remote processor, including taking the necessary steps to perform a single operation.

[0107] In some embodiments, memory 152 may contain instructions 153 (e.g., program logic) that can be used by processor 151 to perform various functions of vehicle 100, including those described above. Memory 152 may also include additional instructions, such as instructions for sending data to, receiving data from, interacting with, and / or controlling one or more of the mobility system 110, sensing system 120, control system 130, and peripheral devices 140.

[0108] For example, in addition to instruction 153, memory 152 may also store data, such as road maps, route information, vehicle position, direction, speed, and other such vehicle data, as well as other information. This information can be used by vehicle 100 and computer system 150 during operation of vehicle 100 in autonomous, semi-autonomous, and / or manual modes.

[0109] User interface 170 can be used to provide information to or receive information from users of vehicle 100. Optionally, user interface 170 may include one or more input / output devices within a set of peripheral devices 140, such as wireless communication system 141, on-board computer 142, microphone 143, and speaker 144.

[0110] In embodiments of this application, computer system 150 can control the functions of vehicle 100 based on input received from various subsystems (e.g., mobility system 110, sensing system 120, and control system 130) and from user interface 170. For example, computer system 150 can utilize input from control system 130 to control braking unit 133 to avoid obstacles detected by sensing system 120 and obstacle avoidance system 136. In some embodiments, computer system 150 is operable to provide control over many aspects of vehicle 100 and its subsystems.

[0111] Alternatively, one or more of these components may be installed separately from or associated with vehicle 100. For example, memory 152 may exist partially or completely separately from vehicle 100. The components may be communicatively coupled together in a wired and / or wireless manner.

[0112] Optionally, the components described above are merely examples. In actual applications, components in each of the above modules may be added or removed as needed. Figure 1 This should not be construed as a limitation on the embodiments of this application.

[0113] Autonomous vehicles traveling on roads, such as vehicle 100 above, can identify objects in their surrounding environment to determine adjustments to their current speed. These objects can be other vehicles, traffic control equipment, or other types of objects. In some examples, each identified object can be considered independently, and based on the object's individual characteristics, such as its current speed, acceleration, and distance from the vehicle, the speed adjustment to be made by the autonomous vehicle can be determined.

[0114] Optionally, the autonomous vehicle 100 or the computing device associated with the autonomous vehicle 100 (such as...) Figure 1 The computer system 112, computer vision system 140, and data storage device 114 can predict the behavior of the identified objects based on the characteristics of the identified objects and the state of the surrounding environment (e.g., traffic, rain, ice on the road, etc.). Optionally, each identified object depends on the behavior of each other, so all identified objects can also be considered together to predict the behavior of a single identified object. The vehicle 100 can adjust its speed based on the predicted behavior of the identified objects. In other words, the autonomous vehicle can determine what steady state the vehicle will need to adjust to (e.g., accelerate, decelerate, or stop) based on the predicted behavior of the objects. In this process, other factors can also be considered in determining the speed of the vehicle 100, such as the lateral position of the vehicle 100 in the road, the curvature of the road, the proximity of static and dynamic objects, etc.

[0115] The 3D target detection model provided in this application embodiment can be deployed on, for example, Figure 1 In the vehicle 100 shown, objects in the surrounding environment are detected based on data collected by the sensing system 120. The method in this embodiment can improve the diversity of training samples, which is beneficial to improving the accuracy of the model, that is, improving the accuracy of the detection results, and thus improving driving safety.

[0116] The aforementioned vehicle 100 can be a traditional car, a new energy vehicle, or an intelligent car. A traditional car refers to a vehicle powered by fuels such as diesel fuel. A new energy vehicle refers to a newly emerging vehicle powered by new energy sources such as electricity or natural gas. An intelligent car refers to a vehicle equipped with intelligent control units and other intelligent devices. The vehicle types of the aforementioned vehicle 100 may include, for example, sedans, trucks, buses, engineering vehicles, and public transport buses. This application embodiment does not impose any particular limitation. In this application embodiment, various types of vehicles traveling on roads are mainly used as examples for description.

[0117] Figure 2 A schematic flowchart illustrating the vehicle control process in an autonomous driving system is shown. Specifically, the vehicle control process includes the following steps:

[0118] S1: Collect sensor data.

[0119] For example, step S1 can be performed by Figure 1 The sensor system 120 in the middle is executed.

[0120] Specifically, perception data includes data collected from the driving environment by various devices such as cameras and lidar.

[0121] For example, a lidar can acquire information returned from the external environment at a frequency of 10 fps.

[0122] For example, the camera can capture information about the external environment at a speed of 25fps or 30fps. For instance, the camera can... Figure 1 Camera 125.

[0123] S2: Perform target detection.

[0124] The 3D target detection model obtained in this embodiment can be used to perform step S2.

[0125] For example, the target to be detected refers to a target in the driving environment. The target to be detected can include dynamic targets and static targets. For example, dynamic targets include pedestrians or vehicles. Static targets include traffic signs, lane lines, or drivable areas (free space).

[0126] S3: Perform target tracking.

[0127] For example, step S3 can be performed by Figure 1 The computer vision system 134 in the middle is executed.

[0128] Target tracking can be performed based on the results of target detection.

[0129] For example, target tracking can be used to detect the speed of a target or predict the trajectory of a target.

[0130] S4: Integrates obstacle information from multiple sensors.

[0131] Each sensor has its own advantages in perception. Multi-sensor fusion allows each sensor to play its role, achieving a fusion result that is superior to the result of any single sensor. Autonomous driving systems typically use multiple sensors to redundantly detect 3D targets in the environment simultaneously, with each sensor obtaining its own detection result for the same target. This step involves fusing the target detection results from all sensors, removing duplicate results, and outputting the deduplicated final detection result.

[0132] S5: Planning and Control.

[0133] Path planning and vehicle driving status control are based on obstacle information.

[0134] The 3D target detection model in this application embodiment can be deployed on the computing node (such as an onboard computer) of the vehicle-mounted perception device to improve the accuracy of 3D target detection.

[0135] Since the embodiments of this application involve the application of neural networks, for ease of understanding, the relevant terms and concepts of neural networks that may be involved in the embodiments of this application will be introduced below.

[0136] (1) Neural Network

[0137] Neural networks can be composed of neural units, which can refer to units represented by x. s The arithmetic unit that takes an intercept of 1 as input can output the following:

[0138]

[0139] Where s = 1, 2, ..., n, n is a natural number greater than 1, W s For x s The weights are denoted by b, where b is the bias of the neural unit.

[0140] f represents the activation function of a neural network, used to introduce nonlinear characteristics and convert the input signal into the output signal. The output signal of this activation function can be used as the input to the next layer. For example, the activation function can be ReLU, tanh, or sigmoid.

[0141] A neural network is a network formed by connecting multiple individual neural units, meaning that the output of one neural unit can be the input of another. The input of each neural unit can be connected to the local receptive field of the previous layer to extract features from the local receptive field, which can be a region composed of several neural units.

[0142] (2) Deep Neural Networks

[0143] A deep neural network (DNN), also known as a multilayer neural network, can be understood as a neural network with multiple hidden layers. Based on the position of the layers, the internal neural network of a DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. The layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer.

[0144] Although DNNs seem complex, the operation of each layer is actually not complicated. Simply put, it involves the following linear relationship expression: in, It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is also relatively large. The definitions of these parameters in DNNs are as follows: Taking the coefficient W as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W is located, while the subscript corresponds to the third layer index 2 of the output and the second layer index 4 of the input.

[0145] In summary, the coefficient from the k-th neuron in layer L-1 to the j-th neuron in layer L is defined as...

[0146] It's important to note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by vectors from many layers).

[0147] (3) Convolutional Neural Network

[0148] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons in a CNN that performs convolutional processing on the input signal. In a convolutional layer of a CNN, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature planes, each composed of rectangularly arranged neural units. Neural units on the same feature plane share weights, which are called the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The convolutional kernel can be formalized as a matrix of random size, and during the training process of the CNN, the kernel can learn appropriate weights. Furthermore, the direct benefit of shared weights is reducing the connections between layers in the CNN, while also reducing the risk of overfitting.

[0149] (4) Loss Function

[0150] In training deep neural networks, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value and update the weight vector of each layer based on the difference. (Of course, there's usually a pre-configuration process before the first update, where parameters are pre-configured for each layer.) For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network can predict the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss. Generally, a smaller loss indicates higher training quality, while a larger loss indicates lower training quality. Similarly, smaller loss fluctuations result in more stable training, while larger loss fluctuations lead to less stable training.

[0151] (5) Backpropagation algorithm

[0152] Neural networks can employ backpropagation (BP) to correct the parameters of the neural network model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates error loss; this error loss information is then propagated back to update the parameters of the neural network model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the neural network model, such as the weight matrix.

[0153] For example, the loss value generated during each training iteration of a neural network model is passed layer by layer from back to front. At each layer, the update amount of that layer's parameters is calculated (partial derivative operation), and this update amount is related to the gradient. The value of the gradient and the value of the loss are linearly and positively correlated.

[0154] Single-modal 3D object detection refers to schemes that perform 3D object detection based on a single modality of data, such as image data or laser data. Multimodal 3D object detection refers to schemes that perform 3D object detection based on fused image data and laser data. Multimodal 3D object detection has higher accuracy than single-modal 3D object detection and is gradually becoming the mainstream detection scheme. Effective data augmentation methods can increase the diversity of training samples, thereby improving the performance of the trained model, such as improving the model's accuracy and robustness.

[0155] A series of effective data augmentation methods have been developed for unimodal 3D object detection. For example, point clouds and 3D bounding boxes of objects from other scenes are pasted into the current scene to increase the diversity of training samples and improve model accuracy. However, the training samples required for multimodal 3D object detection include data from multiple modalities, and these data must be consistent across modalities. Data augmentation methods used in unimodal 3D object detection cannot be applied to the training process of multimodal 3D object detection. Existing data augmentation methods used in the training process of multimodal 3D object detection are limited to operations such as rotation, translation, and scaling, which cannot effectively increase the number of training samples and thus cannot improve model performance.

[0156] This application proposes a data augmentation method that performs modal synchronization resampling on image frames and point cloud frames. The resampled point cloud frames (i.e., intermediate point cloud frames in the following text) and image frames (i.e., intermediate image frames in the following text) are processed separately according to the occlusion relationship between objects, which ensures the consistency of the projection relationship between point cloud frames and image frames, can improve the diversity of training samples, and is conducive to improving the performance of the trained 3D object detection model.

[0157] To better describe the methods of the embodiments of this application, the data enhancement apparatus of the embodiments of this application will be described below with reference to the accompanying drawings.

[0158] Figure 3 An embodiment of this application is shown, such as a data enhancement apparatus. Figure 3 As shown, the device 300 includes a target region extraction module 310, a sampling module 320, a point cloud addition module 330, a point cloud filtering module 340, an image block processing module 350, and an image block addition module 360.

[0159] The target region extraction module 310, the point cloud addition module 330, and the point cloud filtering module 340 belong to the point cloud processing unit. The image block processing module 350 and the image block addition module 360 ​​belong to the image processing unit.

[0160] The target region extraction module 310 is used to extract the target region from the initial point cloud frame. The target region extraction module 310 is an optional module.

[0161] For example, the target region in the initial point cloud frame can be a planar region.

[0162] For example, the target region extraction module 310 can extract planar regions from the initial point cloud frame by fitting the point cloud plane.

[0163] Optionally, the target area can be a road surface area.

[0164] The sampling module 320 is used to sample point clouds and image patches of at least one sample from the sampling pool. The sampling pool includes point clouds and image patches of multiple candidate samples. These multiple candidate sample point clouds and image patches can be point clouds and image patches of labeled samples extracted from paired point cloud frames and image frames using the ground truth values ​​of labeled samples.

[0165] In this context, at least one sample is used to indicate at least one object.

[0166] Sampling module 320 is optional. Point clouds and image patches of the samples can also be obtained in other ways, as described in step S420 below.

[0167] The point cloud adding module 330 is used to add the point cloud of at least one sample to the initial point cloud frame to obtain an intermediate point cloud frame.

[0168] When the device 300 includes a target region extraction module 310, the point cloud addition module 330 can add the point cloud of the at least one sample to the target region in the initial point cloud frame.

[0169] Optionally, the region in which the point cloud of the at least one sample is located in the initial point cloud frame can be indicated by the 3D bounding box of the at least one sample. The region in which the at least one sample is located in the initial point cloud frame is the target position of the point cloud of the at least one sample in the initial point cloud frame, and the point cloud of the sample is the point cloud that falls within the 3D bounding box of the sample. See step S420 below for a detailed description.

[0170] The point cloud filtering module 340 is used to filter points in the intermediate point cloud frame that are occluded by objects in the intermediate point cloud frame, to obtain the target point cloud frame. Alternatively, the point cloud filtering module 340 is used to filter points occluded by objects in the intermediate point cloud frame based on the occlusion relationships between objects in the intermediate point cloud frame. See step S430 below for a detailed description.

[0171] The image block processing module 350 is used to add the image block of the at least one sample to the initial image frame.

[0172] Optionally, the region where the image patch of the at least one sample is located in the initial image frame can be obtained by projecting the point cloud of the at least one sample into the region where the point cloud is located in the initial point cloud frame. The region where the image patch of the at least one sample is located in the initial image frame is the target position of the image patch of the at least one sample in the initial image frame.

[0173] Specifically, based on the camera parameters of the initial image frame, the area where the point cloud of the at least one sample is located in the initial point cloud frame is projected into the initial image frame to obtain the area where the image block of the at least one sample is located in the initial image frame.

[0174] In this case, the image block processing module 350 can further be used to process at least one sample image block to obtain a processed image block of at least one sample, and to translate the processed image block of at least one sample to the area where the image block of at least one sample is located in the initial image frame.

[0175] For example, the image block processing module 350 performs scaling processing on the image block of at least one sample to obtain an image block of the same size as the area where the image block of the at least one sample is located in the initial image frame, and performs translation processing on the image block obtained after scaling processing to translate the image block of the at least one sample after scaling processing to the area where the image block of the at least one sample is located in the initial image frame.

[0176] For a detailed description, please refer to step S420 below.

[0177] The image patch adjustment module 360 ​​is used to adjust the coverage order of image patches of objects in the intermediate image frame according to the occlusion relationship between objects in the intermediate point cloud frame, so as to obtain the target image frame.

[0178] For a detailed description, please refer to step S440 below.

[0179] The scheme in this embodiment increases the diversity of training samples and improves the performance of the trained model, i.e., improves detection accuracy and reduces false detection rate. Simultaneously, by filtering occluded point clouds based on the occlusion relationships between objects in intermediate point cloud frames, the rationality of the target point cloud frame is ensured. Furthermore, by adjusting the coverage order of image patches of objects based on the occlusion relationships in intermediate point cloud frames, the rationality of the target image frame is further guaranteed, while also ensuring consistency between the target point cloud frame and the target image frame.

[0180] The following is combined with Figures 4 to 8 The data enhancement method of the embodiments of this application will be described.

[0181] Figure 4 This application illustrates a data augmentation method 400 provided in an embodiment of the present application. The method of this embodiment can be applied to various computing devices, which can be terminal devices or servers. Terminal devices may include, but are not limited to, vehicles, personal computers, cameras, server computers, handheld or laptop devices, mobile devices, mobile robots, or drones. Mobile devices may include mobile phones, tablets, personal digital assistants, or media players.

[0182] In one possible implementation, the data augmentation method of this application embodiment is executed by the terminal device to increase the training samples. In another possible implementation, the terminal device can send the data to other devices, such as a cloud server or other terminal devices, and the other devices can execute the data augmentation method of this application embodiment. Furthermore, the other devices can send the processed results to the terminal device. This application embodiment only uses the application of method 400 in the field of autonomous driving as an example to illustrate method 400, and does not limit the application field of the solution in this application embodiment.

[0183] The training samples obtained through method 400 can be used to train a multimodal 3D object detection model.

[0184] For example, method 400 can be derived from... Figure 3 The device 300 is executed.

[0185] Method 400 includes steps S410 to S440, which are described below.

[0186] S410, acquire the initial point cloud frame and the initial image frame.

[0187] There is a correspondence between the initial point cloud frame and the initial image frame.

[0188] In this embodiment of the application, the image frames and point cloud frames with a corresponding relationship can be understood as paired image frames and point cloud frames.

[0189] The objects in the initial point cloud frame are the same as those in the initial image frame.

[0190] For example, the initial point cloud frame and the initial image frame can be obtained at the same time or during the same period based on the same scene.

[0191] For example, the initial point cloud frame can be data acquired by LiDAR.

[0192] For example, the initial image frame can be data captured by a camera. For instance, the initial image frame can be an RGB image.

[0193] Alternatively, the initial point cloud frame and the initial image frame can also be obtained after processing by method 400. That is, the target point cloud frame and the target image frame obtained after processing by method 400 are used again as the initial point cloud frame and the initial image frame, and method 400 is executed again.

[0194] S420, add the point cloud of at least one sample to the initial point cloud frame to obtain an intermediate point cloud frame; add the image patch of at least one sample to the initial image frame to obtain an intermediate image frame. The at least one sample is used to indicate at least one object.

[0195] For example, step S420 can be performed by Figure 3 The point cloud adding module 330 and the image patch processing module 350 are executed. Specifically, the point cloud adding module 330 adds the point cloud of at least one sample to the initial point cloud frame to obtain an intermediate point cloud frame. The image patch processing module 350 adds the image patch of at least one sample to the initial image frame to obtain an intermediate image frame.

[0196] For example, a sample can be used to indicate an object, the point cloud of the sample is the point cloud of the object indicated by the sample, that is, the point cloud of the sample can represent the set of points on the surface of the object, and the image patch of the sample is the image patch of the object indicated by the sample, that is, the image patch of the sample refers to the image of the object included in the picture.

[0197] Step S420 can be understood as adding the point cloud of the object indicated by at least one sample to the initial point cloud frame to obtain an intermediate point cloud frame; and adding the image block of the object indicated by at least one sample to the initial image frame to obtain an intermediate image frame.

[0198] For example, the at least one sample includes, Figure 6 The sample shown is used to indicate a cyclist. Adding the point cloud of this cyclist to the initial point cloud frame yields the following result: Figure 6 The intermediate point cloud frame is shown in (a). Adding the image patch of the cyclist to the initial image frame yields the result shown in (a). Figure 6 The intermediate image frame shown in (e) is an example.

[0199] Optionally, the point cloud of at least one sample and the image patch of at least one sample can be the point cloud and image patch of the labeled sample extracted from at least one point cloud frame and at least one image frame, respectively. Each image frame in the at least one image frame and each point cloud frame in the at least one point cloud frame are in one-to-one correspondence.

[0200] In other words, the at least one image frame and the at least one point cloud frame include at least one set of paired image frames and point cloud frames. The point cloud and image patch of a sample are extracted from a set of paired image frames and point cloud frames.

[0201] The point cloud and image patch of the at least one sample can be extracted from a set of paired image frames and point cloud frames, or it can be extracted from multiple sets of paired image frames and point cloud frames respectively.

[0202] For example, paired image frames and point cloud frames can be obtained based on image frames and point cloud frames acquired at the same time or during the same period based on the same scene.

[0203] The paired image frame and point cloud frame contain labeled samples. A labeled sample refers to a sample with a ground truth label. The image patch and point cloud containing the labeled samples can be referred to as a 2D image-3D point cloud pair. The ground truth label can also be called a label.

[0204] For example, the labeled truth value can be the pre-labeled real data of the sample.

[0205] The labeled samples on a point cloud frame are used to indicate objects within that frame. The ground truth of each labeled sample includes its pre-labeled 3D bounding box, which is the 3D bounding box corresponding to the region where the object indicated by the sample is located. In other words, the 3D bounding box of the labeled sample is the 3D bounding box used to indicate the region of the point cloud containing the object. The point cloud containing the labeled sample is the point cloud that falls within the 3D bounding box of the labeled sample. The 3D bounding box of the labeled sample can be understood as the bounding box of the point cloud containing the labeled sample.

[0206] For example, the 3D bounding box of the labeled sample can be manually labeled.

[0207] Alternatively, the point cloud of a sample can be the point cloud of the labeled sample extracted from the point cloud frames in the paired point cloud frames and image frames using the ground truth values ​​of the labeled sample. That is, the point cloud of a sample can be the point cloud within the 3D bounding box of the labeled sample extracted from the point cloud frames in the paired point cloud frames and image frames using the 3D bounding box of the labeled sample.

[0208] For example, adding the point cloud of a sample to an initial point cloud frame may include adding the point cloud of the sample and its label to the initial point cloud frame.

[0209] For example, add the 3D bounding box of the sample and the point cloud falling within the 3D bounding box to the initial point cloud frame.

[0210] Annotated samples on an image frame are used to indicate objects within that frame. The ground truth of annotated samples includes the pre-annotated 2D bounding box of the sample, i.e., the 2D bounding box corresponding to the region where the object indicated by the sample is located. Alternatively, the 2D bounding box of the sample is the 2D bounding box indicating the region where the image patch containing the object is located. The image patch of the sample is the image patch that falls within the 2D bounding box of the sample. The 2D bounding box of a sample can be understood as the bounding box of the image patch of the sample.

[0211] For example, the 2D bounding box of the labeled sample can be manually labeled.

[0212] For example, the 2D bounding box of the labeled sample can be obtained by projecting the 3D bounding box corresponding to the area where the labeled object is located on the point cloud frame onto the image frame.

[0213] An image patch for a labeled sample can be an image patch within the region of the 2D bounding box of that labeled sample in an image frame. For example, an image patch obtained by cropping the 2D bounding box of the labeled sample from the image frame is an image patch for that labeled sample.

[0214] In other words, the image patch of a sample can be the image patch of the labeled sample extracted from the image frame of the paired point cloud frame and image frame using the ground truth of the labeled sample. That is, the image patch of a sample can be the image patch within the 2D bounding box of the labeled sample extracted from the image frame of the paired point cloud frame and image frame using the 2D bounding box of the labeled sample.

[0215] For example, adding an image patch of a sample to an initial image frame may include adding the image patch of the sample and its label to the initial image frame.

[0216] For example, the 2D bounding box of the sample and the image patch falling within the 2D bounding box are added to the initial image frame.

[0217] For example, a labeled sample is used to indicate car 1#. The 3D bounding box corresponding to the area where car 1# is located in the point cloud frame can be manually labeled, and the point cloud of car 1# is the point cloud falling within the 3D bounding box. The 2D bounding box corresponding to the area where car 1# is located in the image frame can be obtained by projecting the 3D bounding box corresponding to the area where car 1# is located, and the image patch of car 1# is the image area falling within the 2D bounding box.

[0218] Furthermore, the point cloud and image patch of the at least one sample can be obtained by sampling in a sampling pool. The sampling pool includes point clouds and image patches of multiple candidate samples.

[0219] For example, this step can be performed by Figure 3 The sampling module 320 in the middle is executed.

[0220] The point clouds and image patches of these multiple candidate samples can be extracted from the point clouds and image patches of the labeled samples in paired point cloud frames and image frames using the ground truth values ​​of the labeled samples. Specific extraction methods can be found in the description above and will not be repeated here.

[0221] For example, adding the point cloud of at least one sample to an initial point cloud frame may include pasting the point cloud of at least one sample to the initial point cloud frame.

[0222] Furthermore, point clouds of different samples from at least one sample are added to different regions of the initial point cloud frame.

[0223] For example, the point clouds of different samples from at least one sample are pasted into different regions of the initial point cloud frame.

[0224] Since the intermediate point cloud frame is obtained by adding the point cloud of at least one sample to the initial point cloud frame, the area where the point cloud of at least one sample is added in the initial point cloud frame is the area of ​​the point cloud of at least one sample in the intermediate point cloud frame. That is, "the area where the point cloud of at least one sample is located in the initial point cloud frame" and "the area where the point cloud of at least one sample is located in the intermediate point cloud frame" can be understood as the same area.

[0225] In other words, the point clouds of the different samples do not overlap in the intermediate point cloud frame. For example, the 3D bounding boxes of the different samples do not overlap in the intermediate point cloud frame.

[0226] Optionally, there is no overlap between the point cloud region of at least one sample in the intermediate point cloud frame and the point cloud region of the object in the initial point cloud frame.

[0227] The region containing the point cloud of a sample can be indicated by the sample's 3D bounding box. The region containing the point cloud of an object in the initial point cloud frame can be indicated by a label on the initial point cloud frame. The label on the initial point cloud frame includes a 3D bounding box used to indicate the region containing the object in the initial point cloud frame.

[0228] Optionally, the point cloud of the at least one sample does not overlap with the area in the initial point cloud and the area in the point cloud of the object in the initial point cloud, including: in a bird's-eye view (BEV), the 3D bounding box of the at least one sample and the 3D bounding box of the area in the object in the initial point cloud frame do not overlap in the intermediate point cloud frame.

[0229] In the actual point cloud frames, there will be no overlapping of multiple objects. In the solution of this application embodiment, there is no overlapping area between the point cloud of the sample and the point cloud of the object in the initial point cloud frame. This can avoid the unreasonable situation of multiple objects colliding in three-dimensional space in the intermediate point cloud frame, thus ensuring the rationality of the intermediate point cloud frame. In turn, it ensures the rationality of the target point cloud frame obtained based on the intermediate point cloud frame. Using the target point cloud frame as a training sample can improve the performance of the trained 3D object detection model.

[0230] In other words, the point cloud of at least one sample can be added to any position in the initial point cloud frame, as long as there is no overlapping area between the point cloud of at least one sample and the point cloud of the object in the initial point cloud frame.

[0231] Optionally, the region in which the point cloud of at least one sample is located in the intermediate point cloud frame is the same as the region in which the point cloud of at least one sample is located in the point cloud frame of the original scene, where the point cloud frame of the original scene is a point cloud frame acquired based on the scene where the at least one object indicated by the at least one sample is located. As mentioned above, the point cloud and image patch of a sample are extracted from paired point cloud frames and image frames. The paired point cloud frames and image frames are point cloud frames and image frames acquired based on the original scene. The region in the point cloud frame of the original scene where the point cloud of the at least one sample is located can also be understood as the region in the point cloud frame of the paired point cloud frames and image frames where the point cloud of the at least one sample is located.

[0232] As mentioned above, the solution of this application embodiment can be applied to the field of autonomous driving. In this case, the area of ​​the object / sample in the point cloud frame in this application embodiment can be the area of ​​the object / sample in the vehicle coordinate system under different scenarios, or the area of ​​the object / sample relative to the vehicle under different scenarios.

[0233] For example, if the sample is car 1#, and car 1# is located to the left and in front of itself when it is captured, then the area of ​​car 1#'s point cloud in the original scene's point cloud frame is the area to the left and in front of car 1#. The position where car 1#'s point cloud is added to the initial point cloud frame is the area to the left and in front of car 2#. Car 1# and car 2# can be the same car or different cars.

[0234] This means adding the point cloud of the car located to the left front of car #1 in one point cloud frame to the position located to the left front of car #2 in the initial point cloud frame.

[0235] Specifically, the point cloud of the at least one sample is added to the area indicated by the coordinates of the 3D bounding box corresponding to the at least one sample in the initial point cloud frame.

[0236] The coordinates of the 3D bounding box of the sample can be the coordinates in the vehicle coordinate system at the time the sample was acquired. By using the coordinates of the 3D bounding box corresponding to the sample as the coordinates in the vehicle coordinate system at the time the initial point cloud frame was acquired, the region of the sample's point cloud in the initial point cloud frame can be obtained.

[0237] Alternatively, the point cloud of the sample can be directly pasted into the area indicated by the coordinates of the 3D bounding box of the sample in the initial point cloud frame.

[0238] In the scheme of this application embodiment, the region where the point cloud of at least one sample is located in the intermediate point cloud frame is the same as the region where the point cloud of at least one sample is located in the point cloud frame of the original scene. This satisfies the sparsity of the point cloud in the intermediate point cloud frame, avoids discrepancies with reality, and further improves the rationality of the intermediate point cloud frame. This, in turn, ensures the rationality of the target point cloud frame obtained based on the intermediate point cloud frame. Using the target point cloud frame as a training sample can improve the performance of the trained 3D object detection model. For example, objects at different distances from the vehicle have different sparsity in the point cloud frame. Objects closer to the vehicle have denser point clouds, while objects farther away have sparser point clouds. The region where the sample's point cloud is located in the intermediate point cloud frame is the same as the region where the sample's point cloud is located in the point cloud frame of the original scene, ensuring the consistency of the distance between the object indicated by the sample and the vehicle, thereby guaranteeing the sparsity of the point cloud in the intermediate point cloud frame. For example, the sparsity of the point cloud of an object located in different directions from the vehicle is different. In the point cloud frame, the right side of the object located in front of the vehicle to the left is denser, while the left side of the object located in front of the vehicle to the right is denser. The area where the sample's point cloud is located in the intermediate point cloud frame is the same as the area where the sample's point cloud is located in the point cloud frame of the original scene. This can ensure the consistency of the orientation between the object indicated by the sample and the vehicle, thereby ensuring the sparsity of the point cloud in the intermediate point cloud frame.

[0239] Furthermore, the point cloud of at least one sample is added to the target region in the initial point cloud frame.

[0240] For example, target region extraction can be performed by Figure 3 The target region extraction module 310 is executed.

[0241] For example, the target region in the initial point cloud frame can be a planar region.

[0242] For example, the target region extraction module 310 can extract planar regions from the initial point cloud frame by fitting the point cloud plane.

[0243] For example, method 400 can be applied to the field of autonomous driving, and the target area can be a road surface area. Accordingly, at least one frame sample is a sample on the road surface area. For example, the object indicated by the at least one frame sample can include dynamic objects, such as pedestrians or vehicles on the road surface.

[0244] For example, the road surface area can be extracted from the initial point cloud frame by fitting the point cloud plane.

[0245] In the field of autonomous driving, more attention is usually paid to objects in the road surface area. Adding the point cloud of the sample to the road surface area can increase the number of objects in the road surface area, while ensuring the rationality of the intermediate point cloud frame. This, in turn, ensures the rationality of the target point cloud frame obtained based on the intermediate point cloud frame. Using the target point cloud frame as a training sample can improve the performance of the trained 3D object detection model.

[0246] The following example illustrates the process of adding point clouds of at least one sample to an initial point cloud frame. It should be understood that the following is merely an example and does not constitute a limitation on the scope of the embodiments of this application.

[0247] S11, multiple sample point clouds and image patches are obtained by sampling multiple candidate samples in the sampling pool, and these multiple samples are samples on the road surface area.

[0248] S12, add the point clouds of the multiple samples to the road surface area in the initial point cloud frame. The positions of the point clouds of the multiple samples in the original scene point cloud frame are the same as the positions of the point clouds of the multiple samples in the initial point cloud frame.

[0249] S13, delete samples from multiple samples that overlap with the area where the point cloud of the object in the initial point cloud is located, and the remaining sample among the multiple samples is the at least one sample.

[0250] For example, the region in which the image patch of the at least one sample is located in the initial image frame is the same as the region in which the image patch of the at least one sample is located in the original scene image frame. The original scene image frame refers to an image frame acquired based on the scene in which the object indicated by the at least one sample is located. As previously described, the point cloud and image patch of a sample are extracted from paired point cloud frames and image frames. The paired point cloud frames and image frames are point cloud frames and image frames acquired based on the original scene. The region in which the image patch of the at least one sample is located in the original scene image frame is the region in which the image patch of the at least one sample is located in the image frame of the paired point cloud frames and image frames.

[0251] Specifically, an image patch of the at least one sample is added to the area indicated by the coordinates of the 2D bounding box of the at least one sample in the initial image frame.

[0252] The coordinates of the 2D bounding box corresponding to the sample can be the coordinates in the vehicle coordinate system at the time the sample was acquired. By using the coordinates of the 2D bounding box corresponding to the sample as the coordinates in the vehicle coordinate system at the time the initial image frame was acquired, the region where the image patch of the sample is located in the initial image frame can be obtained.

[0253] Alternatively, the image patch of the sample can be directly pasted into the area indicated by the coordinates of the 2D bounding box of the sample in the initial image frame.

[0254] For example, the image patch of the first sample in at least one sample is located in a first region in the initial image frame, which can be the region indicated by the coordinates of the 2D bounding box of the first sample in the initial image frame.

[0255] Due to camera extrinsic fluctuations, the position of the point cloud obtained by the above method in the initial point cloud frame may not completely correspond to the position of the image patch of the sample in the initial image frame. In other words, due to camera extrinsic fluctuations, the 3D bounding box corresponding to the sample in the initial point cloud frame and the 2D bounding box corresponding to the sample in the initial image frame do not completely correspond, which will affect the training effect of the 3D object detection model.

[0256] Optionally, the region in which the image patch of the at least one sample is located in the intermediate image frame is obtained by projecting the region in which the point cloud of the at least one sample is located in the intermediate point cloud frame onto the initial image frame.

[0257] For example, the point cloud of the first sample in the at least one sample is added to the first region in the initial point cloud frame, the image patch of the first sample is added to the first region in the initial image frame, and the first region in the initial image frame is obtained by projecting the first region in the initial point cloud frame onto the initial image frame.

[0258] In other words, based on the camera parameters and projection relationships of the initial image frame, the 3D bounding box corresponding to the sample in the initial point cloud frame is projected onto the initial image frame, and the resulting region is the area where the sample's image patch is located in the initial image frame. This is equivalent to obtaining a new 2D bounding box for the sample's image patch in the initial image frame.

[0259] The size of the region obtained by projection transformation may not match the original size of the sample image patch. The sample image patch can be preprocessed, and then the preprocessed sample image patch can be added to the corresponding region.

[0260] Specifically, adding at least one sample image block to the initial image frame includes: adding at least one sample image block to the initial image frame, processing the at least one sample image block, and translating the processed at least one sample image block to the area where the at least one sample image block is located in the initial image frame.

[0261] For example, this step can be performed by Figure 3 The image block processing module 350 in the middle is executed.

[0262] For example, the point cloud of the first sample in the at least one sample is added to the first region in the initial point cloud frame, the first region is projected into the initial image frame to obtain the first region in the initial image frame, the image block of the first sample is scaled to obtain an image block of the same size as the first region in the initial image frame, and the scaled image block is translated to the first region in the initial image frame.

[0263] In this embodiment, the position of the sample image patch in the initial image frame is obtained based on the position of the sample point cloud in the initial point cloud frame. This avoids the influence of camera extrinsic jitter and ensures the consistency between the region of the sample point cloud in the initial point cloud frame and the region of the sample image patch in the initial image frame, thereby improving the accuracy of the training samples and ensuring the training effect of the 3D object detection model.

[0264] Figure 5 The image shows the road surface area in the initial point cloud frame and the point cloud of the added samples. Figure 5 (a) shows a schematic diagram of the initial image frame and the initial point cloud frame. Figure 5 (b) shows the point cloud of the road surface region extracted from the initial point cloud frame. Figure 5 (c) shows a point cloud with 7 samples added to the initial point cloud frame, and an image patch with 7 samples added to the initial image frame. Figure 5 As shown in (c), the point clouds of the seven added samples are all located in the road surface region of the initial point cloud frame, and there is no overlap between the regions where the point clouds of the seven added samples are located and the regions where the point clouds of the objects in the initial point cloud frame are located. It should be understood that... Figure 5 The number of additional samples in (c) is for illustrative purposes only; more or fewer samples may be added.

[0265] S430: Filter out points in the intermediate point cloud frame that are occluded by objects in the intermediate point cloud frame to obtain the target point cloud frame.

[0266] Alternatively, based on the occlusion relationship between objects in the intermediate point cloud frame, the occluded points are filtered out to obtain the target point cloud frame.

[0267] For example, step S430 can be performed by Figure 3 The point cloud filtering module 340 in the middle is executed.

[0268] The occlusion relationship between objects in the intermediate point cloud frame can also be understood as the occlusion relationship between the point cloud of at least one sample and the point cloud in the initial point cloud frame.

[0269] The occlusion relationship between objects in a midpoint cloud frame refers to the occlusion relationship between objects determined from the observation point. The observation point can also be understood as the origin in the midpoint cloud frame, or in other words, the shooting point or acquisition point.

[0270] For example, the solution of this application embodiment can be applied to the field of autonomous driving. In this case, the point cloud frame and the image frame can be collected on the vehicle, the observation point is the vehicle position, and the origin of the point cloud frame is the origin of the vehicle coordinate system.

[0271] For example, Figure 15 The diagram illustrates the vehicle's driving process. The vehicle is equipped with a LiDAR 11 and a camera 12. The LiDAR 11 is used to acquire point cloud frames, and the coordinates of the points in the point cloud frames are the coordinates in the vehicle's coordinate system. The camera 12 is used to acquire image frames. It should be understood that... Figure 15 The positions of the lidar 11 and camera 12 shown are for illustrative purposes only. In practical applications, the lidar and camera can also be placed in other locations on the vehicle.

[0272] exist Figure 15 At the time shown, the point cloud frame acquired by LiDAR 11 includes point clouds of objects 21, 22, and 23, while the image frame acquired by camera 12 includes image blocks of objects 21, 22, and 23. The origin of the point cloud frame acquired by LiDAR 11 is the vehicle's position, meaning that... Figure 15 The observation point of the point cloud frame acquired by lidar 11 at the time shown is... Figure 15 The position of the vehicle at the indicated moment. From this observation point, part of object 23 is obscured by object 21.

[0273] The objects in the intermediate point cloud frame may include the object indicated by at least one sample as well as the objects in the initial point cloud frame.

[0274] For example, a 3D box in the intermediate point cloud frame can correspond to an object.

[0275] After adding at least one sample image patch to the initial image frame, there may be overlap between the image patches of the object. If the sample image patch is directly pasted into the initial image frame, it may cause inconsistencies between the image and the point cloud.

[0276] For example, such as Figure 6 As shown, the sample is used to indicate a cyclist; the cyclist's point cloud is pasted onto an initial point cloud frame, which includes the car's point cloud. In... Figure 6 In the intermediate point cloud frame shown in (a), from the observer's perspective, i.e., from the perspective of the car as the origin, the point cloud of the cyclist is added behind the point cloud of the car, meaning the cyclist is partially occluded by the car. For example... Figure 6 As shown in (d), the image patch of the cyclist was added to Figure 6 Within the dashed box of the initial image frame shown in (d), we obtain the following: Figure 6 In the intermediate image frame shown in (e), there is an overlap between the image patch of the cyclist and the image patch of the car. Points in the point cloud are projected onto... Figure 6 If a point falls within the overlapping area of ​​the image frame shown in (e), meaning the point corresponds to a pixel in the overlapping area, then the 3D point may capture incorrect image information, indicating an inconsistency between the point cloud frame and the image frame. Specifically, this inconsistency includes inconsistencies between foreground and background points.

[0277] A foreground point refers to a point located within a 3D bounding box. If a foreground point in a point cloud is projected onto an image frame and falls within the overlapping area of ​​the image frame—that is, if the foreground point corresponds to a pixel in the overlapping area—then the foreground point may capture incorrect image information. For example, some foreground points in a car's point cloud might not correspond to pixels in the overlapping area when projected onto a bounding box. Figure 6 In the image shown in (e), it may fall into the image patch of a cyclist, incorrectly capturing the image information of the cyclist. In other words, there is an incorrect correspondence between the points in the point cloud and the pixels in the image.

[0278] Background points refer to points that are not located within a 3D bounding box. If a background point in a point cloud is projected onto an image frame and falls within the overlapping area of ​​the image frame, that is, if a background point corresponds to a pixel in the overlapping area, then the background point may capture incorrect image information. Figure 6 If some background points in (a) are projected onto, as shown Figure 6 In the image shown in (e), it is possible to fall within the image patch of a bicycle, incorrectly capturing the image information of a cyclist. In other words, there is an incorrect correspondence between the points in the point cloud and the pixels in the image.

[0279] Specifically, the occlusion relationship between objects can be determined based on the objects' distance information. This distance information indicates the distance between the object and the origin.

[0280] From the observer's perspective, objects closer to the origin can block objects farther from the origin.

[0281] Optionally, points occluded by objects in the intermediate point cloud frame include: points located within the viewing range of objects in the intermediate point cloud frame, and whose distance from the observation point in the intermediate point cloud frame is greater than the distance between the object in the intermediate point cloud frame and the observation point.

[0282] For example, the objects in the intermediate point cloud frame are processed sequentially from near to far. During the processing of the objects in the intermediate point cloud frame, points in the intermediate point cloud frame that are occluded by the objects in the intermediate point cloud frame are filtered out.

[0283] The order from near to far refers to the order in which the distance from the origin increases.

[0284] In other words, processing objects in the intermediate point cloud frame in order of proximity means processing the points occluded by the object closest to the origin first, and processing the points occluded by the object farthest from the origin last. This improves processing efficiency.

[0285] The following section uses an object (the first object) in the intermediate point cloud frame as an example to illustrate step S430. The processing method for other objects in the intermediate point cloud frame is the same as that for this object.

[0286] If the object being processed (the first object) is an object in the initial point cloud frame, then only the points in the point cloud of at least one sample added to the initial point cloud frame need to be processed. If the object being processed is an object indicated by at least one sample, then the points in the intermediate point cloud frame need to be processed, that is, both the background points within the object's field of view and the foreground points occluded by the object need to be filtered.

[0287] Optionally, if the first object is an object in the initial point cloud frame, filter the first 3D point in the intermediate point cloud frame (an example of a point occluded by an object in the intermediate point cloud frame).

[0288] The first 3D point is a point in the point cloud of at least one sample, and the first 3D point is located within the view range of the first object, and the distance between the first 3D point and the origin is greater than the distance between the first object and the origin.

[0289] Optionally, if the first object is the object indicated by at least one sample, the second and third 3D points in the intermediate point cloud frame (an example of points occluded by objects in the intermediate point cloud frame) are filtered out.

[0290] The second 3D point is a background point, meaning the second 3D point is not within any 3D frame; and the second 3D point is within the viewpoint of the first object.

[0291] The third 3D point is the foreground point, meaning it is within the 3D frame; and the third 3D point is within the field of view of the first object; the distance between the third 3D point and the origin is greater than the distance between the first object and the origin; and the third 3D point does not belong to the first object.

[0292] The field of view of an object can be represented in a variety of ways.

[0293] For example, the viewing range of an object can be represented by coordinates in a spherical coordinate system.

[0294] The coordinates of 3D points in a point cloud are usually in a Cartesian coordinate system. We need to convert the coordinates in the Cartesian coordinate system to the spherical coordinate system.

[0295] The coordinates of a point p in a point cloud in a rectangular coordinate system are (x, y, z). Converting this to a spherical coordinate system gives...

[0296] Where r represents the distance from point p to the origin, and θ represents the angle between the line connecting the origin and point p and the z-axis. This represents the angle between the projection of the line connecting the origin and point p onto the plane formed by the x and y axes and the positive x-axis.

[0297] The coordinates in a spherical coordinate system satisfy the following formula:

[0298] θ = arccos(z / r),

[0299] The viewing range of an object can be determined by the coordinates of θ and φ in the 3D bounding box corresponding to the object. The range of angles is represented.

[0300] The set of coordinates of the eight vertices of the 3D bounding box corresponding to the object can be represented as: The viewing angle range V of this object can be expressed as:

[0301]

[0302] θ min =min(θ) k ), k = 1, 2, ..., 8;

[0303] θ max =max(θ) k ), k = 1, 2, ..., 8;

[0304]

[0305]

[0306] For any point in the point cloud frame If θ q ∈[θ min ,θ max ],and Then point q is located within the object's field of view.

[0307] The following is combined with Figure 7 The processing flow of step S430 is illustrated with an example. It should be understood that... Figure 7 The execution order of the steps in the processing flow is for illustrative purposes only and does not constitute a limitation on the solution in the embodiments of this application.

[0308] Figure 7 The processing flow specifically includes steps S701 to S711. Steps S701 and S711 are described below.

[0309] S701, extract N objects from the intermediate point cloud frame, where N is an integer greater than or equal to 1.

[0310] Extracting N objects from a midpoint cloud frame can be understood as extracting the 3D bounding boxes corresponding to the N objects in the midpoint cloud frame.

[0311] N objects can be represented as object i, where the value of i ranges from [1, N] and i is an integer.

[0312] S702, sort the N objects.

[0313] For example, the N objects are sorted in ascending order of their distance from the origin. That is, the N objects are sorted from closest to furthest. The sorted N objects can be represented as object i, where i ranges from [1, N] and is an integer. The smaller the value of i, the smaller the distance between object i and the origin. For example, when N is an integer greater than 1, object 1 has the smallest distance from the origin, and object N has the largest distance.

[0314] Processing points occluded by objects in intermediate point cloud frames sequentially from near to far can improve processing efficiency.

[0315] It should be understood that step S702 is an optional step.

[0316] S703, calculate the viewing range of object i.

[0317] Object i is the object currently being processed.

[0318] S704, filtering point clouds.

[0319] Specifically, points within the view range of object i are selected from the intermediate point cloud frame. These points are the points to be filtered.

[0320] S705, determine whether object i is the object indicated by the sample.

[0321] In other words, determine whether object i is an object in the initial point cloud frame.

[0322] If object i is the object indicated by the sample, proceed to step S706; if object i is not the object indicated by the sample, proceed to step S711.

[0323] Alternatively, if object i is not an object in the initial point cloud frame, proceed to step S706; if object i is an object in the initial point cloud frame, proceed to step S711.

[0324] S706, determine whether the point to be filtered is a background point.

[0325] In other words, it determines whether the point to be filtered is a foreground point.

[0326] If the point to be filtered is a background point, proceed to step S707; if the point to be filtered is not a background point, proceed to step S709.

[0327] Alternatively, if the point to be filtered is not a foreground point, proceed to step S707; if the point to be filtered is a foreground point, proceed to step S709.

[0328] S707, filter the points to be filtered.

[0329] Proceed to step S708.

[0330] S708, determine whether i is less than or equal to N.

[0331] If i is less than or equal to N, increment i by 1 and proceed to steps S703 and S705 to update the i values ​​in steps S703 and S705; otherwise, end the current processing flow.

[0332] S709, determine whether the point to be filtered is located behind object i.

[0333] In other words, it determines whether the distance between the point to be filtered and the origin is greater than the distance between object i and the origin.

[0334] If the point to be filtered is located behind object i, proceed to step S707; otherwise, proceed to step S710.

[0335] S710, retain the point to be filtered.

[0336] Proceed to step S704 until all points within the view range of object i have been processed.

[0337] S711, determine whether the point to be filtered is a point in the point cloud of the sample.

[0338] For example, determine whether the point to be filtered is a point pasted into the initial point cloud frame, that is, whether the point to be processed is a pasted point.

[0339] If the point to be filtered is a point in the point cloud of the sample, proceed to step S709; if the point to be filtered is not a point in the point cloud of the sample, proceed to step S710.

[0340] Figure 6 and Figure 8 The diagram shows the filtering results for two scenarios.

[0341] Figure 6 (a) is a schematic diagram of the intermediate point cloud frame obtained by directly pasting the sample point cloud into the initial point cloud frame. Figure 6 (b) shows the method by Figure 7 The method described above identifies the points to be filtered and the points to be retained in the intermediate point cloud frame obtained from the previous method. Figure 6 (c) shows the filtered result, i.e., a schematic diagram of the target point cloud frame.

[0342] Figure 8 (a) is a schematic diagram of the intermediate point cloud frame obtained by directly pasting the sample point cloud into the initial point cloud frame. Figure 8 (b) shows the method by Figure 7 The filtered result obtained by the method described in the diagram is a schematic diagram of the target point cloud frame.

[0343] S440, based on the occlusion relationship between objects in the intermediate point cloud frame, adjust the coverage order of image blocks of objects in the intermediate image frame to obtain the target image frame.

[0344] For example, step S440 can be performed by Figure 3 The image block adjustment module 360 ​​in the system is executed.

[0345] The occlusion relationships between objects can be determined based on the object's distance information. This distance information indicates the distance between the object and the origin. In other words, the occlusion relationships between objects in the intermediate point cloud frame are determined based on the distance between the object in the intermediate point cloud frame and the viewpoint. This ensures that the occlusion relationships in the target point cloud frame and the target image frame are consistent.

[0346] Specifically, step S440 includes: sequentially covering the image blocks of the objects in the intermediate image frame in order of increasing distance between the objects in the intermediate point cloud frame and the observation point.

[0347] That is, the image patch of the object that is closer to the observation point is overlaid on the image patch of the object that is farther away from the observation point.

[0348] Alternatively, extract image blocks of all objects from the intermediate image frames, and then paste the image blocks of the objects in order from farthest to closest.

[0349] For example, such as Figure 6 In the intermediate point cloud frame shown, the car is closer to the cyclist than to the cyclist. By first pasting the image patch of the cyclist onto the intermediate image frame, and then pasting the image patch of the car, the following result is obtained: Figure 6 The target image frame shown in (f) has a partially obscured image block of a cyclist.

[0350] The target point cloud frame is the enhanced initial point cloud frame, and the target image frame is the enhanced initial image frame. Both the target point cloud frame and the target image frame can be used as training samples to train a multimodal 3D object detection model.

[0351] The target point cloud frame and target image frame obtained by the scheme in this application embodiment can be used as training samples. Compared with using the initial image frame and initial point cloud frame as training samples, the scheme of this application increases the diversity of training samples and improves the performance of the trained model, for example, improving the recognition rate and reducing the false detection rate. At the same time, based on the occlusion relationship between objects in the intermediate point cloud frame, occluded point clouds are filtered out, ensuring the rationality of the target point cloud frame. In addition, based on the occlusion relationship between objects in the intermediate point cloud frame, the coverage order between the image blocks of the objects is adjusted, which further ensures the rationality of the target image frame, and at the same time, it can ensure the consistency of the projection relationship between the target point cloud frame and the target image frame.

[0352] Furthermore, there is no overlap between the region where the sample's point cloud is located in the initial point cloud frame and the region where the object's point cloud is located in the initial point cloud frame, ensuring that there is no collision between objects in three-dimensional space, thus ensuring the rationality of the target point cloud frame.

[0353] As mentioned earlier, the training samples obtained through method 400 can be used to train a multimodal 3D object detection model.

[0354] This application provides a training method for a 3D object detection model. The 3D object detection model is trained based on the training samples obtained by method 400, which can improve the performance of the model.

[0355] Figure 9 The present application illustrates a training method 900 for a 3D object detection model provided in an embodiment of this application. Figure 9 The method shown can be executed by a training device for a neural network model. This training device can be a cloud service device or a terminal device, such as a computer or server with sufficient computing power to execute the training method for a 3D object detection model. It can also be a system composed of cloud service devices and terminal devices.

[0356] The training data in Method 900 includes data based on... Figure 4 To avoid unnecessary repetition, the descriptions of methods 900 will be omitted when introducing method 900 below.

[0357] Method 900 includes steps S910 to S920. Steps S910 to S920 are described in detail below.

[0358] S910, obtain training samples.

[0359] The training samples include target point cloud frames and labels on the target point cloud frames, as well as target image frames and labels on the target image frames. The labels on the target point cloud frames include 3D bounding boxes indicating the area where the point cloud of an object is located in the target point cloud frame, and the labels on the target image frames include 2D bounding boxes indicating the area where the point cloud of an object is located in the target image frame.

[0360] The target point cloud frame is obtained by filtering out points in the intermediate point cloud frame that are occluded by objects in the intermediate point cloud frame. The target image frame is obtained by adjusting the coverage order of the image patches of the objects in the intermediate image frame according to the occlusion relationship between the objects in the intermediate point cloud frame. The intermediate point cloud frame is obtained by adding the point cloud of at least one sample to the initial point cloud frame. The intermediate image frame is obtained by adding the image patch of the at least one sample to the initial image frame. There is a correspondence between the initial point cloud frame and the initial image frame. At least one sample is used to indicate at least one object.

[0361] Image frames and point cloud frames with a corresponding relationship can be understood as paired image frames and point cloud frames.

[0362] There is a correspondence between the initial point cloud frame and the initial image frame, and correspondingly, there is also a correspondence between the target point cloud frame and the target image frame.

[0363] The method for obtaining the initial point cloud frame and the initial image frame can be referred to the description in step S410 above, and will not be repeated here.

[0364] For example, the target point cloud frame can be as follows Figure 6 As shown in (c), the corresponding target image frame can be as follows: Figure 6 As shown in (f).

[0365] Adding the point cloud of a sample to the initial point cloud frame can also be understood as adding the point cloud of the object indicated by the sample to the initial point cloud frame.

[0366] Adding an image patch of a sample to the initial image frame can also be understood as adding an image patch of the object indicated by the sample to the initial image frame.

[0367] For example, the at least one sample includes, Figure 6 The sample shown is used to indicate a cyclist. Adding the point cloud of the cyclist to the initial point cloud frame yields the following result: Figure 6 The intermediate point cloud frame is shown in (a). Adding the image patch of the cyclist to the initial image frame yields the result shown in (a). Figure 6 The intermediate point cloud frame shown in (e).

[0368] The 3D bounding box used to indicate the region where the point cloud of an object is located in the target point cloud frame can be understood as a bounding box used to indicate the region where the point cloud of an object is located in the target point cloud frame. In other words, the points in the target point cloud frame that fall within this 3D bounding box are the point clouds of that object in the target point cloud frame. For example, Figure 6 The diagram shows a schematic of a 3D frame.

[0369] The 2D bounding box used to indicate the region where the point cloud of an object is located in the target image frame can be understood as a bounding box used to indicate the region where the point cloud of the object is located in the target image frame. Alternatively, the image patch in the target image frame that falls within this 2D bounding box is the image patch in the target image frame that includes the object. For example, Figure 6 The diagram shows a schematic of a 2D box.

[0370] In this embodiment of the application, the label can also be referred to as the truth value label.

[0371] For example, the labels on the target point cloud frame can be pre-annotated. The labels on the target image frame can also be pre-annotated. For instance, the labels on the target point cloud frame can be manually annotated.

[0372] Alternatively, the labels on the target point cloud frame can be determined based on the labels on intermediate point cloud frames. The labels in the intermediate point cloud frames are determined based on the labels on the initial point cloud frame and the labels of the point clouds of the at least one sample. The labels on the target image frame can be determined based on the labels on intermediate image frames. The labels in the intermediate image frames are determined based on the labels on the initial image frame and the labels of the image patches of the at least one sample.

[0373] In this embodiment of the application, the label on the point cloud frame includes a 3D bounding box for indicating the area where the object is located, and the label on the image frame includes a 2D bounding box for indicating the area where the object is located.

[0374] In this case, adding the point cloud of at least one sample to the initial point cloud frame can be understood as adding the point cloud of at least one sample and its label to the initial point cloud frame. Adding the image patch of at least one sample to the initial point cloud frame can be understood as adding the image patch of at least one sample and its label to the initial image frame. For example, adding the point cloud of at least one sample and the 3D bounding box of the sample to the initial point cloud frame. The 3D bounding box of the sample is the bounding box of the point cloud of the object indicated by the sample. Adding the image patch of at least one sample and the 2D bounding box of the sample to the initial image frame. The 2D bounding box of the sample is the bounding box of the image patch of the object indicated by the sample.

[0375] Optionally, the point cloud of at least one sample and the image patch of at least one sample can be the point cloud and image patch of the labeled sample extracted from at least one point cloud frame and at least one image frame, respectively.

[0376] Each image frame in the at least one image frame and each point cloud frame in the at least one point cloud frame are in one-to-one correspondence.

[0377] In other words, the at least one image frame and the at least one point cloud frame include at least one set of paired image frames and point cloud frames. The point cloud and image patch of a sample are extracted from a set of paired image frames and point cloud frames.

[0378] The point cloud and image patch of the at least one sample can be extracted from a set of paired image frames and point cloud frames, or it can be extracted from multiple sets of paired image frames and point cloud frames respectively.

[0379] The paired point cloud frame and image frame include labeled samples. Labeled samples refer to samples with labeled ground truth values.

[0380] The point cloud of a sample can be the point cloud of the labeled sample extracted from paired point cloud frames and image frames using the ground truth of the labeled sample. The image patch of a sample can be the image patch of the labeled sample extracted from paired point cloud frames and image frames using the ground truth of the labeled sample.

[0381] Furthermore, the point cloud and image patch of the at least one sample can be obtained by sampling in a sampling pool. The sampling pool includes point clouds and image patches of multiple candidate samples. The point clouds and image patches of the multiple candidate samples can be point clouds and image patches of labeled samples extracted from paired point cloud frames and image frames using the ground truth values ​​of labeled samples.

[0382] For a detailed description of the sample acquisition method, please refer to step S420 above, which will not be repeated here.

[0383] An intermediate point cloud frame is obtained by adding the point cloud of at least one sample to an initial point cloud frame, including: the intermediate point cloud frame is obtained by adding the point clouds of different samples from the at least one sample to different regions of the initial point cloud frame respectively.

[0384] For example, intermediate point cloud frames are obtained by pasting the point clouds of different samples from at least one sample into different regions of the initial point cloud.

[0385] In other words, the positions of point clouds from different samples do not overlap in the intermediate point cloud frame. For example, the 3D bounding boxes of the different samples do not have overlapping areas in the intermediate point cloud frame.

[0386] Optionally, there is no overlap between the point cloud region of at least one sample in the intermediate point cloud frame and the point cloud region of the object in the initial point cloud frame.

[0387] For example, the region where the point cloud of a sample is located can be indicated by the 3D bounding box of the sample. The region where the point cloud of an object in the initial point cloud frame is located can be indicated by a label on the initial point cloud frame. The label on the initial point cloud frame includes a 3D bounding box used to indicate the region where the object is located in the initial point cloud frame.

[0388] Optionally, the point cloud region of at least one sample in the intermediate point cloud frame does not overlap with the point cloud region of the object in the initial point cloud frame, including: on BEV, the 3D bounding box of the at least one sample does not overlap with the 3D bounding box of the object region in the initial point cloud frame in the intermediate point cloud frame.

[0389] It should be noted that, since the intermediate point cloud frame is obtained by adding the point cloud of the sample to the initial point cloud frame, in this embodiment, the area where the point cloud of the sample is located on the initial point cloud frame and the area where the point cloud of the sample is located on the intermediate point cloud frame can be understood to have the same meaning.

[0390] In this way, there is no overlap between the point cloud of the sample and the point cloud of the object in the initial point cloud frame. This avoids the unreasonable situation of multiple objects colliding in three-dimensional space in the intermediate point cloud frame, ensuring the rationality of the intermediate point cloud frame. In turn, it ensures the rationality of the target point cloud frame obtained based on the intermediate point cloud frame. Using the target point cloud frame as a training sample can improve the performance of the trained 3D object detection model.

[0391] Optionally, the area in which the point cloud of at least one sample is located in the intermediate point cloud frame is the same as the area in which the point cloud of at least one sample is located in the point cloud frame of the original scene, and the point cloud frame of the original scene is a point cloud frame collected based on the scene where the at least one object indicated by the at least one sample is located.

[0392] As mentioned earlier, the point cloud and image patch of a sample are extracted from paired point cloud frames and image frames. The paired point cloud frames and image frames are those acquired based on the original scene. The region of the point cloud of the at least one sample in the point cloud frame of the original scene can also be understood as the region of the point cloud of the at least one sample in the point cloud frame of the paired point cloud frames and image frames.

[0393] Specifically, if the area where the point cloud of at least one sample is located in the point cloud frame of the original scene is indicated by the coordinates of the 3D bounding box of the at least one sample, then the area where the point cloud of at least one sample is located in the intermediate point cloud frame can also be indicated by the coordinates of the 3D bounding box of the at least one sample.

[0394] In other words, the point cloud of at least one sample is added to the area indicated by the coordinates of the 3D bounding box of the at least one sample in the initial point cloud frame.

[0395] In this way, the region in which the point cloud of at least one sample is located in the intermediate point cloud frame is the same as the region in which the point cloud of at least one sample is located in the point cloud frame of the original scene, which satisfies the sparsity of the point cloud in the intermediate point cloud frame, avoids the situation that does not match the actual situation, and further improves the rationality of the intermediate point cloud frame. In turn, it ensures the rationality of the target point cloud frame obtained based on the intermediate point cloud frame. Using the target point cloud frame as a training sample can improve the performance of the trained 3D object detection model.

[0396] Optionally, the point cloud of the at least one sample is located in the target region of the intermediate point cloud frame within the intermediate point cloud frame.

[0397] In this way, the point cloud of the sample can be placed in the target region, increasing the number of objects in the target region to obtain training samples that better meet the requirements.

[0398] For example, the target region in the initial point cloud frame can be a planar region.

[0399] For example, planar regions can be extracted from the initial point cloud frame by fitting the point cloud plane.

[0400] Optionally, the target area is the road surface area.

[0401] Accordingly, at least one frame sample is a sample on the road surface area. For example, the object indicated by the at least one frame sample may include dynamic objects, such as pedestrians or vehicles on the road surface.

[0402] For example, method 400 can be applied to the field of autonomous driving. In the field of autonomous driving, more attention is usually paid to objects in the road surface area. Adding the point cloud of the sample to the road surface area can increase the number of objects in the road surface area, while ensuring the rationality of the intermediate point cloud frame, thereby ensuring the rationality of the target point cloud frame obtained based on the intermediate point cloud frame. Using the target point cloud frame as a training sample can improve the performance of the trained 3D object detection model.

[0403] Similar to the method for determining the region where the point cloud of at least one sample is located, the region where the image patch of at least one sample is located in the intermediate image frame can be the same as the region where the image patch of at least one sample is located in the original scene image frame. The original scene image frame refers to the image frame acquired based on the scene where the object indicated by the at least one sample is located.

[0404] As mentioned earlier, the point cloud and image patch of a sample are extracted from paired point cloud frames and image frames. The paired point cloud frames and image frames are those acquired based on the original scene. The position of the image patch of at least one sample in the image frame of the original scene can also be understood as the position of the image patch of at least one sample in the image frame of the paired point cloud frames and image frames.

[0405] Specifically, if the area where the image block of at least one sample is located in the image frame of the original scene is indicated by the coordinates of the 2D bounding box of the at least one sample, then the area where the image block of at least one sample is located in the intermediate image frame can also be indicated by the coordinates of the 2D bounding box of the at least one sample.

[0406] In other words, the image block of at least one sample is added to the area indicated by the coordinates of the 2D bounding box of the at least one sample in the initial image frame.

[0407] Due to camera extrinsic jitter, the position of the point cloud of the sample obtained by the above method in the initial point cloud frame may not completely correspond to the position of the image patch of the sample in the initial image frame.

[0408] Optionally, the region in which the image patch of the at least one sample is located in the intermediate image frame is obtained by projecting the region in which the point cloud of the at least one sample is located in the intermediate point cloud frame onto the initial image frame.

[0409] For example, the point cloud of the first sample in the at least one sample is added to the first region in the initial point cloud frame, the image patch of the first sample is added to the first region in the initial image frame, and the first region in the initial image frame is obtained by projecting the first region in the initial point cloud frame onto the initial image frame.

[0410] In this way, by determining the region of the sample's image patch in the initial image frame based on the region of the sample's point cloud in the initial point cloud frame, the influence of camera extrinsic jitter can be avoided, ensuring the consistency between the position of the sample's point cloud in the initial point cloud frame and the position of the sample's image patch in the initial image frame. This improves the accuracy of the training samples and, consequently, ensures the training effect of the 3D object detection model.

[0411] Optionally, points occluded by objects in the intermediate point cloud frame include: points located within the viewing range of objects in the intermediate point cloud frame, and whose distance from the observation point in the intermediate point cloud frame is greater than the distance between the object in the intermediate point cloud frame and the observation point.

[0412] Optionally, the object in the intermediate point cloud frame includes a first object. If the first object is an object in the initial point cloud frame, the point occluded by the first object includes a first 3D point in the intermediate point cloud frame. The first 3D point is a point in the point cloud of at least one sample, and the first 3D point is located within the view range of the first object. The distance between the first 3D point and the origin is greater than the distance between the first object and the origin.

[0413] Optionally, the objects in the intermediate point cloud frame include a first object. If the first object is an object in the initial point cloud frame, the points occluded by the first object include the second 3D points and the third 3D points in the intermediate point cloud frame.

[0414] The second 3D point is a background point, and the second 3D point is located within the viewpoint of the first object.

[0415] The third 3D point is the foreground point and is located within the field of view of the first object; the distance between the third 3D point and the origin is greater than the distance between the first object and the origin; and the third 3D point does not belong to the first object.

[0416] For specific filtering methods, please refer to the description in step S430, which will not be repeated here.

[0417] Optionally, the occlusion relationship between objects in the intermediate point cloud frame is determined based on the distance between the objects in the intermediate point cloud frame and the observation point, and the target image frame is obtained by adjusting the coverage order of the image blocks of the objects in the intermediate image frame according to the occlusion relationship between the objects in the intermediate point cloud frame, including: the target image frame is obtained by sequentially covering the image blocks of the objects in the intermediate image frame in order from farthest to closest distance between the objects in the intermediate point cloud frame and the observation point.

[0418] For example, extract image patches of all objects from the intermediate image frame, and paste the image patch of the object farthest from the vehicle onto the intermediate image frame first, and then paste the image patch of the object closest to the observation point, in order of distance between the objects in the intermediate point cloud frame and the observation point.

[0419] S920 trains the 3D object detection model based on training samples to obtain a trained 3D object detection model.

[0420] The 3D object detection model can be a neural network model; for example, the 3D object detection model can include a convolutional neural network.

[0421] Specifically, the target point cloud frame and the target image frame are used as inputs to the 3D target detection model, and the labels on the target point cloud frame and the labels on the target image frame are used as the target outputs of the 3D target detection model to train the model until training is complete, resulting in a trained 3D target detection model. The specific training process can adopt the training methods in existing solutions, and this application embodiment does not limit it in this way.

[0422] A well-trained 3D object detection model can be used to achieve 3D object detection.

[0423] According to the scheme of this application embodiment, the training samples are obtained by enhancing the initial image frame and the initial point cloud frame. Compared with using the initial image frame and the initial point cloud frame as training samples, the scheme of this application can improve the diversity of training samples, thereby improving the performance of the trained model, that is, improving the detection accuracy and reducing the false detection rate. Moreover, based on the occlusion relationship between objects in the intermediate point cloud frame, occluded point clouds are filtered out, ensuring the rationality of the target point cloud frame. At the same time, based on the occlusion relationship between objects in the intermediate point cloud frame, the coverage order between the image blocks of the objects is adjusted, which further ensures the rationality of the target image frame and also ensures the consistency between the target point cloud frame and the target image frame.

[0424] Table 1 shows a performance comparison between the model trained using the method of this application and the model trained using existing methods, assuming the same 3D object detection model structure. Alternatively, Table 1 can also be interpreted as a performance comparison between the model trained on baseline training samples and training samples enhanced by the data augmentation method of this application.

[0425] Table 1

[0426]

[0427] Table 1 shows the mean average precision (MAP), nuScenes detection score (NDS), and detection accuracy for various objects, including cars, trucks, construction vehicles (CVs), buses, trailers, barriers, motorcycles, bicycles, pedestrians, and traffic cones, respectively, based on the method of this application and models trained using existing methods. The data in Table 1 are expressed as a percentage (%). As shown in Table 1, the mAP of the model obtained based on the method of this application is improved by 5 percentage points, indicating that the performance of the model obtained based on the method of this application is significantly better than that of the model obtained using existing methods.

[0428] Tables 2 and 3 show the performance comparison results of the model obtained based on the method of this application and the models obtained by other schemes on the nuScenes autonomous driving dataset.

[0429] Table 2

[0430] method mAP NDS Car Truck CV Bus Pointpillars 30.5 45.3 68.4 23.0 4.1 28.2 3DSSD 42.6 56.4 81.2 47.2 12.6 61.4 Pointpainting 46.4 58.1 77.9 35.8 15.8 36.2 CBGS 52.8 63.3 81.1 48.5 10.5 54.9 Centerpoint 60.3 67.3 85.2 53.5 20.0 63.6 This application 66.8 71.0 87.5 57.3 28.0 65.2

[0431] Table 3

[0432] method Trailer Barrier Motor. Bicycle Ped. TC Pointpillars 23.4 38.9 27.4 1.1 59.7 30.8 3DSSD 30.5 47.9 36.0 8.6 70.2 31.1 Pointpainting 37.3 60.2 41.5 24.1 73.3 62.4 CBGS 42.9 65.7 51.5 22.3 80.1 70.9 Centerpoint 56.0 71.1 59.5 30.7 84.6 78.4 This application 60.7 72.6 74.3 50.9 87.9 83.6

[0433] Tables 2 and 3 show the performance comparison results of the Pointpillars model, the 3D single-stage object detection (3DSSD) model, the Pointpainting model, the class-balanced grouping and sampling for point cloud 3D object detection (CBGS) method, the centerpoint model, and the model obtained by the method of this application. As can be seen from Tables 2 and 3, the 3D detection accuracy of the model obtained by the method of this application is significantly better than that of the models obtained by other methods.

[0434] Table 4 shows the performance comparison results of the model obtained based on the method of this application and the models obtained by other schemes on the Waymo autonomous driving dataset.

[0435] Table 4

[0436]

[0437] Table 4 shows the comparison results of the centerpoint model and the model obtained by the method of this application under difficulty level 1 and difficulty level 2 conditions in multiple categories such as vehicles, pedestrians, and cyclists. It also compares the mAP and mean average precision weighted by heading (mAPH) under difficulty level 1 conditions in all categories, and the mAP and mAPH under difficulty level 2 conditions. As shown in Table 4, the scheme of this application has significant performance gains in the detection accuracy of various targets. Furthermore, the 3D detection accuracy of the model obtained by the method of this application is significantly better than that of models obtained by other methods.

[0438] Figure 10 The illustration shows a schematic flowchart of a 3D target detection method 1000 provided in an embodiment of this application. The method can be executed by a device or equipment capable of performing 3D target detection. For example, the device can be a cloud service device or a terminal device, such as a computer, server, automobile, or vehicle-mounted sensing device with LiDAR and camera, etc., whose computing power is sufficient to execute the 3D target detection method. It can also be a system composed of cloud service devices and terminal devices.

[0439] Figure 10 The trained 3D object detection model used in the 3D object detection method 1000 can be obtained through the above... Figure 9 The method described in the previous section is used to construct the method. Method 1100 includes steps S1110 to S1120. The specific implementation of method 1100 can be referred to the aforementioned method 900. To avoid unnecessary repetition, repeated descriptions will be appropriately omitted when introducing method 1100 below.

[0440] S1110, acquire the image frame to be processed and the point cloud frame to be processed, and there is a correspondence between the image frame to be processed and the point cloud frame to be processed.

[0441] For example, the point cloud frame to be processed and the image frame to be processed can be obtained based on the same scene at the same time or during the same period.

[0442] For example, the point cloud frame to be processed can be data collected by an onboard LiDAR.

[0443] For example, the image frame to be processed can be data acquired by an onboard camera. For instance, the image frame to be processed can be an RGB image.

[0444] S1120: The trained 3D object detection model is used to perform 3D object detection on the image frame and point cloud frame to be processed, and the detection results are obtained.

[0445] The trained 3D object detection model is obtained by training on training samples. The training samples include: target point cloud frames and labels on the target point cloud frames, target image frames and labels on the target image frames. The labels on the target point cloud frames include 3D bounding boxes indicating the regions where the point cloud of the object in the target point cloud frame is located. The labels on the target image frames include 2D bounding boxes indicating the regions where the image patches of the object in the target image frame are located. The target point cloud frames are obtained by filtering out points in intermediate point cloud frames that are occluded by objects in the intermediate point cloud frames. The target image frames are obtained by adjusting the coverage order of the image patches of the objects in the intermediate image frames according to the occlusion relationship between the objects in the intermediate point cloud frames. The intermediate point cloud frames are obtained by adding the point cloud of at least one sample to the initial point cloud frame. The intermediate image frames are obtained by adding the image patches of at least one sample to the initial image frame. There is a correspondence between the initial image frames and the initial point cloud frames. At least one sample is used to indicate at least one object.

[0446] The 3D object detection model can be a neural network model; for example, the 3D object detection model can include a convolutional neural network.

[0447] According to the scheme of this application embodiment, the trained 3D object detection model is trained by the method in the first aspect, wherein the training samples are obtained by enhancing the initial image frame and the initial point cloud frame. Compared with using the initial image frame and the initial point cloud frame as training samples, the scheme of this application can improve the diversity of training samples, thereby improving the performance of the trained model, that is, improving the detection accuracy and reducing the false detection rate. Moreover, based on the occlusion relationship between objects in the intermediate point cloud frame, occluded point clouds are filtered out, ensuring the rationality of the target point cloud frame. At the same time, based on the occlusion relationship between objects in the intermediate point cloud frame, the coverage order between the image blocks of the objects is adjusted, which further ensures the rationality of the target image frame and also ensures the consistency between the target point cloud frame and the target image frame.

[0448] Optionally, there is no overlap between the point cloud region of at least one sample in the intermediate point cloud frame and the point cloud region of the object in the initial point cloud frame.

[0449] Optionally, the area in which the point cloud of at least one sample is located in the intermediate point cloud frame is the same as the area in which the point cloud of at least one sample is located in the point cloud frame of the original scene, and the point cloud frame of the original scene is a point cloud frame acquired based on the scene where at least one object indicated by at least one sample is located.

[0450] Optionally, the region in which the image patch of at least one sample is located in the intermediate image frame is obtained by projecting the region in which the point cloud of at least one sample is located in the intermediate point cloud frame onto the initial image frame.

[0451] Optionally, points occluded by objects in the intermediate point cloud frame include: points located within the viewing range of objects in the intermediate point cloud frame, and whose distance from the observation point in the intermediate point cloud frame is greater than the distance between the object in the intermediate point cloud frame and the observation point.

[0452] Optionally, the occlusion relationship between objects in the intermediate point cloud frame is determined based on the distance between the objects in the intermediate point cloud frame and the observation point, and the target image frame is obtained by adjusting the coverage order of the image blocks of the objects in the intermediate image frame according to the occlusion relationship between the objects in the intermediate point cloud frame, including: the target image frame is obtained by sequentially covering the image blocks of the objects in the intermediate image frame in order from farthest to closest distance between the objects in the intermediate point cloud frame and the observation point.

[0453] Optionally, the area where the point cloud of at least one sample is located in the intermediate point cloud frame is the target area in the intermediate point cloud frame.

[0454] Optionally, the target area is the road surface area.

[0455] The following is combined with Figures 11 to 14 The apparatus of the embodiments of this application will be described below. It should be understood that the apparatus described below is capable of performing the methods of the foregoing embodiments of this application. To avoid unnecessary repetition, repeated descriptions will be appropriately omitted when describing the apparatus of the embodiments of this application below.

[0456] Figure 11 This is a schematic block diagram of a training device for a 3D object detection model according to an embodiment of this application. Figure 11 The training device 3000 for the 3D object detection model shown includes an acquisition unit 3010 and a processing unit 3020.

[0457] The acquisition unit 3010 and the processing unit 3020 can be used to execute the training method of the 3D target detection model of the present application embodiment, specifically, they can be used to execute method 900.

[0458] The acquisition unit 3010 is used to acquire training samples, which include: target point cloud frames and labels on the target point cloud frames, target image frames and labels on the target image frames. The labels on the target point cloud frames include 3D bounding boxes indicating the area where the point cloud of an object is located in the target point cloud frame, and the labels on the target image frames include 2D bounding boxes indicating the area where the image patch of an object is located in the target image frame. The target point cloud frames are obtained by filtering out points in intermediate point cloud frames that are occluded by objects in the intermediate point cloud frames. The target image frames are obtained by adjusting the coverage order of the image patches of objects in the intermediate image frames according to the occlusion relationship between objects in the intermediate point cloud frames. The intermediate point cloud frames are obtained by adding the point cloud of at least one sample to the initial point cloud frame, and the intermediate image frames are obtained by adding the image patch of at least one sample to the initial image frame. There is a correspondence between the initial image frames and the initial point cloud frames, and at least one sample is used to indicate at least one object.

[0459] The processing unit 3020 is used to train the 3D object detection model based on the training samples to obtain the trained 3D object detection model.

[0460] Optionally, as an embodiment, there is no overlap between the point cloud region of at least one sample in the intermediate point cloud frame and the point cloud region of the object in the initial point cloud frame.

[0461] Optionally, as an embodiment, the area in which the point cloud of at least one sample is located in the intermediate point cloud frame is the same as the area in which the point cloud of at least one sample is located in the point cloud frame of the original scene, and the point cloud frame of the original scene is a point cloud frame collected based on the scene where at least one object indicated by at least one sample is located.

[0462] Optionally, as an embodiment, the region in which the image patch of at least one sample is located in the intermediate image frame is obtained by projecting the region in which the point cloud of at least one sample is located in the intermediate point cloud frame onto the initial image frame.

[0463] Optionally, as an embodiment, the points occluded by objects in the intermediate point cloud frame include: points located within the viewing range of the objects in the intermediate point cloud frame, and whose distance from the observation point in the intermediate point cloud frame is greater than the distance between the objects in the intermediate point cloud frame and the observation point.

[0464] Optionally, as an embodiment, the occlusion relationship between objects in the intermediate point cloud frame is determined based on the distance between the objects in the intermediate point cloud frame and the observation point, and the target image frame is obtained by adjusting the coverage order of the image blocks of the objects in the intermediate image frame according to the occlusion relationship between the objects in the intermediate point cloud frame, including: the target image frame is obtained by sequentially covering the image blocks of the objects in the intermediate image frame in order from farthest to closest distance between the objects in the intermediate point cloud frame and the observation point.

[0465] Optionally, as an embodiment, the area where the point cloud of at least one sample is located in the intermediate point cloud frame is the target area in the intermediate point cloud frame.

[0466] Optionally, as an example, the target area is the road surface area.

[0467] Figure 12 This is a schematic block diagram of the 3D target detection device 4000 provided in the embodiments of this application. Figure 12 The 3D target detection device 4000 shown includes an acquisition unit 4010 and a processing unit 4020.

[0468] The acquisition unit 4010 and the processing unit 4020 can be used to execute the 3D target detection method of the embodiments of this application, for example, they can be used to execute method 1100.

[0469] Specifically, the acquisition unit 4010 is used to acquire the image frame to be processed and the point cloud frame to be processed, and there is a correspondence between the image frame to be processed and the point cloud frame to be processed.

[0470] The processing unit 4020 is used to perform 3D object detection on the image frame and the point cloud frame to be processed using a trained 3D object detection model, and obtain the detection result. The trained 3D object detection model is obtained by training the 3D object detection model based on training samples.

[0471] The training samples include: target point cloud frames and labels on the target point cloud frames, target image frames and labels on the target image frames, the labels on the target point cloud frames include 3D bounding boxes indicating the area where the point cloud of the object in the target point cloud frame is located, and the labels on the target image frames include 2D bounding boxes indicating the area where the image patch of the object in the target image frame is located.

[0472] The target point cloud frame is obtained by filtering out points in the intermediate point cloud frame that are occluded by objects in the intermediate point cloud frame. The target image frame is obtained by adjusting the coverage order of the image patches of the objects in the intermediate image frame according to the occlusion relationship between the objects in the intermediate point cloud frame. The intermediate point cloud frame is obtained by adding the point cloud of at least one sample to the initial point cloud frame. The intermediate image frame is obtained by adding the image patch of at least one sample to the initial image frame. There is a correspondence between the initial image frame and the initial point cloud frame. At least one sample is used to indicate at least one object.

[0473] Optionally, as an embodiment, there is no overlap between the point cloud region of at least one sample in the intermediate point cloud frame and the point cloud region of the object in the initial point cloud frame.

[0474] Optionally, as an embodiment, the area in which the point cloud of at least one sample is located in the intermediate point cloud frame is the same as the area in which the point cloud of at least one sample is located in the point cloud frame of the original scene, and the point cloud frame of the original scene is a point cloud frame collected based on the scene where at least one object indicated by at least one sample is located.

[0475] Optionally, as an embodiment, the region in which the image patch of at least one sample is located in the intermediate image frame is obtained by projecting the region in which the point cloud of at least one sample is located in the intermediate point cloud frame onto the initial image frame.

[0476] Optionally, as an embodiment, the points occluded by objects in the intermediate point cloud frame include: points located within the viewing range of the objects in the intermediate point cloud frame, and whose distance from the observation point in the intermediate point cloud frame is greater than the distance between the objects in the intermediate point cloud frame and the observation point.

[0477] Optionally, as an embodiment, the occlusion relationship between objects in the intermediate point cloud frame is determined based on the distance between the objects in the intermediate point cloud frame and the observation point, and the target image frame is obtained by adjusting the coverage order of the image blocks of the objects in the intermediate image frame according to the occlusion relationship between the objects in the intermediate point cloud frame, including: the target image frame is obtained by sequentially covering the image blocks of the objects in the intermediate image frame in order from farthest to closest distance between the objects in the intermediate point cloud frame and the observation point.

[0478] Optionally, as an embodiment, the area where the point cloud of at least one sample is located in the intermediate point cloud frame is the target area in the intermediate point cloud frame.

[0479] Optionally, as an example, the target area is the road surface area.

[0480] It should be noted that the training device 3000 and device 4000 mentioned above are embodied in the form of functional units. The term "unit" here can be implemented in software and / or hardware, and there is no specific limitation on this.

[0481] For example, a "unit" can be a software program, a hardware circuit, or a combination of both that implements the above functions. The hardware circuit may include an application-specific integrated circuit (ASIC), electronic circuitry, a processor (e.g., a shared processor, a proprietary processor, or a group processor) and memory for executing one or more software or firmware programs, integrated logic circuitry, and / or other suitable components that support the described functions.

[0482] Therefore, the units of the various examples described in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0483] Figure 13 This is a schematic diagram of the hardware structure of the training device for the 3D object detection model provided in the embodiments of this application. Figure 13 The training device 5000 for the 3D object detection model shown (specifically, the device 5000 can be a computer device) includes a memory 5001, a processor 5002, a communication interface 5003, and a bus 5004. The memory 5001, processor 5002, and communication interface 5003 are interconnected via the bus 5004.

[0484] The memory 5001 can be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 5001 can store programs. When the program stored in the memory 5001 is executed by the processor 5002, the processor 5002 executes the various steps of the training method for the 3D object detection model of this application embodiment. Specifically, the processor 5002 can execute the steps described above... Figure 9 Step S920 in the method shown.

[0485] The processor 5002 may be a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits, used to execute relevant programs to implement the training method of the 3D object detection model of the method embodiment of this application.

[0486] The processor 5002 can also be an integrated circuit chip with signal processing capabilities. In implementation, each step of the training method for the 3D object detection model of this application can be completed through the integrated logic circuits in the hardware of the processor 5002 or through software instructions.

[0487] The processor 5002 described above can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 5001, and the processor 5002 reads the information in memory 5001 and combines it with its hardware to complete the task. Figure 11 The training apparatus shown includes units that are required to perform functions, or to perform the methods described in this application. Figure 9 The training method of the 3D object detection model is shown.

[0488] The communication interface 5003 uses a transceiver device, such as, but not limited to, a transceiver, to enable communication between the device 5000 and other devices or communication networks. For example, initial point cloud frames and initial image frames can be acquired through the communication interface 5003.

[0489] Bus 5004 may include a pathway for transmitting information between various components of device 5000 (e.g., memory 5001, processor 5002, communication interface 5003).

[0490] Figure 14 This is a schematic diagram of the hardware structure of the 3D target detection device according to an embodiment of this application. Figure 14 The 3D target detection device 6000 shown includes a memory 6001, a processor 6002, a communication interface 6003, and a bus 6004. The memory 6001, processor 6002, and communication interface 6003 are interconnected via the bus 6004.

[0491] The memory 6001 can be a ROM, static storage device, or RAM. The memory 6001 can store programs, and when the program stored in the memory 6001 is executed by the processor 6002, the processor 6002 and the communication interface 6003 are used to execute the various steps of the 3D target detection method of this application embodiment. Specifically, the processor 6002 can execute the steps described above... Figure 10 Step S1120 in the method shown.

[0492] The processor 6002 may be a general-purpose CPU, microprocessor, ASIC, GPU, or one or more integrated circuits, used to execute relevant programs to achieve the functions required by the units in the 3D target detection device of this application embodiment, or to execute the 3D target detection method of this application method embodiment.

[0493] The processor 6002 can also be an integrated circuit chip with signal processing capabilities. In implementation, each step of the 3D target detection method of this application embodiment can be completed by the integrated logic circuitry in the processor 6002 or by software instructions.

[0494] The processor 6002 described above can also be a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 6001. The processor 6002 reads the information in memory 6001 and, in conjunction with its hardware, completes the functions required by the units included in the 3D target detection device of the embodiments of this application, or executes the 3D target detection method of the method embodiments of this application.

[0495] The communication interface 6003 uses a transceiver device, such as, but not limited to, a transceiver, to enable communication between the device 6000 and other devices or communication networks. For example, point cloud frames and image frames to be processed can be acquired through the communication interface 6003.

[0496] Bus 6004 may include a pathway for transmitting information between various components of device 6000 (e.g., memory 6001, processor 6002, communication interface 6003).

[0497] It should be noted that although only the memory, processor, and communication interface are shown in the above-described devices 5000 and 6000, those skilled in the art should understand that in specific implementations, devices 5000 and 6000 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that devices 5000 and 6000 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that devices 5000 and 6000 may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 13 and Figure 14 All the devices shown.

[0498] This application provides a computer-readable medium storing program code for execution by a device, the program code including a training method or a 3D object detection method for executing the data augmentation method or 3D object detection model in the embodiments of this application.

[0499] This application provides a computer program product containing instructions that, when run on a computer, cause the computer to execute the data augmentation method or the training method or the 3D object detection method of the embodiment of this application.

[0500] This application provides a chip, which includes a processor and a data interface. The processor reads instructions stored in a memory through the data interface. This application provides a data augmentation method, a training method for a 3D object detection model, or a 3D object detection method.

[0501] Optionally, as one implementation, the chip may further include a memory storing instructions, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used for the data augmentation method or the training method of the 3D object detection model or the 3D object detection method in the embodiments of this application.

[0502] The aforementioned chip can be a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).

[0503] This application provides a terminal, which includes any one of device 3000, device 4000, device 5000 or device 6000.

[0504] The aforementioned terminal can be a vehicle, and the aforementioned device can be installed on the vehicle or can be independent of the vehicle. For example, it can be controlled by using a drone, other vehicles, robots, etc.

[0505] In other possible scenarios, the terminal could also be a drone or a robot. This application uses a vehicle as an example, but the solution can be extended to other possible terminal types.

[0506] This application provides a wearable device, which includes any one of device 3000, device 4000, device 5000, or device 6000. The wearable device can be augmented reality (AR) glasses or AR helmets, etc.

[0507] It should be understood that the processor in the embodiments of this application can be a central processing unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0508] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0509] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0510] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0511] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0512] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0513] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0514] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0515] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0516] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0517] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0518] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0519] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A training method for a 3D object detection model, characterized in that, include: Acquire training samples, the training samples including: target point cloud frames and labels on the target point cloud frames, target image frames and labels on the target image frames, the labels on the target point cloud frames including 3D bounding boxes for indicating the area where the point cloud of an object is located in the target point cloud frame, and the labels on the target image frames including 2D bounding boxes for indicating the area where the image patch of an object is located in the target image frame. The target point cloud frame is obtained by filtering out points in the intermediate point cloud frame that are occluded by objects in the intermediate point cloud frame. The target image frame is obtained by adjusting the coverage order of image patches of objects in the intermediate image frame according to the occlusion relationship between objects in the intermediate point cloud frame. The intermediate point cloud frame is obtained by adding the point cloud of at least one sample to the initial point cloud frame. The intermediate image frame is obtained by adding the image patch of the at least one sample to the initial image frame. There is a correspondence between the initial image frame and the initial point cloud frame. The at least one sample is used to indicate at least one object. The 3D object detection model is trained based on the training samples to obtain a trained 3D object detection model.

2. The method according to claim 1, characterized in that, There is no overlap between the point cloud region of the at least one sample in the intermediate point cloud frame and the point cloud region of the object in the initial point cloud frame.

3. The method according to claim 1 or 2, characterized in that, The area in which the point cloud of the at least one sample is located in the intermediate point cloud frame is the same as the area in which the point cloud of the at least one sample is located in the point cloud frame of the original scene, and the point cloud frame of the original scene is a point cloud frame collected based on the scene in which the at least one object indicated by the at least one sample is located.

4. The method according to claim 1 or 2, characterized in that, The region in which the image patch of the at least one sample is located in the intermediate image frame is obtained by projecting the point cloud of the at least one sample into the region in the intermediate point cloud frame.

5. The method according to claim 1 or 2, characterized in that, The points occluded by objects in the intermediate point cloud frame include: points located within the viewing range of objects in the intermediate point cloud frame, and whose distance from the observation point in the intermediate point cloud frame is greater than the distance between the object in the intermediate point cloud frame and the observation point.

6. The method according to claim 1 or 2, characterized in that, The occlusion relationship between objects in the intermediate point cloud frame is determined based on the distance between the objects in the intermediate point cloud frame and the observation point. The target image frame is obtained by adjusting the coverage order of image patches of objects in the intermediate image frame according to the occlusion relationship between objects in the intermediate point cloud frame, including: The target image frame is obtained by sequentially covering the image blocks of the objects in the intermediate image frame in order of increasing distance between the objects in the intermediate point cloud frame and the observation point.

7. The method according to claim 1 or 2, characterized in that, The region in which the point cloud of at least one sample is located in the intermediate point cloud frame is the target region in the intermediate point cloud frame.

8. The method according to claim 7, characterized in that, The target area is the road surface area.

9. A 3D target detection method, characterized in that, include: Acquire image frames and point cloud frames to be processed, wherein there is a corresponding relationship between the image frames and point cloud frames to be processed; A trained 3D object detection model is used to perform 3D object detection on the image frame and point cloud frame to be processed, and the detection results are obtained. The trained 3D object detection model is obtained by training on training samples. The training samples include: target point cloud frames and labels on the target point cloud frames, target image frames and labels on the target image frames, the labels on the target point cloud frames include 3D bounding boxes for indicating the area where the point cloud of an object is located in the target point cloud frame, and the labels on the target image frames include 2D bounding boxes for indicating the area where the image patch of an object is located in the target image frame. The target point cloud frame is obtained by filtering out points in the intermediate point cloud frame that are occluded by objects in the intermediate point cloud frame. The target image frame is obtained by adjusting the coverage order of image patches of objects in the intermediate image frame according to the occlusion relationship between objects in the intermediate point cloud frame. The intermediate point cloud frame is obtained by adding the point cloud of at least one sample to the initial point cloud frame. The intermediate image frame is obtained by adding the image patch of the at least one sample to the initial image frame. There is a correspondence between the initial image frame and the initial point cloud frame. The at least one sample is used to indicate at least one object.

10. The method according to claim 9, characterized in that, There is no overlap between the point cloud region of the at least one sample in the intermediate point cloud frame and the point cloud region of the object in the initial point cloud frame.

11. The method according to claim 9 or 10, characterized in that, The area in which the point cloud of the at least one sample is located in the intermediate point cloud frame is the same as the area in which the point cloud of the at least one sample is located in the point cloud frame of the original scene, and the point cloud frame of the original scene is a point cloud frame collected based on the scene in which the at least one object indicated by the at least one sample is located.

12. The method according to claim 9 or 10, characterized in that, The region in which the image patch of the at least one sample is located in the intermediate image frame is obtained by projecting the point cloud of the at least one sample into the region in the intermediate point cloud frame.

13. The method according to claim 9 or 10, characterized in that, The points occluded by objects in the intermediate point cloud frame include: points located within the viewing range of objects in the intermediate point cloud frame, and whose distance from the observation point in the intermediate point cloud frame is greater than the distance between the object in the intermediate point cloud frame and the observation point.

14. The method according to claim 9 or 10, characterized in that, The occlusion relationship between objects in the intermediate point cloud frame is determined based on the distance between the objects in the intermediate point cloud frame and the observation point. The target image frame is obtained by adjusting the coverage order of image patches of objects in the intermediate image frame according to the occlusion relationship between objects in the intermediate point cloud frame, including: The target image frame is obtained by sequentially covering the image blocks of the objects in the intermediate image frame in order of increasing distance between the objects in the intermediate point cloud frame and the observation point.

15. The method according to claim 9 or 10, characterized in that, The region in which the point cloud of at least one sample is located in the intermediate point cloud frame is the target region in the intermediate point cloud frame.

16. The method according to claim 15, characterized in that, The target area is the road surface area.

17. A data augmentation method, characterized in that, include: Acquire the initial point cloud frame and the initial image frame, and there is a corresponding relationship between the initial point cloud frame and the initial image frame; Add the point cloud of at least one sample to the initial point cloud frame to obtain the intermediate point cloud frame; An image patch of at least one sample is added to an initial image frame to obtain an intermediate image frame, wherein at least one sample is used to indicate at least one object; Filter out points in the intermediate point cloud frame that are occluded by objects in the intermediate point cloud frame to obtain the target point cloud frame; Based on the occlusion relationship between objects in the intermediate point cloud frame, the covering order of the image blocks of the objects in the intermediate image frame is adjusted to obtain the target image frame.

18. A training device for a 3D object detection model, characterized in that, include: An acquisition unit is used to acquire training samples, the training samples including: a target point cloud frame and a label on the target point cloud frame, a target image frame and a label on the target image frame, the label on the target point cloud frame including a 3D bounding box for indicating the area where the point cloud of an object in the target point cloud frame is located, and the label on the target image frame including a 2D bounding box for indicating the area where the image patch of an object in the target image frame is located. The target point cloud frame is obtained by filtering out points in the intermediate point cloud frame that are occluded by objects in the intermediate point cloud frame. The target image frame is obtained by adjusting the coverage order of image patches of objects in the intermediate image frame according to the occlusion relationship between objects in the intermediate point cloud frame. The intermediate point cloud frame is obtained by adding the point cloud of at least one sample to the initial point cloud frame. The intermediate image frame is obtained by adding the image patch of the at least one sample to the initial image frame. There is a correspondence between the initial image frame and the initial point cloud frame. The at least one sample is used to indicate at least one object. The processing unit is used to train the 3D object detection model based on the training samples to obtain the trained 3D object detection model.

19. The apparatus according to claim 18, characterized in that, There is no overlap between the point cloud region of the at least one sample in the intermediate point cloud frame and the point cloud region of the object in the initial point cloud frame.

20. The apparatus according to claim 18 or 19, characterized in that, The area in which the point cloud of the at least one sample is located in the intermediate point cloud frame is the same as the area in which the point cloud of the at least one sample is located in the point cloud frame of the original scene, and the point cloud frame of the original scene is a point cloud frame collected based on the scene in which the at least one object indicated by the at least one sample is located.

21. The apparatus according to claim 18 or 19, characterized in that, The region in which the image patch of the at least one sample is located in the intermediate image frame is obtained by projecting the point cloud of the at least one sample into the region in the intermediate point cloud frame.

22. The apparatus according to claim 18 or 19, characterized in that, The points occluded by objects in the intermediate point cloud frame include: points located within the viewing range of objects in the intermediate point cloud frame, and whose distance from the observation point in the intermediate point cloud frame is greater than the distance between the object in the intermediate point cloud frame and the observation point.

23. The apparatus according to claim 18 or 19, characterized in that, The occlusion relationship between objects in the intermediate point cloud frame is determined based on the distance between the objects in the intermediate point cloud frame and the observation point. The target image frame is obtained by adjusting the coverage order of image patches of objects in the intermediate image frame according to the occlusion relationship between objects in the intermediate point cloud frame, including: The target image frame is obtained by sequentially covering the image blocks of the objects in the intermediate image frame in order of increasing distance between the objects in the intermediate point cloud frame and the observation point.

24. The apparatus according to claim 18 or 19, characterized in that, The region in which the point cloud of at least one sample is located in the intermediate point cloud frame is the target region in the intermediate point cloud frame.

25. The apparatus according to claim 24, characterized in that, The target area is the road surface area.

26. A 3D target detection device, characterized in that, include: An acquisition unit is used to acquire an image frame to be processed and a point cloud frame to be processed, wherein there is a corresponding relationship between the image frame to be processed and the point cloud frame to be processed. The processing unit is used to perform 3D object detection on the image frame and the point cloud frame to be processed using a trained 3D object detection model, and obtain the detection result. The trained 3D object detection model is obtained by training the 3D object detection model based on training samples. The training samples include: target point cloud frames and labels on the target point cloud frames, target image frames and labels on the target image frames, the labels on the target point cloud frames include 3D bounding boxes for indicating the area where the point cloud of an object is located in the target point cloud frame, and the labels on the target image frames include 2D bounding boxes for indicating the area where the image patch of an object is located in the target image frame. The target point cloud frame is obtained by filtering out points in the intermediate point cloud frame that are occluded by objects in the intermediate point cloud frame. The target image frame is obtained by adjusting the coverage order of image patches of objects in the intermediate image frame according to the occlusion relationship between objects in the intermediate point cloud frame. The intermediate point cloud frame is obtained by adding the point cloud of at least one sample to the initial point cloud frame. The intermediate image frame is obtained by adding the image patch of the at least one sample to the initial image frame. There is a correspondence between the initial image frame and the initial point cloud frame. The at least one sample is used to indicate at least one object.

27. The apparatus according to claim 26, characterized in that, There is no overlap between the point cloud region of the at least one sample in the intermediate point cloud frame and the point cloud region of the object in the initial point cloud frame.

28. The apparatus according to claim 26 or 27, characterized in that, The area in which the point cloud of the at least one sample is located in the intermediate point cloud frame is the same as the area in which the point cloud of the at least one sample is located in the point cloud frame of the original scene, and the point cloud frame of the original scene is a point cloud frame collected based on the scene in which the at least one object indicated by the at least one sample is located.

29. The apparatus according to claim 26 or 27, characterized in that, The region in which the image patch of the at least one sample is located in the intermediate image frame is obtained by projecting the point cloud of the at least one sample into the region in the intermediate point cloud frame.

30. The apparatus according to claim 26 or 27, characterized in that, The points occluded by objects in the intermediate point cloud frame include: points located within the viewing range of objects in the intermediate point cloud frame, and whose distance from the observation point in the intermediate point cloud frame is greater than the distance between the object in the intermediate point cloud frame and the observation point.

31. The apparatus according to claim 26 or 27, characterized in that, The occlusion relationship between objects in the intermediate point cloud frame is determined based on the distance between the objects in the intermediate point cloud frame and the observation point. The target image frame is obtained by adjusting the coverage order of image patches of objects in the intermediate image frame according to the occlusion relationship between objects in the intermediate point cloud frame, including: The target image frame is obtained by sequentially covering the image blocks of the objects in the intermediate image frame in order of increasing distance between the objects in the intermediate point cloud frame and the observation point.

32. The apparatus according to claim 26 or 27, characterized in that, The region in which the point cloud of at least one sample is located in the intermediate point cloud frame is the target region in the intermediate point cloud frame.

33. The apparatus according to claim 32, characterized in that, The target area is the road surface area.

34. A data augmentation device, characterized in that, include: The acquisition unit is used to acquire the initial point cloud frame and the initial image frame, and there is a corresponding relationship between the initial point cloud frame and the initial image frame. Processing unit, used for: Add the point cloud of at least one sample to the initial point cloud frame to obtain the intermediate point cloud frame; An image patch of at least one sample is added to an initial image frame to obtain an intermediate image frame, wherein at least one sample is used to indicate at least one object; Filter out points in the intermediate point cloud frame that are occluded by objects in the intermediate point cloud frame to obtain the target point cloud frame; Based on the occlusion relationship between objects in the intermediate point cloud frame, the covering order of the image blocks of the objects in the intermediate image frame is adjusted to obtain the target image frame.

35. A training device for a 3D object detection model, characterized in that, It includes a processor and a memory, the memory being used to store program instructions, and the processor being used to invoke the program instructions to perform the method of any one of claims 1 to 8.

36. A 3D target detection device, characterized in that, It includes a processor and a memory, the memory being used to store program instructions, and the processor being used to invoke the program instructions to perform the method of any one of claims 9 to 16.

37. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code for execution by the device, the program code including methods for performing any one of claims 1 to 8 or claims 9 to 16.

38. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as claimed in any one of claims 1 to 8 or 9 to 16.

Images