Multi-modal data joint enhancement method and system based on depth consistency

Abstract

The invention relates to the field of three-dimensional target detection, in particular to a multi-modal data joint enhancement method based on depth consistency, and is used for solving the problem of unbalanced category distribution. The method comprises the steps that a three-dimensional target object is generated based on RGB pixel blocks corresponding to a target object, and target object point cloud in the three-dimensional target object is copied to sample point cloud to be enhanced; obtaining a to-be-enhanced RGB sample image corresponding to the to-be-enhanced sample point cloud, and projecting a target object point cloud in the to-be-enhanced sample point cloud in the to-be-enhanced RGB sample image to obtain an enhanced RGB sample image; according to the current distance between the center point of the point cloud of the target object and the radar, determining the current point cloud density by using the distance-density linear relationship; and resampling the target object point cloud based on the current point cloud density to obtain an enhanced sample point cloud with consistent density. The scheme can be used for generating application scenes belonging to long-tail categories, so that category distribution is balanced.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional target detection, and in particular to a multi-modal data joint enhancement method and system based on depth consistency. Background Art

[0002] For multimodal data (point cloud and RGB images) collected by autonomous vehicles, it is necessary to analyze some extreme cases or long-tail cases, such as cones. Because there are fewer cones in the samples, this class is ignored. And these cases are often difficult to collect, but in the process of training the autonomous driving system, these cases are often more important. These cases with low probability of occurrence will lead to long-tail problems, which in turn affects the generalization ability of the autonomous driving system. That is, in autonomous driving training, there are too few samples for extreme cases, and the vehicle will assume that this situation will not occur, thus affecting the overall performance. Summary of the invention

[0003] The purpose of this case is to propose a multimodal data joint enhancement method and system based on deep consistency, using pre-collected multimodal data sets to automatically expand to obtain multimodal data that conforms to the distribution of real samples, so as to achieve the purpose of balancing the category distribution of multimodal data sets. This method can be applied to the expansion of long-tail cases in autonomous driving scenarios.

[0004] To achieve the above objectives, the specific technical solution of this case is as follows.

[0005] In the first aspect, this case proposes a multimodal data joint enhancement method based on depth consistency, which is used to obtain multimodal enhanced data, and the multimodal data enhancement data includes an enhanced RGB sample image and its corresponding enhanced sample point cloud. The steps include: generating a three-dimensional target object based on the RGB pixel block corresponding to the target object, and copying the target object point cloud in the three-dimensional target object to the sample point cloud to be enhanced; obtaining the RGB sample image to be enhanced corresponding to the sample point cloud to be enhanced, and projecting the target object point cloud in the sample point cloud to be enhanced on the RGB sample image to be enhanced to obtain an enhanced RGB sample image; according to the current distance between the center point of the target object point cloud and the radar, determining the current point cloud density using the distance-density linear relationship; resampling the target object point cloud based on the current point cloud density to obtain an enhanced sample point cloud with consistent density.

[0006] In an implementation of the above technical solution, after copying the target object point cloud in the three-dimensional target object to the sample point cloud to be enhanced, the method performs posture adjustment on the target object point cloud, and the posture adjustment includes adjustment of direction and angle.

[0007] In one implementation of the above technical solution, a three-dimensional target object is generated based on the RGB pixel blocks corresponding to the target object, and the steps include: obtaining a first sample point cloud having the target object and its corresponding first RGB sample image; performing instance segmentation based on the first RGB sample image to obtain a number of RGB pixel blocks including the target object, and then obtaining a point cloud block corresponding to each RGB pixel block in the first sample point cloud; and performing shearing based on the RGB pixel blocks corresponding to the target object, and using the Nerf algorithm to generate the three-dimensional target object.

[0008] In one implementation of the above technical solution, the point cloud block corresponding to each RGB pixel block in the first sample point cloud is obtained, and the steps include: projecting the first sample point cloud into the first RGB sample image, and obtaining the position of each point in the first sample point cloud in the first RGB sample image; for each RGB pixel block, based on the position of each point in the first sample point cloud in the first RGB sample image, determining the point cloud block corresponding to the RGB pixel block in the first sample point cloud.

[0009] In one implementation of the above technical solution, the distance-density function, the determination step includes: randomly selecting two object point cloud blocks from the first sample point cloud, obtaining the distance d between the center of each object point cloud block and the radar, and obtaining the density ρ of each object point cloud block; based on the distance-density linear relationship d=αρ+β, obtaining a set of two-variable linear equations about α and β, and then determining the distance-density linear relationship.

[0010] In one implementation of the above technical solution, the density ρ of the object point cloud block, the acquisition step includes: voxelizing the object point cloud block to obtain a point cloud number matrix PN in n grids, whose distribution satisfies PN~N(μ,σ 2 ), N(μ,σ 2 ) is a Gaussian distribution, μ is the mean of the number of point clouds in each grid, σ 2 is the variance of the number of point clouds; PN i is the number of point clouds in the i-th grid.

[0011] In one implementation of the above technical solution, the resampling method includes uniform sampling, farthest point sampling, and random sampling.

[0012] On the second aspect, this case proposes an autonomous driving system, which includes a target detection and classification model, and the target detection and classification model is trained using multimodal enhanced data obtained by any of the above methods.

[0013] On the third aspect, the present case proposes a computer-readable storage medium storing a computer program that can be loaded by a processor and execute any of the above methods.

[0014] Beneficial technical effects of this case: Compared with simulation-based data enhancement methods, this method avoids the problem of domain shift caused by multimodal data enhancement. Compared with methods based on manual collection, the collection efficiency is higher and the quality of samples can be better controlled. Compared with single-modality data enhancement, this method can effectively utilize multimodal data information. This method can generate more extreme cases to balance data distribution. When applied to autonomous driving, it can enhance the generalization performance of the autonomous driving system. When used as an autonomous driving data set for autonomous driving training, it can improve the accuracy of target detection in the autonomous driving system, thereby improving its safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0016] Figure 1 , one A schematic diagram of a method flow chart in an embodiment of the present invention.

[0017] Figure 2 , one Schematic diagram of the camera coordinate system, radar coordinate system, and pixel coordinate system in this implementation manner.

[0018] Figure 3 , one Schematic diagram of projecting a point cloud into an RGB image in one implementation manner.

[0019] Figure 4 , one Schematic diagram of instance segmentation results of a sample RGB image by a pre-trained model in one implementation manner.

[0020] Figure 5 , one Schematic diagram of the point cloud block corresponding to the sample RGB image in the first implementation manner.

[0021] Figure 6 , one A high-fidelity three-dimensional object schematic diagram generated by a sample RGB image using the Nerf algorithm in this implementation manner.

[0022] Figure 7 , one Schematic diagram of copying an object RGB to another sample RGB in an implementation manner.

[0023] Figure 8 , one Schematic diagram of copying the object point cloud to the sample point cloud in this implementation manner.

[0024] Fig. 9 , one Schematic diagram of object point cloud resampling results in one implementation manner. DETAILED DESCRIPTION

[0025] To address the long-tail problem of autonomous driving datasets in existing technologies, existing methods include:

[0026] (1) Driving scenario simulation: By building a simulation platform, extreme scenarios are constructed in the simulation environment to enrich the data. However, the data collected by this method is distributed differently from the data in the real scene, which will cause domain shift.

[0027] (2) Based on manual large-scale data collection: By collecting a large number of extreme cases, the data distribution will be enriched. However, this method consumes a lot of time and resources to collect data. Manual data collection requires a lot of manpower and time investment, especially when large-scale data collection is required.

[0028] (3) Single-modality data enhancement: For RGB image data, image data is increased by random cropping, random flipping, color jittering, etc. For point cloud data: new point cloud data is obtained by copying and pasting, GAN network generation, and CAD modeling, thereby generating a large number of samples. Single-modality data enhancement methods are not applicable to autonomous driving scenarios and cannot meet the requirements for the development of autonomous driving scenarios towards multi-modal data.

[0029] Based on this, this case proposes a multimodal data joint enhancement method based on deep consistency, which uses pre-collected data with long-tail categories to enhance the data set, expand the small sample class, increase the number of long-tail categories, and achieve the purpose of balancing the category distribution of the multimodal data set. The flow chart of the method is shown in the figure. Figure 1 As shown, the steps include:

[0030] Generate a three-dimensional target object based on the RGB pixel blocks corresponding to the target object, and copy the target object point cloud in the three-dimensional target object to the sample point cloud to be enhanced;

[0031] Obtaining the RGB sample image to be enhanced corresponding to the sample point cloud to be enhanced, projecting the target object point cloud in the sample point cloud to be enhanced onto the RGB sample image to be enhanced, and obtaining an enhanced RGB sample image;

[0032] According to the current distance between the center point of the target object point cloud and the radar, the current point cloud density is determined using the linear relationship between distance and density;

[0033] The sample point cloud to be enhanced is resampled based on the current point cloud density to obtain an enhanced sample point cloud with consistent density.

[0034] Figure 1 The operations in the flowchart may not be performed in order. Instead, the operations may be performed in reverse order or simultaneously. In addition, one or more other operations may be added to the flowchart. One or more operations may be removed from the flowchart.

[0035] The following is a clear and complete description of how to use a small sample to expand and obtain multimodal data that conforms to the distribution of real samples in this case to achieve the purpose of balancing the category distribution of the multimodal data set. Obviously, the described implementation is only a part of the implementation of this case, not all of the implementations. Based on the implementation in this case, all other implementations obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0036] In this case, point cloud data refers to spatial coordinate point data, and may also include texture data, intensity data, etc. The data set includes multimodal data for autonomous driving, and in some implementations, also includes multimodal data acquired in indoor scenes. Multimodal data includes point cloud data and RGB images, and point cloud data is acquired by a radar camera and RGB image data is acquired by multiple cameras.

[0037] First, establish the camera coordinate system, lidar coordinate system, and pixel coordinate system. The position relationship is as follows Figure 2 As shown, the laser radar coordinate system is O L X L Y L Z L , the camera coordinate system is O C X C Y C Z C The camera obtains the image, and the pixel coordinate system is O l The aforementioned laser radar model can be 32-line, 64-line, synthetic aperture, etc., and this case does not limit the radar model.

[0038] Three-dimensional point cloud data can be generated by the following methods: 1) image generation based method, 2) radar acquisition based method, and 3) depth camera based method.

[0039] A multimodal data with a target object is obtained, and the multimodal data includes a sample point cloud and a corresponding RGB sample image. The RGB sample image can be one or more. In the following method, one is taken as an example. When there are multiple samples, they can be processed in sequence. By performing statistical analysis on the acquired data set in advance, the categories in the data set that belong to the long-tail distribution can be determined, and the class belonging to the long-tail distribution is taken as the target class, and the object belonging to this class is the target object.

[0040] The acquired sample point cloud with the target object is used as the first sample point cloud, and its corresponding sample image is used as the first RGB sample image.

[0041] When the point cloud position of the target object is recorded as P and projected into the pixel coordinate system, the coordinates of the projected pixel point are the coordinates of the point in the three-dimensional point cloud in the image ( Figure 3 ), the coordinates of the projected pixel are denoted as (u, v), and the calculation formula for the corresponding pixel is:

[0042]

[0043] In the formula, sample P is the point cloud data of the object point cloud P, P rect Represents the intrinsic parameter matrix of the camera, R rect represents the correction matrix of the camera, It is the external parameter matrix that transforms the radar coordinate system to the camera coordinate system.

[0044] By projecting the first sample point cloud onto the corresponding first RGB sample image, the position of each point in the first sample point cloud in the first RGB sample image can be obtained. Figure 3 , Figure 3 The lower part is a diagram showing the coordinates of the point cloud data after being projected into the image.

[0045] Using a pre-trained instance segmentation network (such as SAM, Mask-RCNN), the first sample RGB image data is predicted and processed to obtain multiple instance pixel blocks (see Figure 4 The number of instances depends on the scenario. Figure 4 , where the example pixel blocks include cars, trees, billboards, and portraits of people in billboards, etc. According to the example pixel blocks and the obtained projection positions, the point cloud blocks corresponding to each RGB pixel block in the first sample point cloud can be obtained.

[0046] The point cloud data corresponding to each instance pixel block is used as the object point cloud block, and the object point cloud block is bound to the RGB pixel block (see Figure 5 Example), establish a one-to-one correspondence, so that according to the projection position of the point cloud in the image, the position and posture of the object in the RGB image can be accurately determined. For each RGB pixel block, the corresponding point cloud block is obtained through the corresponding projection position. By binding the object point cloud block with the RGB pixel block, the geometric shape information and visual features of the object are obtained at the same time.

[0047] Use the instance segmentation network to obtain the corresponding RGB pixel block. The instance segmentation result of the target class is represented as obj RGB =seg(.sample RGB ), obj RGBThat is, the pixel block corresponding to the target class, sample RGB Represents sample RGB image data, and seg represents the instance segmentation step.

[0048] When the projected point cloud is within the instance pixel block obtained by segmentation, the pairing process of the point cloud and the instance pixel block can be expressed as: P , obj RGB}, obj P Represents the object point cloud data corresponding to the target object, (x, y, z)∈obj P , obj RGB Represents the instance pixel block corresponding to the target object, (u, v)∈obj RGB Indicates the projection position of the point cloud in the RGB image. The point cloud projected into the RGB pixel block of an instance is an object point cloud block.

[0049] Next, randomly select object point cloud blocks from two sample point clouds, and establish a distance-density linear relationship in the sample point cloud based on the distance between the object point cloud block and the radar and the point cloud density. The specific steps include:

[0050] The object point cloud block randomly selected from the sample point cloud is denoted as P, and the object point cloud block P is voxelized to obtain the point cloud number matrix PN in n grids, PN = {PN 0 , PN 1 , ..., PN n-1}. PN i is the number of point clouds in the i-th grid, i = 0, 1, ..., n-1. Assume that the point cloud satisfies the Gaussian distribution, that is, PN ~ N (μ, σ 2 ), μ is the mean value of the number of point clouds in each grid, σ 2 is the variance of the number of point clouds,

[0051] The probability density can be expressed as The sum of the probability densities of all points is the point cloud density of the object point cloud block, which can be expressed as the density evaluation function: The function exp(x) represents the natural exponential function.

[0052] Based on the density evaluation function, a distance-density linear function is established as follows:

[0053] d = αρ + β, d represents the distance from the center of the point cloud block to the radar, α and β are the slope and intercept respectively. α and β are unknown quantities to be determined.

[0054] Get an object point cloud block P with K points, then the distance between the center of the object point cloud block and the radar is the coordinate of the i-th point in the object point cloud P. The density of the object point cloud block The function exp(x) represents the natural exponential function, which can obtain a binary equation of the distance-density linear function d=αρ+β, where α and β are unknowns.

[0055] Based on another object point cloud block, another binary equation of the distance-density linear function can be obtained in the same way. The values ​​of α and β can be solved by using two binary equations to obtain the distance-density function.

[0056] Then, the RGB pixel blocks corresponding to the target object are cut and the Nerf algorithm is used to generate a high-fidelity 3D target object (see Figure 6 For example), the object can be scaled or rotated to generate objects with different postures. The specific algorithm of Nerf is not limited, and it can be Make-It-3D, SinNeRF, MCC, Stable Diffusion and other algorithms. The generated three-dimensional target object is mesh data, which consists of dense point cloud and color, and is a total of six-dimensional data of X, Y, Z, R, G, and B. X, Y, and Z are position information, and R, G, and B are color information. The dense point cloud provides geometric shape information of the object, while the color adds real texture and details to the surface of the object. This high-fidelity three-dimensional object representation can provide more accurate and detailed input data for further computer vision tasks such as object recognition and scene reconstruction. By using the Nerf algorithm to generate such a three-dimensional object, it is possible to enrich the representation of object categories in the data set and improve the generalization performance of the algorithm. The generated high-fidelity three-dimensional target object is represented as 3Dobj P，RGB , then: 3Dobj P，RGB =Nerf(obj RGB ).

[0057] The object point cloud in the 3D target object is copied to the sample point cloud to be enhanced. This process can be expressed as newsample P =copyandpaste(3Dobj P，RGB ), copyandpaste indicates a copy operation. By rotating and copying the target object point cloud to the sample point cloud to be enhanced, the posture of the object at different angles and directions can be simulated, and the posture of the target object point cloud can be adjusted. The sample point cloud to be enhanced can be the original first sample point cloud or other sample point clouds.

[0058] Since the target object point cloud is a dense point cloud, it can be directly projected into an RGB image, thereby providing more comprehensive data information (see Figure 7Example). The target object point cloud is projected on the RGB sample image to be enhanced corresponding to the sample point cloud to obtain the enhanced RGB sample image. This process can be expressed as: newsample RGB =proj(3Dobj P，RGB |newsample P ), 3Dobj P，RGB |newsample P It means that only the target object data in the copied object point cloud is projected, and proj represents the point cloud projection process.

[0059] After the point cloud is copied, the position information of the point cloud will change in real time. Through the point cloud position information, the current distance between the new radar and the point cloud center of the target object can be obtained, such as Figure 8 shown. Figure 8 The bright block in is the copied target object point cloud. The current distance between the new radar and the target object point cloud center in the new position can be directly obtained in the point cloud data. Finally, according to the current distance d′ between the radar and the target object point cloud center, and the distance-density linear function of α and β that have been determined, the current point cloud density can be solved Based on the point cloud density ρ′, the sample point cloud to be enhanced that copies the target object point cloud is resampled to obtain an enhanced sample point cloud with consistent density. Fig. 9 As shown in the figure, the bright block disappears, indicating that the density of the point cloud is consistent after resampling. This process can be expressed as:

[0060] newsample P =resampling score(PN′) (3Dobj P，RGB )

[0061] Among them, 3Dobj P，RGB It means that only the point cloud data of the newly added objects in the sample point cloud to be enhanced is resampled, without affecting other data of the original objects in the new sample point cloud. You can use uniform sampling, farthest point sampling, random sampling and other methods to sample point cloud data of a certain density.

[0062] In the resampling process, the object point cloud is sampled according to the learned distance-density function, which can achieve appropriate supplementation of sparse areas and appropriate simplification of dense areas. That is, in the same 3D point cloud depth, the density should be consistent with the original data density. This can keep the distribution of the resampled sample data similar to that of the original data, while reducing the amount of data and computational complexity. By resampling the object point cloud, a set of new sample data with more representativeness and completeness is obtained. These data can be used to train and evaluate machine learning models, thereby improving the generalization ability and stability of the model. In addition, resampling can also provide more reliable and effective input for subsequent data processing and analysis tasks.

[0063] This case does not limit the setting of single-modal enhancement. You can use only RGB images or point cloud data for enhancement with only slight changes.

[0064] Starting from the development needs of high efficiency and multimodality of autonomous driving, this case proposes a multimodal data joint enhancement method based on deep consistency, realizes the preprocessing of multimodal data with imbalanced categories, and generates more extreme scenes of autonomous driving. It is a more effective and scientific technical means. When these enhanced data sets are used for autonomous driving training, the accuracy of target detection of autonomous driving systems can be improved, thereby improving their safety. This method can also be used in the fields of mapping and remote sensing observation.

[0065] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method of the present disclosure can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structures used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits, etc. However, for the present disclosure, software program implementation is a better implementation method in most cases.

[0066] Although the embodiments of the present invention are described above in conjunction with the accompanying drawings, the present invention is not limited to the above specific embodiments and application fields, and the above specific embodiments are only illustrative and instructive, rather than restrictive. A person of ordinary skill in the art can also make many forms under the guidance of this specification and without departing from the scope of protection of the claims of the present invention, all of which belong to the protection of the present invention.

Claims

1. A multimodal data joint enhancement method based on deep consistency, characterized in that: The method is used to obtain multimodal enhanced data, wherein the multimodal data enhanced data includes an enhanced RGB sample image and its corresponding enhanced sample point cloud, and the steps include: Generate a three-dimensional target object based on the RGB pixel blocks corresponding to the target object, and copy the target object point cloud in the three-dimensional target object to the sample point cloud to be enhanced; Obtaining the RGB sample image to be enhanced corresponding to the sample point cloud to be enhanced, projecting the target object point cloud in the sample point cloud to be enhanced onto the RGB sample image to be enhanced, and obtaining an enhanced RGB sample image; According to the current distance between the center point of the target object point cloud and the radar, the current point cloud density is determined using the distance-density linear relationship; The target object point cloud is resampled based on the current point cloud density to obtain an enhanced sample point cloud with consistent density.

2. The method according to claim 1, characterized in that The method performs posture adjustment on the target object point cloud after copying the target object point cloud in the three-dimensional target object to the sample point cloud to be enhanced, wherein the posture adjustment includes adjustment of direction and angle.

3. The method according to claim 1, characterized in that The steps of generating a three-dimensional target object based on the RGB pixel blocks corresponding to the target object include: Acquire a first sample point cloud having a target object and a first RGB sample image corresponding thereto; Perform instance segmentation based on the first RGB sample image to obtain a number of RGB pixel blocks including the target object, and then obtain a point cloud block corresponding to each RGB pixel block in the first sample point cloud; Based on the RGB pixel blocks corresponding to the target object, the three-dimensional target object is generated using the Nerf algorithm.

4. The method according to claim 3, characterized in that Obtaining a point cloud block corresponding to each RGB pixel block in the first sample point cloud, the steps include: Projecting the first sample point cloud into the first RGB sample image, and obtaining the position of each point in the first sample point cloud in the first RGB sample image; For each RGB pixel block, based on the position of each point in the first sample point cloud in the first RGB sample image, a point cloud block corresponding to the RGB pixel block in the first sample point cloud is determined.

5. The method according to claim 1, characterized in that Distance-density function, the determination steps include: Randomly select two object point cloud blocks from the first sample point cloud, obtain the distance d between the center of each object point cloud block and the radar, and obtain the density ρ of each object point cloud block; Based on the distance-density linear relationship d=αρ+β, a set of two-variable linear equations about α and β is obtained, and then the distance-density linear relationship is determined.

6. The method according to claim 5, characterized in that The density ρ of the object point cloud block is obtained by: The object point cloud block is voxelized to obtain the point cloud number matrix PN in n grids, whose distribution satisfies PN~N(μ,σ 2 ), N(μ,σ 2 ) is a Gaussian distribution, μ is the mean of the number of point clouds in each grid, σ 2 is the variance of the number of point clouds; PN i is the number of point clouds in the i-th grid.

7. The method according to claim 1, characterized in that Resampling methods include uniform sampling, farthest point sampling, and random sampling.

8. An automatic driving system, characterized in that: The system includes a target detection classification model, and the target detection classification model is trained using multimodal enhanced data obtained by any method in claims 1 to 7 above.

9. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 7.

Images