A data enhancement method and system for monocular 3D target detection

Abstract

The application discloses a kind of data enhancement method and system for monocular 3D target detection, respectively construct scene database and object database;Randomly sample scene in scene database, randomly sample position in the accessible area of scene, randomly sample object in object database, after collision, occlusion detection, insert random sampling position, the reorganization image and depth map are obtained by fusing object and scene;Based on depth map, reorganization image is converted to 3D space, and random disturbance is carried out to camera pose, and training image under different visual angle is generated;Random recombination of object, scene and camera pose is synthesized into new image constantly generated, as the training data of neural network, the performance of monocular 3D target detection model is improved;The present application can efficiently utilize existing data resources, significantly improve model performance;It is beneficial to the application of camera-based scene perception algorithm in the application of automatic driving industry, and it is beneficial to the rapid landing and development of relevant industry.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and safety auxiliary driving, and particularly relates to a data enhancement method and system for monocular 3D target detection. BACKGROUND

[0002] Monocular three-dimensional (3D) target detection uses a single RGB image to calculate the 3D properties of objects in the scene, such as the 3D position, orientation and size of the object. Compared with high-cost laser radar solutions and multi-view solutions, monocular 3D detection provides a low-cost solution for automatic driving scene perception. However, due to the ill-posed nature of monocular 3D vision, the monocular 3D detection task is more challenging.

[0003] Due to complex and intertwined visual changes from diverse scenes, object categories and camera poses, training a robust deep learning-based monocular 3D model requires a large amount of labeled data. This can indeed help the network learn a robust feature representation, thereby alleviating the ill-posed problem. However, compared with traditional 2D target detection, 3D target detection is more difficult and expensive to label. Limited training data can result in low accuracy of monocular 3D detection models and difficulty in applying them to real scenes. Therefore, developing an efficient data utilization solution is a meaningful but challenging task for monocular 3D detection.

[0004] Existing works have made various efforts to alleviate the problem of limited training data. Image-level augmentation is a basic solution. Copy-paste methods can generate more target labels for monocular 3D detection, but due to the fixed occupation of existing objects in the scene, these methods are limited when inserting the scene position. Semi-supervised solutions directly alleviate the problem by directly expanding the training data size, but are limited by the quality of pseudo-labels.

[0005] In addition, camera pose is an important factor affecting monocular 3D detection data. Existing methods attempt to learn camera pose online and correct feature maps to enhance the robustness of the model under pose perturbation. However, this method does not consider how to directly reduce the sensitivity of the model to the camera pose through data perturbation. SUMMARY

[0006] The technical problem to be solved by the present application is to provide a data enhancement method and system for monocular 3D target detection, which fully utilizes limited data and enhances the robustness of the model, to solve the technical problems of limited monocular 3D detection data and the tight entanglement of objects, scenes and camera poses in the collected data, and is beneficial to the application of camera-based scene perception algorithms in the automatic driving industry, and is beneficial to the rapid landing and development of related industries.

[0007] The application adopts the following technical solutions:

[0008] A data enhancement method for monocular 3D target detection, comprising the following steps:

[0009] Collecting laser radar and image data, obtaining the label of the object by full labeling or sparse labeling; decomposing the training image into a 3D object point cloud model with texture information, and removing all the obtained objects from the scene to construct an empty scene; constructing an object database using all the obtained objects, and constructing a scene database using all the original scenes and empty scenes;

[0010] Randomly sampling a scene in the obtained scene database, randomly sampling a position in the reachable area of the scene, moving an object in the obtained object database to the sampled position, inserting the object into the randomly sampled position after collision and occlusion detection, and fusing the object and the scene to obtain a reorganized image;

[0011] Converting the obtained reorganized image into 3D space based on the depth map, randomly perturbing the camera pose, and re-rendering the scene image;

[0012] Combining the re-rendered scene image by random reorganization of objects, scenes and camera poses as training data for neural network training;

[0013] When the parameters of the neural network are updated to the maximum number of iterations or meet the termination condition, the student network is retained for real scenes, and the camera data is used as input when facing real scenes, and the position, size and orientation of each target are obtained by inference according to the trained network parameters of the student network, completing the three-dimensional positioning of the target.

[0014] Preferably, constructing an empty scene specifically comprises:

[0015] First, use a depth completion model to obtain a dense depth map of the scene, and use an object segmentation model to obtain a foreground object mask;

[0016] Then, the edge distortion depth of the foreground part of the depth map is repaired, and a 3D object point cloud model with texture information is extracted using the camera intrinsic parameters;

[0017] The foreground object is eliminated to obtain an image and a depth map of the empty scene.

[0018] Preferably, when correcting the edge distortion depth of the foreground object, the distortion depth correction method is as follows:

[0019]

[0020] wherein, represents the distorted depth value; represents the corrected depth value; is the anchor point depth value; The average value of the absolute values of the depth distances of the anchor point and the five laser radar image projection points closest to the anchor point in the image is calculated.

[0021] Preferably, when eliminating the foreground object, object elimination is performed using an object elimination model, and for the depth map, the depth of the foreground region is replaced:

[0022]

[0023] wherein, is the ground depth map, is the scene depth map, is the empty scene depth map, represents the pixel position of the foreground region, represents the position of the background pixel used to replace the corresponding foreground pixel.

[0024] Preferably, the object and the scene are fused to obtain a reorganized image and a depth map, specifically:

[0025] The reachable area of the scene is generated using laser radar data. First, an existing method for generating a sparse reachable area is used to obtain a reachable area indication map in the bird's eye view, and then the indication map is converted to a polar coordinate representation, wherein the origin of the coordinate system is the position of the body camera. Then, the reachable area indication map is completed by row in the polar coordinate representation, and then converted back to the Cartesian coordinate system to obtain a dense reachable area indication map.

[0026] Before fusing the object and the scene, it is determined whether the object collides in 3D space using a 3D bounding box, and the degree of occlusion of each object foreground region is determined in the 2D image using the foreground depth. Objects that collide and are severely occluded are screened out and not subjected to subsequent fusion.

[0027] After screening the object, the textured 3D model of the object is projected onto the image plane through the camera intrinsic parameters, and then depth blanking is performed pixel by pixel according to the dense depth map of the scene, and the texture with a smaller depth value is retained for display, thereby obtaining the fused scene image and depth map.

[0028] Preferably, the object is moved from the original position in the bird's eye view to the sampling position When the object is moved from the original position in the bird's eye view to the sampling position , the position coordinates of the object are converted from to

[0029]

[0030] wherein, is the ground equation of the scene.

[0031] Preferably, the perturbation of camera pose is randomly performed, and the re-rendering of the scene image is specifically:

[0032] The scene image after the fusion of the scene and the object and the scene depth The scene point cloud is converted to a 3D space using the camera intrinsic parameters ;

[0033] Then, the perturbation angles of the camera pitch angle and the roll angle are sampled and The translation perturbation of the camera along the axis is sampled , so as to obtain the perturbed rotation and translation

[0034] Finally, the coordinates of the scene point cloud and the 3D labels of the object are updated .

[0035] Preferably, the perturbed rotation and translation are as follows:

[0036]

[0037] wherein, is the perturbed rotation matrix, is the perturbed translation vector, is the perturbed pitch angle, is the perturbed yaw angle, is the perturbed depth value.

[0038] 9. The data enhancement method for monocular 3D target detection according to claim 1, characterized in that, in the training process, under the condition of sparse labeling, a sparse object database and an empty scene database are used to synthesize training data for pre-training, and then a complete labeled scene is introduced for mixed training.

[0039] In a second aspect, an embodiment of the present application provides a data enhancement system for monocular 3D target detection, comprising:

[0040] A data module collects laser radar and image data, and obtains the labeling of the object through complete labeling or sparse labeling; the training image is decomposed into a 3D object point cloud model with texture information, and all objects obtained are removed from the scene to construct an empty scene; an object database is constructed using all the obtained objects, and a scene database is constructed using all the original scenes and the empty scene;

[0041] A recombination module randomly samples a scene in the obtained scene database, randomly samples a position in the reachable area of the scene, moves an object randomly sampled in the obtained object database to the sampled position, inserts the object into the randomly sampled position after collision and occlusion detection, and fuses the object and the scene to obtain a recombined image;

[0042] a rendering module, converting the obtained recombined image into a 3D space based on the depth map, randomly perturbing the camera pose, and re-rendering the scene image;

[0043] a training module, synthesizing the re-rendered scene image from random combinations of objects, scenes and camera poses as training data for neural network training;

[0044] an output module, when the parameters of the neural network are updated to the maximum number of iterations or meet the termination condition, retaining the student network for real scenes, using camera data as input, and using the trained network parameters of the student network to infer the positions, sizes and orientations of the targets to complete the three-dimensional positioning of the targets.

[0045] In a third aspect, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the above-mentioned data augmentation method for monocular 3D target detection when executing the computer program.

[0046] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium including a computer program, and the computer program implements the steps of the above-mentioned data augmentation method for monocular 3D target detection when executed by a processor.

[0047] In a fifth aspect, a chip includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the above-mentioned data augmentation method for monocular 3D target detection when executing the computer program.

[0048] In a sixth aspect, an embodiment of the present application provides an electronic device including a computer program, and the computer program implements the steps of the above-mentioned data augmentation method for monocular 3D target detection when executed by the electronic device.

[0049] Compared with the prior art, the present application has at least the following beneficial effects:

[0050] A data augmentation method for monocular 3D target detection is divided into data acquisition, data decomposition and data recombination training stages; in the data acquisition stage, laser radar data and image data are acquired, and objects in the scene are fully labeled or sparsely labeled. In the decomposition stage, the image and depth map of the original scene are decomposed into empty scenes and textured object 3D models, and a scene database and an object database are constructed; in the data recombination training stage, new training data are continuously synthesized by randomly combining scenes, objects and camera poses, which can fully utilize limited data and enhance the generalization performance of the network.

[0051] Further, in the case of sparse annotation, for an instance in a video sequence, only the closest to the camera is annotated, and these samples can be used to obtain training samples at further positions through position resampling, thereby significantly reducing the annotation cost. At the same time, the complete annotated video sequence can be flexibly increased to flexibly control the annotation cost.

[0052] Further, when decomposing objects and scenes, using a textured object 3D point cloud model can enable low-cost editing of the 3D position changes of the object. Repairing the edge distortion depth of the foreground part of the depth map can make the object 3D model more accurate and available.

[0053] Further, backgrounding the foreground part of the depth map to eliminate the object can make the camera pose perturbation feasible.

[0054] Further, based on the laser radar data, the accessible area and the obstacle area can be preliminarily divided in the bird's eye view. Due to the sparse scanning characteristics of the laser radar, many areas are not classified. Completing the accessible area indication map under the bird's eye view makes it possible to fully consider the complete scene accessible area when sampling the position, and completing the indication map under the polar coordinate considers the scanning characteristics of the laser radar, making the completion process simpler and more feasible.

[0055] Further, when resampling the position of the object, the ground equation of the scene is considered, and the Y coordinate of the object model and the object label is adjusted, which makes the object fit the ground when moving to a new position, and is more consistent with the physical characteristics of the real traffic scene.

[0056] Further, before object-scene fusion, collision detection between objects in 3D space and occlusion detection between objects in 2D images are performed. Collision detection makes the object relationship consistent with reality, and occlusion detection excludes objects that are severely occluded from fusion because these objects will bring noise and affect network training. Diversified object-scene fusion can reduce the overfitting of the network to specific object semantics and increase the utilization of the scene, fully exploiting the structural information of the scene.

[0057] Further, the classic image-level data augmentation does not consider the camera pose, but the camera pose has a great influence on the detection result, because the monocular 3D detection of the traffic scene will use the ground plane prior to calculate the depth of the object based on the vertical position of the object in the image. Therefore, converting the fused scene to 3D space for camera pose perturbation and obtaining new training images can make the network insensitive to the camera pose, thereby improving the robustness of the network.

[0058] Further, the training data obtained by fusing the object and the empty scene only can maximize the flexibility of the object-scene combination, but the synthetic data and the real data have domain differences, which can affect the network performance. Therefore, the training data obtained by mixing and fusing the object and the empty scene and the original scene can achieve a compromise between maximizing the flexibility and reducing the domain difference.

[0059] Further, please supplement the purpose or benefit of setting according to the content of claim 8, and give a principle analysis.

[0060] Further, please supplement the purpose or benefit of setting according to the content of claim 9, and give a principle analysis.

[0061] It can be understood that the beneficial effects of the above-mentioned second aspect to the sixth aspect can be referred to the related description in the first aspect, which will not be repeated here.

[0062] In summary, based on the collected laser radar data and image data, the data is fully labeled or sparsely labeled, and the data is fully decomposed to obtain a scene database and an object database. In training, the object is randomly inserted into the reachable area of the scene, and then the camera pose is disturbed to obtain new training images for network training; the limited labeled data is fully utilized, and the labeling cost can be significantly reduced, and it is suitable for various basic networks.

[0063] The technical solutions of the present application will be further described in detail below with the help of the accompanying drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0064] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiments of the present application will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0065] Figure 1 The overall flowchart of the present application is shown in the figure;

[0066] Figure 2 The schematic diagram of data decomposition and recombination proposed by the present application is shown in the figure;

[0067] Figure 3 The comparative schematic diagram of sparse labeling and full labeling proposed by the present application is shown in the figure;

[0068] Figure 4 The comparative schematic diagram of the training image and the original training image obtained by recombination of the present application is shown in the figure;

[0069] Figure 5The effect diagram of data enhancement designed for the application under four different annotation ratios for three basic models

[0070] Figure 6 The schematic diagram of a computer device provided by an embodiment of the application is shown in FIG. 1.

[0071] Figure 7 The block diagram of a chip provided by an embodiment of the application is shown in FIG. 2. DETAILED DESCRIPTION

[0072] The technical solutions in the embodiments of the application will be clearly and completely described in connection with the drawings in the embodiments of the application. Obviously, the described embodiments are some but not all of the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the application.

[0073] In the description of the application, it should be understood that the terms “include” and “contain” indicate the presence of described features, whole, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, whole, steps, operations, elements, components and / or sets thereof.

[0074] It should also be understood that the terms used in the specification of the application are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in the specification and the appended claims of the application, unless the context clearly indicates otherwise, the singular forms “a”, “an” and “the” are intended to include the plural forms.

[0075] It should be further understood that the term “and / or” used in the specification and the appended claims of the application means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations, for example, A and / or B can represent three cases of A alone, A and B together, and B alone. In addition, the character “ / ” in the application generally represents an “or” relationship between the front and rear associated objects.

[0076] It should be understood that although the terms first, second, third, etc. may be used in the embodiments of the application to describe the preset ranges, etc., these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from each other. For example, the first preset range can also be referred to as the second preset range, and similarly, the second preset range can also be referred to as the first preset range without departing from the scope of the embodiments of the application.

[0077] Depending on the context, the word "if' as used herein can be interpreted to mean "when" or "while" or "in response to determining" or "in response to detecting." Similarly, the phrase "if it is determined" or "if [a stated condition or event] is detected" can be interpreted to mean "upon determining" or "in response to determining" or "upon detecting [the stated condition or event]" or "in response to detecting [the stated condition or event]."

[0078] Various structural diagrams according to the disclosed embodiments of the present application are shown in the accompanying drawings. These diagrams are not drawn to scale in which certain details are exaggerated for clarity and others omitted. The shapes and relative sizes of the various regions, layers, and the relative positions of these in the drawings are shown for example only and can deviate in practice due to manufacturing tolerances or technical limitations, and regions / layers with different shapes, sizes, relative positions can be additionally designed by those skilled in the art according to actual needs.

[0079] The present application provides a data enhancement method for monocular 3D target detection, based on collected lidar data and image data, through full annotation or sparse annotation, constructing object 3D model database, constructing scene database; in the training process, randomly sampling scenes, randomly sampling positions in the reachable area of the scene, randomly sampling objects in the object database, after collision and occlusion detection, inserting the sampling position, and fusing with the scene to obtain the fusion image; converting the fusion image back to 3D space, sampling the disturbed camera pose, and then rendering to obtain a new image for network training; in the training process, mixing sampling empty scenes and original scenes and objects for reorganization; the model trained is used for inferring the 3D position of the object from a single image in a real scene.

[0080] Embodiment 1

[0081] The present application provides a data enhancement method for monocular 3D target detection, including the following steps:

[0082] S1, collecting lidar and image data, based on the lidar and image data collected by the data collection vehicle, obtaining object annotation through full annotation or sparse annotation;

[0083] Please refer to Figure 3 In the case of sparse annotation, for an instance in a video sequence, only the nearest to the camera is annotated, and then the complete annotated video sequence is flexibly increased to control the annotation cost.

[0084] S2, decompose the training image into a 3D object point cloud model with texture information, and remove all objects from the scene to construct an empty scene; construct an object database using all the obtained objects, and construct a scene database using all the obtained original scenes and empty scenes;

[0085] First, the dense depth map of the scene is obtained using a depth completion model, and the foreground object mask is obtained using an object segmentation model; then, the edge distortion depth of the foreground part of the depth map is repaired, and the 3D object point cloud model with texture information is extracted using the camera intrinsic parameter; the foreground object is eliminated to obtain the image and depth map of the empty scene.

[0086] When correcting the edge distortion depth of the foreground object, the distortion depth correction method is as follows:

[0087]

[0088] wherein, represents the distorted depth value; represents the corrected depth value; is the anchor point depth value, which is the depth value of the laser radar image projection point closest to the distorted depth position; calculated from the anchor point and the average of the absolute values of the depth distances of the five laser radar image projection points closest to the anchor point in the image.

[0089] When eliminating the foreground object, for the image, the object elimination model is used for object elimination, and for the depth map, the depth of the foreground region is replaced using the following formula:

[0090]

[0091] wherein, is the ground depth map, which can be calculated from the ground equation of the scene, is the scene depth map, is the empty scene depth map, represents the pixel position of the foreground region, represents the position of the background pixel used to replace the corresponding foreground pixel, which is determined by the following formula:

[0092]

[0093] wherein, is the image column in a certain row coordinate, is the value of and the current scanning pixel coordinate are the same.

[0094] S3, randomly sample a scene in the scene database, randomly sample a position in the reachable area of the scene, randomly sample an object in the object database to move to the sampled position, insert the object into the randomly sampled position after collision and occlusion detection, and fuse the object and the scene to obtain a reorganized image and depth map;

[0095] The reachable region of a scene is generated using lidar data. A reachable region indicator map is first obtained in an overhead view using an existing method for generating a sparse reachable region. The indicator map is then converted to a polar coordinate representation, with the origin of the coordinate system being the location of the body camera. The reachable region indicator map is then completed in rows in the polar coordinate representation, and then converted back to a Cartesian coordinate system to obtain a dense reachable region indicator map.

[0096] When the object is moved from the original position in the overhead view to the sampling position , the position coordinates of the object are converted from to , where is the ground equation of the scene. The coordinates of each point of the 3D model of the object are updated using the following formula:

[0097]

[0098]

[0099] Before fusing the object and the scene, it is determined whether the object collides with the scene in 3D space using a 3D bounding box, and the degree of occlusion of the foreground region of each object is determined in a 2D image using foreground depth. Objects that collide and are severely occluded are filtered out and not fused.

[0100] After the objects are filtered, the 3D model of the object with texture is projected onto the image plane using the camera intrinsic parameters, and then depth blanking is performed pixel by pixel based on the dense depth map of the scene, and the texture with a smaller depth value is retained for display, thereby obtaining the fused scene image and depth map.

[0101] S4, convert the reorganized image to 3D space based on the depth map, randomly perturb the camera pose, and re-render the scene image;

[0102] The scene image after fusing the scene and the object and the scene depth are converted to 3D space using the camera intrinsic parameters to obtain the scene point cloud .

[0103] Then, the perturbation angles of the sampling camera pitch angle and roll angle and are sampled, and the translation perturbation of the sampling camera along the axis is sampled , so as to obtain the perturbed rotation and translation, which are represented by the following two formulas:

[0104]

[0105] Then, the scene point cloud coordinates and the 3D label of the object are updated ​The following two formulas are used for updating:

[0106]

[0107] S5, images are constantly composed of random combinations of objects, scenes and camera poses as training data for neural network training;

[0108] During the training process, the required scenes of the same batch of training data are obtained by mixing sampling from the original scenes and the empty scenes in a certain proportion.

[0109] In particular, in the sparse annotation setting, first, the sparse object database and the empty scene database are used to synthesize training data for pre-training, and then the complete annotated scene is introduced for mixed training.

[0110] S6, when the parameters of the neural network are updated to the maximum number of iterations or meet the termination condition, the student network is retained for real scenes, and when facing real scenes, camera data is used as input, and the position, size and orientation of each target are obtained according to the trained network parameters of the student network. Complete the three-dimensional positioning of the target.

[0111] Those skilled in the art can understand that various aspects of the present application can be implemented as a system, a method or a program product. Therefore, various aspects of the present application can be implemented in the form of a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, which can be collectively referred to as "circuitry", "module" or "platform" here.

[0112] Embodiment 2

[0113] The present application provides a data enhancement system for monocular 3D target detection, which can be used to realize the above-mentioned data enhancement method for monocular 3D target detection. Specifically, the data enhancement system for monocular 3D target detection includes a data module, a recombination module, a rendering module, a training module and an output module.

[0114] The data module collects laser radar and image data, and obtains the annotation of the object through complete annotation or sparse annotation; the training image is decomposed into a 3D object point cloud model with texture information, and all objects obtained are removed from the scene to construct an empty scene; an object database is constructed using all the obtained objects, and a scene database is constructed using all the original scenes and empty scenes.

[0115] The recombination module randomly samples a scene in the obtained scene database, randomly samples a position in the reachable area of the scene, moves an object randomly sampled in the obtained object database to the sampled position, inserts the object into the randomly sampled position after collision and occlusion detection, and fuses the object and the scene to obtain a recombined image;

[0116] a rendering module, converting the obtained recombined image into a 3D space based on the depth map, randomly perturbing the camera pose, and re-rendering the scene image;

[0117] a training module, synthesizing the re-rendered scene image from random recombination of objects, scenes and camera poses as training data for neural network training;

[0118] an output module, when the parameters of the neural network are updated to the maximum number of iterations or meet the termination condition, retaining the student network for real scenes, using camera data as input when facing real scenes, and using the trained network parameters of the student network to infer the positions, sizes and orientations of each target to complete the three-dimensional positioning of the target.

[0119] Embodiment 3

[0120] The application provides a terminal device, which comprises a processor and a memory, the memory is used for storing a computer program, the computer program comprises program instructions, and the processor is used for executing the program instructions stored in the computer storage medium. The processor can be a central processing unit (CPU), and can also be other general-purpose processors, graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components and the like, which are the computing core and control core of the terminal, and are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions to implement a corresponding method flow or a corresponding function. The processor in the embodiment of the application can be used for the operation of the data enhancement method for monocular 3D target detection, which comprises the following steps:

[0121] Collecting laser radar and image data, obtaining the label of the object through full labeling or sparse labeling; decomposing the training image into a 3D object point cloud model with texture information, and removing all the obtained objects from the scene to construct an empty scene; constructing an object database using all the obtained objects, and constructing a scene database using all the original scenes and empty scenes; randomly sampling a scene in the obtained scene database, randomly sampling a position in the reachable area of the scene, moving an object in the obtained object database to the sampled position after collision and occlusion detection, and inserting the object into the randomly sampled position to obtain a reorganized image by fusing the object and the scene; converting the obtained reorganized image into 3D space based on a depth map, randomly perturbing the camera pose, and re-rendering the scene image; combining the re-rendered scene image by random reorganization of objects, scenes and camera poses as training data for neural network training; when the update of the parameters of the neural network reaches the maximum number of iterations or meets the termination condition, the student network is retained for real scenes, and when facing real scenes, the camera data is used as input, the position, size and orientation of each target are obtained by inference according to the trained network parameters of the student network, and the three-dimensional positioning of the target is completed.

[0122] Please refer to Figure 6 , the terminal device is a computer device, and the computer device 60 of the embodiment includes a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61, and the computer program 63 implements the method for data enhancement for monocular 3D target detection in the embodiment when executed by the processor 61. To avoid repetition, details are not repeated here. Alternatively, the computer program 63 implements the functions of each model / unit in the data enhancement system for monocular 3D target detection in the embodiment when executed by the processor 61. To avoid repetition, details are not repeated here.

[0123] The computer device 60 can be a desktop computer, a notebook computer, a palm computer, and a cloud server, etc. The computer device 60 can include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art can understand that Figure 6 The computer device 60 is only an example and does not constitute a limitation on the computer device 60, and can include more or fewer components than shown, or combine certain components, or different components, for example, the computer device can also include an input / output device, a network access device, a bus, etc.

[0124] The processor 61 can be a central processing unit (CPU), and can also be other general-purpose processors, graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0125] The memory 62 can be an internal storage unit of the computer device 60, such as a hard disk or a memory of the computer device 60. The memory 62 can also be an external storage device of the computer device 60, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 60.

[0126] Further, the memory 62 can include both an internal storage unit and an external storage device of the computer device 60. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 can also be used to temporarily store data that has been output or will be output.

[0127] Please refer to Figure 7 , the terminal device is an electronic device 600, and the electronic device 600 is in the form of a general-purpose computing device. The components of the electronic device can include but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0128] Among them, the storage unit stores program codes, and the program codes can be executed by the processing unit 610, so that the processing unit 610 executes the steps of various exemplary embodiments according to the present application described in the method part of the specification. For example, the processing unit 610 can execute the steps as shown in Figure 1 .

[0129] Storage 620 can include a readable medium, such as volatile memory (RAM) 6201 and / or cache memory 6202 in the form of a volatile, non- transitory storage unit, and / or a read only memory (ROM) 6203.

[0130] Storage 620 can also include program / utility 6204 having a set of programs / modules 6205, including operating system, one or more application programs, other program modules, and program data, each of which or a combination thereof, can include implementation of a network environment, for example, in each of or a combination of these examples.

[0131] Bus 630 can represent one or more of several types of bus structures, including a storage bus or bus for storage controller, peripheral bus, graphics bus, processor or local bus using any of a variety of bus structures.

[0132] Electronic device 600 can also communicate with one or more external devices 700 such as a keyboard or pointing device, using one or more communication interfaces 650. Communication interfaces 650 can also enable communication with one or more devices that enable a user to interact with electronic device 600, such as a Bluetooth device, and / or enable communication with one or more devices that enable electronic device 600 to communicate with one or more other computing devices. Such communication can be facilitated, for example, by an I / O interface 650. Additionally, electronic device 600 can communicate with one or more networks, such as a local area network (LAN), a wide area network (WAN), and / or the public network, such as the Internet, using a network adapter 660. Network adapter 660 can be any of a variety of modems, including cable modems, telephone modems, and wireless modems, and the like. It will be appreciated that, while not shown, additional hardware and / or software modules can be used in conjunction with electronic device 600 such as, for example, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0133] Example 4

[0134] The present application further provides a storage medium, specifically a computer readable storage medium (Memory), which is a memory device in the terminal device, used for storing programs and data. It can be understood that the computer readable storage medium herein can include the built-in storage medium in the terminal device, and of course can also include the expansion storage medium supported by the terminal device, and can be any tangible medium containing or storing programs, which can be used by or in combination with an instruction execution system, device or apparatus. The computer readable storage medium provides a storage space, which stores the operating system of the terminal. Moreover, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, which can be one or more computer programs (including program codes). It should be noted that more specific examples (non-exhaustive list) of the computer readable storage medium herein include: an electrical connection with one or more conductive wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0135] The computer readable storage medium further includes a data signal carried in the baseband or as a part of a carrier wave, in which readable program codes are borne. Such a propagated data signal can take various forms, including but not limited to electro-magnetic signal, optical signal or any suitable combination of the above. The readable storage medium can also be any readable medium other than the readable storage medium, which can send, propagate or transmit programs for use by or in combination with an instruction execution system, device or apparatus. The program codes contained on the readable storage medium can be transmitted by any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0136] The program code for performing the operations of the present application can be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, C++, etc., and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. In the case of involving a remote computing device, the remote computing device can be connected to the user's computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, connected to the Internet through an Internet service provider).

[0137] The one or more instructions in the computer-readable storage medium can be loaded and executed by the processor to implement the corresponding steps of the data enhancement method for monocular 3D target detection in the above embodiments; the one or more instructions in the computer-readable storage medium are loaded and executed by the processor to implement the following steps:

[0138] Collecting laser radar and image data, obtaining the label of the object through full labeling or sparse labeling; decomposing the training image into a 3D object point cloud model with texture information, and removing all the obtained objects from the scene to construct an empty scene; constructing an object database using all the obtained objects, and constructing a scene database using all the original scenes and the empty scene; randomly sampling a scene in the obtained scene database, randomly sampling a position in the reachable area of the scene, moving an object randomly sampled in the obtained object database to the sampled position, and inserting the object into the sampled position after collision and occlusion detection to obtain a reorganized image by fusing the object and the scene; converting the obtained reorganized image into a 3D space based on a depth map, randomly perturbing the camera pose, and re-rendering the scene image; combining the re-rendered scene image by random reorganization of the object, the scene and the camera pose to serve as training data for neural network training; when the update of the parameters of the neural network reaches the maximum number of iterations or meets the termination condition, retaining the student network for real scenes, using camera data as input when facing real scenes, and performing inference according to the trained network parameters of the student network to obtain the position, size and orientation of each target, thereby completing the three-dimensional positioning of the target.

[0139] To make the objects, technical solutions, and advantages of the embodiments of the present application clearer, the following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application but not all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0140] To verify the effectiveness of the present application, experiments were conducted on the KITTI test set.

[0141] KITTI dataset is a widely used computer vision algorithm evaluation dataset for autonomous driving scenarios, jointly created by Karlsruhe Institute of Technology in Germany and Toyota Technical Institute in the United States. KITTI contains real image data collected in various scenes such as urban areas, rural areas and highways, covering various degrees of occlusion and truncation. The dataset contains 141 video sequences, including 7481 images for training and 7518 images for testing.

[0142] According to the recognized division method, the training data is divided into a training set containing 96 video sequences and 3712 images, and a validation set containing 45 video sequences and 3769 images.

[0143] The dataset is mainly used for detecting cars, pedestrians and cyclists, and is divided into three levels of difficulty according to the difficulty of detection: easy, medium and difficult. The standard evaluation method is to compare the average precision of each method under 3D view and BEV two different views.

[0144] Please refer to Figure 5 The method of the present application can be applied to various labeling scale settings and various basic models in a plug-in manner. In order to verify the effectiveness of the method, three representative high-precision monocular 3D detection models are selected as basic models. Figure 5 It is shown that the method designed by the present application can significantly improve the precision of the basic model under various labeling scale settings, and is not limited by the effect of the basic model itself. At the same time, the method designed by the present application can achieve the effect of training the basic model under complete labeling with only 10% labeling.

[0145] Further, the present embodiment compares the results with the currently best published method on the test set, and the comparison results are shown in Table 1:

[0146] Table 1

[0147]

[0148] The results show that the method of the present application achieves the best effect under 3D and BEV two different views, effectively proving the effectiveness of the module of the present application.

[0149] Please refer to Figure 4 In order to more intuitively show the new training samples constructed by the enhancement method of the present application, the original training images and the reorganized training images are compared. The present application greatly increases the flexibility of object, scene and camera combination, greatly improves the diversity of data, and greatly increases the utilization efficiency of limited data, so as to train a robust monocular 3D detection model.

[0150] In summary, the data enhancement method and system for monocular 3D target detection can be applied to various basic models without modification, can greatly improve the detection accuracy of the basic model without increasing the inference cost, and has good universality. The object and scene in the original data are completely decomposed for the first time, and the object, scene and camera pose are freely recombined to generate a large amount of new training data, which can significantly improve the performance and robustness of the model.

[0151] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is exemplified, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the application. The specific working process of the units and modules in the system can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0152] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in a certain embodiment can be referred to the related description of other embodiments.

[0153] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the disclosed embodiments of the present application can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0154] In the embodiments of the present application, it should be understood that the disclosed apparatus / terminal and method can be implemented in other manners. For example, the embodiments of the apparatus / terminal described above are merely schematic, and the division of the modules or units is merely logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the units can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0155] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0156] In addition, each functional unit in the various embodiments of the present application can be integrated into a processing unit, or each unit can be a physically independent unit, or two or more units can be integrated into a unit. The integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0157] The integrated module / unit, if implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, all or part of the flow of the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. When the processor executes the computer program, the steps of each method embodiment described above can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the computer readable medium can include or exclude content according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0158] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps in one or more flow or blocks

[0159] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps in one or more flow or blocks

[0160] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks ​ means for functionally implementing the steps in one or more flow or blocks

[0161] The above merely provides the technical idea of the present application, and cannot be used to limit the protection scope of the present application. Any modification made according to the technical idea of the present application, and on the basis of the technical solutions, falls within the protection scope of the claims of the present application.

Claims

1. A data augmentation method for monocular 3D object detection, characterized in that, Comprise the following steps: Collecting laser radar and image data, obtaining the label of the object through full annotation or sparse annotation; decompose the training image into a 3D object point cloud model with texture information, and remove all the objects obtained from the scene to construct an empty scene; construct an object database using all the objects obtained, and construct a scene database using all the original scenes and empty scenes; The construction of the empty scene is specifically: First, use the depth completion model to obtain a dense depth map of the scene, and use the object segmentation model to obtain the foreground object mask; Then, the edge distortion depth of the foreground part of the depth map is repaired, and the 3D object point cloud model with texture information is extracted using the camera intrinsic parameter; The foreground object is eliminated to obtain the image and depth map of the empty scene; Randomly sample the scene in the obtained scene database, randomly sample the position in the reachable area of the scene, and move the object in the obtained object database to the sampling position after collision and occlusion detection, and insert it into the randomly sampled position. The object and the scene are fused to obtain the reorganized image, and the object and the scene are fused to obtain the reorganized image and depth map, which is specifically: Use the laser radar data to generate the reachable area of the scene, first use the existing method to generate a sparse reachable area to obtain the reachable area indication map under the bird's eye view, then convert the indication map to polar coordinates to represent the reachable area, and then complete the indication map in the polar coordinate system. Complete, and then convert it back to the Cartesian coordinate system to obtain a dense reachable area indication map; Before fusing the object and the scene, first use the 3D annotation box in the 3D space to judge whether the object collides, and use the foreground depth in the 2D image to judge the degree of occlusion of each object foreground area. The objects that collide and are severely occluded will be screened out and not fused later; After screening the object, project the 3D model with texture of the object to the image plane through the camera intrinsic parameter, and then perform pixel-by-pixel depth blanking according to the dense depth map of the scene. The texture with smaller depth value is retained for display, thereby obtaining the fused scene image and depth map; Convert the obtained reorganized image to 3D space based on the depth map, randomly perturb the camera pose, and re-render the scene image, which is specifically: The scene image after fusing the scene and the object and scene depth Convert to 3D space using camera intrinsic to get scene point cloud ; Then, perturbation angles of the pitch and roll angles of the sampling camera are sampled and , the translation perturbation of the sampling camera along axis is sampled , so as to acquire the rotation and translation perturbation; Update the scene point cloud coordinates and 3D labels of objects ; The re-rendered scene image is composed of random recombination of objects, scenes and camera poses, which is used as training data for neural network training; When the parameters of the neural network are updated to the maximum number of iterations or meet the termination condition, the student network is retained for real scenes. When facing real scenes, use camera data as input, and use the network parameters trained by the student network to infer the position, size and orientation of each target to complete the three-dimensional positioning of the target.

2. The data augmentation method for monocular 3D object detection according to claim 1, wherein, When correcting the edge distortion depth of the foreground object, the distortion depth correction method is as follows: wherein, represents a distorted depth value; represents a corrected depth value; is an anchor depth value; is calculated from the anchor and the average of the absolute values of the depth distances of the five closest laser radar image projection points to the anchor in the image.

3. The data augmentation method for monocular 3D object detection according to claim 1, wherein, When eliminating the foreground object, use the object elimination model to eliminate the object. For the depth map, replace the depth of the foreground area: wherein, is the ground depth map, is the scene depth map, is the empty scene depth map, represents the pixel position of the foreground region, represents the position of the background pixel used to replace the corresponding foreground pixel.

4. The data augmentation method for monocular 3D object detection according to claim 1, wherein, The object is moved from the original position in the bird's eye view to the sampling position The position coordinates of the object are converted from to , the coordinates of the individual points of the 3D model of the object are updated as follows: wherein is the scene ground equation.

5. The data augmentation method for monocular 3D object detection according to claim 1, wherein, The rotation and translation of the perturbation are as follows: wherein, is a rotation matrix of the disturbance, is a translation vector of the disturbance, is a pitch angle of the disturbance, is a yaw angle of the disturbance, is a depth value of the disturbance.

6. The data augmentation method for monocular 3D object detection according to claim 1, wherein, During training, under the condition of sparse annotation, first use the sparse object database and the empty scene database to synthesize training data for pre-training, and then introduce the complete annotation scene for mixed training.

7. A data augmentation system for monocular 3D object detection, characterized in that, Comprise: The data module collects laser radar and image data, obtains the labels of objects through full labeling or sparse labeling, decomposes the training image into a 3D object point cloud model with texture information, removes all the obtained objects from the scene to construct an empty scene, specifically: first, a depth completion model is used to obtain a dense depth map of the scene, and an object segmentation model is used to obtain a foreground object mask; then, the edge distortion depth of the foreground part of the depth map is repaired, and a 3D object point cloud model with texture information is extracted using the camera intrinsic parameter; the foreground object is removed to obtain the image and depth map of the empty scene; an object database is constructed using all the obtained objects, and a scene database is constructed using all the original scenes and empty scenes; The reorganization module randomly samples a scene in the obtained scene database, randomly samples a position in the reachable area of the scene, moves an object in the obtained object database to the sampled position after collision and occlusion detection, and inserts the object into the randomly sampled position to fuse the object and the scene to obtain a reorganized image, specifically: The reachable area of the scene is generated using laser radar data, first, an existing method for generating a sparse reachable area is used to obtain a reachable area indication map under a bird's eye view, then the indication map is converted to a polar coordinate representation, wherein the origin of the coordinate system is the position of the camera, then the reachable area indication map is completed in rows in the polar coordinate representation, and finally converted back to the Cartesian coordinate system to obtain a dense reachable area indication map; Before fusing the object and the scene, it is first determined whether the object collides in the 3D space using a 3D label box, and the degree of occlusion of the foreground area of each object is determined in the 2D image using the foreground depth, and the objects that collide and are severely occluded are screened out and not fused subsequently; After screening the objects, the 3D model with texture of the object is projected onto the image plane through the camera intrinsic parameter, and then the depth is blanked out pixel by pixel according to the dense depth map of the scene, and the texture with a smaller depth value is retained for display, thereby obtaining the fused scene image and depth map; The rendering module converts the obtained reorganized image to the 3D space based on the depth map, randomly perturbs the camera pose, and re-renders the scene image, specifically: Scene image after fusing scene and object and scene depth Convert to 3D space using camera intrinsic to get scene point cloud Then, sample perturbation angles of camera pitch and roll angles and Sample translation perturbation of camera along axis , so as to obtain perturbed rotation and translation Update the scene point cloud coordinates and 3D labels of objects ; The training module combines the re-rendered scene image by random reorganization of objects, scenes and camera poses as training data for neural network training; The output module retains the student network for real scenes when the parameters of the updated neural network reach the maximum number of iterations or meet the termination condition, and uses the camera data as input to infer the position, size and orientation of each target according to the trained network parameters of the student network to complete the three-dimensional positioning of the target.

Images