Data enhancement method and system for monocular 3D target detection

Abstract

The invention discloses a data enhancement method and system for monocular 3D target detection. The method comprises the following steps: respectively constructing a scene database and an object database; randomly sampling a scene in the scene database, randomly sampling a position in a reachable area of the scene, randomly sampling an object in the object database, inserting the random sampling position after collision and shielding detection, and fusing the object and the scene to obtain a recombined image and a depth map; based on the depth map, the recombined image is converted to a 3D space, random disturbance is carried out on the camera attitude, and training images at different visual angles are generated; the object, the scene and the camera attitude are randomly recombined to form a continuously generated new image which serves as training data of a neural network, and the performance of a monocular 3D target detection model is improved; existing data resources can be efficiently utilized, and the model performance is remarkably improved; the application of a camera-based scene perception algorithm in the automatic driving industry is facilitated, and the rapid landing and development of related industries are facilitated.

Description

Background Art

[0002] Monocular three-dimensional (3D) object detection uses a single RGB image to calculate the 3D properties of objects in the scene, such as the 3D position, orientation, and size of the objects. Compared with the high-cost LiDAR solution and multi-view solution, monocular 3D detection provides a low-cost solution for autonomous driving scene perception. However, due to the pathological nature of monocular 3D vision, the monocular 3D detection task is more challenging.

[0003] Due to the complex and entangled visual variations from diverse scenes, object types, and camera poses, training a robust deep learning-based monocular 3D model requires a large amount of annotated data. This can indeed help the network learn robust feature representations, thereby alleviating the pathological problem. However, compared with traditional 2D object detection, annotation for 3D object detection is more difficult and expensive. Limited training data may result in low accuracy of monocular 3D detection models and difficulty in applying them to real scenes. Therefore, developing a data-efficient solution is a meaningful but challenging task for monocular 3D detection.

[0004] Existing works have made various efforts to alleviate the limited training data. Image-level enhancement is the basic solution. Copy-paste methods can generate more target labels for monocular 3D detection, but these methods are limited when inserting scene positions due to the fixed occupancy of existing objects in the scene. Semi-supervised solutions directly alleviate this problem by directly expanding the scale of training data, but are limited by the quality of pseudo-labels.

[0005] In addition, camera pose is an important factor affecting monocular 3D detection data. Existing methods attempt to learn camera pose online and correct feature maps to enhance the robustness of the model under pose disturbances. However, this method does not consider how to directly reduce the sensitivity of the model to camera pose by means of data perturbation. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a data enhancement method and system for monocular 3D target detection in view of the deficiencies in the above-mentioned prior art, so as to make full use of limited data and enhance the robustness of the model, so as to solve the technical problems that monocular 3D detection data is limited and the objects, scenes and camera postures in the collected data are tightly entangled, which is conducive to the application of camera-based scene perception algorithms in the autonomous driving industry and the rapid implementation and development of related industries.

[0007] The present invention adopts the following technical solutions: A data enhancement method for monocular 3D object detection comprises the following steps: Collect lidar and image data, obtain object annotations through full annotation or sparse annotation; decompose the training image into a 3D object point cloud model with texture information, and remove all the obtained objects from the scene to construct an empty scene; use all the obtained objects to build an object database, and use all the original scenes and empty scenes to build a scene database; Randomly sample scenes in the obtained scene database, randomly sample positions in the reachable area of ​​the scene, randomly sample objects in the obtained object database and move them to the sampling positions, insert the random sampling positions after collision and occlusion detection, and fuse the objects and scenes to obtain a reconstructed image; Based on the depth map, the reconstructed image is converted into 3D space, the camera posture is randomly perturbed, and the scene image is re-rendered; The re-rendered scene images are synthesized by random reorganization of objects, scenes and camera poses as training data for neural network training; When the parameters of the updated neural network reach the maximum number of iterations or meet the termination conditions, the student network is retained for real scenes. When facing real scenes, the camera data is used as input, and the position, size and orientation of each target are inferred based on the network parameters trained by the student network to complete the three-dimensional positioning of the target.

[0008] Preferably, constructing an empty scene is specifically as follows: First, use the depth completion model to obtain a dense depth map of the scene, and use the object segmentation model to obtain the foreground object mask; Then, the edge distortion depth of the foreground part of the depth map is repaired, and the camera intrinsic parameters are used to extract the 3D object point cloud model with texture information; Eliminate the foreground objects to obtain the image and depth map of the empty scene.

[0009] Preferably, when correcting the distorted depth of the foreground object edge, the correction method of the distorted depth is as follows:

[0010] in, Depth value representing distortion; Indicates the corrected depth value; is the anchor point depth value; It is calculated by the average of the absolute values ​​of the depth distances between the anchor point and the five LiDAR image projection points closest to the anchor point in the image.

[0011] Preferably, when removing the foreground object, an object removal model is used to remove the object, and for the depth map, the depth of the foreground area is replaced:

[0012] in, is the ground depth map, is the scene depth map, is the empty scene depth map, represents the pixel position of the foreground area, Represents the position of the background pixel that is used to replace the corresponding foreground pixel.

[0013] Preferably, the object and the scene are fused to obtain the reconstructed image and the depth map as follows: Generate the reachable area of ​​the scene using LiDAR data. First, use the existing method of generating sparse reachable areas to obtain a reachable area indication map from a bird's-eye view. Then convert the indication map to polar coordinates, where the origin of the coordinate system is the location of the camera. Then, complete the polar coordinates row by row, and then convert back to the Cartesian coordinate system to obtain a dense reachable area indication map. Before fusing objects and scenes, use 3D annotation boxes in 3D space to determine whether objects collide. Use foreground depth in 2D images to determine the degree of occlusion of the foreground area of ​​each object. Objects that collide or are severely occluded will be screened out and not be fused later. After screening the objects, the textured 3D model of the objects is projected onto the image plane through the camera intrinsic parameters. Then, according to the dense depth map of the scene, the depth is hidden pixel by pixel, and the textures with smaller depth values ​​are retained for display, thus obtaining the fused scene image and depth map.

[0014] Preferably, the object is moved from its original position in the bird's-eye view Move to sampling position When , the position coordinates of the object are given by Convert to , the coordinates of each point of the object 3D model Updated as follows:

[0015] in, is the scene ground equation.

[0016] Preferably, the camera posture is randomly disturbed and the scene image is re-rendered as follows: The scene image after the scene and object are fused and scene depth Use the camera's intrinsic parameters to convert to 3D space to get the scene point cloud ; Then, sample the perturbation angles of the camera pitch and roll angles and , the sampling camera along Translational perturbation of the axis , thereby obtaining the rotation and translation of the disturbance; Finally update the scene point cloud coordinates and object 3D labels .

[0017] Preferably, the rotation and translation of the perturbation are as follows:

[0018] in, is the perturbed rotation matrix, is the translation vector of the disturbance, is the pitch angle of the disturbance, is the yaw angle of the disturbance, is the depth of the disturbance.

[0019] 9. The data enhancement method for monocular 3D target detection according to claim 1 is characterized in that, during the training process, under the sparse annotation setting, a sparse object database and an empty scene database are first used to synthesize training data for pre-training, and then a fully annotated scene is introduced for hybrid training.

[0020] In a second aspect, an embodiment of the present invention provides a data enhancement system for monocular 3D target detection, comprising: The data module collects lidar and image data, obtains object annotations through full annotation or sparse annotation; decomposes the training image into a 3D object point cloud model with texture information, removes all the obtained objects from the scene, and constructs an empty scene; uses all the obtained objects to build an object database, and uses all the original scenes and empty scenes to build a scene database; The recombinant module randomly samples scenes from the obtained scene database, randomly samples positions in the reachable area of ​​the scene, randomly samples objects from the obtained object database and moves them to the sampling positions. After collision and occlusion detection, the random sampling positions are inserted, and the objects and scenes are fused to obtain a recombinant image. The rendering module converts the reconstructed image into 3D space based on the depth map, randomly perturbs the camera posture, and re-renders the scene image; The training module synthesizes the re-rendered scene images by random recombination of objects, scenes and camera poses as training data for neural network training; The output module, when the parameters of the updated neural network reach the maximum number of iterations or meet the termination conditions, retains the student network for real scenes. When facing real scenes, the camera data is used as input, and the position, size and orientation of each target are inferred based on the network parameters trained by the student network to complete the three-dimensional positioning of the target.

[0021] In a third aspect, a computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned data enhancement method for monocular 3D target detection when executing the computer program.

[0022] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned data enhancement method for monocular 3D target detection.

[0023] In a fifth aspect, a chip comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned data enhancement method for monocular 3D target detection when executing the computer program.

[0024] In a sixth aspect, an embodiment of the present invention provides an electronic device, including a computer program, which, when executed by the electronic device, implements the steps of the above-mentioned data enhancement method for monocular 3D target detection.

[0025] Compared with the prior art, the present invention has at least the following beneficial effects: A data enhancement method for monocular 3D target detection is divided into data acquisition, data decomposition and data reorganization training stages. In the data acquisition stage, lidar data and image data are acquired, and objects in the scene are fully or sparsely labeled. In the decomposition stage, the image and depth map of the original scene are decomposed into an empty scene and a 3D model of an object with texture, and a scene database and an object database are constructed. In the data reorganization training stage, scenes, objects and camera postures are continuously randomly combined to synthesize new training data, so that limited data can be fully utilized and the generalization performance of the network can be enhanced.

[0026] Furthermore, in the case of sparse annotation, for an instance in a video sequence, it is only annotated when it is closest to the camera. These samples can be resampled to obtain training samples at a farther position, which can significantly reduce the annotation cost. At the same time, the fully annotated video sequence can be flexibly increased to flexibly control the annotation cost.

[0027] Furthermore, when decomposing objects and scenes, using a textured 3D point cloud model of an object can make it possible to edit the 3D position change of the object at a low cost. Repairing the edge distortion depth of the foreground part of the depth map can make the object 3D model more accurate and usable.

[0028] Furthermore, backgrounding the foreground part of the depth map to remove objects can make camera pose perturbations feasible.

[0029] Furthermore, based on the LiDAR data, the reachable area and obstacle area can be preliminarily divided in the bird's-eye view. Due to the sparse scanning characteristics of the LiDAR, many areas are not classified. Completing the reachable area indicator map from the bird's-eye view allows the complete scene reachable area to be fully considered when sampling positions. At the same time, completing the indicator map in polar coordinates takes into account the scanning characteristics of the LiDAR, making the completion process simpler and more feasible.

[0030] Furthermore, when resampling the object position, the ground equation of the scene is taken into account, and the Y coordinates of the object model and the object label are adjusted. This ensures that when the object moves to a new position, the object as a whole fits the ground, which is more in line with the physical characteristics of real traffic scenes.

[0031] Furthermore, before object-scene fusion, collision detection between objects is performed in 3D space and occlusion detection between objects is performed in 2D images. Collision detection makes the relationship between objects consistent with reality, and occlusion detection prevents objects that are severely occluded from being included in the fusion, because these objects will bring noise and affect network training. Diversified object-scene fusion can reduce the network's overfitting of specific object semantics, increase the utilization of scenes, and fully explore the structural information of scenes.

[0032] Furthermore, classic image-level data enhancement does not consider the camera's posture, but the camera's posture has a great impact on the detection results, because the monocular 3D detection of traffic scenes uses the ground plane prior to calculate the depth of the object based on the vertical position of the object in the image. Therefore, converting the fused scene to 3D space to perturb the camera's posture and then obtaining new training images can make the network insensitive to the camera's posture, thereby improving the robustness of the network.

[0033] Furthermore, training data obtained by fusing objects only with empty scenes can maximize the flexibility of object-scene combinations, but synthetic data and real data have domain differences, which will affect network performance. Therefore, mixing and fusing objects with empty scenes and original scenes to obtain training data can achieve a compromise between maximizing combination flexibility and reducing domain differences.

[0034] Furthermore, please provide additional explanation of the purpose or benefit of the setting based on the content of claim 8 and provide an analysis of the principle.

[0035] Furthermore, please provide additional explanation of the purpose or benefit of the setting based on the content of claim 9 and provide an analysis of the principle.

[0036] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0037] In summary, the present invention fully or sparsely annotates the data based on the collected lidar data and image data; and then completely decomposes the data to obtain a scene database and an object database. During training, objects are randomly inserted into the reachable area of ​​the scene, and then the camera posture is perturbed to obtain new training images for network training; it makes full use of limited annotated data and can significantly reduce the annotation cost, and is applicable to a variety of basic networks.

[0038] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 It is the overall flow chart of the present invention; Figure 2 A schematic diagram of data decomposition and reorganization proposed by the present invention; Figure 3 A schematic diagram showing the comparison between sparse annotation and complete annotation proposed in the present invention; Figure 4 A schematic diagram showing the comparison between the training image reconstructed by the present invention and the original training image; Figure 5 Schematic diagram of the effect of data enhancement designed for the present invention on three basic models at four different annotation scales Figure 6 A schematic diagram of a computer device provided by an embodiment of the present invention; Figure 7 The block diagram of a chip provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] In the description of the present invention, it should be understood that the terms “include” and “comprises” indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0043] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0044] It should be further understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects are in an "or" relationship.

[0045] It should be understood that, although the terms first, second, third, etc. may be used to describe preset ranges, etc. in the embodiments of the present invention, these preset ranges should not be limited to these terms. These terms are only used to distinguish preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0046] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0047] Various structural schematic diagrams of the embodiments disclosed in the present invention are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0048] The present invention provides a data enhancement method for monocular 3D target detection. Based on the collected laser radar data and image data, a 3D model database of an object and a scene database are constructed through complete or sparse annotation. During the training process, scenes are randomly sampled, positions are randomly sampled in a reachable area of ​​the scene, and objects are randomly sampled in an object database. The sampled positions are inserted after collision and occlusion detection, and the objects are fused with the scene to obtain a fused image. The fused image is converted back to 3D space, the disturbed camera posture is sampled, and then a new image is rendered to be used for network training. During the training process, mixed sampled empty scenes and original scenes are recombined with objects. The trained model is used to infer the 3D position of an object from a single image in a real scene.

[0049] Example 1 The present invention provides a data enhancement method for monocular 3D target detection, comprising the following steps: S1. Collecting lidar and image data, and obtaining object annotations through complete annotation or sparse annotation based on the lidar and image data collected by the data collection vehicle; See also Figure 3 ,In the sparse annotation case, for an instance in a video sequence, it is only annotated when it is closest to the camera, and then the ,completely annotated video sequences are flexibly increased to control the ,annotation cost.

[0050] S2, decomposing the training image into a 3D object point cloud model with texture information, and removing all objects from the scene to construct an empty scene; using all the obtained objects to construct an object database, and using all the obtained original scenes and empty scenes to construct a scene database; First, the depth completion model is used to obtain a dense depth map of the scene, and the object segmentation model is used to obtain the foreground object mask; then the edge distortion depth of the foreground part of the depth map is repaired, and the camera intrinsic parameters are used to extract the 3D object point cloud model with texture information; the foreground object is eliminated to obtain the image and depth map of the empty scene.

[0051] When correcting the distorted depth of the foreground object edge, the correction method of the distorted depth is as follows:

[0052] in, Depth value representing distortion; Indicates the corrected depth value; is the anchor point depth value, which is the depth value of the LiDAR image projection point closest to the distorted depth position; It is calculated by the average of the absolute values ​​of the depth distances between the anchor point and the five LiDAR image projection points closest to the anchor point in the image.

[0053] When removing the foreground object, for the image, the object removal model is used to remove the object, and for the depth map, the depth of the foreground area is replaced by the following formula:

[0054] in, is the ground depth map, which can be calculated by the ground equation of the scene. is the scene depth map, is the empty scene depth map, represents the pixel position of the foreground area, represents the position of the background pixel used to replace the corresponding foreground pixel, which is determined by the following formula:

[0055] in, is an image The coordinate of a row in the column, is the value and coordinates of the current scanned pixel same.

[0056] S3, randomly sample scenes in the scene database, randomly sample positions in the reachable area of ​​the scene, randomly sample objects in the object database and move them to the sampling positions, insert the random sampling positions after collision and occlusion detection, and fuse the objects and scenes to obtain a reconstructed image and depth map; The reachable area of ​​the scene is generated using lidar data. The existing method for generating sparse reachable areas is first used to obtain a reachable area indication map from a bird's-eye view. The indication map is then converted to polar coordinates, where the origin of the coordinate system is the location of the camera. The polar coordinates are then completed row by row and then converted back to the Cartesian coordinate system to obtain a dense reachable area indication map.

[0057] Move the object from its original position in the bird's eye view Move to sampling position When , the position coordinates of the object are given by Convert to ,in, is the ground equation of the scene. The coordinates of each point of the object 3D model Use the following formula to update:

[0058]

[0059] Before fusing objects and scenes, we first use 3D annotation boxes in 3D space to determine whether objects collide. In 2D images, we use foreground depth to determine the degree of occlusion of the foreground area of ​​each object. Objects that collide or are severely occluded will be screened out and not fused later.

[0060] After screening the objects, the textured 3D model of the objects is projected onto the image plane through the camera intrinsic parameters. Then, according to the dense depth map of the scene, the depth is hidden pixel by pixel, and the textures with smaller depth values ​​are retained for display, thus obtaining the fused scene image and depth map.

[0061] S4, converting the reconstructed image into 3D space based on the depth map, randomly perturbing the camera posture, and re-rendering the scene image; The scene image after the scene and object are fused and scene depth Use the camera's intrinsic parameters to convert to 3D space to get the scene point cloud ; Then, sample the perturbation angles of the camera pitch and roll angles and , the sampling camera along Translational perturbation of the axis , thereby obtaining the rotation and translation of the disturbance, which is expressed by the following two equations:

[0062] Then, the scene point cloud coordinates and the 3D labels of the objects Use the following two formulas to update:

[0063] S5, images are continuously synthesized by random reorganization of objects, scenes and camera poses as training data for neural network training; During the training process, the required scenes of the same batch of training data are sampled from the original scenes and empty scenes in a certain proportion.

[0064] In particular, under the sparse annotation setting, the sparse object database and the empty scene database are first used to synthesize the training data for pre-training, and then the fully annotated scenes are introduced for hybrid training.

[0065] S6. When the parameters of the updated neural network reach the maximum number of iterations or meet the termination conditions, the student network is retained for real scenes. When facing real scenes, the camera data is used as input, and the position, size and orientation of each target are inferred based on the network parameters trained by the student network to complete the three-dimensional positioning of the target.

[0066] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "platforms".

[0067] Example 2 The present invention provides a data enhancement system for monocular 3D target detection, which can be used to implement the above-mentioned data enhancement method for monocular 3D target detection. Specifically, the data enhancement system for monocular 3D target detection includes a data module, a reorganization module, a rendering module, a training module and an output module.

[0068] The data module collects lidar and image data, obtains object annotations through full annotation or sparse annotation; decomposes the training image into a 3D object point cloud model with texture information, removes all the obtained objects from the scene, and constructs an empty scene; constructs an object database using all the obtained objects, and constructs a scene database using all the original scenes and empty scenes; The recombinant module randomly samples scenes from the obtained scene database, randomly samples positions in the reachable area of ​​the scene, randomly samples objects from the obtained object database and moves them to the sampling positions. After collision and occlusion detection, the random sampling positions are inserted, and the objects and scenes are fused to obtain a recombinant image. The rendering module converts the reconstructed image into 3D space based on the depth map, randomly perturbs the camera posture, and re-renders the scene image; The training module synthesizes the re-rendered scene images by random recombination of objects, scenes and camera poses as training data for neural network training; The output module, when the parameters of the updated neural network reach the maximum number of iterations or meet the termination conditions, retains the student network for real scenes. When facing real scenes, the camera data is used as input, and the position, size and orientation of each target are inferred based on the network parameters trained by the student network to complete the three-dimensional positioning of the target.

[0069] Example 3 The present invention provides a terminal device, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, graphics processors (GPU), tensor processors (TPU), digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the data enhancement method for monocular 3D target detection, including: Collect lidar and image data, obtain object annotations through complete or sparse annotation; decompose the training image into a 3D object point cloud model with texture information, and remove all the objects obtained from the scene to construct an empty scene; use all the objects obtained to build an object database, and use all the original scenes and empty scenes to build a scene database; randomly sample scenes in the obtained scene database, randomly sample positions in the reachable area of ​​the scene, randomly sample objects in the obtained object database and move them to the sampling positions, insert the random sampling positions after collision and occlusion detection, and fuse the objects and scenes to obtain a reconstructed image; based on the depth map, convert the obtained reconstructed image into 3D space, randomly perturb the camera posture, and re-render the scene image; synthesize the re-rendered scene image by random recombination of objects, scenes and camera postures as training data for neural network training; when the parameters of the neural network are updated to the maximum number of iterations or the termination condition is met, retain the student network for real scenes. When facing real scenes, use camera data as input, and infer the position, size and orientation of each target based on the network parameters trained by the student network to complete the three-dimensional positioning of the target.

[0070] See also Figure 6, the terminal device is a computer device, and the computer device 60 of this embodiment includes: a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When the computer program 63 is executed by the processor 61, the data enhancement method for monocular 3D target detection in the embodiment is implemented. To avoid repetition, it is not described one by one here. Alternatively, when the computer program 63 is executed by the processor 61, the functions of each model / unit in the data enhancement system for monocular 3D target detection in the embodiment are implemented. To avoid repetition, it is not described one by one here.

[0071] The computer device 60 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art will appreciate that Figure 6 This is only an example of the computer device 60 and does not constitute a limitation of the computer device 60. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device may also include input and output devices, network access devices, buses, etc.

[0072] The processor 61 may be a central processing unit (CPU), or other general-purpose processors, graphics processing units (GPU), tensor processing units (TPU), digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0073] The memory 62 may be an internal storage unit of the computer device 60, such as a hard disk or memory of the computer device 60. The memory 62 may also be an external storage device of the computer device 60, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the computer device 60.

[0074] Furthermore, the memory 62 may include both an internal storage unit of the computer device 60 and an external storage device. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 may also be used to temporarily store data that has been output or is to be output.

[0075] See also Figure 7 The terminal device is an electronic device 600, which is in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0076] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention described in the above method section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .

[0077] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0078] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0079] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0080] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, etc.

[0081] Example 4 The present invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the terminal device and extended storage media supported by the terminal device, and can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by a processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that more specific examples (non-exhaustive list) of the computer-readable storage medium here include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0082] Computer readable storage media also include data signals propagated in baseband or as part of a carrier wave, which carry readable program codes. Such propagated data signals can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium can also be any readable medium other than a readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the readable storage medium can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0083] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0084] One or more instructions stored in a computer-readable storage medium may be loaded and executed by a processor to implement the corresponding steps of the data enhancement method for monocular 3D object detection in the above embodiment; one or more instructions in the computer-readable storage medium are loaded and executed by the processor as follows: Collect lidar and image data, obtain object annotations through complete or sparse annotation; decompose the training image into a 3D object point cloud model with texture information, and remove all the objects obtained from the scene to construct an empty scene; use all the objects obtained to build an object database, and use all the original scenes and empty scenes to build a scene database; randomly sample scenes in the obtained scene database, randomly sample positions in the reachable area of ​​the scene, randomly sample objects in the obtained object database and move them to the sampling positions, insert the random sampling positions after collision and occlusion detection, and fuse the objects and scenes to obtain a reconstructed image; based on the depth map, convert the obtained reconstructed image into 3D space, randomly perturb the camera posture, and re-render the scene image; synthesize the re-rendered scene image by random recombination of objects, scenes and camera postures as training data for neural network training; when the parameters of the neural network are updated to the maximum number of iterations or the termination condition is met, retain the student network for real scenes. When facing real scenes, use camera data as input, and infer the position, size and orientation of each target based on the network parameters trained by the student network to complete the three-dimensional positioning of the target.

[0085] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings here can usually be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0086] To verify the effectiveness of the present invention, experiments were conducted on the KITTI test set.

[0087] The KITTI dataset was jointly created by the Karlsruhe Institute of Technology and Toyota Research Institute of America. It is currently a widely used dataset for evaluating computer vision algorithms in autonomous driving scenarios. KITTI contains real image data collected from multiple scenes such as urban areas, rural areas, and highways, covering various degrees of occlusion and truncation. The dataset contains 141 video sequences, including 7481 images for training and 7518 images for testing.

[0088] The present invention divides the training data into a training set including 96 video sequences and 3712 images and a verification set including 45 video sequences and 3769 images according to a generally recognized division method.

[0089] The dataset is mainly used to detect cars, pedestrians and cyclists, and is divided into three levels according to the difficulty of detection: easy, medium and difficult. The standard evaluation method is to compare the average accuracy of each method in two different views: 3D view and BEV.

[0090] See also Figure 5 The method of the present invention can be applied to a variety of annotation scale settings and a variety of basic models in a plug-in manner. In order to verify the effectiveness of the method, three representative high-precision monocular 3D detection models are selected as basic models. Figure 5 This shows that the method designed by the present invention can significantly improve the accuracy of the basic model under various annotation ratio settings, and is not limited by the effect of the basic model itself. At the same time, the method designed by the present invention can achieve the effect of training the basic model under complete annotation with only 10% annotation.

[0091] Furthermore, this embodiment is compared with the currently best published method on the test set, and the comparison results are shown in Table 1: Table 1

[0092] The results show that the method of the present invention achieves the best results in both 3D and BEV views, which effectively proves the effectiveness of the module of the present invention.

[0093] See also Figure 4 In order to more intuitively demonstrate the new training samples constructed by the enhancement method of the present invention, the original training images and the reconstructed training images are compared. The present invention greatly increases the flexibility of the combination of objects, scenes and cameras, greatly improves the diversity of data, and greatly increases the efficiency of the use of limited data, so that a robust monocular 3D detection model can be trained.

[0094] In summary, the data enhancement method and system for monocular 3D target detection of the present invention can be applied to a variety of basic models without modification, and can greatly improve the detection accuracy of the basic model without increasing the inference cost, and has good versatility. The present invention completely decomposes the objects and scenes in the original data for the first time, and freely reorganizes the objects, scenes and camera postures to generate a large amount of new training data, which can significantly improve the performance and robustness of the model.

[0095] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0096] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0097] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0098] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0099] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0100] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0101] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0102] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0103] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0104] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0105] The above contents are only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A data enhancement method for monocular 3D object detection, characterized in that: The following steps are involved: Collect lidar and image data, obtain object annotations through full annotation or sparse annotation; decompose the training image into a 3D object point cloud model with texture information, and remove all the obtained objects from the scene to construct an empty scene; use all the obtained objects to build an object database, and use all the original scenes and empty scenes to build a scene database; Randomly sample scenes in the obtained scene database, randomly sample positions in the reachable area of ​​the scene, randomly sample objects in the obtained object database and move them to the sampling positions, insert the random sampling positions after collision and occlusion detection, and fuse the objects and scenes to obtain a reconstructed image; Based on the depth map, the reconstructed image is converted into 3D space, the camera posture is randomly perturbed, and the scene image is re-rendered; The re-rendered scene images are synthesized by random reorganization of objects, scenes and camera poses as training data for neural network training; When the parameters of the updated neural network reach the maximum number of iterations or meet the termination conditions, the student network is retained for real scenes. When facing real scenes, the camera data is used as input, and the position, size and orientation of each target are inferred based on the network parameters trained by the student network to complete the three-dimensional positioning of the target.

2. The data enhancement method for monocular 3D target detection according to claim 1, characterized in that: Constructing an empty scene is as follows: First, use the depth completion model to obtain a dense depth map of the scene, and use the object segmentation model to obtain the foreground object mask; Then, the edge distortion depth of the foreground part of the depth map is repaired, and the camera intrinsic parameters are used to extract the 3D object point cloud model with texture information; Eliminate the foreground objects to obtain the image and depth map of the empty scene.

3. The data enhancement method for monocular 3D target detection according to claim 2, characterized in that: When correcting the distorted depth of the foreground object edge, the correction method of the distorted depth is as follows: in, Depth value representing distortion; Indicates the corrected depth value; is the anchor point depth value; It is calculated by the average of the absolute values ​​of the depth distances between the anchor point and the five LiDAR image projection points closest to the anchor point in the image.

4. The data enhancement method for monocular 3D target detection according to claim 2, characterized in that: When removing foreground objects, use the object removal model to remove objects. For the depth map, replace the depth of the foreground area: in, is the ground depth map, is the scene depth map, is the empty scene depth map, represents the pixel position of the foreground area, Represents the position of the background pixel that is used to replace the corresponding foreground pixel.

5. The data enhancement method for monocular 3D target detection according to claim 1, characterized in that: The object and scene are fused to obtain the reconstructed image and depth map as follows: Generate the reachable area of ​​the scene using LiDAR data. First, use the existing method of generating sparse reachable areas to obtain a reachable area indication map from a bird's-eye view. Then convert the indication map to polar coordinates, where the origin of the coordinate system is the location of the camera. Then, complete the polar coordinates row by row, and then convert back to the Cartesian coordinate system to obtain a dense reachable area indication map. Before fusing objects and scenes, use 3D annotation boxes in 3D space to determine whether objects collide. Use foreground depth in 2D images to determine the degree of occlusion of the foreground area of ​​each object. Objects that collide or are severely occluded will be screened out and not be fused later. After screening the objects, the textured 3D model of the objects is projected onto the image plane through the camera intrinsic parameters. Then, according to the dense depth map of the scene, the depth is hidden pixel by pixel, and the textures with smaller depth values ​​are retained for display, thus obtaining the fused scene image and depth map.

6. The data enhancement method for monocular 3D target detection according to claim 5, characterized in that: Move the object from its original position in the bird's eye view Move to sampling position When , the position coordinates of the object are given by Convert to , the coordinates of each point of the object 3D model Updated as follows: in, is the scene ground equation.

7. The data enhancement method for monocular 3D target detection according to claim 1, characterized in that: Randomly perturb the camera posture and re-render the scene image as follows: The scene image after the scene and object are fused and scene depth Use the camera's intrinsic parameters to convert to 3D space to get the scene point cloud ; Then, sample the perturbation angles of the camera pitch and roll angles and , the sampling camera along Translational perturbation of the axis , thereby obtaining the rotation and translation of the disturbance; Finally update the scene point cloud coordinates and object 3D labels .

8. The data enhancement method for monocular 3D target detection according to claim 7, characterized in that: The rotation and translation of the perturbation are as follows: in, is the perturbed rotation matrix, is the translation vector of the disturbance, is the pitch angle of the disturbance, is the yaw angle of the disturbance, is the depth of the disturbance.

9. The data enhancement method for monocular 3D target detection according to claim 1, characterized in that: During the training process, under the sparse annotation setting, the sparse object database and the empty scene database are first used to synthesize the training data for pre-training, and then the fully annotated scene is introduced for hybrid training.

10. A data enhancement system for monocular 3D target detection, characterized in that: include: The data module collects lidar and image data and obtains object annotations through full annotation or sparse annotation; Decompose the training image into a 3D object point cloud model with texture information, and remove all the obtained objects from the scene to construct an empty scene; use all the obtained objects to build an object database, and use all the original scenes and the empty scenes to build a scene database; The recombinant module randomly samples scenes from the obtained scene database, randomly samples positions in the reachable area of ​​the scene, randomly samples objects from the obtained object database and moves them to the sampling positions. After collision and occlusion detection, the random sampling positions are inserted, and the objects and scenes are fused to obtain a recombinant image. The rendering module converts the reconstructed image into 3D space based on the depth map, randomly perturbs the camera posture, and re-renders the scene image; The training module synthesizes the re-rendered scene images by random recombination of objects, scenes and camera poses as training data for neural network training; The output module, when the parameters of the updated neural network reach the maximum number of iterations or meet the termination conditions, retains the student network for real scenes. When facing real scenes, the camera data is used as input, and the position, size and orientation of each target are inferred based on the network parameters trained by the student network to complete the three-dimensional positioning of the target.

Images