A method and device for weakly supervised monocular vehicle spatial location prediction
By employing a weakly supervised monocular vehicle spatial location prediction method and optimizing it with object detection networks and RefineNet, the problems of high-level semantic feature extraction and occlusion detection in monocular vehicle detection are solved, achieving high-precision vehicle spatial location prediction with inexpensive cameras.
Patent Information
- Application Number
- CN202111199528.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-14
- Publication Date
- 2026-01-27
- Estimated Expiration
- 2041-10-14
AI Technical Summary
In existing technologies, monocular vehicle detection methods have difficulty effectively extracting high-level semantic features of objects, have insufficient occlusion detection capabilities, and suffer from high costs for LiDAR or alignment and registration issues with multi-camera detection, resulting in insufficient accuracy in predicting vehicle spatial position.
We adopt a weakly supervised monocular vehicle spatial position prediction method. The 2D image coordinates of the vehicle's grounding point are obtained through a pre-trained target detection network. The inverse projection is used to generate a bird's-eye view. The spatial position is optimized using a RefineNet neural network. The method is trained with a small amount of 3D spatial annotation information to improve detection accuracy.
It enables vehicle spatial location prediction using inexpensive cameras with low annotation and data acquisition costs, improving occlusion detection and spatial location inference accuracy.
Smart Images
Figure CN114202745B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of spatial location prediction technology, and in particular to a method and apparatus for predicting the spatial location of a weakly supervised monocular vehicle. Background Technology
[0002] In existing technologies, monocular vehicle detection in autonomous driving scenarios mainly involves monocular 2D object detection methods and 3D object detection methods.
[0003] Monocular 2D object detection methods detect objects in an image and output the object's 2D bounding box and category. These methods typically employ manual feature extraction, which often struggles to extract high-level semantic features and is insufficient for occlusion detection. Following the success of deep learning methods in object classification, researchers have begun using deep neural networks for object detection.
[0004] Furthermore, the goal of 3D object detection is to obtain the 3D bounding box of an object in space. Current academic research methods typically include LiDAR-based 3D object detection, camera-image-based 3D object detection, and hybrid LiDAR and image-based 3D object detection. Camera-based 3D object detection is further divided into monocular 3D object detection using only one camera and binocular / multi-view 3D object detection using multiple cameras. However, LiDAR is extremely expensive and difficult to use on a large scale, while using multiple cameras involves alignment and registration issues.
[0005] Therefore, monocular 3D target detection has become a cost-effective and easy-to-use technical solution in the industry. Summary of the Invention
[0006] The present invention aims to at least partially solve one of the technical problems in the related art.
[0007] Therefore, the first objective of this invention is to propose a weakly supervised monocular vehicle spatial position prediction method, which enables the use of a small amount of 3D spatial annotation information to train the neural network RefineNet, thereby significantly optimizing the object spatial position inference effect.
[0008] The second objective of this invention is to provide a device for predicting the spatial position of a weakly supervised monocular vehicle.
[0009] The third objective of this invention is to provide a computer device.
[0010] A fourth objective of this invention is to provide a non-transitory computer-readable storage medium.
[0011] To achieve the above objectives, a first aspect of the present invention proposes a method for weakly supervised monocular vehicle spatial position prediction, comprising:
[0012] Acquire a 2D image of the target vehicle, and detect the coordinate information of the grounding point of the target vehicle according to a pre-trained target detection network;
[0013] Based on the coordinate information of the grounding point, the 2D image of the target vehicle is inversely projected to obtain the coordinate information of the grounding point in the 3D coordinate system, and a bird's-eye view of the target vehicle is generated based on the coordinate information in the 3D coordinate system.
[0014] The bird's-eye view of the target vehicle is optimized to obtain the spatial location of the target vehicle.
[0015] Optionally, in one embodiment of this application, before detecting the coordinate information of the grounding point of the target vehicle according to the pre-trained target detection network, the method further includes:
[0016] The detection network used for human joint detection is modified by changing the key point output dimension of the detection network to obtain the target detection network.
[0017] Optionally, in one embodiment of this application, the 2D image of the target vehicle is inversely projected based on the coordinate information of the grounding point to obtain the coordinate information of the grounding point in a 3D coordinate system, including:
[0018] The coordinate information of the grounding point is processed to remove distortion.
[0019] Use OpenCV library functions to obtain the coordinate information p of the grounding point after distortion removal. undistorted , will p undistorted Transform into three-dimensional homogeneous coordinates p undistorted-homo Then the ground point P in the camera coordinate system camera The direction is:
[0020]
[0021] Among them, P camera-line Points on a straight line emitted from the camera as the origin toward the ground point, and thus the straight line P in the camera coordinate system. camera-line Transform to the rear axle center coordinate system P body-line :
[0022]
[0023] Obtain the straight line P of the contact point in the coordinate system of the rear axle center of the vehicle body. body-line-homoThen, by applying the ground plane assumption, the precise coordinates of the contact point in the rear axle center coordinate system of the vehicle body are obtained: P body .
[0024] Optionally, in one embodiment of this application, optimizing the bird's-eye view of the target vehicle to obtain the spatial location of the target vehicle includes:
[0025] Using 3D spatial ground truth as supervision information, the RefineNet neural network is trained to optimize spatial positioning;
[0026] An MLP neural network was chosen as the skeleton of RefineNet, with the input being the initial rectangular five-dimensional tensor Rec of the inverse projection. origin ;
[0027] The five-dimensional tensor and the initial input Rec origin Element-wise addition is performed to form a residual structure. The last dimension of the result, rotation, is then transformed back to the range [-π, π] to obtain the optimized five-dimensional tensor Rec. pred Output.
[0028] To achieve the above objectives, a second aspect of the present invention provides an apparatus for weakly supervised monocular vehicle spatial position prediction, comprising:
[0029] The acquisition module is used to acquire a 2D image of the target vehicle and detect the coordinate information of the grounding point of the target vehicle according to a pre-trained target detection network.
[0030] The processing module is used to perform inverse projection on the 2D image of the target vehicle based on the coordinate information of the grounding point, obtain the coordinate information of the grounding point in the 3D coordinate system, and generate a bird's-eye view of the target vehicle based on the coordinate information in the 3D coordinate system.
[0031] The optimization module is used to optimize the bird's-eye view of the target vehicle to obtain the spatial position of the target vehicle.
[0032] Optionally, in one embodiment of this disclosure, it further includes:
[0033] The training module is also used to modify the detection network used for human joint detection, modifying the key point output dimension of the detection network to obtain the target detection network.
[0034] To achieve the above objectives, a third aspect of this application provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the weakly supervised monocular vehicle spatial position prediction method described in the first aspect of this application.
[0035] To achieve the above objectives, a fourth aspect of this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the weakly supervised monocular vehicle spatial position prediction method described in the first aspect of this application.
[0036] The method and apparatus for weakly supervised monocular vehicle spatial position prediction in this invention proposes an adaptive grounding point position prediction method that adjusts key points, thereby improving the detection effect of occluded grounding points. This method can address the issue of errors in vehicle bird's-eye views after obtaining the vehicle's spatial position, and can significantly optimize the object spatial position inference effect by training the RefineNet neural network using only a small amount of 3D spatial annotation information. Additional aspects and advantages of this invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description
[0037] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:
[0038] Figure 1 This is a flowchart of a weakly supervised monocular vehicle spatial position prediction method provided in an embodiment of the present invention.
[0039] Figure 2 Comparison of data plots after direct inverse projection inference and RefineNet optimization.
[0040] Figure 3 Flowchart for inferring the spatial location of vehicles under weak supervision.
[0041] Figure 4 This is a schematic diagram of the structure of a weakly supervised monocular vehicle spatial position prediction device provided in an embodiment of the present invention. Detailed Implementation
[0042] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.
[0043] The following description, with reference to the accompanying drawings, illustrates a method and apparatus for weakly supervised monocular vehicle spatial position prediction according to an embodiment of the present invention.
[0044] Figure 1 This is a flowchart illustrating a method for weakly supervised monocular vehicle spatial position prediction provided in an embodiment of the present invention.
[0045] like Figure 1 As shown, this weakly supervised monocular vehicle spatial location prediction method includes the following steps:
[0046] Step S1: Obtain a 2D image of the target vehicle and detect the coordinate information of the grounding point of the target vehicle according to the pre-trained target detection network.
[0047] In one embodiment of the present invention, before detecting the coordinate information of the grounding point of the target vehicle according to the pre-trained target detection network, the method further includes:
[0048] The detection network used for human joint detection is modified by changing the key point output dimension of the detection network to obtain the target detection network.
[0049] Specifically, it is a simple anchor-free detection method. In the neural network, instead of explicitly setting the anchor of the object bounding box, the center point of the object is used to represent the various attributes of the object. This allows for various tasks such as 2D bounding box detection, human pose estimation, and 3D bounding box detection. CenterNet can be used for human keypoint detection on the COCO dataset.
[0050] In one embodiment of this application, the present invention further improves CenterNet. CenterNet's output for keypoints directly offsets the predicted keypoints corresponding to the object to the keypoints predicted by the bottom-up keypoint detection network. Considering that the keypoint detection network has poor ability to detect occluded keypoints, the present invention adopts an adaptive adjustment method to obtain the final keypoint output:
[0051] hm score =min(max(4hm) confidence -0.3, 0.0), 1.0)
[0052] kps final =hm score ×kps hm +(1-hm score )×kps object
[0053] The expression above, hm confidence It is the value of the heatmap output by the bottom-up key point detection network, kps. hm It's the location of the heatmap, kps object This is the predicted location of each object's keypoints output by CenterNet. The improved CenterNet offers some improvement in detecting occluded grounding points.
[0054] Step S2: Based on the coordinate information of the grounding point, perform inverse projection on the 2D image of the target vehicle to obtain the coordinate information of the grounding point in the 3D coordinate system, and generate a bird's-eye view of the target vehicle based on the coordinate information in the 3D coordinate system.
[0055] In one embodiment of this disclosure, a 2D image of the target vehicle is inversely projected based on the coordinate information of the grounding point to obtain the coordinate information of the grounding point in a 3D coordinate system, including:
[0056] The coordinate information of the grounding point is processed to remove distortion.
[0057] Use OpenCV library functions to obtain the coordinate information p of the grounding point after distortion removal. undistorted , will p undistorted Transform into three-dimensional homogeneous coordinates p undistorted-homo Then the ground point P in the camera coordinate system camera The direction is:
[0058]
[0059] Among them, P camera-line Points on a straight line emitted from the camera as the origin toward the ground point, and thus the straight line P in the camera coordinate system. camera-line Transform to the rear axle center coordinate system P body-line :
[0060]
[0061] Obtain the straight line P of the contact point in the coordinate system of the rear axle center of the vehicle body. body-line-homo Then, by applying the ground plane assumption, the precise coordinates of the contact point in the rear axle center coordinate system of the vehicle body are obtained: P body .
[0062] The upper part yielded a bird's-eye view of the vehicle's ground contact points, which are the positions of the four wheels. Ideally, these points should form a rectangle in the bird's-eye view. However, in a 2D front view, it's impossible for all four wheels to be visible; some wheels will inevitably be obscured by themselves or other vehicles. Furthermore, target vehicles located far from the data acquisition vehicle are too small in the image, making it difficult to accurately determine the wheel ground contact points. These factors make it difficult to precisely identify the ground contact points of many target vehicles in the 2D image. Further inverse projection into space accumulates errors, resulting in the vehicle's four ground contact points appearing as irregular quadrilaterals. To optimize the visualization, this invention uses the Nelder-Mead algorithm to optimize the vehicle's bird's-eye view into a rectangle; please refer to step S3 for a detailed description.
[0063] Step S3: Optimize the bird's-eye view of the target vehicle to obtain the spatial location of the target vehicle.
[0064] In one embodiment of this disclosure, optimizing a bird's-eye view of the target vehicle to obtain its spatial location includes:
[0065] Using 3D spatial ground truth as supervision information, the RefineNet neural network is trained to optimize spatial positioning;
[0066] An MLP neural network was chosen as the skeleton of RefineNet, with the input being the initial rectangular five-dimensional tensor Rec of the inverse projection. origin ;
[0067] Five-dimensional tensor and initial input Rec origin Element-wise addition is performed to form a residual structure. The last dimension of the result, rotation, is then transformed back to the range [-π, π] to obtain the optimized five-dimensional tensor Rec. pred Output.
[0068] Specifically, in one embodiment of this disclosure, when a Rec pred During training, its ground truth is the distance Rec among all labeled objects corresponding to that image. pred Recent object Rec gt This invention uses the L1 loss function to minimize Rec pred and Rec gt The error.
[0069] After optimization, the accuracy and recall of vehicle spatial location prediction were significantly improved. Considering an IOU > 0.3 between the predicted rectangle and the ground truth rectangle as a correct prediction, the accuracy and recall of vehicle spatial location prediction after direct inverse projection and RefineNet optimization are as follows: Figure 2 As shown.
[0070] Specifically, the overall method flowchart of the present invention is as follows: Figure 3 As shown in the upper part, the boxes with lower transparency represent geometric optimization methods, while the boxes with higher transparency represent deep learning methods. Furthermore, the effect diagram of this invention is shown below. Figure 3 The lower half shows the input image on the left and the output aerial view of the car locations on the right. Each yellow rectangle represents the location of a car. For example, the small yellow rectangle on the far right represents the location of the sedan in the image, and the largest yellow rectangle in the middle represents the location of the bus.
[0071] The technical advantages of this application are: it realizes a weakly supervised method for inferring vehicle spatial location using only 2D image annotations. Furthermore, optimizations are made in multiple steps to significantly improve spatial location detection performance. Using this technical solution, a neural network can be trained on datasets with low annotation and acquisition costs, using only inexpensive cameras, to predict vehicle spatial location.
[0072] Figure 4 This is a schematic diagram of the structure of a weakly supervised monocular vehicle spatial position prediction device provided in an embodiment of the present invention.
[0073] like Figure 4 As shown, this weakly supervised monocular vehicle spatial position prediction method includes the following modules:
[0074] The acquisition module is used to acquire a 2D image of the target vehicle and detect the coordinate information of the grounding point of the target vehicle based on a pre-trained target detection network.
[0075] The processing module is used to perform inverse projection on the 2D image of the target vehicle based on the coordinate information of the grounding point, obtain the coordinate information of the grounding point in the 3D coordinate system, and generate a bird's-eye view of the target vehicle based on the coordinate information in the 3D coordinate system.
[0076] The optimization module is used to optimize the bird's-eye view of the target vehicle to obtain the spatial position of the target vehicle.
[0077] In one embodiment of this application, it further includes:
[0078] The training module is used to modify the detection network used for human joint detection by modifying the key point output dimension of the detection network to obtain the target detection network.
[0079] The advantages of this application are: it realizes a weakly supervised method to infer the spatial position of a vehicle using only 2D image annotations, proposes an adaptive grounding point position prediction method with key point confidence adjustment to improve the detection effect of occluded grounding points, and proposes an improved spatial position inference method that uses a small amount of 3D Bounding Box supervision information to significantly improve the spatial position detection effect.
[0080] To achieve the above objectives, a third aspect of this application provides a computer device having a memory, a processor, and a computer program stored in the memory and executable on the processor. When executed by the processor, the computer program implements a weakly supervised monocular vehicle spatial position prediction method as described in the first aspect of this application.
[0081] To achieve the above objectives, a fourth aspect of this application provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a weakly supervised monocular vehicle spatial position prediction method as described in the first aspect of this application.
[0082] Although this application has been disclosed in detail with reference to the accompanying drawings, it should be understood that these descriptions are merely exemplary and not intended to limit the application of this application. The scope of protection of this application is defined by the appended claims and may include various variations, modifications, and equivalents of the invention without departing from the scope and spirit of this application.
[0083] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0084] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.
[0085] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of the invention pertain.
[0086] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.
[0087] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0088] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.
[0089] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.
[0090] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.
Claims
1. A method for predicting the spatial position of a vehicle using a weakly supervised monocular camera, characterized in that, Includes the following steps: A 2D image of the target vehicle is acquired. The coordinates of the target vehicle's grounding points are detected using a pre-trained target detection network. Before detecting the grounding point coordinates using the pre-trained target detection network, the detection network used for human joint detection is modified. The keypoint output dimension of the detection network is adjusted to obtain the target detection network. Specifically, CenterNet is improved. CenterNet's keypoint output directly offsets the predicted keypoints corresponding to the object to the keypoints predicted by the bottom-up keypoint detection network, and an adaptive adjustment method is used to obtain the final keypoint output. in, It is the value of the heatmap output by the bottom-up key point detection network. It's the location on the heat map. It is the predicted location of each object's keypoints output by CenterNet; Based on the coordinate information of the grounding point, the 2D image of the target vehicle is inversely projected to obtain the coordinate information of the grounding point in the 3D coordinate system, and a bird's-eye view of the target vehicle is generated based on the coordinate information in the 3D coordinate system. The bird's-eye view of the target vehicle is optimized to obtain the spatial location of the target vehicle.
2. The method according to claim 1, characterized in that, Based on the coordinate information of the grounding point, a 2D image of the target vehicle is inversely projected to obtain the coordinate information of the grounding point in a 3D coordinate system, including: The coordinate information of the grounding point is processed to remove distortion. Use OpenCV library functions to obtain the coordinates of the grounding point after distortion removal. ,Will Transform into three-dimensional homogeneous coordinates Then the grounding point in the camera coordinate system The direction is: in, Points on a straight line emitted from the camera as the origin toward the ground point, and thus the straight line in the camera coordinate system. Transform to the rear axle center coordinate system of the vehicle body : Obtain the straight line of the contact point in the coordinate system of the rear axle center of the vehicle body. Then, the precise coordinates of the contact point in the rear axle center coordinate system are obtained by applying the ground plane assumption: .
3. A device for predicting the spatial position of a weakly supervised monocular vehicle, characterized in that, include: The acquisition module is used to acquire a 2D image of the target vehicle and detect the coordinate information of the grounding point of the target vehicle according to a pre-trained target detection network. Before detecting the coordinate information of the grounding point of the target vehicle according to the pre-trained target detection network, the detection network used for human joint detection is modified by changing the keypoint output dimension of the detection network to obtain the target detection network. Specifically, CenterNet is improved. CenterNet's keypoint output directly offsets the predicted keypoints corresponding to the object to the keypoints predicted by the bottom-up keypoint detection network, and an adaptive adjustment method is used to obtain the final keypoint output. in, It is the value of the heatmap output by the bottom-up key point detection network. It's the location on the heat map. It is the predicted location of each object's keypoints output by CenterNet; The processing module is used to perform inverse projection on the 2D image of the target vehicle based on the coordinate information of the grounding point, obtain the coordinate information of the grounding point in the 3D coordinate system, and generate a bird's-eye view of the target vehicle based on the coordinate information in the 3D coordinate system. The optimization module is used to optimize the bird's-eye view of the target vehicle to obtain the spatial position of the target vehicle.
4. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the method as described in any one of claims 1-2.
5. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-2.