Method for 3D information correction using deep convolutional neural network

Abstract

A method for 3D information correction using a deep convolutional neural network includes: (1) obtaining current camera image data; (2) inputting the obtained current camera image data; (3) detection phase: using a 3D monocular detection network to detect the image data and obtain a target image detection result; (4) correction phase: using a residual correction network to correct the detection result and obtain corrected image data; (5) outputting the corrected image data. The result of target detection by a monocular camera is optimized, and the information with large errors is fine-tuned in combination with a deep convolutional network, which can well process and correct the errors of monocular detection and obtain accurate detection results, solving the problem that monocular cameras cannot meet the detection accuracy in unmanned driving.

Description

Technical Field

[0001] The present disclosure relates to the fields of autonomous driving and computer vision, and in particular to a method for correcting 3D information using a deep convolutional neural network. Background Art

[0002] Autonomous driving is a research hotspot in the field of transportation vehicles at this stage. The environmental perception module is an indispensable part of unmanned driving systems and intelligent robots. At present, there are many solutions for how to obtain the shape, position and other information of the target vehicle in the camera coordinate system in autonomous driving. The main approach is to obtain 3D point cloud information based on multiple sensors such as depth cameras or lidars, or to form a stereo vision system through multiple cameras to correctly detect the target 3D information. Among them, the solution based on lidar is expensive and has distance limitations. When the target is too far away, it cannot be located; the system based on binocular and multi-camera cameras has complex configuration and calibration, and its depth range and accuracy are limited by the baseline and resolution of the binocular, and visual computing consumes a lot of computing resources; some scholars also use monocular cameras to locate the target, but due to the scale uncertainty of monocular images, the detection accuracy of monocular cameras is relatively low.

[0003] The system configuration cost of the monocular camera target positioning system is much lower than that of the solution using LiDAR, and it is also superior to the solution using binocular and multi-camera in terms of solution complexity, cost, and system computing consumption. Therefore, using a monocular camera to detect the target vehicle is of great practical significance. However, the positioning accuracy of the monocular camera target positioning system is low, and the detection results of the monocular camera directly cannot meet the visual positioning requirements in autonomous driving. Due to the particularity of autonomous driving, the environmental perception system needs to obtain the information of objects in the environment in real time and accurately. If an effective correction method can be used to correct the detection results of the monocular camera so that it meets the visual positioning accuracy requirements of autonomous driving, then this will be of great practical application value.

[0004] There are not many studies on information correction of the detection results of monocular cameras in the prior art. The relevant literature retrieved is as follows:

[0005] 1. Patent: US20200219315A1 'Bounding box estimation and lane vehicle association' This method uses weak prior information to estimate the 3D bounding box of the vehicle in the monocular image. Specifically, this method estimates the specific size of the 3D bounding box through the geometric relationship between the 2D bounding box and the 3D bounding box, and uses the prior information of the radar to estimate the distance of the vehicle. However, it cannot be ignored that the error caused by the calculation of the 2D bounding box will accumulate and be passed to the calculation of the 3D bounding box. When estimating the 3D bounding box from the 2D bounding box, a small measurement observation error can easily lead to a large reconstruction error. In addition, this method also requires prior information from the radar to measure the distance, and its scope of use is also limited. The information processed by the monocular system alone cannot meet the accuracy requirements of autonomous driving.

[0006] 2. Patent WO2018184963A2 'Direct vehicle detection as 3D bounding boxesusing neural network image processing' This method uses two adjacent quadrilaterals on the front and side of the vehicle to define a 3D bounding box. The two bounding boxes share an edge and regress 6 key points to regress the bounding box as well as the depth and orientation angle. However, this method does not consider the problems of occlusion and truncation. When the target vehicle is blocked by obstacles or is about to drive out of the boundary, the number of feature points will decrease, and the accuracy of the regression results will also be affected. The actual driving scenes are complex and changeable, and the autonomous driving system has high requirements for accuracy and model generalization performance. Therefore, this method is difficult to apply in practice.

[0007] 3. Paper: Mousavian A, Anguelov D, Flynn J, et al. 3d bounding boxestimation using deep learning and geometry[C] / / Proceedings of the IEEEConference on Computer Vision and Pattern Recognition. 2017: 7074-7082. In this paper, the azimuth of the target is regressed, and then the yaw angle is calculated based on the azimuth of the target and the position information of the target. However, since the estimated position information of the target may also have errors, the accuracy of the yaw angle needs to be further improved. The method cannot meet the needs of autonomous driving.

[0008] 4. Paper: Liu Z, Wu Z, Tóth R. SMOKE: Single-Stage Monocular 3D Object Detection via Keypoint Estimation[C] / / Proceedings of the IEEE / CVF Conferenceon Computer Vision and Pattern Recognition Workshops. 2020: 996-997. Through two branches for regression and classification respectively, 3D target detection in monocular images is realized, and the fitting effect of information such as the size of the 3D bounding box is good. However, the depth information is directly calculated by the regressor and the predefined scale and displacement parameters. The problem of monocular image scale uncertainty is not considered. The accuracy of depth regression needs to be improved. This method cannot be applied to actual autonomous driving.

[0009] Therefore, it is necessary to study effective correction methods to correct the detection results of the monocular camera so that it can meet the visual positioning accuracy requirements in autonomous driving. Summary of the invention

[0010] In response to the above technical problems, the present disclosure proposes a new method for 3D information correction using a deep convolutional neural network in order to study an effective method for correcting detection information of a monocular camera. The method is optimized on the basis of a monocular camera, and the detection results of the monocular camera are fine-tuned and corrected. The method occupies few computing resources and is simple and easy to implement. It can obtain better detection effects without occupying too many computing resources.

[0011] Specifically, for the yaw angle with large error in the detection information, the residual between the actual yaw angle of the target and the yaw angle in the monocular detection result is fitted to achieve angle correction; for the scale uncertainty problem of the monocular image, the depth information z is corrected by combining the deep convolution information and the proportional relationship between scale normalization and perspective projection.

[0012] As an indispensable part of unmanned driving systems and intelligent robots, the environmental perception module needs to obtain information about objects in the environment in real time and accurately. The detection speed using a monocular camera is fast and the cost is low, but the accuracy is relatively low. If an effective method can be taken to correct the detection results of the monocular camera so that it meets the accuracy requirements of the unmanned driving perception system, it will be a very ideal solution. Based on this purpose, the present invention optimizes the results of target detection by a monocular camera, and combines a deep convolutional network to fine-tune the information with large errors, which can well process and correct the errors of monocular detection and obtain accurate detection results. Solve the problem that monocular cameras cannot meet the detection accuracy in unmanned driving.

[0013] To solve the above technical problems, according to one aspect of the present disclosure, a method for 3D information correction using a deep convolutional neural network is provided, comprising:

[0014] Step 1, obtain the current camera image data;

[0015] Step 2: input the acquired current camera image data;

[0016] Step 3, detection stage: use a 3D monocular detection network to detect the image data to obtain a target image detection result;

[0017] Step 4, correction stage: use the residual correction network to correct the detection result to obtain corrected image data;

[0018] Step 5: output the corrected image data.

[0019] Preferably, the camera is a monocular camera.

[0020] Preferably, a monocular camera image is input in real time to obtain the monocular camera real-time image.

[0021] Preferably, the 3D monocular detection network inference steps are:

[0022] The original data set consisting of image data collected in real time by a monocular camera is divided into a training set and a validation set. The training set in the divided data set is used to train the network, and then the validation set is input into the trained network to obtain an evaluation value of the network performance.

[0023] Preferably, the frame matching and error interval calculation steps are:

[0024] According to the matching relationship between the detection result and the true label, the error between the two is calculated, and its mean and variance are counted.

[0025] Preferably, the detection box matching step based on IOU:

[0026] The predicted label (predict label) of the validation set is matched with the true label (GT label) by IOU to obtain the corresponding box.

[0027] Preferably, for each 3D real box (GT box) in the real label (GT label), in the corresponding predicted label (predict label), loop through all boxes and calculate the 3D IOU and 2D IOU values ​​with the 3D real box (GT box), and take the one with the largest 3D IOU as the corresponding predicted box.

[0028] Preferably, if the 3D real box (GT box) does not have a predicted box that has an intersection with it, the one with the largest 2D IOU is taken as the corresponding predicted box, otherwise there is no corresponding predicted box.

[0029] Preferably, the calculation of the 2D IOU and 3D IOU is as shown in formula (1):

[0030] (1)

[0031] Where A and B are two rectangular boxes or 3D boxes.

[0032] Preferably, according to the corresponding relationship of the 3D real box (GT box), the error of the regression quantity, that is, the error between the real value and the predicted value, is calculated, wherein the regression quantity includes the depth z and the yaw angle r_y of the 3D box.

[0033] Preferably, the error obeys a Gaussian distribution.

[0034] Preferably, the mean μ and standard deviation σ of the error Gaussian distribution are calculated, and then the influence of extreme values ​​is removed according to the 3σ principle of the Gaussian distribution to obtain the deviation range of each variable.

[0035] Preferably, generating training data comprises the following steps:

[0036] According to the range of the error, data perturbation is performed on the original training set to generate training data.

[0037] Preferably, the deviation range is added to the yaw angle r_y and the depth z respectively, and the distribution of the deviation is a normal distribution. Then, the 3D frame of the target with the deviation added is drawn in the original picture, and a mask is added to the front thereof to distinguish the direction; then the picture is cropped, and the cropping ratio is expanded in four directions by 0.2 times the target width (weight) or target height (height) according to the 2D frame, wherein the image data is the picture after the forward mask is added and cropped.

[0038] Preferably, the tag data are the sin value and cos value of the disturbance value of the yaw angle r_y and the disturbance value of the depth information z.

[0039] Preferably, generating training data comprises the following steps:

[0040] Randomly translate the cropping box to the left or right by 0.4 target width (weight) and crop it, leaving its label data unchanged;

[0041] Mirror the training data horizontally.

[0042] Preferably, the residual correction network consists of two parts, namely a feature extraction network part and two regression branch network parts.

[0043] Preferably, the feature extraction network uses a ResNet101 network, which is composed of five convolution blocks, wherein the first layer Conv1 is a convolution layer with a convolution kernel size of 7x7 and 64 channels, the second layer Conv2 first passes through a max pooling layer for downsampling, and then is a convolution block, and subsequent Conv3-Conv5 are all composed of different convolution blocks, two fully connected layers, wherein the first fully connected layer Fc1 is used to reduce the number of channels, and the second fully connected layer Fc2 is used to regress the required information.

[0044] Preferably, the image is normalized before being input into the network, and the size of all images is normalized to 64*64. The training batch size (batchsize) is 8 during training, the optimizer is Adam, the initial learning rate is 0.001, and the learning rate is multiplied by 0.8 after every 10 rounds of training. The feature extraction network uses ResNet101. After preprocessing, the image enters the feature extraction layer, and the extracted features enter two branches for angle information and depth information correction. The loss function used for angle correction is smooth L1 loss function, and the mean square error used for depth correction.

[0045] Preferably, the predicted value of the yaw angle output is the sin value and cos value of the correction angle θ, and then the corresponding offset angle is calculated based on the sin value and cos value. Subtracting the disturbance value from the label predicted by the first-level network is the corrected yaw angle.

[0046] Preferably, the value range of the yaw angle r_y is [-π, π].

[0047] Preferably, the yaw angle that is out of range after partial angle correction is processed and mapped into the range of [-π, π].

[0048] Preferably, if r_y+θ is in the range of [-2π, -π], θ is added with 2π, and if r_y+θ is in the range of [π, 2π], θ is subtracted with 2π; as shown in formula (2):

[0049] (2)

[0050] Among them, r_y is the yaw angle, r_y ’ is the corrected yaw angle, θ is the correction angle;

[0051] The activation function of the angle uses the L2 paradigm, transforms the sin and cos values ​​obtained by regression, and then restores the angle value based on the sin and cos values ​​obtained; the transformation formula is shown in formula (3):

[0052] (3)

[0053] Among them, sinx and cosx are the sine and cosine values ​​​​finally used to calculate the angle. sin and O cos is the output value of the fully connected network, O sin is the predicted value of the cosine angle in the network, O cos is the predicted value of the sine angle.

[0054] Preferably, the input of the residual correction network normalizes all input images to a fixed size in advance, and the fixed size is a pixel size of 64x64.

[0055] Preferably, for input images with a size smaller than 64x64, they are enlarged to 64x64, the target size itself remains unchanged, and the target depth value in the visual system becomes smaller; for images with a size larger than 64x64, they are reduced to 64x64, the target size itself remains unchanged, and the target depth value in the visual system becomes larger.

[0056] Preferably, the proportion of the visual change of the object due to magnification is listed according to the area formula of the trapezoid as shown in formula (4) and the corresponding proportion is solved, and then the original size is restored according to the proportion, as shown in formula (5):

[0057] (4)

[0058] (5)

[0059] Where h1 is the required proportional relationship.

[0060] Preferably, the scale of the visual change of the object due to scaling is restored according to formula (6):

[0061] (6).

[0062] Preferably, for targets with a size smaller than 64x64, the corrected depth information is obtained by subtracting the errors z0 and h output by the network from the original depth information z, as shown in formula (7):

[0063] (7)

[0064] where z ’ is the corrected depth information.

[0065] Preferably, for targets with a size greater than 64x64, the corrected depth information is obtained by subtracting the error z0 output by the network from the original depth information z and then subtracting h, as shown in formula (8):

[0066] (8)

[0067] where z ’ is the corrected depth information.

[0068] Preferably, the residual correction network is trained using training data, and then the inference results of the saved first-level detection network are clipped to generate verification data for the residual correction network, and the verification set is input into the residual correction network to obtain the corrected results, which are used to evaluate the performance of the network.

[0069] To solve the above technical problems, according to another aspect of the present disclosure, a processing device for 3D information correction using a deep convolutional neural network is provided, comprising:

[0070] An image acquisition device, which acquires current camera image data;

[0071] A data input device for inputting the acquired current camera image data;

[0072] A detection device detects the image data using a 3D monocular detection network to obtain a target image detection result;

[0073] A correction device, which uses a residual correction network to correct the detection result to obtain corrected image data;

[0074] The output device outputs the corrected image data.

[0075] Beneficial effects of the present disclosure:

[0076] 1. The present disclosure is universal for monocular detection networks, that is, it is universal for any network that uses monocular images for 3D target detection. The present disclosure can fine-tune the detection results based on the first-level monocular detection network, thereby ensuring the accuracy of the detected object information.

[0077] 2. Taking into account the respective limitations of 3D IOU (Intersection over Union) and 2D IOU (Intersection over Union), the present disclosure combines these two calculation methods to perform frame matching, thereby ensuring the correctness of the matching between the detection frame and the real frame, as well as the accuracy of subsequent error statistics.

[0078] 3. Considering the possibility of truncation of vehicles in the image, a large amount of truncated data is obtained by using translation for data enhancement to train the network, ensuring robustness to special scenarios.

[0079] 4. Considering the problem of direction change, the range of the r_y yaw angle is [-π, π]. When it exceeds the critical value, the direction will change and the sign will change at the same time. The present disclosure ensures that the corrected r_y yaw angle is still within the range of [-π, π] by changing the error angle.

[0080] 5. Considering the scale uncertainty of monocular images, the perspective projection technology is used to transform the corrected depth information z, eliminating the visual difference caused by image normalization, so that the depth information is corrected within one scale, which greatly improves the accuracy of the corrected depth information.

[0081] 6. Monocular detection ensures the rapidity of system detection in principle. The data correction method disclosed in the present invention is simple and easy to implement, has good real-time performance, and is very suitable for the use of information perception systems in autonomous driving. It is also suitable for monocular detection in other occasions and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] The accompanying drawings illustrate exemplary embodiments of the present disclosure and are used together with the description thereof to explain the principles of the present disclosure, wherein these drawings are included to provide a further understanding of the present disclosure, and the drawings are included in and constitute a part of this specification. By describing the embodiments of the present disclosure in detail in conjunction with the accompanying drawings, the above and other purposes, features, and advantages of the present disclosure will become more apparent.

[0083] Figure 1 It is the overall processing flow chart;

[0084] Figure 2 It is the training flow chart of the residual correction network;

[0085] Figure 3 It is a schematic diagram of 2D IOU calculation;

[0086] Figure 4 It is a schematic diagram of 3D IOU calculation;

[0087] Figure 5 is a Gaussian distribution graph;

[0088] Figure 6 is a training image sample;

[0089] Figure 7 is a sample of labeled data;

[0090] Figure 8 It is the residual correction network processing flow chart;

[0091] Fig. 9 This is the yaw angle out of range diagram after r_y is added to θ;

[0092] Fig.10 It is the principle of perspective;

[0093] Fig.11 These are scale images after normalization of different scales, where (a) is a scale image with a size smaller than 64x64, and (b) is a scale image with a size larger than 64x64. DETAILED DESCRIPTION

[0094] The present disclosure is further described in detail below in conjunction with the accompanying drawings and implementations. It is understood that the specific implementations described herein are only used to explain the relevant content, rather than to limit the present disclosure. It should also be noted that, for ease of description, only the parts related to the present disclosure are shown in the accompanying drawings.

[0095] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0096] Unless otherwise specified, the exemplary embodiments / embodiments shown will be understood as providing exemplary features of various details of some ways in which the technical concept of the present disclosure can be implemented in practice. Therefore, unless otherwise specified, the features of the various embodiments / embodiments can be combined, separated, interchanged and / or rearranged without departing from the technical concept of the present disclosure.

[0097] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment / method or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments / methods or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments / methods or examples described in this specification and the features of the different embodiments / methods or examples, unless they are contradictory.

[0098] like Figure 1 As shown, the method for 3D information correction using a deep convolutional neural network disclosed in the present invention generally includes the following steps:

[0099] Step 1, obtain the current camera image data;

[0100] Step 2: input the acquired current camera image data;

[0101] Step 3, detection stage: use a 3D monocular detection network to detect the image data to obtain a target image detection result;

[0102] Step 4, correction stage: use the residual correction network to correct the detection result to obtain corrected image data;

[0103] Step 5: output the corrected image data.

[0104] Preferably, the camera is a monocular camera, and the monocular camera image is input in real time to obtain the real-time image of the monocular camera.

[0105] Specifically, the following steps are also included:

[0106] 1) 3D monocular detection network inference

[0107] The original data set composed of image data collected in real time by the monocular camera is divided into a training set and a validation set. There are two division methods: 3dop and subcnn. You can choose according to the needs of different detection networks. The training set in the divided data set is used to train the network, and then the validation set is input into the trained network to obtain the evaluation value of the network performance, and save the inference result of the validation set.

[0108] 2) Box matching and error interval calculation

[0109] like Figure 2 As shown in the figure, in order to count the error distribution of the 3D detection network, we first find the matching relationship between the detection result and the true label in the previous step, and then calculate the error between the two, and count their mean and variance.

[0110] The specific steps include:

[0111] 2.1) Detection box matching based on IOU

[0112] Perform IOU matching on the predicted label (predict label) of the validation set obtained in the previous step with the real label (GT label) of the real validation set to obtain the corresponding box. Specifically, for each 3D real box (GT box) in the real label (GTlabel) of a single sample, in the corresponding predicted label (predict label), loop through all boxes and calculate the 3D IOU and 2D IOU values ​​with the 3D real box (GT box), and take the one with the largest 3D IOU as the corresponding predicted box. Considering that there may be no intersection between boxes due to inaccurate prediction depth, that is, all 3D IOUs are 0, if there is no predicted box that intersects with a certain 3D real box (GT box), then take the one with the largest 2D IOU as the corresponding predicted box, otherwise there is no corresponding predicted box. The calculation of 2D IOU and 3D IOU is as follows: Figure 3 and Figure 4 As shown, the calculation is shown in formula (1):

[0113] (1)

[0114] Where A and B are two rectangular boxes or 3D boxes.

[0115] The combined use of 2D and 3D IOU for frame matching can ensure that the corresponding frames are matched together without omissions or incorrect matches, ensuring the accuracy of subsequent statistical errors, thereby making the results obtained by the correction network more accurate.

[0116] 2.2) Calculate the error interval

[0117] According to the corresponding relationship of the 3D frame obtained in the previous step, the error of the regression quantity is calculated, that is, the error between the true value and the predicted value, where the regression quantity includes the depth z and the yaw angle r_y of the 3D frame. The error follows a Gaussian distribution, such as Figure 5 As shown, the mean μ and standard deviation σ are calculated, and then the influence of extreme values ​​is removed according to the 3σ principle of Gaussian distribution to obtain the deviation range of each variable, as shown in Table 1.

[0118] Table 1. Deviation range of each variable (unit: yaw angle is radians, the rest are meters)

[0119]

[0120] 3) Generate training data

[0121] Generating training data comprises the following steps:

[0122] 3.1) According to the error range obtained in the previous step, make data perturbations on the original training set to generate training data.

[0123] Specifically, the deviation range obtained in step 2) is added to the yaw angle r_y and the depth z respectively, and the distribution of the deviation is normal distribution. Then the 3D frame of the target with the deviation is drawn in the original image, and a mask is added to its front to distinguish the direction. Then the image is cropped, and the cropping ratio is expanded by 0.2 times the target width (weight) or target height (height) in each of the four directions according to the 2D frame. The image data is the image after adding the forward mask (mask) and cropping. Figure 6 shown.

[0124] The label data is the sin and cos values ​​of the perturbation value of the yaw angle r_y and the perturbation value of the depth information z. The label sample file is as follows Figure 7 Each line represents a sample. The first item *.png represents the name of the picture. The second and third items are the sine and cosine values ​​of the yaw angle disturbance value, respectively. The fourth item represents the disturbance value of the depth information z.

[0125] 3.2) To address the problem of less truncated data, the above cropping box is randomly translated to the left or right by 0.4 target width (weight) and cropped, and its label data remains unchanged.

[0126] 3.3) To solve the problem of unbalanced deflection direction, the training data obtained in the previous two steps is horizontally mirrored. At this time, due to the change in deflection direction, the sin value in the label data changes its sign, while the cos value and the perturbation value of z remain unchanged. Using horizontal mirroring can effectively solve the problem of unbalanced deflection direction.

[0127] 4) Design and training of residual correction network

[0128] 4.1) Network structure and parameters:

[0129] Network structure: The residual correction network consists of two parts, namely the feature extraction network part and the two regression branch network parts. As shown in Table 2:

[0130] Table 2: Residual correction network structure

[0131]

[0132] The feature extraction network uses the ResNet101 network, which is composed of five convolution blocks, where the first layer Conv1 is a convolution layer with a convolution kernel size of 7x7 and 64 channels. The second layer Conv2 first passes through the max pooling layer for downsampling, and then is a convolution block. The subsequent Conv3-Conv5 are all composed of different convolution blocks, whose sizes are shown in the table. Next are two fully connected layers. The first fully connected layer Fc1 is used to reduce the number of channels, and the second fully connected layer Fc2 is used to regress the required information.

[0133] Network parameters: The images are normalized before being input into the network. All images are normalized to 64*64. The batch size is 8 during training. The optimizer is Adam. The initial learning rate is 0.001. After every 10 rounds of training, the learning rate is multiplied by 0.8. The feature extraction network uses ResNet101. After preprocessing, the image enters the feature extraction layer. The extracted features enter two branches for angle information and depth information correction. The loss function used for angle correction is smooth L1 loss function, and the mean square error used for depth correction.

[0134] 4.2) Correction method of yaw angle r_y based on angle transformation

[0135] The predicted value of the yaw angle output is the sin and cos values ​​of the disturbance θ, and then the corresponding offset angle is calculated based on the sin and cos values. The perturbation value is subtracted from the label predicted by the first-level network to get the corrected yaw angle. The range of the yaw angle r_y is [-π, π]. Some angles may exceed the range after correction. Fig. 9 As shown, where r_y is the yaw angle and θ is the correction angle.

[0136] After partial angle correction, the yaw angle that is out of range is processed and mapped to the range of [-π, π]. If r_y+θ is in the range of [-2π, -π], θ is added with 2π; if r_y+θ is in the range of [π, 2π], θ is subtracted with 2π; as shown in formula (2):

[0137] (2)

[0138] Among them, r_y is the yaw angle, r_y ’ is the corrected yaw angle, θ is the correction angle. This method further normalizes the range of the yaw angle based on the network-corrected yaw angle, eliminates the ambiguity caused by the periodicity of trigonometric functions, and reduces the complexity of target information processing by the subsequent driving system.

[0139] The activation function of the angle uses the L2 paradigm, transforms the sin and cos values ​​obtained by regression, and then restores the angle value based on the sin and cos values ​​obtained. The transformation formula is shown in formula (3):

[0140] (3)

[0141] Among them, sinx and cosx are the sine and cosine values ​​​​finally used to calculate the angle. sin and O cos is the output value of the fully connected network, O sin is the predicted value of the cosine angle in the network, O cos is the predicted value of the sine angle. Since the sine and cosine values ​​can only be between [-1, 1], they are normalized using formula (3) to make their values ​​within the correct range. Then, the angle θ can be corrected and calculated based on the obtained sine and cosine values. The corrected yaw angle can be obtained based on formula (2). This method improves the accuracy of the correction angle θ and further ensures the accuracy of the yaw angle correction result.

[0142] 4.3) Correction method of depth information z based on perspective principle

[0143] The input of the residual correction network will pre-normalize the input images to a fixed size, that is, the pixel size is 64x64. For images larger or smaller than this size, a visual error will occur where the near ones are larger and the far ones are smaller, as shown in Figure (10).

[0144] In this regard, the present disclosure handles it as follows: for images smaller than 64x64, they are enlarged to 64x64, the target size itself remains unchanged, and the target depth value in the visual system becomes smaller; for images larger than 64x64, they are reduced to 64x64, the target size itself remains unchanged, and the target depth value in the visual system becomes larger; the relationship between these two scales can be expressed as Fig.11 Indicates. Where a is the length in the perspective view, b is the actual length, and k is the scale, whose size is k=(w*h) / 64*64. Where h is the distance from the perspective view of the object to the object, h1 is the distance from the object to the perspective view with a scale of 64*64, and h2 is the distance from the perspective view of the object to the perspective view with a scale of 64*64.

[0145] according to Fig.11 The proportional relationship in sub-figure (a) of FIG. 1 can be obtained by listing the equation shown in formula (4) according to the area formula of the trapezoid and solving the corresponding proportion. Then, the original size can be restored according to the proportion. Among them, h1 in formula (5) is the proportional relationship we need.

[0146] (4)

[0147] (5)

[0148] Analogy Fig.11 The proportional relationship in sub-image (b) in the figure, the scale of the visual change of the object caused by scaling can be restored according to formula (6),

[0149] (6)

[0150] For targets smaller than 64x64, the corrected depth information is obtained by subtracting the network output errors z0 and h from the original depth information z, as shown in formula (7):

[0151] (7)

[0152] where z ’ is the corrected depth information.

[0153] For targets with a size larger than 64x64, the corrected depth information is obtained by subtracting the network output error z0 from the original depth information z and then subtracting h, as shown in formula (8):

[0154] (8)

[0155] where z ’ is the corrected depth information.

[0156] This method corrects the depth information z by placing objects at different distances on the same scale, thereby solving the problem of inaccurate depth error estimation caused by objects that are larger when they are near and smaller when they are far away, and ensuring the accuracy of the depth information z correction result.

[0157] 4.4) Training and inference of residual correction network

[0158] The residual correction network is trained using the training data obtained in the above step 3), and then the inference results of the first-level detection network saved in step 1) are cropped to generate verification data for the residual correction network. The verification set is input into the residual correction network to obtain the corrected result, and the network performance is evaluated using it.

[0159] 4.5) Application of residual correction network

[0160] The detection results of the first-level 3D monocular detection network are cropped according to the method in step 3). When cropping, the name of the cropped image is added after the corresponding label for fusion of the correction results. Then the cropped image is input into the residual correction network for inference. The inferred result will be corrected according to the corresponding image name and the corresponding label. The final output label is the corrected detection result.

[0161] Experiments show that the present disclosure can well handle and correct the error of monocular detection and obtain more accurate detection results. It can be seen that the present disclosure can quickly realize the judgment of the vehicle corner posture without marking the vehicle posture, and still has good performance under low-speed congested road conditions; it not only overcomes the shortcomings of the existing prediction through neural networks that require a large amount of vehicle posture data calibration and cannot guarantee real-time performance, but also solves the problem that the traditional historical trajectory calculation method is only applicable to medium and high speed states and cannot be applied to low speed states; the present disclosure uses wheels as the main basis for posture judgment, performs posture estimation under a bird's-eye view, improves the accuracy of posture calculation, and can perform real-time and accurate posture detection of targets within a large range; the method is simple and easy to implement, has high measurement accuracy, and is suitable for practical application in automatic driving or assisted driving; the monocular detection method adopted by the present disclosure ensures that the system detection is fast in principle, and the data correction method adopted is simple and easy to implement, has good real-time performance, is very suitable for the use needs of information perception systems in automatic driving, and is also suitable for monocular detection in other occasions, and has broad application prospects.

[0162] So far, the technical solutions of the present disclosure have been described in conjunction with the preferred implementation methods shown in the accompanying drawings. However, those skilled in the art should understand that the above implementations are only for the purpose of clearly illustrating the present disclosure, and are not intended to limit the scope of the present disclosure. The protection scope of the present disclosure is obviously not limited to these specific implementations. Without departing from the principles of the present disclosure, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present disclosure.

Claims

1. A method for 3D information correction using a deep convolutional neural network, characterized in that: The steps include: Step 1, obtain the current camera image data; Step 2: input the acquired current camera image data; Step 3, detection stage: use a 3D monocular detection network to detect the image data to obtain a target image detection result; Step 4, correction stage: use the residual correction network to correct the detection result to obtain corrected image data; Step 5, outputting the corrected image data; The camera is a monocular camera; Input a monocular camera image in real time to obtain the real-time image of the monocular camera; These include: 3D monocular detection network inference steps: The original data set consisting of image data collected in real time by a monocular camera is divided into a training set and a validation set. The training set in the divided data set is used to train the network, and then the validation set is input into the trained network to obtain the evaluation value of the network performance; Box matching and error interval calculation steps: According to the matching relationship between the detection result and the true label, the error between the two is calculated, and the mean and variance are counted; IOU-based detection frame matching steps: Perform IOU matching on the predicted labels of the validation set and the true labels to obtain the corresponding boxes; For each 3D real box in the real label, in the corresponding predicted label, loop through all boxes and calculate the 3D IOU and 2D IOU values ​​with the 3D real box, and take the one with the largest 3D IOU as the corresponding predicted box.

2. The method for 3D information correction using a deep convolutional neural network according to claim 1, characterized in that: If there is no prediction box that intersects the 3D real box, the one with the largest 2D IOU is taken as the corresponding prediction box, otherwise there is no corresponding prediction box.

3. The method for 3D information correction using a deep convolutional neural network according to claim 1, characterized in that: The calculation of the 2D IOU and 3D IOU is shown in formula (1): （1） A and B are two rectangular boxes or 3D boxes.

4. The method for 3D information correction using a deep convolutional neural network according to claim 1, characterized in that: According to the correspondence of the 3D real frame, the error of the regression quantity, that is, the error between the real value and the predicted value, is calculated, wherein the regression quantity includes the depth z and the yaw angle r_y of the 3D frame.

5. The method for 3D information correction using a deep convolutional neural network according to claim 4, characterized in that: The error follows a Gaussian distribution.

6. The method for 3D information correction using a deep convolutional neural network according to claim 5, characterized in that: Calculate the mean μ and standard deviation σ of the error Gaussian distribution, and then remove the influence of extreme values ​​according to the 3σ principle of Gaussian distribution to obtain the deviation range of each variable.

7. The method for 3D information correction using a deep convolutional neural network according to claim 6, characterized in that: Including generating training data, Generating training data comprises the following steps: According to the range of the error, data perturbation is performed on the original training set to generate training data.

8. The method for 3D information correction using a deep convolutional neural network according to claim 7, characterized in that: The deviation range is added to the yaw angle r_y and the depth z respectively, and the distribution of the deviation is a normal distribution. Then, the 3D frame of the target with the deviation is drawn in the original picture, and a mask is added to the front of it to distinguish the direction; then the picture is cropped, and the cropping ratio is expanded by 0.2 times the target width or target height in four directions according to the 2D frame, wherein the image data is the picture after adding the forward mask and cropping.

9. The method for 3D information correction using a deep convolutional neural network according to claim 7, characterized in that: The tag data is the sin value and cos value of the perturbation value of the yaw angle r_y and the perturbation value of the depth information z.

10. The method for 3D information correction using a deep convolutional neural network according to claim 8, characterized in that: Generating training data comprises the following steps: Randomly translate the cropping box to the left or right by 0.4 target width and crop it, leaving its label data unchanged; Mirror the training data horizontally.

11. The method for 3D information correction using a deep convolutional neural network according to claim 1, characterized in that: The residual correction network consists of two parts, namely a feature extraction network part and two regression branch network parts.

12. The method for 3D information correction using a deep convolutional neural network according to claim 11, characterized in that: The feature extraction network uses the ResNet101 network, which is composed of five convolution blocks, wherein the first layer Conv1 is a convolution layer with a convolution kernel size of 7x7 and 64 channels, the second layer Conv2 first passes through the max pooling layer for downsampling, and then is a convolution block, and the subsequent Conv3-Conv5 are all composed of different convolution blocks and two fully connected layers, wherein the first fully connected layer Fc1 is used to reduce the number of channels, and the second fully connected layer Fc2 is used to regress the required information.

13. The method for 3D information correction using a deep convolutional neural network according to claim 11, characterized in that: The images are normalized before being input into the network. The size of all images is normalized to 64*64. The training batch size is 8, the optimizer is Adam, the initial learning rate is 0.001, and the learning rate is multiplied by 0.8 after every 10 rounds of training. The feature extraction network uses ResNet101. After preprocessing, the image enters the feature extraction layer. The extracted features enter two branches for angle information and depth information correction. The loss function used for angle correction is smooth L1 loss function, and the mean square error is used for depth correction.

14. The method for 3D information correction using a deep convolutional neural network according to claim 11, characterized in that: The predicted value of the yaw angle output is the sin and cos values ​​of the correction angle θ. Then the corresponding offset angle is calculated based on the sin and cos values. The corrected yaw angle is obtained by subtracting the disturbance value from the label predicted by the first-level network.

15. The method for 3D information correction using a deep convolutional neural network according to claim 14, characterized in that: The value range of the yaw angle r_y is [-π, π].

16. The method for 3D information correction using a deep convolutional neural network according to claim 14, characterized in that: The yaw angles that are out of range after partial angle correction are processed and mapped into the range of [-π, π].

17. The method for 3D information correction using a deep convolutional neural network according to claim 16, characterized in that: If r_y+θ is in the range of [-2π, -π], then θ is added with 2π; if r_y+θ is in the range of [π, 2π], then θ is subtracted with 2π; as shown in formula (2): （2） Among them, r_y is the yaw angle, r_y ’ is the corrected yaw angle, θ is the correction angle; The activation function of the angle uses the L2 paradigm, transforms the sin and cos values ​​obtained by regression, and then restores the angle value based on the obtained sin and cos values; the transformation formula is shown in formula (3): （3） Among them, sinx and cosx are the sine and cosine values ​​​​finally used to calculate the angle. sin and O cos is the output value of the fully connected network, O sin is the predicted value of the cosine angle in the network, O cos is the predicted value of the sine angle.

18. The method for 3D information correction using a deep convolutional neural network according to claim 17, characterized in that: The input of the residual correction network normalizes all input images to a fixed size in advance, and the fixed size is 64x64 pixels.

19. The method for 3D information correction using a deep convolutional neural network according to claim 18, characterized in that: For input images with a size smaller than 64x64, they are enlarged to 64x64, the target size itself does not change, and the target depth value in the visual system becomes smaller; for images with a size larger than 64x64, they are reduced to 64x64, the target size itself does not change, and the target depth value in the visual system becomes larger.

Images