3D target detection method and system based on visual image

Abstract

The invention provides a 3D target detection method and system based on a visual image, and the method comprises the steps: S1, collecting and preprocessing a visual image, and obtaining a preprocessed visual image; s2, performing feature extraction on the preprocessed visual image to obtain a feature map; s3, obtaining a feature fusion feature map based on the feature pyramid of the feature map; s4, performing target prediction and target regression on a detection head part of the feature fusion feature map to obtain a prediction confidence coefficient and a bounding box of the target; s5, whether the IOU of the target bounding box is larger than a preset threshold value or not is judged, and if yes, the corresponding bounding box is filtered out; if not, not processing; and S6, based on the bounding box obtained in the step S5, calculating and outputting 3D information of the obstacle through a regression network. According to the small target detection method based on the visual image, the recognition effect of a farther or smaller target object in an unmanned scene and other scenes is improved, and the depth information of the target object can be obtained.

Description

Technical Field

[0001] The present invention belongs to the field of unmanned vehicle target detection technology, specifically, to a method and system for 3D target detection based on visual images. More specifically, it relates to real-time 3D detection of small objects such as pedestrians by unmanned vehicles in industrial scenarios. Background Art

[0002] With the rapid development of autonomous driving technology, environmental perception, the "eyes" of autonomous vehicles, has long been a key focus in autonomous driving technology both domestically and internationally. Detecting smaller objects, such as pedestrians, has always been a key and challenging task, especially when obstructed or at a distance, such as pedestrians or even smaller objects.

[0003] Patent document CN115984268A discloses a machine vision-based target detection method, apparatus, electronic device, and medium. The method includes determining an image of a target to be detected; determining at least two target sliding windows for the image to be detected; segmenting the image to be detected using the target sliding windows to obtain corresponding segmented sub-images to be detected, wherein the at least two target sliding windows include at least two sliding windows of different preset sizes; and performing defect detection on the target to be detected using the segmented sub-images to be detected, which are segmented using the at least two target sliding windows. However, this method is computationally expensive, making it difficult to achieve real-time target detection.

[0004] Patent document CN111582093A discloses a method for automatic detection of small targets in high-resolution images based on computer vision and deep learning. The main steps include: first, decomposing the original small target detection task at different scales to obtain a multi-scale task group. Then, low-resolution detectors are trained at different scales and applied to detection to obtain detection results at different scales. Finally, these detection results are fused to obtain the final detection result for small targets. However, this method also has limited effect on targets with smaller pixels, increases calculation time, and cannot calculate target depth.

[0005] Patent document CN116721331A discloses a small target detection method based on an improved Faster R-CNN algorithm. This approach first uses a multi-scale auxiliary feature network to extract shallow features containing spatial and detail information from downsampled images. A combination of ResNet50 and FPN is used as the backbone feature extraction network to extract these features. A fusion module then fuses these backbone and shallow features into a new feature map, ensuring that the feature map retains spatial information and small target information. Finally, the feature map is fed into a classification and regression network. Feature maps for candidate boxes are extracted using RPN. ROIAlign is used instead of ROIPooling to extract feature maps of interest, and target classification and location are performed on the candidate boxes of interest. This approach is unable to detect small targets and obtain depth information.

[0006] This problem needs to be solved urgently. Summary of the Invention

[0007] In view of the defects in the prior art, the purpose of the present invention is to provide a 3D target detection method and system based on visual images.

[0008] A 3D object detection method based on visual images provided by the present invention includes:

[0009] Step S1: collecting and preprocessing visual images to obtain preprocessed visual images;

[0010] Step S2: feature extracting the preprocessed visual image to obtain a feature map;

[0011] Step S3: obtaining a feature fusion feature map based on the feature pyramid of the feature map;

[0012] Step S4: target prediction and target regression The feature fusion feature map is used to obtain the prediction confidence and the target bounding box;

[0013] Step S5: Determine whether the IOU of the target bounding box is greater than a preset threshold. If yes, filter out the corresponding bounding box; if no, do not process it.

[0014] Step S6: Based on the bounding box obtained in step S5, the 3D information of the obstacle is calculated and output through the regression network.

[0015] Preferably, in step S1, the preprocessing is to sample the visual image at every other pixel in the width direction and the height direction respectively through the Focus module to obtain four preprocessing feature maps; and then splice the four preprocessing feature maps in the channel dimension to obtain a preprocessed visual image.

[0016] Preferably, the step S2 includes:

[0017] Step S2.1: Based on CSPNet and DCN, feature extraction and preprocessing of visual images;

[0018] Step S2.2: Obtain feature maps of different scales based on a network structure composed of multiple layers of CBL and CSPNet; the CBL includes convolutional layers, batch normalization layers, and Leaky ReLU activation functions;

[0019] In the step S2.1, it includes: using a backbone network based on CSPNet to process the preprocessed visual image as an input image to extract features of different scales;

[0020] The deformation of the pre-processed visual image is adjusted through the DCN module of the convolutional layer.

[0021] Preferably, the step S3 includes:

[0022] Step S3.1: Based on the feature pyramid of the feature map, the deep network is upsampled through the FPN network structure, and then the shallow network features are fused to obtain the semantic features of the feature fusion feature map;

[0023] Step S3.2: Based on the PANet network structure, shallow network features are fused into deep network features, and the positioning information of the feature fusion feature map is output;

[0024] Step S3.3: Output the feature fusion feature map to the detection head;

[0025] The detection head part includes multiple detection layers, and each detection layer corresponds to a feature fusion feature map of one scale; the detection head part can generate output feature fusion feature maps of four different scales, i.e., different resolutions;

[0026] The numbers of grids of the four different resolutions are 20×20, 40×40, 80×80 and 160×160 respectively;

[0027] In step S4, the “target” refers to an object that the detection model is expected to identify, i.e., a target obstacle;

[0028] In the step S5, it includes:

[0029] Step S5.1: Compare the predicted confidence score with the bounding box of the target obstacle to determine whether the predicted confidence score of the bounding box of the target obstacle is less than 0.3. If the result is yes, the corresponding bounding box is filtered out to obtain the remaining bounding box. If the result is no, no processing is performed.

[0030] Step S5.2: Sort the remaining bounding boxes in ascending or descending order according to the confidence level, and then calculate and obtain the IOU between the bounding boxes. Determine whether the IOU value is greater than 0.7. If the result is yes, filter out the corresponding bounding box; if the result is no, do not process it.

[0031] Preferably, the step S6 includes:

[0032] Step S6.1: Assume that the ground is level and the roll and pitch angles of the target obstacle are 0;

[0033] Step S6.2: Predict and obtain the size vector of the target obstacle using the ResNet18 model; the size vector includes length, width, height, and orientation angle;

[0034] Step S6.3: Based on the size vector, using a pinhole camera model, calculate the center position, horizontal coordinates, and vertical coordinates of the 3D bounding box of the target obstacle;

[0035] The mathematical expression of the center position is:

[0036]

[0037] Where Z is the center position of the 3D bounding box of the target obstacle on the z axis; f y is the vertical focal length of the pinhole camera’s intrinsic parameters; H is the height of the target obstacle; h is the pixel height of the target obstacle in the image;

[0038] The mathematical expressions of the horizontal coordinate and the vertical coordinate are as follows from top to bottom:

[0039]

[0040] Where u and v are the pixel horizontal and vertical coordinates of the center of the target obstacle in the image plane respectively; c x 、c y are the horizontal and vertical coordinates of the optical center of the pinhole camera in the image plane; f x 、f y are the horizontal and vertical focal lengths of the pinhole camera, respectively; X, Y, and Z are the three-dimensional coordinates of the center of the 3D frame of the target obstacle in the pinhole camera coordinate system, and the 3D information of the target obstacle.

[0041] A 3D object detection system based on visual images provided by the present invention includes:

[0042] Module M1: collecting and preprocessing visual images to obtain preprocessed visual images;

[0043] Module M2: Feature extraction of the preprocessed visual image to obtain a feature map;

[0044] Module M3: obtains feature fusion feature map based on feature pyramid of feature map;

[0045] Module M4: Target Prediction and Target Regression The feature fusion feature map detection head part is used to obtain the prediction confidence and the target bounding box;

[0046] Module M5: Determine whether the IOU of the target bounding box is greater than the preset threshold. If the result is yes, the corresponding bounding box is filtered out; if the result is no, no processing is performed;

[0047] Module M6: Based on the bounding box obtained by module M5, the 3D information of the obstacle is calculated and output through the regression network.

[0048] Preferably, in the module M1, the preprocessing is to sample the visual image at every other pixel in the width direction and the height direction respectively through the Focus module to obtain four preprocessing feature maps; and then splice the four preprocessing feature maps in the channel dimension to obtain a preprocessed visual image.

[0049] Preferably, the module M2 includes:

[0050] Module M2.1: Feature extraction and preprocessing of visual images based on CSPNet and DCN;

[0051] Module M2.2: Obtain feature maps of different scales based on the network structure of a multi-layer CBL and CSPNet combination; the CBL includes convolutional layers, batch normalization layers, and Leaky ReLU activation functions;

[0052] In the module M2.1, it includes: using a CSPNet-based backbone network to process a pre-processed visual image as an input image to extract features of different scales;

[0053] The deformation of the pre-processed visual image is adjusted through the DCN module of the convolutional layer.

[0054] Preferably, the module M3 includes:

[0055] Module M3.1: Based on the feature pyramid of the feature map, the deep network is upsampled through the FPN network structure, and then the shallow network features are fused to obtain the semantic features of the feature fusion feature map;

[0056] Module M3.2: Based on the PANet network structure, it fuses shallow network features into deep network features and outputs the positioning information of the feature fusion feature map;

[0057] Module M3.3: outputs the feature fusion feature map to the detection head;

[0058] The detection head part includes multiple detection layers, and each detection layer corresponds to a feature fusion feature map of one scale; the detection head part can generate output feature fusion feature maps of four different scales, i.e., different resolutions;

[0059] The numbers of grids of the four different resolutions are 20×20, 40×40, 80×80 and 160×160 respectively;

[0060] In the module M4, the “target” refers to the object that the detection model is expected to identify, i.e., the target obstacle;

[0061] The module M5 includes:

[0062] Module M5.1: Predicted confidence and target obstacle bounding box, determine whether the predicted confidence of the target obstacle bounding box is less than 0.3; if the result is yes, filter out the corresponding bounding box to obtain the remaining bounding box; if the result is no, do not process it;

[0063] Module M5.2: Sort the remaining bounding boxes in ascending or descending order according to the confidence level, and then calculate and obtain the IOU between the bounding boxes. Determine whether the IOU value is greater than 0.7. If the result is yes, the corresponding bounding box is filtered out. If the result is no, it is not processed.

[0064] Preferably, the module M6 includes:

[0065] Module M6.1: Assume the ground is level and the roll and pitch angles of the target obstacle are 0;

[0066] Module M6.2: Predict and obtain the size vector of the target obstacle through the ResNet18 model; the size vector includes length, width, height and orientation angle;

[0067] Module M6.3: Based on the size vector, calculate the center position, horizontal coordinates, and vertical coordinates of the 3D bounding box of the target obstacle using a pinhole camera model;

[0068] The mathematical expression of the center position is:

[0069]

[0070] Where Z is the center position of the 3D bounding box of the target obstacle on the z axis; f y is the vertical focal length of the pinhole camera’s intrinsic parameters; H is the height of the target obstacle; h is the pixel height of the target obstacle in the image;

[0071] The mathematical expressions of the horizontal coordinate and the vertical coordinate are as follows from top to bottom:

[0072]

[0073] Where u and v are the pixel horizontal and vertical coordinates of the center of the target obstacle in the image plane respectively; c x 、c y are the horizontal and vertical coordinates of the optical center of the pinhole camera in the image plane; f x 、f y are the horizontal and vertical focal lengths of the pinhole camera, respectively; X, Y, and Z are the three-dimensional coordinates of the center of the 3D frame of the target obstacle in the pinhole camera coordinate system, and the 3D information of the target obstacle.

[0074] Compared with the prior art, the present invention has the following beneficial effects:

[0075] 1. This invention proposes a small target detection method based on visual images, which improves the recognition effect of distant or small targets in scenes such as unmanned driving, and can obtain the depth information of the target.

[0076] 2. The present invention can be applied to other application scenarios that require high-precision detection of smaller targets.

[0077] 3. The present invention uses visual images as input and uses the YOLOv5 model as the backbone to optimize the model structure of the backbone, mainly including: introducing variable convolution to improve the feature extraction capability of the target edge, adding a 160*160 feature map with a smaller receptive field, improving the feature extraction capability of small targets, and improving the model's recognition effect on targets such as pedestrians; adding Resnet18 and geometric features to predict / calculate the 3D features of the target to obtain the 3D information of the target; through this algorithm model, the recognition effect of small targets and the problem of visual depth prediction can be effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0079] Figure 1 This is a flowchart of the workflow provided by the present invention. DETAILED DESCRIPTION

[0080] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0081] The present invention provides a small target detection method based on visual images, which can be run in real time and can obtain depth information. The purpose is to provide a method, system and steps for realizing roadbed environment perception of unmanned transport vehicles based on laser radar, so as to provide effective safety protection for unmanned vehicles driving in industrial environments.

[0082] The present invention provides a real-time 3D object detection method based on visual images, comprising: performing downsampling and feature enhancement based on RGB feature input of visual images by operations such as Focus; a feature extraction network based on a backbone network, namely a CrossStage PartialNetwork (CSPNet) and a deformable convolution layer (DCN); and achieving feature fusion of different receptive fields based on a Feature Pyramid Network (FPN) and a Path Aggregation Network (PANet).

[0083] Target detection and regression based on different feature maps; Non-Maximum Suppression of overlapping boxes, also known as NMS filtering; and 3D information of obstacles is inferred based on ResNet18 network regression and geometric relationships.

[0084] Specifically, based on the original YOLOV5 network structure, the present invention adopts DCN in Backbone to replace the convolution layer of a specific layer in CSPNet, namely Convolution, to improve the feature extraction capability for targets with complex and changeable morphology; adds a 160×160 Feature Map to improve the recognition capability for small targets with pixels less than 8×8 at a resolution of 640×640; and adds a ResNet18 network to regress 3D information based on 2D detection results.

[0085] In other words, according to the present invention, a small target detection method based on visual images, the process of its software system includes:

[0086] Step 1: Collect and preprocess visual images;

[0087] Specifically, in step 1, the visual image, i.e., the visual image RGB feature;

[0088] Specifically, step 1 includes:

[0089] Step 1.1: The Focus module samples the visual image at every other pixel in width and height to obtain four feature maps with half the resolution but increased number of channels.

[0090] Step 1.2: Concatenate the four feature maps in the channel dimension to form a new feature map, i.e., the preprocessed visual image. The visual image is downsampled and feature enhanced.

[0091] Specifically, the purpose of this processing is to achieve efficient downsampling and feature extraction without losing key information, thereby accelerating the overall detection algorithm.

[0092] Step 2: Feature extraction of the preprocessed visual image;

[0093] In the step 2, it includes:

[0094] Step 2.1: Feature extraction based on CSPNet and DCN;

[0095] In step 2.1, feature extraction is specifically implemented as follows, including:

[0096] Step 2.1.1: Use the CSPNet-based backbone network to process the input image and extract multi-scale, high-quality feature representations;

[0097] Step 2.1.2: Introduce the DCN module in a specific convolutional layer to make the convolution operation adaptive and able to adjust to the deformation of the object. Deformation refers to non-rigid changes in the geometry, posture, scale, and perspective of the object in the image.

[0098] Step 2.2: Based on the multi-layer CBL and CSPNet network structure, feature maps with different receptive fields are obtained. The receptive field represents the area of the original input image corresponding to a pixel in a feature map of a certain layer in a convolutional neural network. The receptive field is divided into shallow and deep layers: the shallow layer is used to capture details and accurately locate the image; the deep layer is used to extract semantic meaning and accurately classify the image.

[0099] The CBL is a combination of a convolutional layer, a batch normalization layer, and a Leaky ReLU activation function;

[0100] The batch normalization layer, i.e. Batch Normalization;

[0101] The convolutional layer, i.e., Convolutional Layer;

[0102] These feature maps of different scales will serve as the input of the Neck network, i.e., step 3, for subsequent feature fusion processing.

[0103] Step 3: Based on the feature pyramid, i.e., FPN, and the path aggregation network, i.e., PAnet, feature fusion of different receptive fields is achieved;

[0104] In step 3, it includes:

[0105] Step 3.1: Use the FPN network structure to upsample the deep network and then fuse it with the shallow network features with a larger receptive field, i.e., concat, to improve the semantic features;

[0106] In step 3.1, the deep feature maps of the backbone network are upsampled step by step using the FPN network structure and fused with the shallow feature maps with relatively small receptive fields but higher resolutions, that is, the semantic richness of the output features is improved through the feature concatenation operation.

[0107] Step 3.2: Based on the PANet network structure, the shallow network features are integrated into the deep network features to enhance the positioning features.

[0108] Specifically, the two network structures in step 3 are designed to combine high-level semantic features with low-level positioning details to achieve full integration of features from different receptive fields.

[0109] In step 3.2, the PANet network structure is used to transfer shallow features from bottom to top step by step, and then fuse them with deep features again to strengthen the positioning features of the target. That is, by propagating the fine-grained features of the low layer upward to the high layer, the deep layer output features also contain fine positioning information.

[0110] Finally, multiple sets of fused feature maps are output. These feature maps contain both semantic information and positioning information at different scales and are passed to the detection head for target prediction and regression.

[0111] Step 4: Target detection prediction is performed based on different feature maps. The head part contains multiple detection layers. Each detection layer corresponds to a feature map of a certain scale, which is used to regress and output the target bounding box parameters and classification confidence at that scale.

[0112] In step 4, it includes:

[0113] Step 4.1: The head part contains multiple detection layers, each detection layer corresponds to a feature map of a different scale; four output feature maps of different resolutions are generated, with grid sizes of approximately 20×20, 40×40, 80×80, and 160×160, respectively, corresponding to the detection of objects of approximately 32×32, 16×16, 8×8, and 4×4 pixels in the original image;

[0114] Step 4.2: For each scale feature map provided by Neck, the Head part adjusts and expands the number of feature channels through a 1×1 convolution to:

[0115] (N+5)*A

[0116] Where N is the number of target categories; A is the number of anchor boxes preset for each grid cell; and +5 corresponds to the horizontal coordinate, vertical coordinate, width, height, and confidence level of the predicted box center, respectively. The confidence level can be directly obtained through the 1x1 convolution in step 4.2.

[0117] In other words, the purpose of step 4.2 is to obtain the final result, namely the horizontal coordinate, vertical coordinate, width, height and confidence of the center point of the prediction box.

[0118] In this implementation case, the number of anchor boxes on each detection layer is A=4, that is, 4 anchor boxes with different aspect ratios are preset to generate the bounding box of the candidate target and its classification result.

[0119] Step 5: Filter the boxes with larger IOU (Intersection over Union) based on NMS.

[0120] In said step 5, it includes:

[0121] Step 5.1: Based on the predicted confidence, preliminarily filter the boxes with a confidence level less than 0.3;

[0122] Step 5.2: Sort the remaining target boxes based on confidence, calculate the IOU between different boxes, and filter out the target boxes with IOU greater than 0.7 and confidence less than 0.3.

[0123] Step 6: Calculate the 3D obstacle information based on Resnet18 and geometric relationships.

[0124] Specifically, step 6 includes:

[0125] Step 6.1: Assuming the ground is level, the roll and pitch angles of the obstacle are 0;

[0126]

[0127] Rot={pitch,roll,yaw}Rot={0,0,yaw}

[0128] Where: Dim represents the size information of the obstacle 3D frame; Rot represents the rotation angle information of the obstacle 3D frame;

[0129] L, W, and H are the length, width, and height of the target obstacle 3D frame in the camera coordinate system, respectively;

[0130] pitch, roll, and yaw are the pitch, roll, and yaw angles of the 3D box of the target obstacle, respectively.

[0131] Step 6.2: Predict the length, width, height, and orientation angle of the obstacle based on ResNet18 network regression;

[0132] Input the visual image and the result of step 6 into the trained ResNet18 model, and output the predicted three-dimensional size vector of the obstacle, namely the length, width, height, and orientation angle.

[0133] Specifically, this information is combined with the camera imaging model in step 6.3 to calculate the spatial position of the target relative to the camera. By extracting depth features and predicting the target scale, the system can understand the true size of the target in the image, providing a key basis for subsequent distance and position calculations.

[0134] Step 6.3: Based on the pinhole camera model, calculate the depth information of the obstacle, that is, the center position of the obstacle's 3D frame:

[0135]

[0136] Where Z is the center position of the obstacle 3D frame on the z-axis; f y is the vertical focal length of the camera intrinsic parameter; H is the height of the obstacle, that is, the height of the predicted value obtained in step 6.2; h is the pixel height of the obstacle in the image.

[0137] Then, the horizontal and vertical coordinates of the obstacle are obtained by combining the camera internal parameters, that is, (X, Y). The mathematical expression is:

[0138]

[0139] Where u and v are the pixel horizontal and vertical coordinates of the obstacle center in the image plane respectively; c x 、c y are the horizontal and vertical coordinates of the optical center of the camera in the image plane; f x 、f y are the horizontal and vertical focal lengths of the camera respectively; X, Y, and Z are the three-dimensional coordinates of the center of the obstacle's 3D frame in the camera coordinate system.

[0140] In other words, a small target detection method based on visual images provided by the present invention includes:

[0141] Step S1: Use Focus operation to downsample and enhance features;

[0142] Step S2: Feature extraction based on CSPnet and deformable convolution as the backbone;

[0143] Step S3: Based on the feature pyramid, i.e., FPN, and the path aggregation network, i.e., PAnet, feature fusion of different receptive fields is achieved;

[0144] Step S4: perform regression prediction based on different feature maps;

[0145] Step S5: Filter the frames with larger IOU based on NMS.

[0146] Step S6: Calculate the 3D obstacle information based on Resnet18 and geometric relationships.

[0147] Specifically, step S2 includes the following steps:

[0148] Step S2.1: Feature extraction based on CSPnet and Deformable conv;

[0149] Step S2.2: Based on the multi-layer CSP+CBL network structure, feature maps under different receptive fields are obtained and sent to the neck part for feature fusion;

[0150] Specifically, step S3 includes the following steps:

[0151] Step S3.1: Use the FPN network structure to upsample the deep network and then fuse it with the shallow network features with a larger receptive field, i.e. concat, to improve the semantic features;

[0152] Step S3.2: Based on the PAnet network structure, shallow features are integrated into deep network features to enhance positioning features.

[0153] Specifically, step S4 includes the following steps:

[0154] Step S4.1: The head part outputs feature map sizes of 20*20, 40*40, 80*80, and 160*160, corresponding to targets of 32*32, 16*16, 8*8, and 4*4 pixels respectively;

[0155] Step S4.2: Head expands the number of channels of the feature maps of different scales obtained in Neck through 1×1 convolution. The number of feature channels after expansion is (number of categories + 5) × the number of anchors on each detection layer. Among them, 5 corresponds to the horizontal coordinate, vertical coordinate, width, height and confidence of the center point of the prediction box; the four detection layers in Head correspond to the four feature maps of different sizes obtained in Neck. Each grid on the feature map is preset with four anchors of different aspect ratios, which can save all the position information and classification information based on the anchor prior box in the channel dimension of the feature map for prediction and regression of targets.

[0156] In the step S5, it includes:

[0157] Step S5.1: Based on the predicted confidence, preliminarily filter the boxes with a confidence level less than 0.3;

[0158] Step S5.2: Sort the remaining target boxes based on confidence, calculate the IOU between different boxes, and filter out the target boxes with IOU greater than 0.7 and low confidence.

[0159] Specifically, step S6 includes:

[0160] Step S6.1: Assuming the ground is level, the roll angle and pitch angle of the obstacle are 0;

[0161] Step S6.2: Predict the length, width, height, and orientation angle of the obstacle based on Resnet18 network regression;

[0162] Step S6.3: Based on the pinhole camera model, the depth information of the obstacle is calculated, and the longitudinal and lateral coordinates (cx, cy) of the obstacle are obtained by combining the camera intrinsic parameters.

[0163] The present invention also provides a 3D target detection system based on visual images. The 3D target detection system based on visual images can be implemented by executing the process steps of the 3D target detection method based on visual images. That is, those skilled in the art can understand the 3D target detection method based on visual images as a preferred implementation of the 3D target detection system based on visual images.

[0164] Those skilled in the art will appreciate that, in addition to implementing the system and its various devices, modules, and units provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same functions of the system and its various devices, modules, and units provided by the present invention in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; the devices, modules, and units for implementing various functions can also be considered as both software modules implementing the method and structures within the hardware component.

[0165] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A 3D object detection method based on visual images, characterized in that: include: Step S1: collecting and preprocessing visual images to obtain preprocessed visual images; Step S2: feature extracting the preprocessed visual image to obtain a feature map; Step S3: obtaining a feature fusion feature map based on the feature pyramid of the feature map; Step S4: target prediction and target regression The feature fusion feature map is used to obtain the prediction confidence and the target bounding box; Step S5: Determine whether the IOU of the target bounding box is greater than a preset threshold. If yes, filter out the corresponding bounding box; if no, do not process it. Step S6: Based on the bounding box obtained in step S5, the 3D information of the obstacle is calculated and output through the regression network.

2. The 3D target detection method based on visual images according to claim 1, characterized in that: In the step S1, the preprocessing is to sample the visual image at every other pixel in the width direction and the height direction respectively through the Focus module to obtain four preprocessing feature maps; and then splice the four preprocessing feature maps in the channel dimension to obtain a preprocessed visual image.

3. The 3D target detection method based on visual images according to claim 2, characterized in that: In the step S2, it includes: Step S2.1: Based on CSPNet and DCN, feature extraction and preprocessing of visual images; Step S2.2: Obtain feature maps of different scales based on a network structure composed of multiple layers of CBL and CSPNet; the CBL includes convolutional layers, batch normalization layers, and Leaky ReLU activation functions; In the step S2.1, it includes: using a backbone network based on CSPNet to process the preprocessed visual image as an input image to extract features of different scales; The deformation of the pre-processed visual image is adjusted through the DCN module of the convolutional layer.

4. The 3D target detection method based on visual images according to claim 1, characterized in that: In the step S3, it includes: Step S3.1: Based on the feature pyramid of the feature map, the deep network is upsampled through the FPN network structure, and then the shallow network features are fused to obtain the semantic features of the feature fusion feature map; Step S3.2: Based on the PANet network structure, shallow network features are fused into deep network features, and the positioning information of the feature fusion feature map is output; Step S3.3: Output the feature fusion feature map to the detection head; The detection head part includes multiple detection layers, and each detection layer corresponds to a feature fusion feature map of one scale; the detection head part can generate output feature fusion feature maps of four different scales, i.e., different resolutions; The numbers of grids of the four different resolutions are 20×20, 40×40, 80×80 and 160×160 respectively; In step S4, the “target” is the target obstacle; In the step S5, it includes: Step S5.1: Compare the predicted confidence score with the bounding box of the target obstacle to determine whether the predicted confidence score of the bounding box of the target obstacle is less than 0.

3. If the result is yes, the corresponding bounding box is filtered out to obtain the remaining bounding box. If the result is no, no processing is performed. Step S5.2: Sort the remaining bounding boxes in ascending or descending order according to the confidence level, and then calculate and obtain the IOU between the bounding boxes. Determine whether the IOU value is greater than 0.

7. If the result is yes, filter out the corresponding bounding box; if the result is no, do not process it.

5. The 3D target detection method based on visual images according to claim 4, characterized in that: In the step S6, it includes: Step S6.1: Assume that the ground is level and the roll and pitch angles of the target obstacle are 0; Step S6.2: Predict and obtain the size vector of the target obstacle using the ResNet18 model; the size vector includes length, width, height, and orientation angle; Step S6.3: Based on the size vector, using a pinhole camera model, calculate the center position, horizontal coordinates, and vertical coordinates of the 3D bounding box of the target obstacle; The mathematical expression of the center position is: Where Z is the center position of the 3D bounding box of the target obstacle on the z axis; f y is the vertical focal length of the pinhole camera’s intrinsic parameters; H is the height of the target obstacle; h is the pixel height of the target obstacle in the image; The mathematical expressions of the horizontal coordinate and the vertical coordinate are as follows from top to bottom: Where u and v are the pixel horizontal and vertical coordinates of the center of the target obstacle in the image plane respectively; c x 、c y are the horizontal and vertical coordinates of the optical center of the pinhole camera in the image plane; f x 、f y are the horizontal and vertical focal lengths of the pinhole camera, respectively; X, Y, and Z are the three-dimensional coordinates of the center of the 3D frame of the target obstacle in the pinhole camera coordinate system, and the 3D information of the target obstacle.

6. A 3D object detection system based on visual images, characterized in that: include: Module M1: collects and preprocesses visual images to obtain preprocessed visual images; Module M2: Feature extraction of the preprocessed visual image to obtain a feature map; Module M3: obtains feature fusion feature map based on feature pyramid of feature map; Module M4: Target Prediction and Target Regression The feature fusion feature map detection head part is used to obtain the prediction confidence and the target bounding box; Module M5: Determine whether the IOU of the target bounding box is greater than the preset threshold. If the result is yes, the corresponding bounding box is filtered out; if the result is no, no processing is performed; Module M6: Based on the bounding box obtained by module M5, the 3D information of the obstacle is calculated and output through the regression network.

7. The 3D object detection system based on visual images according to claim 6, characterized in that: In the module M1, the preprocessing is to sample the visual image at every other pixel in the width direction and the height direction through the Focus module to obtain four preprocessing feature maps; and then splice the four preprocessing feature maps in the channel dimension to obtain a preprocessed visual image.

8. The 3D object detection system based on visual images according to claim 7, characterized in that: The module M2 includes: Module M2.1: Feature extraction and preprocessing of visual images based on CSPNet and DCN; Module M2.2: Obtain feature maps of different scales based on the network structure of a multi-layer CBL and CSPNet combination; the CBL includes convolutional layers, batch normalization layers, and Leaky ReLU activation functions; In the module M2.1, it includes: using a CSPNet-based backbone network to process a pre-processed visual image as an input image to extract features of different scales; The deformation of the pre-processed visual image is adjusted through the DCN module of the convolutional layer.

9. The 3D object detection system based on visual images according to claim 8, characterized in that: The module M3 includes: Module M3.1: Based on the feature pyramid of the feature map, the deep network is upsampled through the FPN network structure, and then the shallow network features are fused to obtain the semantic features of the feature fusion feature map; Module M3.2: Based on the PANet network structure, it fuses shallow network features into deep network features and outputs the positioning information of the feature fusion feature map; Module M3.3: outputs the feature fusion feature map to the detection head; The detection head part includes multiple detection layers, and each detection layer corresponds to a feature fusion feature map of one scale; the detection head part can generate output feature fusion feature maps of four different scales, i.e., different resolutions; The numbers of grids of the four different resolutions are 20×20, 40×40, 80×80 and 160×160 respectively; In the module M4, the “target” is the target obstacle; The module M5 includes: Module M5.1: Predicted confidence and target obstacle bounding box, determine whether the predicted confidence of the target obstacle bounding box is less than 0.3; if the result is yes, filter out the corresponding bounding box to obtain the remaining bounding box; if the result is no, do not process it; Module M5.2: Sort the remaining bounding boxes in ascending or descending order according to the confidence level, and then calculate and obtain the IOU between the bounding boxes. Determine whether the IOU value is greater than 0.

7. If the result is yes, the corresponding bounding box is filtered out. If the result is no, it is not processed.

10. The 3D object detection system based on visual images according to claim 9, characterized in that: The module M6 includes: Module M6.1: Assume the ground is level and the roll and pitch angles of the target obstacle are 0; Module M6.2: Predict and obtain the size vector of the target obstacle through the ResNet18 model; the size vector includes length, width, height and orientation angle; Module M6.3: Based on the size vector, calculate the center position, horizontal coordinates, and vertical coordinates of the 3D bounding box of the target obstacle using a pinhole camera model; The mathematical expression of the center position is: Where Z is the center position of the 3D bounding box of the target obstacle on the z axis; f y is the vertical focal length of the pinhole camera’s intrinsic parameters; H is the height of the target obstacle; h is the pixel height of the target obstacle in the image; The mathematical expressions of the horizontal coordinate and the vertical coordinate are as follows from top to bottom: Where u and v are the pixel horizontal and vertical coordinates of the center of the target obstacle in the image plane respectively; c x 、c y are the horizontal and vertical coordinates of the optical center of the pinhole camera in the image plane; f x 、f y are the horizontal and vertical focal lengths of the pinhole camera, respectively; X, Y, and Z are the three-dimensional coordinates of the center of the 3D frame of the target obstacle in the pinhole camera coordinate system, that is, the 3D information of the target obstacle.

Images