3D object detection method and system based on visual images

Abstract

This invention provides a 3D target detection method and system based on visual images, comprising: Step S1: acquiring and preprocessing visual images to obtain preprocessed visual images; Step S2: extracting features from the preprocessed visual images to obtain feature maps; Step S3: obtaining a feature fusion feature map based on the feature pyramid of the feature maps; Step S4: predicting and regressing the target by analyzing the detection head portion of the feature fusion feature map to obtain the predicted confidence level and the target's bounding box; Step S5: determining whether the Intersection over Union (IoU) of the target bounding box is greater than a preset threshold; if yes, the corresponding bounding box is filtered out; if no, no processing is performed; Step S6: calculating and outputting the 3D information of the obstacle based on the bounding box obtained in Step S5 through a regression network. This invention's small target detection method based on visual images improves the recognition effect of distant or small targets in scenarios such as autonomous driving, and can obtain the depth information of the target.

Description

Technical Field

[0001] This invention belongs to the field of unmanned vehicle target detection technology, specifically, it relates to a 3D target detection method and system based on visual images. More specifically, it relates to real-time 3D detection of small target objects such as pedestrians by unmanned vehicles in industrial scenarios. Background Technology

[0002] With the rapid development of autonomous driving technology, environmental perception, as the "eyes" of autonomous vehicles, has always been a key focus in the field of autonomous driving technology both domestically and internationally. Detection of smaller targets such as pedestrians has always been a key focus and a challenge, especially for pedestrians who are obstructed, at a distance, or even smaller targets.

[0003] Patent document CN115984268A discloses a target detection method, apparatus, electronic device, and medium based on machine vision. The method includes determining a target image to be detected; determining at least two target sliding windows for the target image; segmenting the target image using the target sliding windows to obtain corresponding segmented images to be detected; the at least two target sliding windows include at least two sliding windows of different preset sizes; and performing defect detection on the target image based on the segmented images obtained from the at least two target sliding windows. However, this method is computationally very time-consuming, making real-time target detection difficult.

[0004] Patent document CN111582093A discloses an automatic small target detection method in high-resolution images based on computer vision and deep learning. The main steps include: first, decomposing the original small target detection task at different scales to obtain multi-scale task groups; then, training low-resolution detectors at different scales and applying them for detection to obtain detection results at different scales; finally, fusing these detection results to obtain the final detection result for small targets. However, this method also offers limited improvement for targets with even smaller pixels, increases computation time, and cannot calculate target depth.

[0005] Patent document CN116721331A discloses a small target detection method based on an improved Faster R-CNN algorithm. This scheme first uses a multi-scale auxiliary feature network to extract shallow features containing spatial and detail information from the downsampled image, and then uses a combination of ResNet50 and FPN as the backbone feature extraction network to extract backbone features. Next, a fusion module merges the backbone features and shallow features into a new feature map, ensuring that spatial and small target information is retained in the feature map. Finally, the feature map is input into a classification and regression network, where RPN extracts candidate bounding box feature maps, and ROIAlign is used instead of ROIPooling to extract feature maps of interest. The candidate bounding boxes of interest are then classified and localized. However, this scheme cannot detect small targets or obtain depth information.

[0006] This problem urgently needs to be solved. Summary of the Invention

[0007] In view of the shortcomings of the prior art, the purpose of this invention is to provide a 3D target detection method and system based on visual images.

[0008] A 3D target detection method based on visual images provided by the present invention includes:

[0009] Step S1: Acquire and preprocess the visual image to obtain a preprocessed visual image;

[0010] Step S2: Extract features from the preprocessed visual image to obtain a feature map;

[0011] Step S3: Based on the feature pyramid of the feature map, obtain the feature fusion feature map;

[0012] Step S4: Target prediction and target regression: The detection head part of the feature map is fused to obtain the predicted confidence and the bounding box of the target;

[0013] Step S5: Determine whether the IOU of the target bounding box is greater than a preset threshold. If the result is yes, filter out the corresponding bounding box; if the result is no, do not process it.

[0014] Step S6: Based on the bounding box obtained in step S5, calculate and output the 3D information of the obstacle through a regression network.

[0015] Preferably, in step S1, the preprocessing involves sampling the visual image at every pixel in both the width and height directions using the Focus module to obtain four preprocessed feature maps; then, the four preprocessed feature maps are stitched together along the channel dimension to obtain a preprocessed visual image.

[0016] Preferably, step S2 includes:

[0017] Step S2.1: Based on CSPNet and DCN, extract features from the visual image and preprocess it;

[0018] Step S2.2: Based on the network structure combining multi-layer CBL and CSPNet, feature maps of different scales are obtained; the CBL includes convolutional layers, batch normalization layers, and the Leaky ReLU activation function;

[0019] In step S2.1, the following steps are included: using a CSPNet-based backbone network to process the preprocessed visual image as the input image and extracting features at different scales;

[0020] The DCN module of the convolutional layer adjusts the deformation of the preprocessed visual image.

[0021] Preferably, step S3 includes:

[0022] Step S3.1: Based on the feature pyramid of the feature map, the deep network is upsampled through the FPN network structure, and then the features of the shallow network are fused to obtain the semantic features of the feature fusion feature map;

[0023] Step S3.2: Based on the PANet network structure, fuse shallow network features into deep network features and output the localization information of the feature fusion feature map;

[0024] Step S3.3: Output the fused feature map to the detection head.

[0025] The detection head includes multiple detection layers, and each detection layer corresponds to a feature fusion feature map at a certain scale; the detection head can generate four different scales, i.e., different resolutions of output feature fusion feature maps.

[0026] The number of grids for the four different resolutions are 20×20, 40×40, 80×80 and 160×160, respectively.

[0027] In step S4, the “target” refers to the object that the detection model is expected to identify, i.e., the target obstacle.

[0028] Step S5 includes:

[0029] Step S5.1: Compare the predicted confidence score with the bounding box of the target obstacle. Determine if the predicted confidence score of the bounding box of the target obstacle is less than 0.3. If the result is yes, filter out the corresponding bounding box and obtain the remaining bounding boxes. If the result is no, do not process it.

[0030] Step S5.2: Sort the remaining bounding boxes in ascending or descending order according to the confidence level, then calculate and obtain the IOU between the bounding boxes, and determine whether the IOU value is greater than 0.7. If the result is yes, the corresponding bounding box is filtered out; if the result is no, no processing is performed.

[0031] Preferably, step S6 includes:

[0032] Step S6.1: Assume the ground is level and the roll and pitch angles of the target obstacle are 0;

[0033] Step S6.2: Predict and obtain the size vector of the target obstacle using the ResNet18 model; the size vector includes length, width, height, and orientation angle;

[0034] Step S6.3: Based on the size vector, calculate the center position, horizontal coordinates, and vertical coordinates of the 3D bounding box of the target obstacle using the pinhole camera model;

[0035] The mathematical expression for the center position is:

[0036]

[0037] Where Z is the center position of the 3D bounding box of the target obstacle on the z-axis; f y is the vertical focal length in the intrinsic parameters of the pinhole camera; H is the height of the target obstacle; h is the pixel height of the target obstacle in the image;

[0038] The mathematical expressions for the horizontal and vertical coordinates, from top to bottom, are as follows:

[0039]

[0040] Where u and v are the pixel coordinates of the center of the target obstacle in the image plane, respectively; c x c y f represents the x-coordinate and y-coordinate of the optical center of the pinhole camera in the image plane, respectively; x f y X, Y, and Z represent the focal lengths of the pinhole camera in the horizontal and vertical directions, respectively; X, Y, and Z are the three-dimensional coordinates of the center of the target obstacle's 3D bounding box in the pinhole camera's coordinate system, representing the 3D information of the target obstacle.

[0041] A 3D target detection system based on visual images provided by the present invention includes:

[0042] Module M1: Acquires and preprocesses visual images to obtain preprocessed visual images;

[0043] Module M2: Feature extraction from the preprocessed visual image to obtain a feature map;

[0044] Module M3: Based on the feature pyramid of the feature map, a feature fusion feature map is obtained;

[0045] Module M4: The detection head part of the feature map fusion for target prediction and target regression, which obtains the prediction confidence and the bounding box of the target;

[0046] Module M5: Determines whether the IOU of the target bounding box is greater than a preset threshold. If the result is yes, the corresponding bounding box is filtered out; if the result is no, no processing is performed.

[0047] Module M6: Based on the bounding box obtained by module M5, it calculates and outputs the 3D information of the obstacle through a regression network.

[0048] Preferably, in module M1, the preprocessing involves sampling the visual image at every pixel in both the width and height directions using the Focus module to obtain four preprocessed feature maps; then, the four preprocessed feature maps are stitched together along the channel dimension to obtain a preprocessed visual image.

[0049] Preferably, module M2 includes:

[0050] Module M2.1: Based on CSPNet and DCN, it performs feature extraction and preprocessing of visual images;

[0051] Module M2.2: Based on the network structure combining multi-layer CBL and CSPNet, feature maps of different scales are obtained; the CBL includes convolutional layers, batch normalization layers, and the Leaky ReLU activation function;

[0052] The module M2.1 includes: using a CSPNet-based backbone network to process the preprocessed visual image as the input image and extracting features at different scales;

[0053] The DCN module of the convolutional layer adjusts the deformation of the preprocessed visual image.

[0054] Preferably, module M3 includes:

[0055] Module M3.1: Feature pyramid based on feature map, which upsamples deep network through FPN network structure and then fuses shallow network features to obtain semantic features of feature fusion feature map;

[0056] Module M3.2: Based on the PANet network structure, it fuses shallow network features into deep network features and outputs the localization information of the feature fusion feature map;

[0057] Module M3.3: Outputs the fused feature map to the detection head.

[0058] The detection head includes multiple detection layers, and each detection layer corresponds to a feature fusion feature map at a certain scale; the detection head can generate four different scales, i.e., different resolutions of output feature fusion feature maps.

[0059] The number of grids for the four different resolutions are 20×20, 40×40, 80×80 and 160×160, respectively.

[0060] In module M4, the "target" refers to the object that the detection model is expected to recognize, i.e., the target obstacle;

[0061] The module M5 includes:

[0062] Module M5.1: The predicted confidence score and the bounding box of the target obstacle are compared. It is determined whether the predicted confidence score of the bounding box of the target obstacle is less than 0.3. If the result is yes, the corresponding bounding box is filtered out and the remaining bounding boxes are obtained. If the result is no, no processing is performed.

[0063] Module M5.2: Sort the remaining bounding boxes in ascending or descending order according to the confidence level, calculate and obtain the IOU between the bounding boxes, and determine whether the IOU value is greater than 0.7. If the result is yes, the corresponding bounding box is filtered out; if the result is no, it is not processed.

[0064] Preferably, module M6 includes:

[0065] Module M6.1: Assuming the ground is level, the roll and pitch angles of the target obstacle are 0;

[0066] Module M6.2: Predicts and obtains the size vector of the target obstacle using the ResNet18 model; the size vector includes length, width, height, and orientation angle;

[0067] Module M6.3: Based on the size vector, using a pinhole camera model, calculate the center position, horizontal coordinates, and vertical coordinates of the 3D bounding box of the target obstacle;

[0068] The mathematical expression for the center position is:

[0069]

[0070] Where Z is the center position of the 3D bounding box of the target obstacle on the z-axis; f y is the vertical focal length in the intrinsic parameters of the pinhole camera; H is the height of the target obstacle; h is the pixel height of the target obstacle in the image;

[0071] The mathematical expressions for the horizontal and vertical coordinates, from top to bottom, are as follows:

[0072]

[0073] Where u and v are the pixel coordinates of the center of the target obstacle in the image plane, respectively; c x c y f represents the x-coordinate and y-coordinate of the optical center of the pinhole camera in the image plane, respectively; x f y X, Y, and Z represent the focal lengths of the pinhole camera in the horizontal and vertical directions, respectively; X, Y, and Z are the three-dimensional coordinates of the center of the target obstacle's 3D bounding box in the pinhole camera's coordinate system, representing the 3D information of the target obstacle.

[0074] Compared with the prior art, the present invention has the following beneficial effects:

[0075] 1. This invention proposes a small target detection method based on visual images, which improves the recognition effect of distant or small targets in scenarios such as autonomous driving, and can obtain the depth information of the target.

[0076] 2. This invention can be applied to other application scenarios that require high-precision detection of smaller targets.

[0077] 3. This invention uses visual images as input and the YOLOv5 model as the backbone. The model structure is optimized for the backbone, primarily by: introducing variable convolution to improve feature extraction of target edges; adding a smaller receptive field (160*160 feature map) to improve feature extraction of small targets and enhance the model's recognition performance for pedestrians and other targets; and adding ResNet18 and geometric features to predict / calculate the 3D features of targets to obtain their 3D information. This algorithm effectively improves the recognition performance for small targets and addresses the issue of visual depth prediction. Attached Figure Description

[0078] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0079] Figure 1 A flowchart illustrating the workflow of this invention. Detailed Implementation

[0080] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0081] This invention provides a small target detection method based on visual images, which can run in real time and obtain depth information. The purpose is to provide a method, system and steps for realizing roadbed environment perception of unmanned transport vehicles based on lidar, so as to provide effective safety assurance for the operation of unmanned vehicles in industrial environments.

[0082] This invention provides a real-time 3D object detection method based on visual images, comprising: downsampling and feature enhancement based on RGB feature input of visual images, performing operations such as Focus; feature extraction network based on a backbone network consisting of a CrossStage Partial Network (CSPNet) and Deformable Convolution (DCN); and feature fusion of different receptive fields based on a Feature Pyramid Network (FPN) and a Path Aggregation Network (PANet).

[0083] Object detection regression is based on different feature maps; non-maximum suppression (NMS) of overlapping boxes is used; and 3D information of obstacles is inferred based on ResNet18 network regression and geometric relationships.

[0084] Specifically, this invention is based on the original YOLOv5 network structure. In the backbone, DCN is used to replace the convolutional layer of a specific layer in CSPNet, i.e., Convolution, to improve the feature extraction capability for objects with complex and varied shapes. A 160×160 Feature Map is added to improve the recognition capability for small objects with pixels smaller than 8×8 at a resolution of 640×640. Based on the 2D detection results, ResNet18 network is added to regress 3D information.

[0085] In other words, the software system flow of the small target detection method based on visual images provided by the present invention includes:

[0086] Step 1: Acquire and preprocess visual images;

[0087] Specifically, in step 1, the visual image refers to the RGB features of the visual image;

[0088] Specifically, step 1 includes:

[0089] Step 1.1: Use the Focus module to sample the visual image every other pixel in the width and height directions to obtain four feature maps with half the resolution but increased number of channels;

[0090] Step 1.2: Concatenate these four feature maps along the channel dimension to form a new feature map, i.e., the preprocessed visual image. Then, perform downsampling and feature enhancement processing on the visual image.

[0091] Specifically, the purpose of this processing is to achieve efficient downsampling and feature extraction without losing key information, thereby accelerating the overall detection algorithm.

[0092] Step 2: Feature extraction from the preprocessed visual image;

[0093] Step 2 includes:

[0094] Step 2.1: Feature extraction based on CSPNet and DCN;

[0095] In step 2.1, feature extraction is specifically implemented as follows:

[0096] Step 2.1.1: Process the input image using a CSPNet-based backbone network to extract multi-scale, high-quality feature representations;

[0097] Step 2.1.2: Introduce a DCN module into a specific convolutional layer to make the convolution operation adaptive, enabling it to adjust to the deformation of the target. Deformation refers to non-rigid changes in the geometric shape, pose, scale, and viewpoint of the target in the image.

[0098] Step 2.2: Based on the network structure combining multiple layers of CBL and CSPNet, feature maps are obtained under different receptive fields. The receptive field represents the size of the region in the original input image corresponding to a pixel in a certain layer of the feature map within the convolutional neural network. The receptive field is divided into shallow and deep layers: shallow layers are used to capture details and provide precise localization; deep layers are used to extract semantics and provide accurate classification.

[0099] The CBL is a combination of convolutional layers, batch normalization layers, and the Leaky ReLU activation function.

[0100] The batch normalization layer is called Batch Normalization.

[0101] The convolutional layer;

[0102] These feature maps at different scales will be used as input to the Neck network, i.e., step 3, for subsequent feature fusion processing.

[0103] Step 3: Based on the feature pyramid, i.e., FPN and the path aggregation network, i.e., PAnet, feature fusion of different receptive fields is achieved;

[0104] Step 3 includes:

[0105] Step 3.1: Utilize the FPN network structure to upsample the deep network and then fuse it with the features of the shallow network with a larger receptive field, i.e., Concat, to improve semantic features;

[0106] In step 3.1, the deep feature maps of the backbone network are upsampled step by step using the FPN network structure and fused with the shallow feature maps with relatively small receptive fields but higher resolution. That is, the semantic richness of the output features is improved by the feature concatenation operation.

[0107] Step 3.2: Based on the PANet network structure, fuse shallow network features into deep network features to enhance localization features;

[0108] Specifically, the two network structures in step 3 aim to combine high-level semantic features with low-level localization details to achieve full fusion of different receptive field features.

[0109] In step 3.2, the PANet network structure is used to pass shallow features from bottom to top and fuse them with deep features again to enhance the target's localization features. That is, by propagating the fine-grained features of the lower layers to the higher layers, the deep output features simultaneously contain fine localization information.

[0110] The final output consists of multiple fused feature maps. These feature maps contain both semantic and localization information at different scales and are passed to the detection head part for target prediction and regression.

[0111] Step 4: Perform target detection prediction based on different feature maps. The Head part contains multiple detection layers, each of which corresponds to a feature map of a certain scale. This is used to regress and output the target bounding box parameters and classification confidence at that scale.

[0112] Step 4 includes:

[0113] Step 4.1: The Head part contains multiple detection layers, each corresponding to a feature map of a certain scale; four output feature maps with different resolutions are generated, with grid sizes of approximately 20×20, 40×40, 80×80 and 160×160 respectively, which are applied to detect targets in the original image with sizes of approximately 32×32, 16×16, 8×8 and 4×4 pixels.

[0114] Step 4.2: For each scale feature map provided by Neck, the Head part uses a 1×1 convolution to adjust and expand the number of feature channels to:

[0115] (N+5)*A

[0116] Where N is the number of target categories; A is the number of anchor boxes preset for each grid cell; +5 corresponds to the x-coordinate, y-coordinate, width, height, and confidence score of the center point of the predicted box, respectively. The confidence score can be directly obtained through the 1x1 convolution in step 4.2.

[0117] In other words, the purpose of step 4.2 is to obtain the final results, namely the x-coordinate, y-coordinate, width, height, and confidence score of the center point of the prediction box.

[0118] In this implementation, the number of anchor boxes A=4 on each detection layer, that is, four anchor boxes with different aspect ratios are preset to generate the bounding boxes of candidate targets and their classification results.

[0119] Step 5: Filter IOUs (Intersection over Union) based on NMS, i.e., the boxes with larger intersections and unions.

[0120] Step 5 includes:

[0121] Step 5.1: Based on the prediction confidence, perform preliminary filtering on boxes with a confidence level less than 0.3;

[0122] Step 5.2: Sort the remaining target boxes based on confidence, calculate the IOU between different boxes, and filter the target boxes with an IOU greater than 0.7 and a confidence score less than 0.3.

[0123] Step 6: Calculate obstacle 3D information based on ResNet18 and geometric relationships.

[0124] Specifically, step 6 includes:

[0125] Step 6.1: Assuming the ground is level, the roll angle and pitch angle of the obstacle are 0;

[0126]

[0127] Rot={pitch,roll,yaw}Rot={0,0,yaw}

[0128] Where: Dim represents the size information of the obstacle 3D bounding box; Rot represents the rotation angle information of the obstacle 3D bounding box;

[0129] L, W, and H are the length, width, and height of the target obstacle's 3D bounding box in the camera coordinate system, respectively.

[0130] pitch, roll, and yaw are the pitch, roll, and yaw angles of the target obstacle's 3D bounding box, respectively.

[0131] Step 6.2: Predict the length, width, height, and orientation angle of the obstacle using ResNet18 network regression;

[0132] The visual image and the result of step 6 are input into the trained ResNet18 model, which outputs the predicted three-dimensional size vector of the obstacle, namely length, width, height, and orientation angle.

[0133] Specifically, this information will be combined with the camera imaging model in step 6.3 to calculate the target's spatial position relative to the camera. By extracting depth features and predicting the target scale, the system can understand the true size of the target in the image, providing crucial information for subsequent distance and position calculations.

[0134] Step 6.3: Based on the pinhole camera model, calculate the depth information of the obstacle, i.e., the center position of the obstacle's 3D bounding box:

[0135]

[0136] Where Z is the center position of the obstacle's 3D bounding box on the z-axis; f y H is the vertical focal length in the camera's intrinsic parameters; H is the height of the obstacle, i.e., the height in the predicted value obtained in step 6.2; h is the pixel height of the obstacle in the image.

[0137] Then, by combining the camera's intrinsic parameters, the horizontal and vertical coordinates of the obstacle, i.e., (X, Y), are obtained. The mathematical expression is:

[0138]

[0139] Where u and v are the pixel coordinates of the obstacle center on the image plane, respectively; c x c y f represents the x-coordinate and y-coordinate of the camera's optical center in the image plane, respectively; x f y X, Y, and Z are the focal lengths of the camera in the horizontal and vertical directions, respectively; X, Y, and Z are the three-dimensional coordinates of the center of the obstacle's 3D bounding box in the camera coordinate system.

[0140] In other words, the small target detection method based on visual images provided by the present invention includes:

[0141] Step S1: Perform downsampling and feature enhancement using the Focus operation;

[0142] Step S2: Feature extraction is performed based on CSPnet and deformable convolution as the backbone;

[0143] Step S3: Based on the feature pyramid, i.e., FPN and the path aggregation network, i.e., PAnet, feature fusion of different receptive fields is achieved;

[0144] Step S4: Perform regression prediction based on different feature maps;

[0145] Step S5: Filter boxes with large IOUs based on NMS.

[0146] Step S6: Calculate obstacle 3D information based on ResNet18 and geometric relationships.

[0147] Specifically, step S2 includes the following steps:

[0148] Step S2.1: Feature extraction based on CSPnet and Deformable conv;

[0149] Step S2.2: Based on the multi-layer CSP+CBL network structure, obtain feature maps under different receptive fields and feed them into the neck part for feature fusion;

[0150] Specifically, step S3 includes the following steps:

[0151] Step S3.1: Utilize the FPN network structure to upsample the deep network and then fuse it with the features of the shallow network with a larger receptive field, i.e., concat, to improve semantic features;

[0152] Step S3.2: Based on the PAnet network structure, fuse shallow features to deep network features to enhance localization features;

[0153] Specifically, step S4 includes the following steps:

[0154] Step S4.1: The Head part outputs feature map sizes of 20*20, 40*40, 80*80 and 160*160 respectively, which correspond to targets of 32*32, 16*16, 8*8 and 4*4 pixels;

[0155] Step S4.2: The Head expands the number of channels of the feature maps at different scales obtained in the Neck using 1×1 convolutions. The expanded number of feature channels is (number of categories + 5) × the number of anchors on each detection layer. Here, 5 corresponds to the x-coordinate, y-coordinate, width, height, and confidence score of the predicted bounding box's center point. The four detection layers in the Head correspond to the four different sizes of feature maps obtained in the Neck. Each grid on the feature map has four anchors with different aspect ratios pre-defined, which can store all positional and classification information based on the anchor prior boxes in the channel dimension of the feature map for target prediction and regression.

[0156] Step S5 includes:

[0157] Step S5.1: Based on the prediction confidence, perform preliminary filtering on boxes with a confidence level less than 0.3;

[0158] Step S5.2: Sort the remaining target boxes based on confidence, calculate the IOU between different boxes, and filter out target boxes with IOU greater than 0.7 and low confidence.

[0159] Specifically, step S6 includes:

[0160] Step S6.1: Assuming the ground is level, the roll angle and pitch angle of the obstacle are 0.

[0161] Step S6.2: Predict the length, width, height, and orientation angle of the obstacle based on ResNet18 network regression;

[0162] Step S6.3: Based on the pinhole camera model, calculate the depth information of the obstacle, and combine the camera intrinsic parameters to obtain the longitudinal and lateral coordinates (cx, cy) of the obstacle.

[0163] The present invention also provides a 3D target detection system based on visual images. The 3D target detection system based on visual images can be implemented by executing the process steps of the 3D target detection method based on visual images. That is, those skilled in the art can understand the 3D target detection method based on visual images as a preferred embodiment of the 3D target detection system based on visual images.

[0164] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0165] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A 3D target detection method based on visual images, characterized in that, include: Step S1: Acquire and preprocess the visual image to obtain a preprocessed visual image; Step S2: Extract features from the preprocessed visual image to obtain a feature map; Step S3: Based on the feature pyramid of the feature map, obtain the feature fusion feature map; Step S4: Target prediction and target regression: The detection head part of the feature map is fused to obtain the predicted confidence and the bounding box of the target; Step S5: Determine whether the IOU of the target bounding box is greater than a preset threshold. If the result is yes, filter out the corresponding bounding box; if the result is no, do not process it. Step S6: Based on the bounding box obtained in step S5, calculate and output the 3D information of the obstacle through a regression network; Step S2 includes: Step S2.1: Based on CSPNet and DCN, extract features from the visual image and preprocess it; Step S2.2: Based on the network structure combining multi-layer CBL and CSPNet, feature maps of different scales are obtained; the CBL includes convolutional layers, batch normalization layers, and the Leaky ReLU activation function; In step S2.1, the following steps are included: using a CSPNet-based backbone network to process the preprocessed visual image as the input image and extracting features at different scales; The deformation of the preprocessed visual image is adjusted through the DCN module of the convolutional layer; Step S3 includes: Step S3.1: Based on the feature pyramid of the feature map, the deep network is upsampled through the FPN network structure, and then the features of the shallow network are fused to obtain the semantic features of the feature fusion feature map; Step S3.2: Based on the PANet network structure, fuse shallow network features into deep network features and output the localization information of the feature fusion feature map; Step S3.3: Output the fused feature map to the detection head. The detection head includes multiple detection layers, and each detection layer corresponds to a feature fusion feature map at a certain scale; the detection head can generate four different scales, i.e., different resolutions of output feature fusion feature maps. The number of grids for the four different resolutions are 20×20, 40×40, 80×80 and 160×160, respectively. In step S4, the "target" refers to the target obstacle. Step S5 includes: Step S5.1: Compare the predicted confidence score with the bounding box of the target obstacle. Determine if the predicted confidence score of the bounding box of the target obstacle is less than 0.

3. If the result is yes, filter out the corresponding bounding box and obtain the remaining bounding boxes. If the result is no, do not process it. Step S5.2: Sort the remaining bounding boxes in ascending or descending order according to the confidence level, then calculate and obtain the IOU between the bounding boxes, and determine whether the IOU value is greater than 0.

7. If the result is yes, the corresponding bounding box is filtered out; if the result is no, no processing is performed.

2. The 3D target detection method based on visual images according to claim 1, characterized in that, In step S1, the preprocessing involves sampling the visual image at every pixel in both the width and height directions using the Focus module to obtain four preprocessed feature maps; then, the four preprocessed feature maps are stitched together along the channel dimension to obtain the preprocessed visual image.

3. The 3D target detection method based on visual images according to claim 1, characterized in that, Step S6 includes: Step S6.1: Assume the ground is level and the roll and pitch angles of the target obstacle are 0; Step S6.2: Predict and obtain the size vector of the target obstacle using the ResNet18 model; the size vector includes length, width, height, and orientation angle; Step S6.3: Based on the size vector, calculate the center position, horizontal coordinates, and vertical coordinates of the 3D bounding box of the target obstacle using the pinhole camera model; The mathematical expression for the center position is: Where Z is the center position of the 3D bounding box of the target obstacle on the z-axis; H is the height of the target obstacle; and h is the pixel height of the target obstacle in the image. The mathematical expressions for the horizontal and vertical coordinates, from top to bottom, are as follows: Where u and v are the pixel coordinates of the center of the target obstacle in the image plane, respectively; These are the x and y coordinates of the optical center of the pinhole camera in the image plane, respectively. X, Y, and Z represent the focal lengths of the pinhole camera in the horizontal and vertical directions, respectively; X, Y, and Z are the three-dimensional coordinates of the center of the target obstacle's 3D bounding box in the pinhole camera's coordinate system, representing the 3D information of the target obstacle.

4. A 3D target detection system based on visual images, characterized in that, include: Module M1: Acquires and preprocesses visual images to obtain preprocessed visual images; Module M2: Feature extraction from the preprocessed visual image to obtain a feature map; Module M3: Based on the feature pyramid of the feature map, a feature fusion feature map is obtained; Module M4: The detection head part of the feature map fusion for target prediction and target regression, which obtains the prediction confidence and the bounding box of the target; Module M5: Determines whether the IOU of the target bounding box is greater than a preset threshold. If the result is yes, the corresponding bounding box is filtered out; if the result is no, no processing is performed. Module M6: Based on the bounding box obtained by module M5, it calculates and outputs the 3D information of obstacles through a regression network; The module M2 includes: Module M2.1: Based on CSPNet and DCN, it performs feature extraction and preprocessing of visual images; Module M2.2: Based on the network structure combining multi-layer CBL and CSPNet, feature maps of different scales are obtained; the CBL includes convolutional layers, batch normalization layers, and the Leaky ReLU activation function; The module M2.1 includes: using a CSPNet-based backbone network to process the preprocessed visual image as the input image and extracting features at different scales; The deformation of the preprocessed visual image is adjusted through the DCN module of the convolutional layer; The module M3 includes: Module M3.1: Feature pyramid based on feature map, which upsamples deep network through FPN network structure and then fuses shallow network features to obtain semantic features of feature fusion feature map; Module M3.2: Based on the PANet network structure, it fuses shallow network features into deep network features and outputs the localization information of the feature fusion feature map; Module M3.3: Outputs the fused feature map to the detection head. The detection head includes multiple detection layers, and each detection layer corresponds to a feature fusion feature map at a certain scale; the detection head can generate four different scales, i.e., different resolutions of output feature fusion feature maps. The number of grids for the four different resolutions are 20×20, 40×40, 80×80 and 160×160, respectively. In module M4, the "target" refers to the target obstacle. The module M5 includes: Module M5.1: The predicted confidence score and the bounding box of the target obstacle are compared. It is determined whether the predicted confidence score of the bounding box of the target obstacle is less than 0.

3. If the result is yes, the corresponding bounding box is filtered out and the remaining bounding boxes are obtained. If the result is no, no processing is performed. Module M5.2: Sort the remaining bounding boxes in ascending or descending order according to the confidence level, calculate and obtain the IOU between the bounding boxes, and determine whether the IOU value is greater than 0.

7. If the result is yes, the corresponding bounding box is filtered out; if the result is no, it is not processed.

5. The 3D target detection system based on visual images according to claim 4, characterized in that, In module M1, the preprocessing involves sampling the visual image at every pixel in both the width and height directions using the Focus module to obtain four preprocessed feature maps; then, the four preprocessed feature maps are stitched together along the channel dimension to obtain the preprocessed visual image.

6. The 3D target detection system based on visual images according to claim 4, characterized in that, The module M6 includes: Module M6.1: Assuming the ground is level, the roll and pitch angles of the target obstacle are 0; Module M6.2: Predicts and obtains the size vector of the target obstacle using the ResNet18 model; the size vector includes length, width, height, and orientation angle; Module M6.3: Based on the size vector, the center position, horizontal coordinates, and vertical coordinates of the 3D bounding box of the target obstacle are calculated using a pinhole camera model; The mathematical expression for the center position is: Where Z is the center position of the 3D bounding box of the target obstacle on the z-axis; H is the height of the target obstacle; and h is the pixel height of the target obstacle in the image. The mathematical expressions for the horizontal and vertical coordinates, from top to bottom, are as follows: Where u and v are the pixel coordinates of the center of the target obstacle in the image plane, respectively; These are the x and y coordinates of the optical center of the pinhole camera in the image plane, respectively. X, Y, and Z are the focal lengths of the pinhole camera in the horizontal and vertical directions, respectively; X, Y, and Z are the three-dimensional coordinates of the center of the target obstacle's 3D bounding box in the pinhole camera's coordinate system, i.e., the 3D information of the target obstacle.

Images