Three-dimensional target detection model generation and detection method, device, equipment and medium

Abstract

The application provides a three-dimensional target detection model generation and detection method, device, equipment and medium, and belongs to the technical field of optoelectronic image recognition. The method comprises the following steps: sequentially connecting a convolution-self-attention parallel branch, a feature guidance module, a multi-scale upsampling structure, a foreground depth clue module constructed based on a deep discretization and continuous estimation mechanism, and a detection head module; connecting a convolution-self-attention double-branch neural network and a loss function layer; inputting a monocular image of a detection target into the convolution-self-attention double-branch neural network to extract two-dimensional information and three-dimensional information of the detection target; inputting training samples and the two-dimensional information and the three-dimensional information into a preset detection model, calculating a loss function value of the detection model, updating network parameters of an initial detection model based on the loss function value, and obtaining a monocular image three-dimensional target detection model after several iterations. The application effectively improves the detection accuracy of three-dimensional targets in optoelectronic monocular images.

Description

Technical Field

[0001] This invention relates to the field of optoelectronic image recognition technology, specifically to a method, apparatus, equipment, and medium for generating and detecting three-dimensional target detection models. Background Technology

[0002] In monocular vision-based 3D object detection tasks, the model needs to predict the object's position information in 3D space from a single 2D image, including key parameters such as category, center position, size, orientation, and depth. Depth estimation, as the bridge connecting the 2D image and 3D space, is a crucial factor affecting the accuracy of 3D object detection. However, due to the physical characteristics of perspective projection, points in 3D space experience irreversible loss of depth information when projected onto the 2D image plane, making monocular depth estimation a typical ill-posed problem. This lack of depth information not only increases the learning difficulty of the model but also significantly limits the overall performance of the detection system.

[0003] Most current mainstream monocular 3D object detection methods are extensions of 2D object detection models, relying on convolutional neural networks (CNNs) to extract image features. However, they face several bottlenecks. First, CNNs struggle to adapt to drastic changes in target scale caused by perspective effects under fixed receptive field conditions, especially in image scenes where objects appear larger in the foreground and smaller in the background, resulting in insufficient modeling of structural details of distant objects. Second, while traditional CNN structures excel at extracting local textures and geometric features, their ability to model global contextual information in images is limited, leading to insufficient capture of semantic relationships and depth cues between objects in complex environments. Furthermore, existing models have poor sensitivity to spatial scene geometry, limiting the accuracy of 3D position inference. In optoelectronic imaging systems (such as low-light, infrared, or embedded vision devices), these challenges are further exacerbated by issues such as imaging quality, noise interference, and lack of parallax information. Therefore, there is an urgent need for an imaging method that can adapt to the imaging characteristics of optoelectronic images to improve the accuracy of 3D object detection in practical applications. Summary of the Invention

[0004] In view of this, it is necessary to provide a method, apparatus, equipment and medium for generating and detecting three-dimensional targets, so as to improve the detection accuracy of three-dimensional targets in optoelectronic monocular images.

[0005] To achieve the above-mentioned technical effects, in a first aspect, the present invention provides a method for generating a three-dimensional target detection model, comprising:

[0006] By sequentially connecting the convolutional-self-attention parallel branch, the feature guidance module, the multi-scale upsampling structure, the foreground depth cue module constructed based on the depth discretization and continuous estimation mechanism, and the detection head module, a convolutional-self-attention dual-branch neural network is obtained.

[0007] The initial detection model is obtained by connecting the convolutional-self-attention dual-branch neural network and the preset loss function layer;

[0008] The monocular image of the target to be detected is input into the convolutional-self-attention dual-branch neural network to extract the two-dimensional and three-dimensional information of the target.

[0009] The preset training samples, along with the two-dimensional and three-dimensional information, are input into the preset detection model. The loss function value of the detection model is calculated, and the network parameters of all modules in the initial detection model are updated based on the loss function value. After several iterations, a monocular image three-dimensional target detection model is obtained, which is used to output the category and spatial geometric parameters of the detected target based on the input monocular image.

[0010] In some embodiments of the present invention, the step of inputting the monocular image of the detected target into the convolutional-self-attention dual-branch neural network to extract the two-dimensional and three-dimensional information of the detected target includes:

[0011] The monocular image is input into a convolutional branch to obtain multi-scale local features;

[0012] The monocular image is input into the self-attention branch to obtain global semantic features;

[0013] Based on the feature guidance module, the multi-scale local features and global semantic features are fused to obtain the main feature map;

[0014] The main feature map is upsampled based on the multi-scale upsampling structure to obtain an enhanced feature map.

[0015] A foreground depth cue feature map is generated based on the foreground depth cue module, and the foreground depth cue feature map is fused with the main feature map to obtain a fused feature map.

[0016] Based on the detection head, the two-dimensional and three-dimensional information of the target to be detected is extracted from the fused feature map.

[0017] In some embodiments of the present invention, the convolutional branch uses the DLA-34 network as the backbone structure and introduces an aggregation module based on a dynamic large kernel;

[0018] The self-attention branch adopts a self-attention branch based on a single-head self-attention mechanism.

[0019] In some embodiments of the present invention, the foreground depth cue module includes: a three-layer convolutional structure;

[0020] The process of generating a foreground depth cue feature map based on the foreground depth cue module to obtain a fused feature map includes:

[0021] The first depth value of the center point of the target two-dimensional detection box is converted into a foreground depth label based on the depth discretization mechanism. The center point of the target two-dimensional detection box includes a discrete foreground depth map generated based on a monocular image.

[0022] Using the foreground depth label as supervision, a foreground depth probability map is generated based on a three-layer convolutional structure and the enhanced feature map, wherein the foreground depth probability map represents the probability that each pixel belongs to each discrete depth category;

[0023] Calculate the second depth value corresponding to each discrete depth category in the foreground depth probability map;

[0024] The second depth value is converted into a continuous depth estimate for each pixel using a probability-weighted approach.

[0025] A foreground depth cue feature map is constructed based on the continuous depth estimates.

[0026] In some embodiments of the present invention, the detection head includes: a two-dimensional detection head and a three-dimensional detection head;

[0027] The extraction of two-dimensional and three-dimensional information of the detected target from the fused feature map based on the detection head includes:

[0028] Predicting heatmaps, center point offsets, and target 2D dimensions based on a 2D detection head;

[0029] Extracting target region features based on ROIAlign;

[0030] By fusing the fused feature map and the target region features, a 3D detection head is used to predict the 3D size, 3D offset, viewing angle, and depth value of the target.

[0031] In some embodiments of the present invention, the training samples include: the category, location, and depth information of the detected target;

[0032] The loss functions include: two-dimensional heatmap loss, two-dimensional offset loss, two-dimensional size loss, three-dimensional offset loss, three-dimensional size loss, angle loss, foreground depth cue supervision loss, and continuous depth regression loss.

[0033] Secondly, the present invention also provides a three-dimensional target detection method, comprising:

[0034] Acquire a monocular image including the target to be detected;

[0035] The monocular image is input into a 3D target detection model to obtain the target's category, 2D position, 3D size, spatial offset, orientation angle, and depth information. The 3D target detection model is obtained according to any one of the 3D target detection model generation methods described above.

[0036] Thirdly, the present invention also provides a three-dimensional target detection model generation device, comprising:

[0037] The feature extraction module is used to input the monocular image of the target to a preset convolutional-self-attention dual-branch neural network to extract the main feature map of the target.

[0038] The feature fusion module is used to generate a foreground depth cue feature map based on a deep discretization and continuous estimation mechanism, and to fuse the foreground depth cue feature map with the main feature map to obtain a fused feature map.

[0039] The information detection module is used to extract two-dimensional and three-dimensional information of the target object from the fused feature map based on the detection head;

[0040] The model training module is used to input preset training samples and the two-dimensional and three-dimensional information into a preset detection model, calculate the loss function value of the detection model, update the network parameters of the detection model based on the loss function value, and after several iterations, obtain a monocular image three-dimensional target detection model for outputting the category, two-dimensional position, three-dimensional size, spatial offset, orientation angle and depth information of the detected target based on the input monocular image.

[0041] Fourthly, the present invention also provides an apparatus comprising:

[0042] Memory, used to store programs;

[0043] A processor, coupled to the memory, is configured to execute the program stored in the memory to implement the steps of the three-dimensional target detection model generation method or the three-dimensional target detection method described in any one of the above method items.

[0044] Fifthly, the present invention also provides a medium comprising:

[0045] Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps of the three-dimensional target detection model generation method or the three-dimensional target detection method described in any one of the above method items.

[0046] The beneficial effects of this invention are as follows: The method for generating a 3D target detection model provided by this invention first utilizes a dual-branch network structure combining convolution and self-attention to acquire image features. Compared with traditional convolutional networks, the dual-branch network structure is more adaptable to scale changes and modeling long-distance dependencies, improving the model's 3D understanding ability in complex optoelectronic image scenes. This invention also introduces the fusion of discretized depth modeling and continuous depth cues, thereby guiding the network to focus on semantically meaningful foreground regions, strengthening depth information expression, and maintaining stable 3D detection performance in complex scenes such as low light and long distances, achieving accurate estimation of 3D size, spatial position, orientation angle, and depth information. Finally, based on the estimated data, a loss function is calculated to train and optimize the parameters of the initial detection model, resulting in a detection model with higher detection accuracy, thus effectively improving the detection accuracy of 3D targets in optoelectronic monocular images. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 A flowchart illustrating an embodiment of the three-dimensional target detection model generation method provided by the present invention;

[0049] Figure 2 for Figure 1 A flowchart illustrating an embodiment of step S103;

[0050] Figure 3 A schematic diagram illustrating the process of extracting image features using a convolutional-self-attention dual-branch neural network provided by this invention;

[0051] Figure 4 The following is a framework diagram of the aggregation module based on dynamic large kernel provided by the present invention;

[0052] Figure 5 A diagram illustrating the self-attention branching framework based on a single-head self-attention mechanism provided by this invention;

[0053] Figure 6 A framework diagram of the feature guidance module provided by this invention;

[0054] Figure 7 for Figure 2 A flowchart illustrating an embodiment of step S205;

[0055] Figure 8 for Figure 1 A flowchart illustrating an embodiment of step S206;

[0056] Figure 9 This is a flowchart illustrating an embodiment of the three-dimensional target detection method provided by the present invention;

[0057] Figure 10 A schematic diagram of an embodiment of the three-dimensional target detection model generation device provided by the present invention.

[0058] Figure 11 This is a schematic diagram of the structure of an embodiment of the device provided by the present invention. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0060] In the description of the embodiments of the present invention, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0061] The terms "first," "second," etc., used in the embodiments of this invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature.

[0062] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0063] This invention provides a method, apparatus, equipment, and medium for generating and detecting three-dimensional target detection models, which will be described below.

[0064] like Figure 1 As shown, in a first aspect, the present invention provides a method for generating a three-dimensional target detection model, comprising:

[0065] S101, the convolutional-self-attention parallel branch, the feature guidance module, the multi-scale upsampling structure, the foreground depth cue module built based on the depth discretization and continuous estimation mechanism, and the detection head module are connected in sequence to obtain the convolutional-self-attention dual-branch neural network;

[0066] S102. Connect the convolutional-self-attention dual-branch neural network and the preset loss function layer to obtain the initial detection model;

[0067] S103. Input the monocular image of the detected target into the convolutional-self-attention dual-branch neural network to extract the two-dimensional and three-dimensional information of the detected target.

[0068] In some embodiments of the present invention, such as Figure 2 Step S103 specifically includes:

[0069] S201. Input the monocular image into the convolution branch to obtain multi-scale local features.

[0070] S202. Input the monocular image into the self-attention branch to obtain global semantic features.

[0071] It should be noted that before inputting the monocular image into the network, the image needs to be resized and standardized. Preferably, the image size is uniformly adjusted to 512×512, and normalization preprocessing is performed so that the image tensor representation is:

[0072] (1)

[0073] In equation (1), and These represent the image's height and width, both being 512.

[0074] In some embodiments of the present invention, the convolutional-self-attention dual-branch neural network includes: a convolutional branch, a self-attention branch, a feature guidance module, and a multi-scale upsampling structure.

[0075] Specifically, to enhance the model's ability to model features of targets at different scales, especially adapting to significant scale changes caused by perspective effects in monocular images, this invention introduces an aggregation module based on a dynamic large kernel into the convolutional branch, and improves upon the DLA-34 network as the backbone structure to construct a feature extraction network with dynamic receptive field adjustment capabilities. The self-attention branch adopts a self-attention branch based on a single-head self-attention mechanism.

[0076] Furthermore, combined Figure 3 The convolutional-self-attention dual-branch neural network architecture shown below illustrates the specific steps for extracting image features using this neural network:

[0077] The input image (Sa1) is first processed through a standard 7×7 convolutional layer, batch normalization (BN), and ReLU activation function, and then input into the improved DLA-34 network for feature extraction. The residual block structure in DLA-34 is shown in the following formula:

[0078] (2)

[0079] In equation (2), It is the input of the residual block. It is the output of the residual block. and This represents two convolutional layers in the residual block.

[0080] Sa2) replaces the traditional aggregation modules. Specifically, the DLA-34 network contains six aggregation modules, with the 3rd and 5th modules having the most input branches and the greatest demand for receptive field adjustment. To balance efficiency and performance, these two modules are preferably replaced with the dynamic large-kernel-based aggregation modules proposed in this invention, while the remaining aggregation modules remain unchanged. The specific structure is as follows: Figure 4 As shown.

[0081] The Sa3 dynamic large kernel aggregation module features feature fusion. This module achieves dynamic receptive field adjustment through multi-scale dilated convolution, channel and spatial attention mechanisms, thereby enhancing feature representation capabilities. Figure 3 As shown, the specific steps include the following:

[0082] Sb1) Multiple input feature maps are concatenated along the channel dimension, then input into a standard convolutional layer and processed by BN and ReLU to obtain preliminary fused features:

[0083] (3)

[0084] Sb2) To achieve multi-scale feature extraction, the fused features are fed into two depthwise separable convolutions with different dilation rates to construct representations with different receptive fields:

[0085] (4)

[0086] (5)

[0087] Sb3) will and Concatenation is performed along the channel dimension to obtain a feature map with multi-scale joint representation. The expression is as follows:

[0088] (6)

[0089] Sb4) splicing feature map Perform average pooling and max pooling operations separately, concatenate them, and then calculate the spatial attention map using a 7×7 convolution. The calculation formula is as follows:

[0090] (7)

[0091] Sb5) Parallel Computation Channel Attention Weight Map The method is to Average pooling and max pooling are performed, and the inputs are processed by a shared multilayer perceptron (MLP), then summed and normalized using the sigmoid activation function, as shown in the following expression:

[0092] (8)

[0093] Sb6) Spatial attention map It is divided into two parts in terms of channel dimension. and The following formulas are used to weight the feature maps at two different scales:

[0094] (9)

[0095] Final fused output feature map The expression is as follows, where, The symbol represents the element-wise multiplication operation:

[0096] (10)

[0097] Sa4) The self-attention branch based on a single-head self-attention mechanism is constructed using an SHViT model-based structure, aiming to compensate for the shortcomings of convolutional branches in modeling long-range contextual dependencies. This structure effectively captures global information in the image while maintaining low computational complexity by introducing a single-head self-attention mechanism. The branch consists of three stages: Stage 1, Stage 2, and Stage 3, with multiple SHViTBlocks stacked in each stage.

[0098] like Figure 5 As shown, each SHViTBlock contains three main modules: a depthwise separable convolutional module, a single-head self-attention module, and a feedforward network module. The depthwise separable convolutional module is used for local feature aggregation, and the single-head self-attention module is used to capture long-range dependencies. Their structure is defined as follows:

[0099] (11)

[0100] In equation (11), the input features Divided along the channel dimension and These are used for attention calculation and residual preservation, respectively. For learnable linear transformation weights, This represents the scaling factor for the attention dimension.

[0101] This self-attention branch consists of three feature extraction stages. The first stage stacks two attention modules, taking image feature blocks embedded after being segmented by an 8×8 sliding window as input, and outputting feature size as the number of channels. The spatial size is one-eighth of the original image; in the second stage, the output of the first stage is downsampled and then three attention modules are stacked, with the output feature size being the number of channels. The spatial size is one-sixteenth of the original image; in the third stage, after further downsampling, three attention modules are stacked, and the output feature size is the number of channels. The spatial size is 1 / 32 of the original image. The spatial size is compressed step by step in three stages, and the global context awareness is enhanced.

[0102] The global features output at each stage will be used as guiding information input into the subsequent feature fusion module, guiding the convolutional branch to focus on the target and its context region during the modeling process, thereby improving the ability to express spatial relationships and structural information in the overall 3D target detection.

[0103] Compared with existing technologies, this invention utilizes a designed convolutional-self-attention dual-branch structure to simultaneously acquire local details and global contextual information, effectively solving the problem that traditional convolutional networks are difficult to adapt to scale changes and model long-distance dependencies, and improving the model's 3D understanding ability in complex optoelectronic image scenes.

[0104] S203. Based on the feature guidance module, multi-scale local features and global semantic features are fused to obtain the main feature map.

[0105] Specifically, the present invention uses a feature guidance module to dynamically guide the local feature response of the convolution branch by utilizing the global context information provided by the self-attention branch, thereby achieving the fusion of local and global information and improving the ability to understand the target structure and its surrounding environment during the target detection process.

[0106] like Figure 6 As shown, the feature guidance module includes: spatial attention mask generation, local feature enhancement, and residual convolution refinement.

[0107] Accordingly, the specific implementation process of step S203 includes: the feature guidance module inputting local features from the convolutional branch. and global features from self-attention branches Feature fusion is achieved through three sub-modules: spatial attention mask generation, feature enhancement, and local detail refinement. Furthermore, the process includes:

[0108] Sc1) Spatial attention mask generation part is based on global features Calculate the spatial attention weights. This process first extracts spatial information through two cascaded 1×1 convolution operations, followed by ReLU activation to obtain the mask image. The calculation formula is as follows:

[0109] (12)

[0110] Sc2) The feature enhancement part utilizes the aforementioned attention mask to modulate the convolutional branch feature map, enhancing information in salient regions while preserving original background features and avoiding interference with non-salient regions. This enhances the model's ability to express details such as edges and structures. The fusion expression is as follows:

[0111] (13)

[0112] In equation (13), This indicates an element-wise multiplication operation.

[0113] Sc3) Local detail refinement employs lightweight residual convolution to perform fine-tuning of the enhanced features, further suppressing background noise interference and improving the quality of the final fused features. The calculation formula for the refinement process is as follows:

[0114] (14)

[0115] S204. Upsample the main feature map based on the multi-scale upsampling structure to obtain the enhanced feature map.

[0116] It should be noted that after the input image undergoes the aforementioned convolutional and attention branches, its feature map downsampling rate is [a certain percentage] of the original input image. Since accurate spatial positioning of targets is required in 3D target detection, direct detection on a small 32×32 feature map will affect the detection accuracy, especially for small targets. Therefore, it is necessary to introduce an upsampling network to restore spatial resolution.

[0117] This invention employs a multi-scale upsampling structure to progressively upsample fused features, thereby improving spatial resolution and enhancing the ability to perceive small targets.

[0118] Preferably, the present invention introduces a feature fusion network with structural enhancement capabilities during the upsampling process, and adopts an IDA structure design that combines deformable convolution and transposed convolution. Resolution restoration and semantic alignment are achieved through multi-scale branch integration, specifically including the following steps:

[0119] To achieve efficient feature map upsampling and fusion, this invention constructs a multi-scale upsampling network with progressive fusion. This network includes three upsampling stages, each doubling the spatial size of the feature map. Information compensation and spatial reconstruction are achieved through a combination of transposed convolution and deformable convolution.

[0120] The Sd2 network employs the following structural approach for fusion at each upsampling stage: First, deformable convolutions are used to process low-resolution features from different scales to enhance their spatial expressiveness; then, spatial scale recovery is achieved through transposed convolutions; finally, 1×1 convolutions connected to fusion nodes are used to align and aggregate features with different channel numbers. This process can be represented as:

[0121] (15)

[0122] In equation (15), Indicates the first i Low-resolution input features of the layer This represents the high-resolution output features after upsampling.

[0123] The Sd3 network performs upsampling three times layer by layer, and achieves the fusion and information alignment of features at adjacent scales in each level. Through learnable deformable convolutional kernels, each level of features contains both rich semantic information and fine spatial distribution, thereby ensuring that small targets are not compressed or flattened during the fusion process.

[0124] The final output feature map (Sd4) is sized relative to the input image, increasing the size to that of the original image. Resolution, i.e., if the input image size is 3× H × W The output feature size of the upsampling network is This provides high-resolution, semantically expressive feature support for subsequent object detection modules.

[0125] S205. Generate a foreground depth cue feature map based on the foreground depth cue module, and fuse the foreground depth cue feature map with the main feature map to obtain a fused feature map.

[0126] It should be noted that, considering that depth information is the most challenging part of 3D object detection in monocular images, this invention designs a foreground depth cue module that introduces depth discretization and continuous estimation mechanisms to guide the model to learn depth cues for the foreground region from 2D visual features. This module discretizes continuous depths and learns the probabilities of different depth categories to extract effective depth representations from foreground objects.

[0127] Specifically, the foreground depth cue module includes: a three-layer convolutional structure; such as Figure 7As shown, step S205 specifically includes:

[0128] S701. Based on a depth discretization mechanism, the first depth value of the center point of the target 2D detection box is converted into a foreground depth label. The center point of the target 2D detection box includes a discrete foreground depth map generated based on a monocular image.

[0129] It should be noted that the foreground depth label is constructed based on the depth information of the center point of the target 2D detection bounding box. The depth of each pixel in the 2D bounding box is set to the depth of the box center. If a pixel belongs to multiple boxes, the depth of the center point with the closest Euclidean distance to that pixel is used as its label. This is to define the depth range... The values ​​within the depth are discretized using various depth discretization methods, including fixed interval (UD), logarithmic interval (LID), and linearly increasing interval (SID). The depth interval for SID is calculated as follows:

[0130] (16)

[0131] In equation (16), Indicates the number of depth categories. For category intervals.

[0132] continuous depth Mapping to discrete depth categories The calculation formula is:

[0133] (17)

[0134] S702. Using foreground depth labels as supervision, a foreground depth probability map is generated based on a three-layer convolutional structure and enhanced feature maps. The foreground depth probability map represents the probability that each pixel belongs to each discrete depth category.

[0135] Preferably, the foreground depth cue module employs a three-layer convolutional structure to process the features extracted from the backbone network. The first two layers are 3×3 convolutional layers, and the last layer is a 1×1 convolutional layer used to output a depth category probability map with a size of [missing information]. .

[0136] S703. Calculate the second depth value corresponding to each discrete depth category in the foreground depth probability map.

[0137] Specifically, the depth value represented by each category is calculated using the following formula. :

[0138] (18)

[0139] S704. The second depth value is converted into a continuous depth estimate for each pixel using a probability-weighted method.

[0140] Specifically, to transform the depth category prediction results into continuous depth values, then based on the probability of each pixel in each category... Calculate the continuously weighted depth estimate :

[0141] (19)

[0142] S705. Construct a foreground depth cue feature map based on continuous depth estimates.

[0143] Compared with existing technologies, this invention utilizes a designed foreground depth cue module to guide the network to focus on semantically meaningful foreground regions by fusing discretized depth modeling with continuous depth cues, thereby enhancing the expression of depth information and maintaining stable 3D detection performance in complex scenarios such as low light and long distances.

[0144] S206. Extract the two-dimensional and three-dimensional information of the target from the fused feature map based on the detection head.

[0145] like Figure 8 Step S206 includes:

[0146] S801, predicts heat map, center point offset and target two-dimensional size based on two-dimensional detection head.

[0147] S802. Extract target region features based on ROIAlign.

[0148] S803. The fused feature map and the target area features are fused together, and a three-dimensional detection head is used to predict the three-dimensional size, three-dimensional offset, viewing angle and depth value of the target.

[0149] Compared with existing technologies, this invention proposes a 3D detection mechanism that integrates normalized coordinate maps, enabling the model to perceive spatial location information at the ROI level and enhance features by combining depth cues. This improves the accuracy of predicting key parameters such as 3D size, orientation, and position, and provides greater scene adaptability and deployment value.

[0150] S104. Input the preset training samples, two-dimensional information, and three-dimensional information into the preset detection model, calculate the loss function value of the detection model, update the network parameters of all modules in the initial detection model based on the loss function value, and after several iterations, obtain a monocular image three-dimensional target detection model for outputting the category and spatial geometric parameters of the detected target based on the input monocular image.

[0151] In some embodiments of the present invention, the training samples include: the category, location, and depth information of the detected target.

[0152] The loss functions include: 2D heatmap loss, 2D offset loss, 2D size loss, 3D offset loss, 3D size loss, angle loss, foreground depth cue supervision loss, and continuous depth regression loss.

[0153] It is understood that the prediction model provided by this invention employs multiple joint loss functions for supervised optimization during training. The total loss function is composed of two-dimensional heatmap loss, two-dimensional offset loss, two-dimensional size loss, three-dimensional offset loss, three-dimensional size loss, angle loss, foreground depth supervision loss, and continuous depth regression loss. The heatmap loss is constructed using a focusing loss function, defined as follows:

[0154] (20)

[0155] in, For predicted values, For the true value, This indicates the number of key points in the heatmap. and Take 2 and 4 respectively.

[0156] Both the two-dimensional center offset loss and the two-dimensional size loss are adopted The loss function is defined as follows:

[0157] (twenty one)

[0158] (twenty two)

[0159] in, This indicates the predicted 2D center point offset. This represents the true value of the 2D center point offset. Indicates the predicted 2D size. Represents the actual value of the 2D dimensions.

[0160] The 3D offset loss and 3D size loss are both calculated using the L1 loss function. The 3D depth prediction part introduces uncertainty supervision, employing the Laplace distribution assumption to construct a negative log-likelihood loss function for 3D depth:

[0161] (twenty three)

[0162] in, For depth prediction mean, This represents the standard deviation of the forecast uncertainty.

[0163] Foreground depth clue monitoring loss uses focused loss to calculate predicted probability. The discrete classification error between the label and the target label is defined as follows:

[0164] (twenty four)

[0165] in, This represents the model's predicted probability for each depth category. Set to 0.25, Set to 2 。

[0166] The continuous depth value regression loss is also constructed using the Laplace log-likelihood loss:

[0167] (25)

[0168] Angle loss is achieved by combining multi-interval classification with residual regression to classify the angle range. The system is uniformly divided into multiple sub-intervals, and joint supervision is performed using multi-branch cross-entropy loss and cosine similarity loss. The loss is defined as follows:

[0169] (26)

[0170] (27)

[0171] The final angle loss can be expressed as:

[0172] (28)

[0173] After calculating the above loss functions, the network parameters are iteratively updated through backpropagation and an optimizer. After multiple training rounds, the final 3D object detection model is obtained.

[0174] In summary, compared with existing technologies, the 3D target detection model generation method provided by this invention first utilizes a constructed dual-branch network structure combining convolution and self-attention to acquire image features. Compared with traditional convolutional networks, the dual-branch network structure is more adaptable to scale changes and modeling long-distance dependencies, improving the model's 3D understanding ability in complex optoelectronic image scenes. Then, by fusing discretized depth modeling with continuous depth cues, the network is guided to focus on semantically meaningful foreground regions, strengthening the expression of depth information and maintaining stable 3D detection performance in complex scenes such as low light and long distances, achieving accurate estimation of 3D size, spatial position, orientation angle, and depth information. Finally, the initial detection model is trained and optimized based on the estimated data to obtain a detection model with higher detection accuracy, thereby effectively improving the detection accuracy of 3D targets in optoelectronic monocular images.

[0175] like Figure 9 Secondly, the present invention also provides a three-dimensional target detection method, comprising:

[0176] S901. Acquire a monocular image including the target to be detected;

[0177] S902. Input the monocular image into the 3D target detection model to obtain the target's category, 2D position, 3D size, spatial offset, orientation angle, and depth information.

[0178] The three-dimensional target detection model is obtained according to any one of the three-dimensional target detection model generation methods described above.

[0179] like Figure 10 Thirdly, the present invention also provides a three-dimensional target detection model generation device 90, comprising:

[0180] The feature extraction module 1010 is used to input the monocular image of the target to a preset convolutional-self-attention dual-branch neural network to extract the main feature map of the target.

[0181] The feature fusion module 1020 is used to generate a foreground depth cue feature map based on a deep discretization and continuous estimation mechanism, and to fuse the foreground depth cue feature map with the main feature map to obtain a fused feature map.

[0182] The information detection module 1030 is used to extract two-dimensional and three-dimensional information of the target object from the fused feature map based on the detection head;

[0183] The model training module 1040 is used to input preset training samples, as well as two-dimensional and three-dimensional information, into a preset detection model, calculate the loss function value of the detection model, update the network parameters of the detection model based on the loss function value, and after several iterations, obtain a monocular image three-dimensional target detection model that outputs the category, two-dimensional position, three-dimensional size, spatial offset, orientation angle, and depth information of the detected target based on the input monocular image.

[0184] like Figure 11 Fourthly, the present invention also provides a device 110, comprising:

[0185] Memory 1110 is used to store programs;

[0186] The processor 1120, coupled to the memory 1110, is used to execute the program stored in the memory to implement the steps of the three-dimensional target detection model generation method or the three-dimensional target detection method described in any one of the above method items.

[0187] Fifthly, the present invention also provides a medium comprising:

[0188] Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps of the three-dimensional target detection model generation method or the three-dimensional target detection method described in any one of the above method items.

[0189] The foregoing has provided a detailed description of a three-dimensional target detection model generation and detection method, apparatus, device, and medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for generating a three-dimensional target detection model, characterized in that, The method comprises the following steps: sequentially connecting a convolution-self-attention parallel branch, a feature guidance module, a multi-scale upsampling structure, a foreground depth clue module constructed based on a deep discretization and continuous estimation mechanism, and a detection head module to obtain a convolution-self-attention dual-branch neural network; connecting the convolution-self-attention dual-branch neural network and a preset loss function layer to obtain an initial detection model; inputting a monocular image of a detection target into the convolution-self-attention dual-branch neural network to extract two-dimensional information and three-dimensional information of the detection target, including: inputting the monocular image into a convolution branch to obtain multi-scale local features; inputting the monocular image into a self-attention branch to obtain global semantic features; fusing the multi-scale local features and the global semantic features based on the feature guidance module to obtain a main feature map; upsampling the main feature map based on the multi-scale upsampling structure to obtain an enhanced feature map; generating a foreground depth clue feature map based on the foreground depth clue module, and fusing the foreground depth clue feature map with the main feature map to obtain a fused feature map; extracting the two-dimensional information and the three-dimensional information of the detection target from the fused feature map based on the detection head module; inputting preset training samples and the two-dimensional information and the three-dimensional information into a preset initial detection model, calculating a loss function value of the initial detection model, updating network parameters of all modules in the initial detection model based on the loss function value, and obtaining a monocular image three-dimensional target detection model for outputting a category and a spatial geometric parameter of a detection target according to an input monocular image after several iterations; the foreground depth clue module comprises three layers of convolution structures; the foreground depth clue module generates a foreground depth clue feature map and fuses the foreground depth clue feature map with a main feature map to obtain a fused feature map, including: converting a first depth value of a target two-dimensional detection frame center point into a foreground depth label based on a depth discretization mechanism, wherein the target two-dimensional detection frame center point comprises a discrete foreground depth map generated based on a monocular image; generating a foreground depth probability map based on three layers of convolution structures and the enhanced feature map under the supervision of the foreground depth label, wherein the foreground depth probability map represents a probability of each pixel belonging to each discrete depth category; calculating a second depth value corresponding to each discrete depth category in the foreground depth probability map; converting the second depth value into a continuous depth estimation value of each pixel based on a probability weighting method; and constructing the foreground depth clue feature map based on the continuous depth estimation value.

2. The three-dimensional object detection model generation method of claim 1, wherein, the convolution branch takes a DLA-34 network as a backbone structure and introduces an aggregation module based on a dynamic large kernel; the self-attention branch adopts a single-head self-attention mechanism.

3. The three-dimensional object detection model generation method of claim 1, wherein, the detection head module comprises a two-dimensional detection head and a three-dimensional detection head; the detection head module extracts the two-dimensional information and the three-dimensional information of the detection target from the fused feature map, including: predicting a heat map, a center point offset, and a target two-dimensional size based on the two-dimensional detection head; extracting a target region feature based on ROIAlign. The fusion feature map and the target region feature are fused, and a three-dimensional detection head is used to predict three-dimensional size, three-dimensional offset, viewing angle and depth value of the detection target.

4. The three-dimensional object detection model generation method of claim 1, wherein, The training sample includes: category, position and depth information of the detection target. The loss function includes: two-dimensional heat map loss, two-dimensional offset loss, two-dimensional size loss, three-dimensional offset loss, three-dimensional size loss, angle loss, foreground depth clue supervision loss and continuous depth regression loss.

5. A three-dimensional object detection method characterized by, The method comprises the following steps: acquiring a monocular image including a detection target; inputting the monocular image into a three-dimensional target detection model to obtain category, two-dimensional position, three-dimensional size, spatial offset, orientation angle and depth information of the target, wherein the three-dimensional target detection model is obtained according to the three-dimensional target detection model generation method in any one of claims 1-4.

6. A three-dimensional object detection model generation device for executing the three-dimensional object detection model generation method according to any one of claims 1 to 4, characterized by The method comprises the following steps: a feature extraction module is configured to input a monocular image of a detection target into a preset convolutional-self-attention dual-branch neural network to extract a main feature map of the detection target; a feature fusion module is configured to generate a foreground depth clue feature map based on a depth discretization and continuous estimation mechanism, and fuse the foreground depth clue feature map with the main feature map to obtain a fusion feature map; an information detection module is configured to extract two-dimensional information and three-dimensional information of the detection target from the fusion feature map based on a detection head; a model training module is configured to input preset training samples and the two-dimensional information and the three-dimensional information into a preset detection model, calculate a loss function value of the detection model, update network parameters of the detection model based on the loss function value, and obtain a monocular image three-dimensional target detection model for outputting category, two-dimensional position, three-dimensional size, spatial offset, orientation angle and depth information of a detection target according to an input monocular image after several iterations.

7. An apparatus, comprising: The method comprises the following steps: a memory is configured to store a program; a processor is coupled to the memory and configured to execute the program stored in the memory to implement the steps of the three-dimensional target detection model generation method in any one of claims 1-4 or the three-dimensional target detection method in claim 5.

8. A computer storage medium, characterized in that The method comprises the following steps: a computer readable program or instruction is stored, and the program or instruction is executed by a processor to implement the steps of the three-dimensional target detection model generation method in any one of claims 1-4 or the three-dimensional target detection method in claim 5.

Images