A three-dimensional target detection method in a two-dimensional image based on instance structure correlation

Abstract

This invention provides a method for 3D object detection in 2D images based on instance structural correlation. The method includes acquiring a set of RGB images from a traffic scene and using these images to construct training and testing sets; constructing a 3D object detection model based on a convolutional neural network, including an RGB image feature extraction backbone network, an instance structural correlation construction module, and a multi-task detection branch; training the 3D object detection model using the training set; calculating the multi-task loss function using a stochastic gradient descent algorithm; updating the parameters of the 3D object detection model until it converges; and using the trained 3D object detection model to predict 3D objects in the 2D RGB image to be detected. This invention innovatively extracts a deep structural model of instances and employs a multi-head attention mechanism to construct structural correlations between instances. Utilizing the updated instance features, more accurate 3D object estimation results can be obtained.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional target detection technology, and in particular to a method for three-dimensional target detection in two-dimensional images based on instance structure correlation. Background Technology

[0002] Imaging 3D targets results in the loss of depth information. Point clouds are sets of points in 3D space mapped back from pixels in a 2D image based on camera parameters. Unlike 2D target detection, 3D target detection requires recovering information such as the object's depth, size, and pose; therefore, obtaining structural information is crucial. Point cloud data is defined in 3D space and thus contains a wealth of structural information, making point cloud-based 3D target detection generally effective. However, point cloud data is computationally intensive, leading to high complexity in such algorithms. Furthermore, noise in the point cloud data can also negatively impact detection performance.

[0003] Another type of 3D object detection method does not rely on point cloud data but directly detects 3D objects from 2D color images. These methods primarily utilize geometric constraints in the 3D scene, training the 3D detection model through an objective function based on these constraints. Because they rely solely on 2D images, these methods are computationally efficient, but the lack of sufficient structural information leads to insufficient detection performance. To improve detection results, some improved methods introduce 2D depth images as additional input. Unlike point cloud-based methods, these methods do not need to directly process point cloud data but instead obtain structural information by processing dimensionality-reduced point cloud data, thus alleviating some complexity. However, the performance of these algorithms heavily depends on depth estimation algorithms and still possesses a certain level of complexity compared to 2D color image-based detection algorithms. Summary of the Invention

[0004] The embodiments of the present invention provide a method for detecting three-dimensional objects in two-dimensional images based on instance structure correlation, so as to effectively detect three-dimensional objects in two-dimensional RGB images.

[0005] To achieve the above objectives, the present invention adopts the following technical solution.

[0006] A method for detecting 3D targets in 2D images based on instance structure correlation, comprising:

[0007] Collect RGB image sets from traffic scenes, and use the RGB image sets to construct training and testing sets;

[0008] A 3D target detection model based on a convolutional neural network is constructed. The 3D target detection model includes an RGB image feature extraction backbone network, an instance structure correlation construction module, and a multi-task detection branch.

[0009] The three-dimensional object detection model is trained using the training set and the test set. The multi-task loss function is calculated using the stochastic gradient descent algorithm to update the parameters in the three-dimensional object detection model until the three-dimensional object detection model converges, thus obtaining the trained three-dimensional object detection model.

[0010] The trained 3D object detection model is used to detect 3D objects in a 2D RGB image.

[0011] Preferably, the acquisition of RGB image sets from traffic scenes, and the construction of training and testing sets using the RGB image sets, includes:

[0012] Collect a set of RGB images of traffic scenes, divide the RGB image set into training set and test set at a 1:1 ratio, classify common objects in traffic scenes in RGB images, label the position, size and corner of common objects, and normalize all RGB images.

[0013] Preferably, the construction of the 3D target detection model based on a convolutional neural network includes an RGB image feature extraction backbone network, an instance structure correlation construction module, and a multi-task detection branch, comprising:

[0014] A 3D target detection model based on a convolutional neural network is constructed. The 3D target detection model includes an RGB image feature extraction backbone network, an instance structure correlation construction module, and a multi-task detection branch. The feature extraction backbone network contains four convolutional modules for extracting multi-level appearance and structural features. The instance structure correlation construction module uses the structural features of instances as input to construct the correlation between instances. The multi-task detection branch is used to classify 3D targets using a classifier and determine the position of 3D targets in 2D and 3D space.

[0015] Preferably, the step of training the 3D object detection model using the training set and the test set, calculating a multi-task loss function using the stochastic gradient descent algorithm, updating the parameters in the 3D object detection model, until the 3D object detection model converges, and obtaining a trained 3D object detection model, includes:

[0016] Step S3-1: Initialize the parameters of the three-dimensional target detection model, specifically including the parameters of the feature extraction backbone network and the convolutional layers, fully connected layers and normalization layers contained in the classifier and regressor;

[0017] Step S3-2: Set the training parameters, including learning rate, impulse, batch size and number of iterations, and use the stochastic gradient descent algorithm to train the 3D object detection model;

[0018] Step S3-3: Calculate the objective function value. For any iteration batch, input all images into the 3D object detection model and obtain the estimated category, location, size, and depth information from the 3D object detection model. Obtain the objective function value through the error with the actual labeled values. Three objective function values ​​are calculated during the training of the 3D object detection model:

[0019]

[0020]

[0021]

[0022] In formula (1), s i With p i Let be the category label and estimated probability of the i-th target, respectively, and in formula (2) With formula (3) gt represents the two-dimensional and three-dimensional estimated bounding boxes of the i-th target, respectively, and N represents the total number of targets.

[0023] Step S3-4: Add the multiple objective function values ​​to obtain the total objective function value, and calculate the partial derivatives with respect to all parameters in the 3D object detection model. Update the parameters using the stochastic gradient descent method.

[0024] Step S3-5: Repeat steps S3-3 and 3-4 to continuously update the parameters of the 3D object detection model until the objective function value no longer decreases during training, the 3D object detection model converges, and finally outputs the trained 3D object detection model parameters.

[0025] Preferably, the step of using the trained 3D object detection model to detect 3D objects in the 2D RGB image to be detected includes:

[0026] The two-dimensional RGB image to be detected is input into a trained three-dimensional object detection model. A backbone feature extraction network extracts the structural feature map R of the two-dimensional RGB image. The structural feature map R has dimensions C×H×W, where C, H, and W represent the dimension, height, and width, respectively. The feature map R is represented as a set of N C-dimensional features: R = [r1, r2, ..., r...]. N ] T Where N = H × W; a position code is added to the structural feature map R, which represents the position information of the object in the two-dimensional image;

[0027] The structural feature map R with added positional encoding is input into the instance structure correlation construction module. The instance structure correlation construction module selects training instance regions based on the overlap between the regions in the structural feature map R and the regions labeled in the training. The features in the training instance regions are then subjected to average pooling to obtain the feature vector n = pool(r1, r2, ..., r...). i ), where r1, r2, ..., r i The feature vector located inside this instance;

[0028] The instance structure correlation building module obtains self-attention maps of different dimensions through multiple self-attention branches, integrates them to obtain the structural correlation between instances, and after obtaining the global instance structure correlation, updates the instance structure features through matrix multiplication.

[0029]

[0030] Among them W v For a trainable parameter matrix, Let j be the normalization coefficient, and let j list all instances related to instance i. For the updated feature vector of instance i,

[0031] The multi-head instance structure correlation construction module with position encoding is represented by a residual network structure as follows:

[0032]

[0033]

[0034] Where n i For the structural features of instance i, p i Its corresponding position code, y i The output features are those updated after correlation. It is a single-head self-attention mechanism, consisting of a trainable parameter matrix. Let δ be the Softmax function, and W be... y It is a trainable parameter matrix;

[0035] The multi-task detection branch obtains the category and location information of the three-dimensional objects in the two-dimensional RGB image to be detected based on the output features updated by correlation.

[0036] As can be seen from the technical solutions provided by the embodiments of the present invention above, the present invention proposes an instance structure correlation construction mechanism for three-dimensional target detection tasks, which is mainly used to strengthen the structural correlation between detected targets, improve the accuracy of target depth estimation, and thus improve the accuracy of three-dimensional target detection.

[0037] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of the invention. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a flowchart illustrating a method for detecting three-dimensional targets in two-dimensional images based on instance structural correlation, provided in an embodiment of the present invention.

[0040] Figure 2 This is a structural diagram of a three-dimensional target detection model provided in an embodiment of the present invention.

[0041] Figure 3 This is a flowchart of a three-dimensional target detection model training process provided in an embodiment of the present invention.

[0042] Figure 4 This is an example of a structural correlation diagram provided in an embodiment of the present invention. Detailed Implementation

[0043] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0044] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or couplings. The term “and / or” as used herein includes any and all combinations of one or more of the associated listed items.

[0045] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0046] To facilitate understanding of the embodiments of the present invention, the following will provide further explanation and description with reference to the accompanying drawings and several specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0047] This invention primarily targets 3D object detection tasks, proposing a correlation construction method based on a graph model. The related technical background mainly includes point cloud-based 3D object detection, 2D image-based 3D object detection, and graph learning methods. This invention also relates to graph learning methods. A graph is a data structure composed of nodes and edges, where edges represent the correlations between nodes, and are the key element of graph learning methods.

[0048] This invention proposes a method for 3D object detection in 2D images based on instance structural correlation. The method uses a single 2D color image as the sole input, obtains the structural features of the objects to be detected through a convolutional neural network model, and then constructs the structural correlations between the objects through a multi-head self-attention mechanism. These structural correlations can help improve the detection accuracy of objects globally. To further enhance the structural correlations between the objects, the method also adds a position encoding module, which constructs structural codes based on the object's position in the 2D image, further improving the accuracy of object detection.

[0049] The method for detecting three-dimensional targets in two-dimensional images based on instance structure correlation in this invention mainly includes the following processing steps:

[0050] (1) Dataset construction: Construct training and testing sets for the 3D object detection model. Specifically, this includes collecting RGB images for training and testing, labeling the categories, positions, depths, and sizes of objects in the training images, and preprocessing the images.

[0051] (2) Construction of 3D target detection model: A one-stage 3D target detection model based on convolutional neural network is constructed, which includes the RGB image feature extraction backbone network, the instance structure correlation construction module, and the multi-task detection branch.

[0052] (3) Training of the three-dimensional target detection model: The parameters in the three-dimensional target detection model are updated until convergence by calculating the multi-task loss function and the stochastic gradient descent algorithm.

[0053] (4) Detecting three-dimensional targets: Detecting three-dimensional objects in the provided color images or video frames.

[0054] A flowchart of a three-dimensional target detection method in a two-dimensional image based on instance structure correlation provided in this embodiment of the invention is shown below. Figure 1 As shown, it specifically includes the following steps:

[0055] Step S1: Construct the training and test sets. Collect a set of RGB images from traffic scenes and divide the RGB image set into a training set and a test set at an approximately 1:1 ratio. Classify common objects in traffic scenes, such as vehicles and pedestrians, in the images. Additionally, label the position, size, and corners of the objects. Finally, normalize all RGB images.

[0056] Step S2: Construct a 3D target detection model. An embodiment of the present invention provides a 3D target detection model as follows: Figure 2 As shown, it mainly includes a feature extraction backbone network, an instance structure correlation construction module, and a multi-task detection branch. The feature extraction backbone network contains four convolutional modules used to extract multi-level appearance and structural features. The instance structure correlation construction module uses the structural features of instances as input to build correlations between instances, thereby improving the accuracy of target depth estimation and thus improving detection accuracy. This is also a key module in this embodiment of the invention. Finally, the multi-task detection branch is used to classify 3D targets using a classifier and determine the positions of 3D targets in 2D and 3D space.

[0057] Step S3: Train the 3D object detection model. After constructing the 3D object detection model, it can be trained using the training set obtained in Step S1. The training process for the 3D object detection model is as follows: Figure 3 As shown, the specific steps include:

[0058] Step S3-1: Initialize model parameters, specifically including the parameters of the feature extraction backbone network and the convolutional, fully connected, and normalization layers contained in the classifier and regressor.

[0059] Step S3-2: Set training parameters. The 3D object detection model in this invention is trained using the Stochastic Gradient Descent algorithm. Before training, relevant training parameters need to be set, including learning rate, impulse, batch size, and number of iterations.

[0060] Step S3-3: Calculate the objective function value. For any given batch of iterations, input all images into the object detection model and obtain its estimated category, location, size, and depth information. The objective function value is then obtained by calculating the error between this estimated and the actual labeled values. Three types of objective function values ​​are calculated during the training of the 3D object detection model:

[0061]

[0062]

[0063]

[0064] In formula (1), s i With p i Let be the category label and estimated probability of the i-th target, respectively, and in formula (2) With formula (3) gt represents the two-dimensional and three-dimensional estimated bounding boxes of the i-th target, respectively, gt represents the actual labeled value, and N represents the total number of targets.

[0065] Step S3-4: Add the multiple objective function values ​​to obtain the total objective function value, calculate the partial derivatives of all parameters in the 3D object detection model, and then update the parameters using the stochastic gradient descent method.

[0066] Step S3-5: Repeat steps S3-3 and 3-4 to continuously update the parameters of the 3D object detection model until the 3D object detection model converges, that is, the objective function value no longer decreases during the training process. Finally, output the parameters of the trained 3D object detection model.

[0067] Step S4: Use the trained 3D object detection model to predict 3D objects in the 2D RGB image.

[0068] After obtaining all the parameters of the trained 3D object detection model, the 2D image to be detected is input into the trained 3D object detection model, and the 3D object detection model detects the objects in the 2D image.

[0069] The key module in the 3D target detection model of this invention is the instance structure correlation construction module. This module can construct structural correlations between instances through depth features, thereby improving the overall accuracy of 3D target detection. A detailed description follows.

[0070] The flowchart of the instance structure correlation construction module in this embodiment of the invention is as follows: Figure 4As shown, its input is the structural features of the instance. It obtains self-attention maps of different dimensions through multiple self-attention branches, then integrates them to obtain the structural correlation between instances, and finally updates the instance structural features through the structural correlation to obtain more accurate instance detection.

[0071] Suppose that the structural feature map R of a 2D RGB input image after passing through a backbone feature extraction network has dimensions C×H×W, where C, H, and W are the dimension, height, and width, respectively. The feature map R can be represented as a set of N C-dimensional features: R = [r1, r2, ..., r...]. N ] T Where N = H × W. Since this embodiment of the invention requires constructing the structural correlation between instances, we need to extract the structural information of the instances. Specifically, we determine the training instances based on the overlap between the regions in the feature map and the regions labeled in the training map. When the overlap between a certain region and a certain training labeled region is higher than 30%, that region in the feature map is determined as a training instance. The features in the training instance regions are then subjected to average pooling to obtain the feature vector n = pool(r1, r2, ..., r...). i ), where r1, r2, ..., r i This is the feature vector located inside this instance.

[0072] After obtaining the feature vectors of all instances in the feature map R, the structural correlations between instances can be constructed. Inspired by the current Transformer model, this embodiment of the invention employs a multi-head self-attention mechanism to construct structural correlations. For each individual self-attention mechanism, the constructed structural correlation can be expressed as:

[0073] A ij =Softmax(W k n i ·W q n j (4)

[0074] Where n i With n j W is the feature vector of the instance structure. k With W q A is a trainable parameter matrix that maps structure vectors to a common space. ij For instance n i With n j The structural correlation matrix between them. For any input image, assuming it contains N training instances, we can construct a structural correlation map of size N×N, which marks the structural correlation between each training instance and the remaining instances.

[0075] After obtaining the global instance structure correlation, we can update the structural features through matrix multiplication:

[0076]

[0077] Among them W v For a trainable parameter matrix, Let j be the normalization coefficient, and let j list all instances related to instance i. Let i be the updated feature vector for instance i. Finally, we use a residual network structure to represent the instance structure correlation module as follows:

[0078]

[0079] Among them W y For a trainable parameter matrix, This is the residual term, y i To output the feature vector. In practical operation, we will use W in the above formula. k W q W v W y It is implemented using a convolution operation without an offset term, with a kernel size of 1×1 and a dimension of 256.

[0080] At this point, we have completed the construction of a structural relevance building module based on a single-head self-attention mechanism. This module can map input structural features to a single common space using a transformation matrix and construct structural relevances. Since the Transformer model has been proven to be a very effective attention learning mechanism for multiple tasks, this embodiment extends the single-head self-attention mechanism to multi-head models to construct better structural relevances. Specifically, we will use the W parameter matrix mentioned above... k W q W v The system is divided into eight parts, each implemented using a 1×1 convolution operation without offset terms, with a dimension of 256 / 8 = 64. Each part acts as a single-head self-attention mechanism, mapping the input structural feature vector to a specific common space and constructing structural correlations. After obtaining the structural correlation matrix generated by each single-head self-attention mechanism, they are accumulated and integrated into a single structural correlation matrix, which is then processed by W... y The residual term in formula (6) can be obtained by transforming the expression.

[0081] To further enhance the structural correlation between instances, we add positional encoding. Positional encoding has a positive effect on the Transformer model, mainly because the Transformer model structure limits its ability to accurately represent the position of instances. Although this embodiment of the invention uses a convolutional neural network capable of acquiring positional information, since the purpose of this embodiment is to acquire the structural information of instances, the introduction of positional encoding can effectively improve the accuracy of the structural information. Specifically, the positional encoding in this embodiment is represented by a one-dimensional trainable parameter vector, the dimension of which is consistent with the number of training instances, and initialized as an all-zero vector. Finally, the multi-head instance structural correlation construction module with positional encoding can be represented as follows:

[0082]

[0083]

[0084] Where n i For the structural features of instance i, p i Its corresponding position code, y i The output features are those updated after correlation. It is a single-head self-attention mechanism, and consists of a trainable parameter matrix. Let δ be the Softmax function. As can be seen, our structural correlation building blocks are entirely composed of learnable parameter matrices, requiring no artificially defined structural relationships or assumptions.

[0085] In the 3D object detection model, the multi-task detection branch obtains the category and location information of the 3D object in the 2D RGB image to be detected based on the output features updated by correlation.

[0086] In summary, this invention innovatively extracts the deep structural model of instances and uses it as input, employing a multi-head attention mechanism to construct structural correlations between instances. These structural correlations can update the deep features of the input instances, resulting in more accurate 3D target estimation. This invention can be embedded into any deep learning-based 3D target detection model, and it also helps to construct low-complexity 3D target detection models, enabling them to achieve satisfactory target detection results without relying on depth data. This has a significant positive impact on applications such as autonomous driving.

[0087] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0088] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

[0089] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for apparatus or system embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The apparatus and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0090] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for detecting three-dimensional targets in two-dimensional images based on instance structural correlation, characterized in that, include: Collect RGB image sets from traffic scenes, and use the RGB image sets to construct training and testing sets; A 3D target detection model based on a convolutional neural network is constructed. The 3D target detection model includes an RGB image feature extraction backbone network, an instance structure correlation construction module, and a multi-task detection branch. The three-dimensional object detection model is trained using the training set and the test set. The multi-task loss function is calculated using the stochastic gradient descent algorithm to update the parameters in the three-dimensional object detection model until the three-dimensional object detection model converges, thus obtaining the trained three-dimensional object detection model. The trained 3D object detection model is used to detect 3D objects in a 2D RGB image to be detected; The aforementioned construction of a 3D object detection model based on a convolutional neural network includes an RGB image feature extraction backbone network, an instance structure correlation construction module, and a multi-task detection branch, comprising: A 3D target detection model based on a convolutional neural network is constructed. The 3D target detection model includes an RGB image feature extraction backbone network, an instance structure correlation construction module, and a multi-task detection branch. The feature extraction backbone network contains four convolutional modules for extracting multi-level appearance and structural features. The instance structure correlation construction module uses the structural features of instances as input to construct the correlation between instances. The multi-task detection branch is used to classify 3D targets using a classifier and determine the position of 3D targets in 2D and 3D space. The process of training the 3D object detection model using the training and test sets, calculating a multi-task loss function using the stochastic gradient descent algorithm, updating the parameters of the 3D object detection model, and continuing until the 3D object detection model converges to obtain a trained 3D object detection model includes: Step S3-1: Initialize the parameters of the three-dimensional target detection model, specifically including the parameters of the feature extraction backbone network and the convolutional layers, fully connected layers and normalization layers contained in the classifier and regressor; Step S3-2: Set the training parameters, including learning rate, impulse, batch size and number of iterations, and use the stochastic gradient descent algorithm to train the 3D object detection model; Step S3-3: Calculate the objective function value. For any iteration batch, input all images into the 3D object detection model and obtain the estimated category, location, size, and depth information from the 3D object detection model. Obtain the objective function value through the error with the actual labeled values. Three objective function values ​​are calculated during the training of the 3D object detection model: ,（1） , （2） ,（3） In formula (1) and The first The category labeling and estimated probability of each target are given in formula (2). Representing the A two-dimensional estimation frame for each target; in formula (3) and Representing the first Centrality of each objective and its 3D estimated bounding box, superscript This represents the actual labeled value. The total number of targets is represented; Step S3-4: The multiple objective function values ​​are added together to obtain the total objective function value, and the partial derivatives of all parameters in the three-dimensional target detection model are calculated respectively. The parameters are then updated using the stochastic gradient descent method. Step S3-5: Repeat steps S3-3 and 3-4 to continuously update the parameters of the 3D object detection model until the objective function value no longer decreases during training, the 3D object detection model converges, and finally outputs the trained 3D object detection model parameters.

2. The method according to claim 1, characterized in that, The process of collecting RGB image sets from traffic scenes and constructing training and testing sets using these RGB image sets includes: Collect a set of RGB images of traffic scenes, divide the RGB image set into training set and test set at a 1:1 ratio, classify common objects in traffic scenes in RGB images, label the position, size and corner of common objects, and normalize all RGB images.

3. The method according to claim 1, characterized in that, The process of using the trained 3D object detection model to detect 3D objects in a 2D RGB image includes: The two-dimensional RGB image to be detected is input into a trained three-dimensional object detection model, and the structural feature map of the two-dimensional RGB image is extracted through the backbone feature extraction network. Structural feature map The dimension is ,in , and The feature map is defined by its dimensions, height, and width. Represented as indivual A set of dimensional features: ,in ; in the structural feature diagram Add a position code to the image; this position code represents the location information of the object in the two-dimensional image. The structural feature map with added positional encoding will be added. The input is fed into the instance structure correlation construction module, which then constructs the instance structure correlation based on the structural feature map. The overlap between the regions in the dataset and the regions labeled in the training dataset is used to select training instance regions. The features within these training instance regions are then subjected to average pooling to obtain the feature vector for that instance. ,in The feature vector located inside this instance; The instance structure correlation building module obtains self-attention maps of different dimensions through multiple self-attention branches, integrates them to obtain the structural correlation between instances, and after obtaining the global instance structure correlation, updates the instance structure features through matrix multiplication. ,（4） in For a trainable parameter matrix, The normalization coefficient is... List all and examples Related examples, For example Updated feature vector, Representation of instances and Attention weights between them; The multi-head instance structure correlation construction module with position encoding is represented by a residual network structure as follows: ，（5） ，（6） in and Examples and Structural features, and Encode its corresponding position. The output features are those updated after correlation. It is a single-head self-attention mechanism, consisting of a trainable parameter matrix. express, For the Softmax function, For trainable parameter matrix, Indicates a cascading operation; The multi-task detection branch obtains the category and location information of the three-dimensional objects in the two-dimensional RGB image to be detected based on the output features updated by correlation.

Images