Three-dimensional object detection method based on multi-level cross-modal self-attention mechanism

Abstract

The present invention provides a three-dimensional target detection method based on a multi-level cross-modal self-attention mechanism. The method comprises constructing a training set and a test set using RGB image data; constructing a three-dimensional target detection model, which comprises an RGB backbone network, a deep backbone network, a classifier and a regressor; training the three-dimensional target detection model using the training set and the test set data, and verifying the training effect using the test set to obtain a trained three-dimensional target detection model; and detecting three-dimensional targets in RGB images using the trained model. The method of the present invention obtains the depth structure information within the global scene range from the depth feature map, and organically combines it with the appearance information to improve the accuracy of the three-dimensional target detection algorithm, thereby effectively detecting the category, position, size and posture information of the three-dimensional objects in the two-dimensional RGB image.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to a three-dimensional target detection method based on a multi-level cross-modal self-attention mechanism. Background Art

[0002] 3D object detection is an important branch in the field of computer vision, and has strong application value in many scenarios such as intelligent transportation, robot vision, 3D reconstruction, virtual reality and augmented reality. The purpose of 3D object detection is to restore the category, position, depth, size and posture of objects in 3D space. According to the different types of processed data, 3D object detection technology is mainly divided into two categories: detection based on 2D images and detection based on point cloud data.

[0003] The process of imaging a three-dimensional object is to map points in the three-dimensional space to a two-dimensional plane after losing the depth information. To detect objects in the three-dimensional space, the lost depth information must be used. This is one of the main differences between three-dimensional target detection and two-dimensional target detection, and it is also the difficulty of three-dimensional target detection. The three-dimensional target detection method based on two-dimensional images can directly obtain depth information from the two-dimensional image and then detect the three-dimensional target. The acquisition of its depth information mainly depends on many constraints such as the geometric constraints in the three-dimensional scene, the shape constraints and semantic constraints of the three-dimensional object. Since the depth information contained in the two-dimensional image is limited, and the constraints are greatly restricted by the scene and the object, the accuracy that can be achieved by this type of three-dimensional target detection method is relatively low.

[0004] Point cloud is a collection of points in three-dimensional space corresponding to pixels in two-dimensional images. Three-dimensional target detection based on point cloud data can obtain depth information by processing point cloud data, which can be further divided into two categories. The first is to directly process point cloud data in three-dimensional space, and directly process point cloud by upgrading the operation of pixels in two-dimensional target detection method to three dimensions. Due to the increase in operation dimension, the computational complexity of this type of method is high, and the noise data in the point cloud will directly affect the detection accuracy of this type of algorithm. Another method first obtains a depth prediction model through point cloud data training, and obtains a two-dimensional depth image through the model, and then obtains depth information through the two-dimensional depth image for three-dimensional target detection. This type of algorithm does not need to directly operate on point cloud data, but reduces the dimension of point cloud data to a two-dimensional depth map, which reduces the computational complexity. At the same time, the depth prediction model can remove some point cloud noise data, so it is widely used in practical applications.

[0005] A 3D target detection method in the prior art includes: after obtaining a 2D depth map, since the depth prediction model obtained by training the point cloud data has the ability to obtain depth information, the method further trains the 3D target detection model through the 2D RGB image based on the depth prediction model. The disadvantages of this method are: for the 3D target detection task, it is not necessary to directly process the 3D point cloud data to obtain the category and position information of the target from the 2D image or video frame, and the point cloud data usually contains a lot of noise.

[0006] Another 3D target detection method in the prior art includes: this method uses a 2D depth image as an independent model input, obtains depth information from the depth image through an additional model, and combines it with a 2D RGB (red, green, blue) image input to perform 3D target detection. The disadvantages of this method are: the depth information that can be obtained from the 2D image is very limited, and the process of obtaining the depth information inevitably uses geometric constraints, so the detection accuracy of this type of algorithm is poor. Summary of the invention

[0007] An embodiment of the present invention provides a three-dimensional target detection method based on a multi-level cross-modal self-attention mechanism to effectively detect the category, position and posture of three-dimensional objects in a two-dimensional RGB image.

[0008] In order to achieve the above object, the present invention adopts the following technical scheme.

[0009] A 3D object detection method based on a multi-level cross-modal self-attention mechanism, comprising:

[0010] Use RGB image data to construct training set and test set data;

[0011] Build a 3D object detection model, which includes an RGB backbone network, a deep backbone network, a classifier, and a regressor;

[0012] The three-dimensional object detection model is trained using the training set and the test set data, and the training effect of the three-dimensional object detection model is verified using the test set. The RGB backbone network and the deep backbone network respectively obtain RGB features and depth features, and the RGB features and the depth features are input into the cross-modal self-attention learning module, and the RGB features are updated. The updated RGB features are used to learn the classifier and the regressor to obtain a trained three-dimensional object detection model;

[0013] The classifier and regressor in the trained three-dimensional target detection model are used to detect the category, position and posture of the three-dimensional object in the two-dimensional RGB image to be identified.

[0014] Preferably, the use of RGB image data to construct training set and test set data includes:

[0015] Collect RGB images, divide the RGB images into training set and test set in a ratio of approximately 1:1, normalize the image data in the training set and the test set, obtain the two-dimensional depth image of the training set image through the depth estimation algorithm, annotate the category of objects in the training set image, and annotate the coordinates of the two-dimensional detection box of the image, as well as the center position, size and rotation angle of the three-dimensional detection box.

[0016] Preferably, the RGB backbone network, deep backbone network, classifier and regressor in the three-dimensional target detection model all include convolutional layers, fully connected layers and normalization layers. The structures of the RGB backbone network and the deep backbone network are consistent, and both include 4 convolutional modules.

[0017] Preferably, the three-dimensional object detection model is trained using the training set and the test set data, the RGB backbone network and the deep backbone network respectively obtain RGB features and depth features, the RGB features and the depth features are input into the cross-modal self-attention learning module, the RGB features are updated, and the classifier and regressor are learned using the updated RGB features to obtain a trained three-dimensional object detection model, including:

[0018] Step S3-1: Initialize the parameters of the convolutional layer, the fully connected layer and the normalization layer included in the RGB backbone network, the deep backbone network, the classifier and the regressor in the three-dimensional object detection model;

[0019] Step S3-2: Setting relevant training parameters of the stochastic gradient descent algorithm, the relevant training parameters including learning rate, impulse, batch size and number of iterations;

[0020] Step S3-3: For any iterative batch, all RGB images and depth images are input into the RGB backbone network and the deep backbone network respectively to obtain multi-level RGB features and depth features, a cross-modal self-attention learning module is constructed, the RGB features and depth features are input into the cross-modal self-attention learning module, a self-attention matrix based on depth information is learned, the RGB features are updated through the self-attention matrix, the updated RGB features are used to learn a classifier and a regressor, and the classifier and the regressor are used for target detection of three-dimensional objects in two-dimensional RGB images.

[0021] The objective function value is obtained by calculating the error between the network estimated value and the actual labeled value. The three objective function values ​​are calculated using formulas (1), (2) and (3) respectively:

[0022]

[0023]

[0024]

[0025] Where s in formula (1) i With p i are the category label and estimated probability of the i-th target respectively. Compared with the formula (3) They represent the 2D estimated box and 3D estimated box of the i-th target, gt represents the actual annotation value, and N represents the total number of targets;

[0026] Step S3-4: Add the three objective function values ​​to obtain a total objective function value, and respectively calculate partial derivatives of all parameters in the three-dimensional object detection model, and update the parameters by stochastic gradient descent method;

[0027] Step S3-5: Repeat steps S3-3 and 3-4 to continuously update the parameters of the three-dimensional object detection model until convergence, and output the parameters of the trained three-dimensional object detection model.

[0028] Preferably, the step of inputting the RGB features and the depth features into a cross-modal self-attention learning module, updating the RGB features, and using the updated RGB features to learn a classifier and a regressor to obtain a trained three-dimensional object detection model includes:

[0029] For any two-dimensional RGB feature map R and two-dimensional depth feature map D, assuming that their dimensions are C×H×W, where C, H, and W are dimension, height, and width respectively, the two-dimensional RGB feature map R and the two-dimensional depth feature map D are represented as a set of N C-dimensional features: R = [r 1 , r 2 , ..., r N ] T and D = [d 1 , d 2 , ..., d N ] T , where N = H × W;

[0030] For the input feature map R, a fully connected graph is constructed, where each feature r i As a node, the edge (r i , r j ) represents the node r i With r j The relationship between them is obtained by learning the edge through the two-dimensional depth feature map D, and the current two-dimensional RGB feature map R is updated, which is specifically expressed as:

[0031]

[0032] in is the normalization parameter, δ is the softmax function, j is all the positions related to i, For the updated RGB features, write the above formula in the form of matrix multiplication:

[0033]

[0034] in is the self-attention matrix, D θ ,D φ With R g The dimensions are all N×C′;

[0035] The feature matrix r of each spatial position i Treat it as a node and search for the same node as r in the entire space. i Associated nodes, for any node i in the deep feature map, the representative features of the number S sampled in all nodes associated with i are:

[0036]

[0037] Where s(n) is the feature vector obtained by sampling, and its dimension is C′, is a sampling function, and the cross-modal self-attention learning module is expressed as:

[0038]

[0039] Where n is the sampled node related to i, δ is the softmax function, d θ (i) = W θ d(i),s φ (i) = W φ s(i),s g (i) = W g s(i). and They are the transformation matrices of the three linear transformations respectively.

[0040] It can be seen from the technical solutions provided by the above embodiments of the present invention that the embodiments of the present invention provide a multi-level cross-modal self-attention learning mechanism for three-dimensional object detection, which obtains the deep structure information within the global scene range from the deep feature map, and organically combines it with the appearance information to improve the accuracy of the three-dimensional object detection algorithm. At the same time, a variety of strategies are adopted to reduce the computational complexity to meet the processing speed requirements of scenes such as unmanned driving.

[0041] Additional aspects and advantages of the present invention will be given in part in the following description, which will become obvious from the following description, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0043] Figure 1 A flow chart of a three-dimensional target detection method based on a cross-modal self-attention mechanism is provided in an embodiment of the present invention.

[0044] Figure 2 A structural diagram of a three-dimensional target detection model provided in an embodiment of the present invention.

[0045] Figure 3 A training flowchart of a three-dimensional object detection model provided in an embodiment of the present invention.

[0046] Figure 4 A structural diagram of a cross-modal self-attention module provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0048] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or coupling. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.

[0049] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0050] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.

[0051] Based on the main shortcomings of the current three-dimensional target detection algorithm, the present invention obtains depth information through a two-dimensional depth map, and formalizes the use of depth information into a cross-modal self-attention module learning problem. The depth information is combined with the appearance information through a cross-modal self-attention mechanism, and the depth information is extracted in a non-iterative manner on a global scale through the self-attention learning mechanism to improve the detection accuracy. When obtaining depth information, the present invention also uses many measures to further reduce the complexity of the calculation to ensure that it can be used in scenarios with real-time processing requirements such as autonomous driving.

[0052] The present invention proposes a three-dimensional target detection method based on a multi-level cross-modal self-attention mechanism, which takes a two-dimensional RGB image and a depth image as input, and combines the appearance information obtained by the two-dimensional RGB image with the structural information obtained by the depth image through the self-attention mechanism to achieve accurate detection results, while avoiding the high amount of computation caused by point cloud processing. In addition, since the self-attention mechanism obtains a large amount of redundant information while obtaining global structural information, this method adopts an improved self-attention mechanism, that is, for a certain area, only the structural information of the area with the highest correlation with it in the global area is calculated, which further reduces the amount of computation while ensuring the detection accuracy.

[0053] The three-dimensional object detection method based on the multi-level cross-modal self-attention mechanism of the embodiment of the present invention includes the following processing steps:

[0054] Dataset construction: Construct training and test sets for the 3D object detection model, including collecting RGB images for training and testing, and extracting the depth information corresponding to the training set images through the depth model. Label the category, 2D coordinates, 3D coordinates, depth, and size of the objects in the training images. Preprocess the image data.

[0055] Construction of three-dimensional object detection model: Construction of a three-dimensional object detection model based on convolutional neural network, including the construction of RGB image feature extraction network, deep image feature extraction network, and cross-modal self-attention learning network.

[0056] 3D object detection model training: By calculating the loss functions of 2D object detection, 3D object detection classification and regression, and the stochastic gradient descent algorithm, the parameters in the 3D object detection model are updated until convergence.

[0057] Detect three-dimensional targets: Detect three-dimensional objects in the provided color images or video frames.

[0058] The processing flow chart of the three-dimensional target detection method based on the multi-level cross-modal self-attention mechanism provided by the embodiment of the present invention is as follows: Figure 1 As shown, it includes the following steps:

[0059] Step S1: Construct a training set and a test set. Collect RGB images and divide them into a training set and a test set in a ratio of approximately 1:1. Since the three-dimensional target detection method provided in the embodiment of the present invention obtains depth information through a two-dimensional depth image rather than the point cloud data used in the traditional method, a two-dimensional depth image is obtained through a depth estimation algorithm for the color image in the training set. In addition, for the objects in the training set image, first mark their categories, and at the same time mark the coordinates of their two-dimensional detection boxes, as well as the center position, size, and rotation angle of the three-dimensional detection box. Finally, the image data in the training set and the test set are normalized.

[0060] Step S2: After obtaining the training set and the test set, a 3D object detection model is constructed. The 3D object detection model includes an RGB backbone network, a deep backbone network, a classifier, and a regressor. The structure of a 3D object detection model provided by an embodiment of the present invention is as follows: Figure 2 As shown. Since we need to extract features from RGB images and depth images respectively during training, we need to build two feature extraction backbone networks. In the embodiment of the present invention, the structures of the RGB backbone network and the depth backbone network are consistent, both of which include 4 convolution modules for extracting multi-level features.

[0061] Step S3: training the 3D object detection model. After the 3D object detection model is constructed, the model can be trained using the training set obtained in step S1, and the training effect of the 3D object detection model can be verified using the test set. Figure 3 As shown, the specific steps include:

[0062] Step S3-1: Initialize the model parameters, including the parameters of the convolutional layers, fully connected layers, and normalization layers included in the RGB backbone network, deep backbone network, classifier, and regressor.

[0063] Step S3-2: Setting training parameters. The three-dimensional object detection model of the embodiment of the present invention is trained using SGD (Stochastic Gradient Descnet, stochastic gradient descent algorithm), and relevant training parameters need to be set before training, including learning rate, impulse, batch size and number of iterations.

[0064] Step S3-3: Calculate the objective function value. For any iterative batch, all RGB images and depth images are input into the RGB backbone network and the deep backbone network respectively to obtain multi-level features, and the updated RGB features are obtained through the cross-modal self-attention learning module. After that, the estimated category, position, posture and depth value of the target object are further obtained through the classifier and regressor. Finally, the objective function value is obtained by calculating the error between the network estimation value and the actual annotation value. Three objective function values ​​are calculated during the training of this model:

[0065]

[0066]

[0067]

[0068] Where s in formula (1) i With p i are the category label and estimated probability of the i-th target respectively. Compared with the formula (3) They represent the 2D estimated box and 3D estimated box of the i-th target respectively, gt represents the actual annotation value, and N represents the total number of targets.

[0069] Step S3-4: Add multiple objective function values ​​to obtain the total objective function value, calculate partial derivatives of all parameters in the model respectively, and then update the parameters by stochastic gradient descent method.

[0070] Step S3-5: Repeat steps S3-3 and 3-4, continuously update the model parameters until convergence, and finally output the model parameters.

[0071] So far, all parameters of the three-dimensional object detection model in the embodiment of the present invention have been obtained, and finally only the objects in the two-dimensional image provided by the user need to be detected.

[0072] Step S4: After obtaining multi-level RGB features and depth features, a cross-modal self-attention learning module is constructed. The module takes RGB features and depth features as inputs at the same time, learns to obtain a self-attention matrix based on depth information, and updates the RGB features through the self-attention matrix to increase the structural information in the RGB features. Finally, the updated RGB features are used to learn the classifier and regressor, which are used for target detection of three-dimensional objects in two-dimensional RGB images, where the classifier can identify the category of the three-dimensional target and the regressor can identify the position and posture of the three-dimensional target.

[0073] The three-dimensional object detection model in the embodiment of the present invention includes an RGB backbone network, a deep backbone network, a classifier and a regressor. After the training is completed, the RGB backbone network has retained the deep structure information through the cross-modal self-attention learning module. When testing, only a two-dimensional RGB image is required, and the deep backbone network is not required to extract deep features.

[0074] The cross-modal self-attention learning module of the embodiment of the present invention can obtain deep structure information through depth map learning and embed it into RGB image features, thereby improving the accuracy of three-dimensional object detection. The following is a detailed introduction.

[0075] The structural flow chart of a cross-modal self-attention learning module provided by an embodiment of the present invention is as follows: Figure 4 As shown in the figure, it mainly includes four submodules: sampling point generation module, multi-level attention learning module, information update module and information fusion module. The core idea of ​​its construction is to obtain a self-attention matrix based on depth information through multi-level deep feature map learning. This self-attention matrix can reflect the structural similarity between different positions within the global image range. The RGB feature map is updated through the self-attention matrix to obtain the structural features within the global image range, and finally improve the accuracy of 3D target detection. Figure 4 In the figure, a two-level deep feature map is used as an example. In actual operation, it can be expanded to multiple levels of deep features.

[0076] For any two-dimensional RGB feature map R and two-dimensional depth feature map D, assume that their dimensions are C×H×W, where C, H, and W are dimension, height, and width respectively. Both the two-dimensional RGB feature map R and the two-dimensional depth feature map D can be represented as a set of N C-dimensional features: R = [r 1 , r 2 , ..., r N ] T and D = [d 1 , d 2 , ..., d N ] T, where N = H × W. For the input feature map R, a fully connected graph is constructed, where each feature r i As a node, the edge (r i , r j ) represents the node r i With r j For the two-dimensional RGB feature map R, its appearance features such as color and texture are more obvious, but the structural information such as depth is insufficient. The cross-modal self-attention learning module in the embodiment of the present invention learns the edge through the two-dimensional depth feature map D, and then updates the current two-dimensional RGB feature map R to increase its structural features, which can be specifically expressed as:

[0077]

[0078] in is the normalization parameter, δ is the softmax function, j is all the positions related to i, is the updated RGB feature. We can further write the above formula in the form of matrix multiplication:

[0079]

[0080] in is the self-attention matrix, D θ ,D φ With R g The dimensions are all N×C′.

[0081] So far, a single-level cross-modal self-attention learning module has been constructed. It can learn the self-attention matrix containing structural information from a single-level deep feature map and update the RGB feature map of the corresponding level. However, as can be seen from the above matrix multiplication formula, the complexity of the operation to update the RGB feature map is O(C′×N 2 ), for 3D object detection, especially for scenes such as unmanned driving, the resolution of the input image or video frame is usually large, so it takes too long to calculate the self-attention matrix A(X), which is not conducive to application scenarios with real-time processing requirements. In the process of constructing the fully connected graph above, we convert the feature matrix r of each spatial position i Treat it as a node and search for the same node as r in the entire space. i The nodes associated with r i There is a high degree of overlap in the associated nodes, so the cross-modal self-attention learning module in the embodiment of the present invention only selects the nodes related to r through a sampling mechanism. iThe nodes with the highest correlation among the associated nodes are used to calculate the self-attention matrix after removing a large number of redundant nodes. This can greatly improve the computational efficiency while ensuring the correlation in all spatial regions. The following is a detailed introduction to the cross-modal self-attention learning module including the sampling mechanism.

[0082] For any node i in the deep feature map, the representative features of S number of samples in all nodes related to i are:

[0083]

[0084] Where s(n) is the feature vector obtained by sampling, and its dimension is C′, is a sampling function. In this way, the cross-modal self-attention learning module in the embodiment of the present invention can be expressed as:

[0085]

[0086] Where n is the sampled node related to i, δ is the softmax function, d θ (i) = W θ d(i),s φ (i) = W φ s(i),s g (i) = W g s(i). and They are the transformation matrices of the three linear transformations. By adding a sampling module, we can reduce the number of nodes from N to S when calculating the self-attention matrix:

[0087]

[0088] And S<<N, which can greatly reduce the computational complexity. For example, for a feature map with a spatial size of 80×80, N is 6400, and in the embodiment of the present invention, the number of selected sampling points is 9.

[0089] The present invention uses the idea of ​​deformable convolution to dynamically select sampling points by estimating the offset. Specifically, for a certain position p in the feature map, the sampling function It can be expressed as:

[0090]

[0091] Where Δp n is the offset obtained by regression. Since the result of the convolution operation usually contains a decimal part, and the coordinate value of the sampling point is only an integer, it is necessary to obtain the integer coordinate value through bilinear interpolation:

[0092]

[0093] where p s =p+Δpn, t is the four adjacent points with integer coordinate values ​​of the calculated sampling point, and K is the bilinear interpolation kernel function.

[0094] In practical applications, for each node in the RGB feature map, we can obtain its offset through linear transformation, and its transformation matrix is The output offset dimension is 2S, which is the offset of the horizontal and vertical coordinates. After bilinear interpolation, the S most representative nodes for each node can be obtained.

[0095] After obtaining the most representative sampling nodes through the deep feature map and calculating the self-attention matrix, the RGB feature map can be updated. In the cross-modal self-attention learning module of the embodiment of the present invention, we use the structure of the residual network to update the RGB feature map, which can be specifically expressed as:

[0096]

[0097] in, The RGB feature in the above formula (7), W y is the linear transformation matrix, is the residual obtained by learning, r i is the original input RGB feature, y i is the final updated RGB feature. This cross-modal self-attention learning module based on the residual network structure can be embedded in any neural network model.

[0098] From the above description, it can be seen that when constructing a single-level cross-modal self-attention learning module, a total of 5 linear transformation matrices are required, namely W in formula (7): φ , W φ With W g , W in formula (11) y , and the linear transformation matrix W used to generate the sampling points s To further reduce the number of parameters, we construct the cross-modal self-attention learning module as a bottleneck structure, that is, W in formula (7) θ , W φ With W g Fusion into a linear transformation matrix W is used to obtain d θ ,s φ With s g In this way, only three linear transformation matrices are needed to construct a single-level cross-modal self-attention learning module. All linear transformations are implemented through 1×1 convolutions, and batch normalization operations need to be added.

[0099] like Figure 4 As shown, the cross-modal self-attention learning module in the embodiment of the present invention can obtain a self-attention matrix containing structural information through multi-level deep feature map learning, and update the RGB feature map, so it is necessary to fuse the multi-level information in the end. The fusion operation can be specifically expressed as:

[0100]

[0101] where j lists all the depth levels, is the linear transformation matrix of the corresponding level, is the updated RGB feature of the corresponding level, which can be calculated by formula (7).

[0102] It is worth noting that in order to further reduce the computational complexity of the embodiment of the present invention, when calculating the self-attention matrix, the feature maps can also be grouped at the spatial and dimensional levels. At the spatial level, for a feature map with a dimension of C×H×W, the feature map can be divided into multiple regions, each region contains multiple feature vectors with a dimension of C×1, and a pooling operation is performed on each region to treat a region as a node, so that matrix operations can be performed on all features in a region, thereby greatly reducing the computational complexity. Similarly, at the dimensional level, all feature channels can be evenly divided into groups, and the feature map dimension of each group is C′×H×W, where C′=C / G. First, the features of each group are calculated, and then the features of all groups are cascaded together to obtain the final feature.

[0103] In summary, the present invention innovatively combines the deep structure information obtained from the depth map with the appearance information obtained from the RGB map through a cross-modal self-attention mechanism to achieve accurate detection results, rather than simply fusing the above two types of information. When obtaining deep structure information, the present invention can consider the correlation between different positions within the global scene range, rather than being limited to the neighborhood range. This is mainly due to the characteristics of the self-attention learning mechanism and the way of learning through multi-level features. In addition, when the present invention obtains the correlation between different positions within the global scene range, only a single operation is performed without iteration, so that the category, position and posture target detection of three-dimensional objects in two-dimensional RGB images can be effectively performed.

[0104] When the cross-modal self-attention mechanism in the present invention obtains the correlation between different positions, it only calculates the self-attention matrix for the positions with high correlation, which can avoid calculating the self-attention matrix between a large number of redundant positions, while ensuring the effect and reducing the computational complexity. In addition, when calculating the self-attention matrix, the deep features can be grouped at the dimensional and spatial levels to further reduce the computational complexity.

[0105] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0106] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention or certain parts of the embodiments.

[0107] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0108] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A three-dimensional object detection method based on a multi-level cross-modal self-attention mechanism, characterized in that: include: Use RGB image data to construct training set and test set data; Build a 3D object detection model, which includes an RGB backbone network, a deep backbone network, a classifier, and a regressor; The three-dimensional object detection model is trained using the training set and the test set data, and the training effect of the three-dimensional object detection model is verified using the test set. The RGB backbone network and the deep backbone network respectively obtain RGB features and depth features, and the RGB features and the depth features are input into the cross-modal self-attention learning module, and the RGB features are updated. The updated RGB features are used to learn the classifier and the regressor to obtain a trained three-dimensional object detection model; Using the classifier and regressor in the trained three-dimensional object detection model to detect the category, position and posture of the three-dimensional object in the two-dimensional RGB image to be identified; The method of using the training set and the test set data to train the three-dimensional object detection model, the RGB backbone network and the deep backbone network respectively obtain RGB features and depth features, input the RGB features and the depth features into the cross-modal self-attention learning module, update the RGB features, and use the updated RGB features to learn the classifier and the regressor to obtain the trained three-dimensional object detection model, including: Step S3-1: Initialize the parameters of the convolutional layer, the fully connected layer and the normalization layer included in the RGB backbone network, the deep backbone network, the classifier and the regressor in the three-dimensional object detection model; Step S3-2: Setting relevant training parameters of the stochastic gradient descent algorithm, the relevant training parameters including learning rate, impulse, batch size and number of iterations; Step S3-3: For any iterative batch, all RGB images and depth images are input into the RGB backbone network and the deep backbone network respectively to obtain multi-level RGB features and depth features, a cross-modal self-attention learning module is constructed, the RGB features and depth features are input into the cross-modal self-attention learning module, a self-attention matrix based on depth information is learned, the RGB features are updated through the self-attention matrix, the updated RGB features are used to learn a classifier and a regressor, and the classifier and the regressor are used for target detection of three-dimensional objects in two-dimensional RGB images. The objective function value is obtained by calculating the error between the network estimated value and the actual labeled value. The three objective function values ​​are calculated using formulas (1), (2) and (3) respectively: Where s in formula (1) i With p i are the category label and estimated probability of the i-th target respectively. Compared with the formula (3) They represent the 2D estimated box and 3D estimated box of the i-th target, gt represents the actual annotation value, and N represents the total number of targets; Step S3-4: Add the three objective function values ​​to obtain a total objective function value, and respectively calculate partial derivatives of all parameters in the three-dimensional object detection model, and update the parameters by stochastic gradient descent method; Step S3-5: Repeat steps S3-3 and 3-4 to continuously update the parameters of the three-dimensional object detection model until convergence, and output the parameters of the trained three-dimensional object detection model; The step of inputting the RGB features and the depth features into the cross-modal self-attention learning module, updating the RGB features, and using the updated RGB features to learn the classifier and the regressor to obtain a trained three-dimensional object detection model includes: For any two-dimensional RGB feature map R and two-dimensional depth feature map D, assuming that their dimensions are C×H×W, where C, H, and W are dimension, height, and width respectively, the two-dimensional RGB feature map R and the two-dimensional depth feature map D are represented as a set of N C-dimensional features: R = [r1, r2, …, r N ] Τ and D=[d1,d2,…,d N ] Τ , where N = H × W; For the input feature map R, a fully connected graph is constructed, where each feature r i As a node, the edge (r i ,r j ) represents the node r i With r j The relationship between them is obtained by learning the edge through the two-dimensional depth feature map D, and the current two-dimensional RGB feature map R is updated, which is specifically expressed as: in is the normalization parameter, δ is the softmax function, j is all the positions related to i, For the updated RGB features, write the above formula in the form of matrix multiplication: in is the self-attention matrix, D θ ,D φ With R g The dimensions are all N×C′; The feature matrix r of each spatial position i Treat it as a node and search for the same node as r in the entire space. i Associated nodes, for any node i in the deep feature map, the representative features of the number S sampled in all nodes associated with i are: Where s(n) is the feature vector obtained by sampling, and its dimension is C ′ , is a sampling function, and the cross-modal self-attention learning module is expressed as: Where n is the sampled node related to i, δ is the softmax function, d θ (i) = W θ d(i),s φ (i) = W φ s(i),s g (i) = W g s(i), and They are the transformation matrices of the three linear transformations respectively.

2. The method according to claim 1, characterized in that: The method of using RGB image data to construct training set and test set data includes: Collect RGB images, divide the RGB images into training set and test set in a 1:1 ratio, normalize the image data in the training set and test set, obtain the two-dimensional depth image of the training set image through the depth estimation algorithm, annotate the category of objects in the training set image, annotate the coordinates of the two-dimensional detection box of the image, and annotate the center position, size and rotation angle of the three-dimensional detection box.

3. The method according to claim 2, characterized in that The RGB backbone network, deep backbone network, classifier and regressor in the three-dimensional object detection model all include convolutional layers, fully connected layers and normalization layers. The structures of the RGB backbone network and the deep backbone network are consistent, and both include 4 convolutional modules.

Images