3D target detection method and device based on space and channel attention mechanism
By introducing a 3D object detection method with spatial and channel attention mechanisms in monocular RGB image object detection, combined with parallel spatial and channel attention mechanism networks, the trade-off problem between detection accuracy and speed in the prior art is solved, and the realization of the realization of the trade-off problem is achieved by Efficient three-dimensional object detection, suitable for real-time autonomous driving applications.
Patent Information
- Application Number
- CN202411395430.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-08
- Publication Date
- 2025-01-10
AI Technical Summary
现有的单目RGB图像目标检测方法在精度和速度之间存在权衡,单阶段方法速度快但精度较低,而双阶段方法虽然精度高但速度较慢,难以在实时检测中达到平衡。
Using a 3D object detection method based on spatial and channel attention mechanism, RGB images are processed through residual networks and two-dimensional detection networks, combined with parallel spatial attention mechanism networks and channel attention mechanism networks, feature maps are extracted and three-dimensional detection boxes are predicted. Its categories are implemented to achieve joint training to improve detection accuracy.
It improves the three-dimensional object detection accuracy of RGB images, achieves an improvement in maintaining speed, and is suitable for real-time autonomous driving applications.
Smart Images
Figure CN119296096A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of autonomous driving technology, and in particular to a 3D target detection method and device based on spatial and channel attention mechanisms. Background Art
[0002] There are two main methods for target detection in monocular RGB images. The first is a single-stage detection method, which directly regresses the detection box coordinates and target category through the network. It is faster but less accurate. The second is a two-stage target detection method, which improves the accuracy by means of coarse regression followed by fine tuning, but is slower. Summary of the invention
[0003] In view of this, the present application provides a 3D target detection method and device based on spatial and channel attention mechanism to solve the above technical problems.
[0004] In a first aspect, an embodiment of the present application provides a 3D object detection method based on a spatial and channel attention mechanism, comprising:
[0005] Get the RGB image captured by the monocular camera on the vehicle;
[0006] The RGB image is processed using the residual network to obtain a feature map; the RGB image is processed using the two-dimensional detection network to obtain N two-dimensional first detection frames;
[0007] Process the feature map and N two-dimensional first detection frames to obtain a first tensor;
[0008] The feature map is processed using a semantic extraction model with a spatial attention mechanism and a channel attention mechanism to obtain a second tensor;
[0009] The first tensor and the second tensor are processed to obtain a three-dimensional detection frame of the RGB image and its corresponding category.
[0010] In a possible implementation, the feature map and N two-dimensional first detection boxes are processed to obtain a first tensor, including:
[0011] Divide each parameter of the N first detection frames by the downsampling multiple in the residual network to obtain N second detection frames;
[0012] Extracting N feature regions corresponding to the N second detection frames from the feature map;
[0013] The first average pooling layer is used to process N feature regions to obtain an N-dimensional tensor X Avg ;
[0014] Use the first fully connected layer to process the N-dimensional tensor X AvgProcessing is performed to obtain the first tensor X of dimension C FC .
[0015] In one possible implementation, the semantic extraction model includes a parallel spatial attention mechanism network, a first convolutional layer and a channel attention mechanism network, an adder, and a second average pooling layer;
[0016] The feature map is processed using a semantic extraction model with a spatial attention mechanism and a channel attention mechanism to obtain a second tensor; including:
[0017] Use the spatial attention mechanism network to process the feature map and obtain the first intermediate tensor;
[0018] The feature map is processed using the first convolutional layer to obtain a second intermediate tensor;
[0019] The feature map is processed using the channel attention mechanism network to obtain the third intermediate tensor;
[0020] Calculate the sum of the first intermediate tensor, the second intermediate tensor and the third intermediate tensor by using an adder to obtain a fourth intermediate tensor;
[0021] The fourth intermediate tensor is processed by a second average pooling layer to obtain a second tensor.
[0022] In a possible implementation, the feature map is processed using a spatial attention mechanism network to obtain a first intermediate tensor; including:
[0023] Use the second convolutional layer to transform the three-dimensional feature map X fea Processing is performed to obtain the first intermediate feature map Y1, and the three-dimensional first intermediate feature map Y1 is processed by the first reshaping layer to obtain a two-dimensional tensor
[0024] Use the third convolutional layer to transform the three-dimensional feature map X fea Processing is performed to obtain a second intermediate feature map Y2, and the three-dimensional second intermediate feature map Y2 is processed by the second reshaping layer to obtain a two-dimensional tensor
[0025] Use the fourth convolutional layer to transform the three-dimensional feature map X fea Processing is performed to obtain a third intermediate feature map Y3, and the three-dimensional third intermediate feature map Y3 is processed by the third reshaping layer to obtain a two-dimensional tensor
[0026] For a two-dimensional tensor Transpose to get a two-dimensional tensor The two-dimensional tensor With a two-dimensional tensor Perform matrix multiplication to obtain matrix Y1,2 ;
[0027] For the matrix Y 1,2 After normalization, we get the matrix
[0028] The two-dimensional tensor With the matrix Perform matrix multiplication to obtain matrix Y 1,2,3 ;
[0029] The matrix Y 1,2,3 Reshape into a three-dimensional feature map
[0030] The three-dimensional feature map With the three-dimensional feature map X fea Add them together to get the first intermediate tensor.
[0031] In a possible implementation, the feature map is processed using a channel attention mechanism network to obtain a third intermediate tensor; including:
[0032] Use the fourth reshape layer to transform the three-dimensional feature map X fea Processing to obtain a two-dimensional tensor
[0033] Use the fifth reshape layer to transform the three-dimensional feature map X fea Processing to obtain a two-dimensional tensor
[0034] Use the sixth reshape layer to transform the three-dimensional feature map X fea Processing to obtain a two-dimensional tensor
[0035] For a two-dimensional tensor Transpose to get a two-dimensional tensor The two-dimensional tensor With a two-dimensional tensor Perform matrix multiplication to obtain matrix Y 6,5 ;
[0036] For the matrix Y 6,5 After normalization, we get the matrix
[0037] The two-dimensional tensor With the matrix Perform matrix multiplication to obtain matrix Y 6,5,4 ;
[0038] The matrix Y 6,5,4 Reshape into a three-dimensional feature map
[0039] The three-dimensional feature map With the three-dimensional feature map X fea Add them together to get the second intermediate tensor.
[0040] In a possible implementation, the first tensor and the second tensor are processed to obtain a three-dimensional detection frame of the RGB image and its corresponding category; including:
[0041] Add the first tensor and the second tensor to get the third tensor;
[0042] The third tensor is processed by the second fully connected layer to obtain a three-dimensional detection frame;
[0043] The third tensor is processed by the third fully connected layer to obtain a one-dimensional tensor; the one-dimensional tensor is processed by the Softmax layer to obtain a one-dimensional category tensor.
[0044] In a possible implementation, the method further includes: a step of jointly training the residual network, the first average pooling layer, the first fully connected layer, the semantic extraction model, the second fully connected layer and the third fully connected layer.
[0045] In a possible implementation, the step of jointly training the residual network, the first average pooling layer, the first fully connected layer, the semantic extraction model, the second fully connected layer, and the third fully connected layer includes:
[0046] Establishing a training set, wherein the training set includes multiple groups of aligned RGB image samples and 3D point cloud data with labeled true detection boxes and categories;
[0047] The residual network is used to process the RGB image samples to obtain feature map samples; the 2D detection network is used to process the RGB image samples to obtain N 2D prediction boxes;
[0048] Process the feature map samples and N two-dimensional prediction boxes to obtain the first tensor sample;
[0049] The feature map samples are processed using a semantic extraction model with a spatial attention mechanism and a channel attention mechanism to obtain a second tensor sample;
[0050] Processing the first tensor sample and the second tensor sample to obtain a three-dimensional prediction box of the RGB image sample and its corresponding category;
[0051] Calculate the detection box loss value using the 3D prediction box of the RGB image sample and the 3D real detection box of the 3D point cloud data in the same group;
[0052] The category loss value is calculated using the cross entropy function using the category of the 3D prediction box of the RGB image sample and the category of the 3D real detection box of the 3D point cloud data in the same group;
[0053] Determine the total loss value based on the detection box loss value and the category loss value;
[0054] Based on the total loss value, update the parameters of the residual network, the first average pooling layer, the first fully connected layer, the semantic extraction model, the second fully connected layer, and the third fully connected layer.
[0055] In a second aspect, an embodiment of the present application provides a 3D object detection device based on a spatial and channel attention mechanism, comprising:
[0056] An acquisition unit, used to acquire an RGB image captured by a monocular camera on the vehicle;
[0057] A first processing unit is used to process the RGB image using a residual network to obtain a feature map; and to process the RGB image using a two-dimensional detection network to obtain N two-dimensional first detection frames;
[0058] A second processing unit is used to process the feature map and the N two-dimensional first detection frames to obtain a first tensor;
[0059] A third processing unit is used to process the feature map using a semantic extraction model with a spatial attention mechanism and a channel attention mechanism to obtain a second tensor;
[0060] A detection unit is used to process the first tensor and the second tensor to obtain a three-dimensional detection frame of the RGB image and its corresponding category
[0061] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of the embodiment of the present application when executing the computer program.
[0062] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method of the embodiment of the present application is implemented.
[0063] This application improves the accuracy of 3D object detection in RGB images. BRIEF DESCRIPTION OF THE DRAWINGS
[0064] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0065] Figure 1A flowchart of a 3D object detection method based on a spatial and channel attention mechanism provided in an embodiment of the present application;
[0066] Figure 2 A schematic diagram of a model of a 3D target detection method provided in an embodiment of the present application;
[0067] Figure 3 A schematic diagram of a spatial attention mechanism network provided for an embodiment of the present application;
[0068] Figure 4 A schematic diagram of a channel attention mechanism network provided for an embodiment of the present application;
[0069] Figure 5 A functional structure diagram of a 3D object detection device based on a spatial and channel attention mechanism provided in an embodiment of the present application;
[0070] Figure 6 A functional structure diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0071] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.
[0072] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.
[0073] First, the design concept of the embodiments of the present application is briefly introduced.
[0074] At present, 3D target detection methods mainly include: 3D target detection based on point cloud, 3D target detection based on camera, 3D target detection based on multimodality, etc. Among them, 3D target detection based on monocular camera is deeply favored by researchers due to its small computational complexity and wide applicability. However, since monocular camera cannot provide depth and point cloud data, the network lacks three-dimensional information, which brings a lot of difficulties to the research. In contrast, the accuracy of point cloud data obtained by LiDAR has been greatly improved compared with RGB images. However, due to the sparsity and hugeness of point cloud data, it is challenging to achieve target detection tasks through point cloud data. At present, there are two mainstream methods for processing detection tasks through point cloud data. One is to directly pass point cloud data into neural network for learning through Poi ntnet framework, and the other is to voxelize point cloud data to realize end-to-end point cloud target detection model. Although these two methods can achieve high-precision detection, considering their huge computational complexity, they are still not ideal for real-time target detection tasks. Taking into account the advantages and disadvantages of both, this application aligns point cloud data and RGB image data, and the network only inputs the RGB image obtained by the monocular camera to achieve low-computation real-time detection.
[0075] At present, there are two methods for detecting targets in RGB images. One is the single-stage detection algorithm, such as the Yolo algorithm and the SSD algorithm, which directly regress the detection box coordinates and target categories through the network. The speed is faster but the accuracy is slightly lower. The other is the two-stage target detection algorithm, such as the RCNN and Fast RCNN algorithms. The Faster RCNN algorithm improves the accuracy by means of coarse regression followed by fine tuning, but the speed is slower.
[0076] In order to improve the ability of single-stage object detection algorithms to extract feature areas, this application introduces spatial and channel attention mechanisms, which can find the locations of parts that convey high-level semantic information in RGB images and meaningful features in the images, thereby improving the three-dimensional object detection accuracy of RGB images.
[0077] After introducing the application scenarios and design concepts of the embodiments of the present application, the technical solutions provided by the embodiments of the present application are described below.
[0078] like Figure 1 As shown, the present application implements a 3D target detection method based on a spatial and channel attention mechanism. With the help of the spatial and channel attention mechanism, the 3D detection frame is predicted for the RGB image collected by the monocular camera. The method includes the following steps:
[0079] Step 101: Obtain an RGB image captured by a monocular camera on the vehicle;
[0080] Among them, the RGB image is represented as Among them, H0, W0 and C0 represent the height, width and number of channels of the RGB image respectively;
[0081] Step 102: Process the RGB image using a residual network to obtain a feature map; process the RGB image using a two-dimensional detection network to obtain N two-dimensional detection frames;
[0082] In this embodiment, the residual network adopts the Restnet18 network, that is:
[0083] X fea =Res18(X input )
[0084] Among them, X fea ∈R H×W×C represents the feature map, H, W, and C represent the height, width, and number of channels of the feature map, respectively.
[0085] In this embodiment, the two-dimensional detection network is YOLOV5; N 2D rectangular detection boxes are represented as: The nth detection box S n =(x n ,y n ,l n ,h n ), where x n ,y n ,l n ,h n Represents the detection box S n The center point coordinates and horizontal and vertical offsets.
[0086] Step 103: Process the feature map and N two-dimensional first detection frames to obtain a first tensor;
[0087] Step 104: Process the feature map using a semantic extraction model with a spatial attention mechanism and a channel attention mechanism to obtain a second tensor;
[0088] Step 105: Process the first tensor and the second tensor to obtain a three-dimensional detection frame of the RGB image and its corresponding category.
[0089] In some embodiments, Figure 2 As shown, step 103 specifically includes:
[0090] The N first detection boxes Divided by the downsampling multiple α in the residual network, we get N features in the feature map X fea The second reduced-size detection box in
[0091] From the feature map X feaExtract N feature regions corresponding to N second detection frames in and Dimension the height and width of the feature area respectively;
[0092] Use the first average pooling layer to transform the feature map X fea The N feature regions are processed to obtain an N-dimensional tensor
[0093] Use the first fully connected layer to process the N-dimensional tensor X Avg Process and get the first C-dimensional tensor
[0094]
[0095] In some embodiments, Figure 2 As shown, the semantic extraction model includes a parallel spatial attention mechanism network, a first convolutional layer and a channel attention mechanism network, an adder, and a second average pooling layer;
[0096] The feature map is processed using a semantic extraction model with a spatial attention mechanism and a channel attention mechanism to obtain a second tensor; including:
[0097] The feature map is processed using the spatial attention mechanism network to obtain a size of R H×W×C The first intermediate tensor of ;
[0098] The feature map is processed using the first convolutional layer to obtain a size of R H×W×C The second intermediate tensor of ;
[0099] The feature map is processed using the channel attention mechanism network to obtain a size of R H×W×C The third intermediate tensor of ;
[0100] Calculate the sum of the first intermediate tensor, the second intermediate tensor and the third intermediate tensor by using an adder to obtain a fourth intermediate tensor;
[0101] The fourth intermediate tensor is processed by the second average pooling layer to obtain a second tensor of C dimension.
[0102] like Figure 3 As shown, in some embodiments, the feature map is processed using a spatial attention mechanism network to obtain a first intermediate tensor; including:
[0103] Use the second convolutional layer to transform the three-dimensional feature map X fea Processing is performed to obtain a size of R H×W×C The first intermediate feature map Y1 of the three-dimensional image is processed by the first reshaping layer to obtain a two-dimensional tensor Among them, the reshape layer does not change the parameters, but only changes the arrangement of the parameters, converting the original Y1∈R H×W×C From a three-dimensional tensor to a two-dimensional tensor Where M = H * W;
[0104] Use the third convolutional layer to transform the three-dimensional feature map X fea Processing is performed to obtain a size of R H×W×C The second intermediate feature map Y2 of the three-dimensional image is processed by the second reshaping layer to obtain a two-dimensional tensor
[0105] Use the fourth convolutional layer to transform the three-dimensional feature map X fea Processing is performed to obtain a size of R H×W×C The third intermediate feature map Y3 of the three-dimensional image is processed by the third reshaping layer to obtain a two-dimensional tensor
[0106] For a two-dimensional tensor Transpose to get a two-dimensional tensor The two-dimensional tensor With a two-dimensional tensor Perform matrix multiplication to obtain matrix Y 1,2 :
[0107]
[0108] For the matrix Y 1,2 Perform Softmax normalization to obtain the matrix
[0109] The two-dimensional tensor With the matrix Perform matrix multiplication to obtain matrix Y 1,2,3 ;
[0110] The matrix Y 1,2,3 Reshape into a three-dimensional feature map
[0111] The three-dimensional feature map With the three-dimensional feature map X fea Add them together to get the first intermediate tensor Y spa :
[0112]
[0113] like Figure 4 As shown, in some embodiments, the feature map is processed using a channel attention mechanism network to obtain a third intermediate tensor; including:
[0114] Use the fourth reshape layer to transform the three-dimensional feature map X fea Processing to obtain a two-dimensional tensor
[0115] Use the fifth reshape layer to transform the three-dimensional feature map X fea Processing to obtain a two-dimensional tensor
[0116] Use the sixth reshape layer to transform the three-dimensional feature map X fea Processing to obtain a two-dimensional tensor Where M = H * W;
[0117] For a two-dimensional tensor Transpose to get a two-dimensional tensor The two-dimensional tensor With a two-dimensional tensor Perform matrix multiplication to obtain matrix Y 6,5 :
[0118]
[0119] For the matrix Y 6,5 Perform Softmax normalization to obtain the matrix
[0120] The two-dimensional tensor With the matrix Perform matrix multiplication to obtain matrix Y 6,5,4 ;
[0121] The matrix Y 6,5,4 Reshape into a three-dimensional feature map
[0122] The three-dimensional feature map With the three-dimensional feature map X fea Add them together to get the second intermediate tensor Y cha :
[0123]
[0124] In some embodiments, Figure 2 As shown, step 105 specifically includes:
[0125] The first tensor X FC and the second tensor Y Arg Add them together to get the third tensor Z;
[0126] The third tensor Z is processed by the second fully connected layer to obtain each 3D detection box Z box , where Z box=(x,y,z,l,h,w,θ),Z box The seven elements in represent the center coordinates, height, width, length and offset angle of the 3D detection box respectively;
[0127] The third tensor Z is processed by the third fully connected layer to obtain a one-dimensional tensor After the Softmax layer, we get a one-dimensional category tensor Z CLS =(c1,c2,······,c k ), Z CLS The k elements in represent the k categories of 3D object detection.
[0128] In some embodiments, the method further includes: a step of jointly training the residual network, the first average pooling layer, the first fully connected layer, the semantic extraction model, the second fully connected layer and the third fully connected layer.
[0129] In some embodiments, the step of jointly training the residual network, the first average pooling layer, the first fully connected layer, the semantic extraction model, the second fully connected layer, and the third fully connected layer includes:
[0130] Establishing a training set, wherein the training set includes multiple groups of aligned RGB image samples and 3D point cloud data with labeled true detection boxes and categories;
[0131] The residual network is used to process the RGB image samples to obtain feature map samples; the 2D detection network is used to process the RGB image samples to obtain N 2D prediction boxes;
[0132] Process the feature map samples and N two-dimensional prediction boxes to obtain the first tensor sample;
[0133] The feature map samples are processed using a semantic extraction model with a spatial attention mechanism and a channel attention mechanism to obtain a second tensor sample;
[0134] Processing the first tensor sample and the second tensor sample to obtain a three-dimensional prediction box of the RGB image sample and its corresponding category;
[0135] Calculate the detection box loss value using the 3D prediction box of the RGB image sample and the 3D real detection box of the 3D point cloud data in the same group;
[0136] The category loss value is calculated using the cross entropy function using the category of the 3D prediction box of the RGB image sample and the category of the 3D real detection box of the 3D point cloud data in the same group;
[0137] Determine the total loss value based on the detection box loss value and the category loss value;
[0138] Based on the total loss value, update the parameters of the residual network, the first average pooling layer, the first fully connected layer, the semantic extraction model, the second fully connected layer, and the third fully connected layer.
[0139] Among them, for the detection box Z box The loss calculation of position is: s represents the true value, a represents the predicted value, then the position loss L loc , size loss L siz , deviation angle loss L θ The calculation formula is as follows:
[0140]
[0141] L θ =|θ s -θ a |
[0142] Detection box Z box The loss of L box The calculation formula is as follows:
[0143] L box =L loc +L siz +L θ
[0144] For the predicted category Z CLS The loss calculation uses the cross entropy function to calculate the loss and predict the category loss L CLS The calculation is as follows:
[0145]
[0146] Based on the same inventive idea, the embodiment of the present application provides a 3D target detection device based on space and channel attention mechanism, see Figure 5 As shown, the 3D object detection device 200 based on the space and channel attention mechanism provided in the embodiment of the present application at least includes:
[0147] An acquisition unit 201 is used to acquire an RGB image captured by a monocular camera on the vehicle;
[0148] The first processing unit 202 is used to process the RGB image using a residual network to obtain a feature map; and process the RGB image using a two-dimensional detection network to obtain N two-dimensional first detection frames;
[0149] A second processing unit 203 is used to process the feature map and the N two-dimensional first detection frames to obtain a first tensor;
[0150] A third processing unit 204 is used to process the feature map using a semantic extraction model with a spatial attention mechanism and a channel attention mechanism to obtain a second tensor;
[0151] The detection unit 205 is used to process the first tensor and the second tensor to obtain a three-dimensional detection frame of the RGB image and its corresponding category.
[0152] It should be noted that the principle of solving the technical problem by the 3D object detection device 200 based on the spatial and channel attention mechanism provided in the embodiment of the present application is similar to the method provided in the embodiment of the present application. Therefore, the implementation of the 3D object detection method 200 based on the spatial and channel attention mechanism provided in the embodiment of the present application can refer to the implementation of the method provided in the embodiment of the present application, and the repeated parts will not be repeated.
[0153] Based on the above embodiments, the present application also provides an electronic device, referring to Figure 6 As shown, the electronic device 300 provided in the embodiment of the present application includes at least: a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program, the 3D target detection method based on the spatial and channel attention mechanism provided in the embodiment of the present application is implemented.
[0154] The electronic device 300 provided in the embodiment of the present application may further include a bus 303 connecting different components (including the processor 301 and the memory 302). The bus 303 represents one or more of several types of bus structures, including a memory bus, a peripheral bus, a local bus, and the like.
[0155] The memory 302 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 3021 and / or a cache memory 3022 , and may further include a read-only memory (ROM) 3023 .
[0156] The memory 302 may also include a program tool 3025 having a set (at least one) of program modules 3024, the program modules 3024 including but not limited to: an operating subsystem, one or more applications, other program modules and program data, each of which or some combination may include the implementation of a network environment.
[0157] The electronic device 300 may also communicate with one or more external devices 304 (e.g., keyboards, remote controls, etc.), and may also communicate with one or more devices that enable a user to interact with the electronic device 300 (e.g., mobile phones, computers, etc.), and / or communicate with any device that enables the electronic device 300 to communicate with one or more other electronic devices 300 (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 305. Furthermore, the electronic device 300 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 306. Figure 6 As shown, the network adapter 306 communicates with other modules of the electronic device 300 via the bus 303. It should be understood that although Figure 6 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 300, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, disk arrays (Redundant Arrays of Independent Disks, RAID) subsystems, tape drives, and data backup storage subsystems.
[0158] It should be noted that Figure 6 The electronic device 300 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0159] The embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions, and when the computer instructions are executed by the processor, the 3D object detection method based on the spatial and channel attention mechanism provided in the embodiment of the present application is implemented. Specifically, the executable program can be built-in or installed in the electronic device 300, so that the electronic device 300 can implement the 3D object detection method based on the spatial and channel attention mechanism provided in the embodiment of the present application by executing the built-in or installed executable program.
[0160] The 3D target detection method based on the spatial and channel attention mechanism provided in the embodiment of the present application can also be implemented as a program product, which includes a program code. When the program product can be run on an electronic device 300, the program code is used to enable the electronic device 300 to execute the 3D target detection method based on the spatial and channel attention mechanism provided in the embodiment of the present application.
[0161] The program product provided in the embodiments of the present application may adopt any combination of one or more readable media, wherein the readable medium may be a readable signal medium or a readable storage medium, and the readable storage medium may be but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination of the above. Specifically, more specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, RAM, ROM, an erasable programmable read-only memory (EPROM), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0162] The program product provided in the embodiment of the present application may adopt a CD-ROM and include program code, and may also be run on a computing device. However, the program product provided in the embodiment of the present application is not limited thereto. In the embodiment of the present application, the readable storage medium may be any tangible medium containing or storing a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device.
[0163] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.
[0164] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.
[0165] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application is described in detail with reference to the embodiments, a person skilled in the art should understand that any modification or equivalent replacement of the technical solution of the present application does not depart from the spirit and scope of the technical solution of the present application and should be included in the scope of the claims of the present application.
Claims
1. A 3D object detection method based on spatial and channel attention mechanism, characterized in that: include: Get the RGB image captured by the monocular camera on the vehicle; Use the residual network to process the RGB image and obtain the feature map; The RGB image is processed using a two-dimensional detection network to obtain N two-dimensional first detection frames; Process the feature map and N two-dimensional first detection frames to obtain a first tensor; The feature map is processed using a semantic extraction model with a spatial attention mechanism and a channel attention mechanism to obtain a second tensor; The first tensor and the second tensor are processed to obtain a three-dimensional detection frame of the RGB image and its corresponding category.
2. The method according to claim 1, characterized in that The feature map and N two-dimensional first detection frames are processed to obtain a first tensor, including: Divide each parameter of the N first detection frames by the downsampling multiple in the residual network to obtain N second detection frames; Extracting N feature regions corresponding to the N second detection frames from the feature map; The first average pooling layer is used to process N feature regions to obtain an N-dimensional tensor X Avg ; Use the first fully connected layer to process the N-dimensional tensor X Avg Processing is performed to obtain the first tensor X of dimension C FC .
3. The method according to claim 2, characterized in that The semantic extraction model includes a parallel spatial attention mechanism network, a first convolutional layer and a channel attention mechanism network, an adder and a second average pooling layer; The feature map is processed using a semantic extraction model with a spatial attention mechanism and a channel attention mechanism to obtain a second tensor; including: Use the spatial attention mechanism network to process the feature map and obtain the first intermediate tensor; The feature map is processed using the first convolutional layer to obtain a second intermediate tensor; The feature map is processed using the channel attention mechanism network to obtain the third intermediate tensor; Calculate the sum of the first intermediate tensor, the second intermediate tensor and the third intermediate tensor by using an adder to obtain a fourth intermediate tensor; The fourth intermediate tensor is processed by a second average pooling layer to obtain a second tensor.
4. The method according to claim 3, characterized in that The feature map is processed using the spatial attention mechanism network to obtain the first intermediate tensor; including: Use the second convolutional layer to transform the three-dimensional feature map X fea Processing is performed to obtain the first intermediate feature map Y1, and the three-dimensional first intermediate feature map Y1 is processed by the first reshaping layer to obtain a two-dimensional tensor Use the third convolutional layer to transform the three-dimensional feature map X fea Processing is performed to obtain a second intermediate feature map Y2, and the three-dimensional second intermediate feature map Y2 is processed by the second reshaping layer to obtain a two-dimensional tensor Use the fourth convolutional layer to transform the three-dimensional feature map X fea Processing is performed to obtain a third intermediate feature map Y3, and the three-dimensional third intermediate feature map Y3 is processed by the third reshaping layer to obtain a two-dimensional tensor For a two-dimensional tensor Transpose to get a two-dimensional tensor The two-dimensional tensor With a two-dimensional tensor Perform matrix multiplication to obtain matrix Y 1,2 ; For the matrix Y 1,2 After normalization, we get the matrix The two-dimensional tensor With the matrix Perform matrix multiplication to obtain matrix Y 1,2,3 ; The matrix Y 1,2,3 Reshape into a three-dimensional feature map The three-dimensional feature map With the three-dimensional feature map X fea Add them together to get the first intermediate tensor.
5. The method according to claim 4, characterized in that The feature map is processed using the channel attention mechanism network to obtain the third intermediate tensor; including: Use the fourth reshape layer to transform the three-dimensional feature map X fea Processing to obtain a two-dimensional tensor Use the fifth reshape layer to transform the three-dimensional feature map X fea Processing to obtain a two-dimensional tensor Use the sixth reshape layer to transform the three-dimensional feature map X fea Processing to obtain a two-dimensional tensor For a two-dimensional tensor Transpose to get a two-dimensional tensor The two-dimensional tensor With a two-dimensional tensor Perform matrix multiplication to obtain matrix Y 6,5 ; For the matrix Y 6,5 After normalization, we get the matrix The two-dimensional tensor With the matrix Perform matrix multiplication to obtain matrix Y 6,5,4 ; The matrix Y 6,5,4 Reshape into a three-dimensional feature map The three-dimensional feature map With the three-dimensional feature map X fea Add them together to get the second intermediate tensor.
6. The method according to claim 5, characterized in that The first tensor and the second tensor are processed to obtain a three-dimensional detection frame of the RGB image and its corresponding category; including: Add the first tensor and the second tensor to get the third tensor; The third tensor is processed by the second fully connected layer to obtain a three-dimensional detection frame; The third tensor is processed by the third fully connected layer to obtain a one-dimensional tensor; the one-dimensional tensor is processed by the Softmax layer to obtain a one-dimensional category tensor.
7. The method according to claim 6, characterized in that The method also includes: a step of jointly training the residual network, the first average pooling layer, the first fully connected layer, the semantic extraction model, the second fully connected layer and the third fully connected layer.
8. The method according to claim 7, characterized in that The step of jointly training the residual network, the first average pooling layer, the first fully connected layer, the semantic extraction model, the second fully connected layer, and the third fully connected layer includes: Establishing a training set, wherein the training set includes multiple groups of aligned RGB image samples and 3D point cloud data with labeled true detection boxes and categories; The residual network is used to process the RGB image samples to obtain feature map samples; the 2D detection network is used to process the RGB image samples to obtain N 2D prediction boxes; Process the feature map samples and N two-dimensional prediction boxes to obtain the first tensor sample; The feature map samples are processed using a semantic extraction model with a spatial attention mechanism and a channel attention mechanism to obtain a second tensor sample; Processing the first tensor sample and the second tensor sample to obtain a three-dimensional prediction box of the RGB image sample and its corresponding category; Calculate the detection box loss value using the 3D prediction box of the RGB image sample and the 3D real detection box of the 3D point cloud data in the same group; The category loss value is calculated using the cross entropy function using the category of the 3D prediction box of the RGB image sample and the category of the 3D real detection box of the 3D point cloud data in the same group; Determine the total loss value based on the detection box loss value and the category loss value; Based on the total loss value, update the parameters of the residual network, the first average pooling layer, the first fully connected layer, the semantic extraction model, the second fully connected layer, and the third fully connected layer.
9. A 3D object detection device based on spatial and channel attention mechanism, characterized in that: include: An acquisition unit, used to acquire an RGB image captured by a monocular camera on the vehicle; A first processing unit is used to process the RGB image using a residual network to obtain a feature map; The RGB image is processed using a two-dimensional detection network to obtain N two-dimensional first detection frames; A second processing unit is used to process the feature map and the N two-dimensional first detection frames to obtain a first tensor; A third processing unit is used to process the feature map using a semantic extraction model with a spatial attention mechanism and a channel attention mechanism to obtain a second tensor; The detection unit is used to process the first tensor and the second tensor to obtain a three-dimensional detection frame of the RGB image and its corresponding category.
10. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 8 when executing the computer program.