Target detection method and system based on radar and vision fusion

Abstract

The invention discloses a target detection method and system based on radar and vision fusion. The method comprises the following steps: S1, acquiring image data and radar point cloud data; s2, processing the data to obtain 2D detection candidate frames of K visual targets and 3D detection candidate frames of N radar targets; s3, mapping each feature matrix to a certain preset dimension space through a dimension expansion network to obtain a dimension-expanded 2D feature vector and a dimension-expanded 3D feature vector; s4, performing coding operation on the 2D feature vector and the 3D feature vector after dimension expansion to obtain a 2D coding information tensor and a 3D coding information tensor; s5, splicing the 2D coding information tensor and the 3D coding information tensor to obtain an integral coding information tensor C; s6, performing decoding operation on the C to obtain a decoding information tensor T; and S7, performing MLP operation on the T to obtain a target detection result. The method has the advantages of high detection precision and the like.

Description

Technical Field

[0001] The present invention mainly relates to the field of radar and vision fusion technology, and specifically relates to a target detection method and system based on radar and vision fusion. Background Art

[0002] Target detection based on radar and vision fusion is a technology that uses 4D millimeter-wave radar and images to achieve target perception. This technology uses the 4D millimeter-wave radar's ability to resist environmental interference and can achieve all-weather perception. At the same time, with the help of image visualization and structuring capabilities, it can extract information such as target type. Compared with a single millimeter-wave radar or camera, it greatly enriches the dimension of target data and improves the accuracy of target detection.

[0003] At present, target detection based on radar and vision fusion is widely used in autonomous driving, security monitoring, environmental perception and target tracking. As target detection applications become more and more widespread, various target detection algorithms emerge in an endless stream. Traditional algorithms often use simple feature splicing or cascading, which may not be able to fully explore the complementary information of the two data, and the high-dimensional data in 4D millimeter-wave radar may not be fully utilized. Summary of the invention

[0004] In view of the technical problems existing in the prior art, the present invention provides a target detection method and system based on radar and vision fusion with high recognition accuracy.

[0005] In order to solve the above technical problems, the technical solution proposed by the present invention is:

[0006] A target detection method based on radar and vision fusion, comprising the steps of:

[0007] S1, obtain image data and radar point cloud data;

[0008] S2, sending the image data and radar point cloud data to the 2D detector and the 3D detector respectively, obtaining the 2D detection candidate boxes of K visual targets and the 3D detection candidate boxes of N radar targets, and representing them in the form of feature matrices;

[0009] S3, mapping the feature matrices of the 2D detection candidate box and the 3D detection candidate box to a preset dimensional space through a dimension expansion network to obtain a 2D feature vector and a 3D feature vector after dimension expansion;

[0010] S4, respectively performing encoding operations on the expanded 2D feature vector and the 3D feature vector to obtain a 2D encoding information tensor and a 3D encoding information tensor;

[0011] S5, concatenating the 2D coding information tensor and the 3D coding information tensor to obtain an overall coding information tensor C;

[0012] S6. Decoding the entire coded information tensor C to obtain a decoded information tensor T;

[0013] S7. Perform MLP operation on the decoded information tensor T to obtain the target detection result.

[0014] Preferably, the specific process of step S3 is:

[0015] For the feature matrix of the 2D detection candidate box, a fully connected neural network with k*m dimensions is passed to obtain a feature vector with K*m dimensions.

[0016] For the feature matrix of the 3D detection candidate box, a fully connected neural network with n*m dimensions is passed to obtain a feature vector with N*m dimensions.

[0017] The calculation formula of the fully connected neural network is: Tw+b; wherein T represents the feature matrix of the 2D or 3D detection candidate box, and w and b represent the parameter matrix.

[0018] Preferably, the specific process of step S4 is:

[0019] S401, input the 2D feature vector and the 3D feature vector into the multi-head attention mechanism network to obtain multi-head attention weight vectors M1 and M2 respectively;

[0020] S402, performing residual and normalization processes on the multi-head attention weight vectors M1 and M2, respectively, to obtain result vectors TK1 and TN1;

[0021] S403, input the result vectors TK1 and TN1 into the cross-head attention mechanism network respectively, and obtain the cross-head attention weight vectors M3 and M4 respectively;

[0022] S404, performing residual and normalization processes on the cross multi-head attention weight vectors M3 and M4, respectively, to obtain result vectors TK2 and TN2, respectively;

[0023] S405, respectively send TK2 and TN2 to the feedforward neural network Feed Forward, and then perform residual and normalization processes on the results to obtain the encoded information tensors TK and TN after an Encoder, with dimensions of K*m and N*m respectively;

[0024] S406: Use the obtained TK and TN as the input of the encoder respectively to perform the next encoder operation, so that after r encoders, the final 2D coding information tensor CK and 3D coding information tensor CN are obtained.

[0025] Preferably, the specific process of step S403 is:

[0026] S4031. Determine the parameter vector of the hth cross attention network W N ;

[0027] S4032, connect TK1 and TN1 respectively Multiply to get the query vector Q k , Q N ,and Multiply to get the key vector K k , K N ,and Multiply to get the value vector V k 、V N ;

[0028] S4033: According to the query vector Q N , key vector K k , value vector V k And the cross attention calculation formula to find Z ki ;

[0029] According to the query vector Q k , key vector K N , value vector V N And the cross attention calculation formula to find Z Ni ;

[0030] S4034, according to Z ki , Z Ni With W K , W N Find the cross attention weight vectors M3 and M4.

[0031] Preferably, the cross attention calculation formula in step S4033 is:

[0032]

[0033] Among them, when calculating Zki, Q N represents the query vector calculated by TN1, K K represents the key vector calculated by TK1, V k represents the value vector calculated by TK1, Denotes the key vector K K The transpose of k Represents vector Q N The number of columns, i.e. the vector dimension;

[0034] Similarly, when calculating ZNi, Q K , K N 、V N Just substitute it into the formula.

[0035] Preferably, the specific process of step S6 is:

[0036] S601. At the first Decoder, determine a (K+N)*m-dimensional starting bit vector cls, all values ​​of which are 0, and then input cls into the multi-head attention mechanism network to obtain a multi-head attention weight vector;

[0037] S602, performing residual and normalization processes on the multi-head attention weight vector to obtain an output vector A;

[0038] S603, input the tensor C and the vector A into the cross-head attention network to obtain the cross-head attention weight vector Ma;

[0039] S604, performing residual and normalization on the cross multi-head attention weight matrix Ma and the vector A to obtain a matrix Mb;

[0040] S605: Send Mb to the feedforward neural network Feed Forward, and then perform residual and normalization processes on the result to obtain the decoded information tensor T.

[0041] Preferably, the specific process of step S603 is:

[0042] S6031. Determine the parameter vector of the hth cross attention network

[0043] S6032. Add vector A and parameter vector Multiply to get the query vector Q A , respectively, the tensor C and the parameter vector Multiply to get the key vector K T , value vector V T ;

[0044] S6033: According to the query vector Q A , key vector K T , value vector V T And the cross attention calculation formula is used to find Zi;

[0045] S6034, according to the Z and parameter matrix of multiple cross attention networks Find the cross multi-head attention weight matrix Ma.

[0046] The present invention also discloses a computer program product, comprising a computer program, wherein the computer program executes the steps of the method described above when executed by a processor.

[0047] The present invention further discloses a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described above are executed.

[0048] The present invention also discloses a target detection system based on radar and vision fusion, comprising a memory and a processor connected to each other, wherein a computer program is stored on the memory, and when the computer program is run by the processor, the steps of the method described above are executed.

[0049] Compared with the prior art, the advantages of the present invention are:

[0050] The target detection method of the present invention combines the information of radar and vision sensors, and utilizes the depth perception capability of radar and the image features of vision to obtain a more comprehensive target description; the accuracy and robustness of target detection can be improved through fusion; the feature vector is mapped to a preset dimensional space through an extended dimensional network, and an encoding operation is performed. The scheme can extract and strengthen important feature information, and provide more powerful support for subsequent decoding and detection; combined with the cross-attention mechanism, it can more effectively highlight key features, strengthen the complementarity between radar and vision, and further improve the overall performance of target detection; the decoded information tensor is processed through the MLP operation to obtain the final target detection result, which helps to achieve accurate recognition of complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 The figure is a flow chart of an embodiment of the target detection method of the present invention. DETAILED DESCRIPTION

[0052] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0053] like Figure 1 As shown, the target detection method based on 4D millimeter wave radar and vision fusion in an embodiment of the present invention specifically includes the following steps:

[0054] S1, acquiring image data and radar point cloud data, and performing filtering preprocessing on the collected radar point cloud data and image data; for example, using Gaussian filtering method to obtain point cloud data and image data without noise points;

[0055] S2, respectively sending the preprocessed image data and radar point cloud data to the 2D detector and the 3D detector to obtain K visual target 2D detection candidate boxes and N radar target 3D detection candidate boxes, which are expressed in the form of feature vectors;

[0056] S3, mapping the feature matrices of the 2D detection candidate box and the 3D detection candidate box to a preset dimensional space through a dimension expansion network to obtain a 2D feature vector and a 3D feature vector after dimension expansion;

[0057] S4, respectively performing encoding operations on the expanded 2D feature vector and the 3D feature vector to obtain a 2D encoding information tensor and a 3D encoding information tensor;

[0058] S5, concatenating the 2D coding information tensor and the 3D coding information tensor to obtain an overall coding information tensor C;

[0059] S6. Decoding the entire coded information tensor C to obtain a decoded information tensor T;

[0060] S7. Perform MLP operation on the decoded information tensor T to obtain a target detection result.

[0061] Specifically, the dimension-expanding network in step S3 is implemented as follows:

[0062] For the feature matrix of the 2D detection candidate box, a feature vector of K*m dimension is obtained through a fully connected network of k*m dimension;

[0063] For the feature matrix of the 3D detection candidate box, a feature vector of N*m dimensions is obtained through a fully connected network of n*m dimensions; where m can be 64 or 128, etc.

[0064] The calculation formula of the fully connected neural network is: Tw+b;

[0065] Among them, T represents the feature matrix of the 2D or 3D detection candidate box, and w and b are parameter matrices.

[0066] Specifically, in step S4, the expanded 2D vector and 3D vector are respectively subjected to r Encoder operations, and the main steps include multi-head self-attention network, residual and normalization, cross multi-head attention network and feedforward neural network.

[0067] The calculation formula of the multi-head self-attention mechanism is as follows:

[0068] M=Concat(Z1,Z2,...Zm)W 0

[0069]

[0070] Where M represents the multi-head attention vector for calculating 2D vector or 3D vector, Z h represents the h-th self-attention vector associated with a 2D vector or a 3D vector, W 0 represents the parameter vector when compressing each self-attention vector, d k Represents vector Q h , K h The number of columns, i.e., the vector dimension, Q h represents the query vector, K hrepresents the key vector, V h represents a value vector, Represents the transpose of the key vector.

[0071] The residual and normalization calculation formulas are:

[0072] LayerNorm(T+Sublayer(T))

[0073] Where T refers to a 2D vector or a 3D vector, and Sublayer(T) represents the result of T calculated by a multi-head attention network.

[0074] The calculation of the cross-head multi-head attention mechanism is similar to the multi-head self-attention mechanism, and the formula is as follows:

[0075] M=Concat(Z1,Z2,...Zm)W 0

[0076]

[0077] Where M is the cross attention vector, W 0 Represents the parameter vector when compressing each cross-attention vector; Q comes from the cross-multi-head attention vector calculated by the 3D vector multiplied by the query vector WQ, K comes from the cross-multi-head attention vector calculated by the 2D vector multiplied by the key vector WK, V comes from the cross-multi-head attention vector calculated by the 2D vector multiplied by the value vector WV, K T represents the transpose of the key vector K, dk represents the number of columns of the vector Q, that is, the vector dimension; or Q comes from the cross-multi-head attention vector calculated by the 2D vector multiplied by the query vector WQ, K comes from the cross-multi-head attention vector calculated by the 3D vector multiplied by the key vector WK, V comes from the cross-multi-head attention vector calculated by the 3D vector multiplied by the value vector WV, K T represents the transpose of the key vector K, and dk represents the number of columns of the vector Q, that is, the vector dimension.

[0078] The calculation formula of the feedforward neural network is: max(0,Tw1+b1)w2+b2; where T is calculated by the cross-headed multi-head attention network.

[0079] Specifically, for the splicing operation in step S5, the specific process is as follows: create a new (K+N)*m-dimensional tensor TKN (that is, the overall coding information tensor C), the first K data of TKN are the coding information tensor CK after passing through r encoders, and the last N data are the coding information tensor CN after passing through r encoders.

[0080] Specifically, in step S6, r Decoder operations are performed on the tensor C after feature splicing. The main steps include multi-head self-attention mechanism, residual and normalization, cross multi-head attention network, and feedforward neural network. The specific calculation formula is the same as the formula mentioned in the Encoder.

[0081] The present invention combines the cross-attention mechanism to fuse image data with radar point cloud data, aiming to better combine the advantages of radar and vision, obtain higher-dimensional target information, enhance the stability of the recognition method, and improve the recognition accuracy.

[0082] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0083] The target detection method based on 4D millimeter wave radar and vision fusion of the present invention specifically comprises the following steps:

[0084] S1, performing filtering preprocessing on the collected radar point cloud data and image data, for example, using Gaussian filtering method to obtain point cloud data and image data without noise points;

[0085] S2. Send the preprocessed image data and radar point cloud data to the 2D detector and the 3D detector respectively to obtain the 2D detection candidate boxes of K visual targets and the 3D detection candidate boxes of N radar targets, which are expressed in the form of feature vectors; for example: (x1, x2, x3...xk) is the feature vector representation of the 2D detection candidate box, and (y1, y2, y3...yn) is the feature vector representation of the 3D detection candidate box;

[0086] Among them, x1, x2, x3...xk include the upper left corner coordinates, lower right corner coordinates, confidence, target category and other information of the 2D detection candidate box, y1, y2, y3...yn include the center point coordinates, length, width, height, confidence, target category and other information of the 3D detection candidate box.

[0087] The 2D detector and 3D detector use the YOLO v5 algorithm and the SECOND algorithm respectively;

[0088] S3, mapping the feature matrices of the 2D detection candidate box and the 3D detection candidate box to a certain dimensional space, such as 64 or 128 dimensions, through the dimension expansion network; the specific process is:

[0089] For the feature matrix of the 2D detection candidate box, a k*m dimensional fully connected neural network is passed through to obtain a K*m dimensional feature vector; for the feature matrix of the 3D detection candidate box, a n*m dimensional fully connected neural network is passed through to obtain an N*m dimensional feature vector, where m can be 64 or 128, etc.; the calculation formula of the above fully connected neural network is: Tw+b; where T represents the feature matrix of the 2D or 3D detection candidate box, and w and b are parameter matrices.

[0090] Among them, the dimension expansion network: maps the feature matrices of the 2D detection candidate box and the 3D detection candidate box to a certain dimensional space through the dimension expansion network, and obtains the 2D feature vector and 3D feature vector in the same dimensional space, which helps the subsequent network to better understand the relationship and interaction between image data and radar data, and is more convenient for cross-attention mechanism to be integrated.

[0091] S4. Perform Encoder operations on the expanded 2D vector and 3D vector respectively. The specific process is as follows:

[0092] S401, input the 2D feature vector and the 3D feature vector into the multi-head attention mechanism network to obtain multi-head attention weight vectors M1 and M2 respectively;

[0093] S402, then, the multi-head attention weight vectors M1 and M2 are subjected to residual and normalization processes respectively, and the result vectors TK1 and TN1 are obtained respectively. Among them, the multi-head attention mechanism network adopts a structure similar to the multi-head attention network in Transformer, which is composed of multiple self-attention network structures.

[0094] S403, input the result vectors TK1 and TN1 obtained by the multi-head attention mechanism network into the cross multi-head attention mechanism network respectively, and obtain the cross multi-head attention weight vectors M3 and M4 respectively;

[0095] Specifically, the cross-head attention mechanism consists of multiple cross-attention networks, which have the same structure but different parameter matrices. The specific process is as follows:

[0096] S4031. Determine the parameter vector of the hth cross attention network W N ;

[0097] S4032, connect TK1 and TN1 respectively Multiply to get the query vector Q k , Q N ,and Multiply to get the key vector K k , K N ,and Multiply to get the value vector Vk 、V N ;

[0098] S4033: According to the query vector Q N , key vector K k , value vector V k And the cross attention calculation formula to find Z ki ;

[0099] According to the query vector Q k , key vector K N , value vector V N And the cross attention calculation formula to find Z Ni ;

[0100] The cross attention calculation formula is as follows:

[0101]

[0102] Among them, when calculating Zki, Q N represents the query vector calculated by TN1, K K represents the key vector calculated by TK1, V k represents the value vector calculated by TK1, Denotes the key vector K K The transpose of k Represents vector Q N The number of columns, i.e. the vector dimension;

[0103] Similarly, when calculating ZNi, just substitute QK, KN, and VN into the formula.

[0104] S4034, then according to Z ki , Z Ni With W K , W N Find the cross attention weight vectors M3 and M4.

[0105] The cross-head attention weight vector is calculated as follows:

[0106] M=Concat(Z1,Z2,...Zp)W 0

[0107] Concat(Z1,Z2,...Zp) means concatenating each cross attention vector, W 0 Represents the parameter vector when compressing each cross-attention vector.

[0108] S404, then, the cross multi-head attention weight vectors M3 and M4 are subjected to residual and normalization processes respectively, and the result vectors TK2 and TN2 can be obtained respectively;

[0109] The residual and normalization calculation formulas are as follows: LayerNorm(T+Sublayer(T));

[0110] Where T refers to the result vectors TK1 and TN1 obtained by the previous multi-head attention mechanism network, and Sublayer(T) represents the result of TK1 or TN1 calculated by the cross multi-head attention mechanism, that is, M3 or M4;

[0111] S405, respectively send TK2 and TN2 to the feedforward neural network Feed Forward, and then perform residual and normalization processes on the results to obtain the encoded information tensors TK and TN after an Encoder, with dimensions of K*m and N*m respectively;

[0112] Among them, the Feed Forward network is a two-layer fully connected layer. The activation function of the first layer is Relu, and the second layer does not use an activation function. The corresponding formula is as follows: max(0,Tw1+b1)w2+b2; among them, T refers to TK2 or TN2 calculated by the cross multi-head attention mechanism, and w1, b1, w2, and b2 represent parameter vectors.

[0113] S406, using the obtained TK and TN as the input of the encoder to perform the next encoder operation, so that after r encoders, the final 2D coding information tensor CK and 3D coding information tensor CN can be obtained;

[0114] Adding a cross-headed multi-head attention network to the Encoder and Decoder mechanism, exchanging the query vector Q calculated by the 2D vector and the 3D vector, and decoding using the K and V calculated by the concatenated feature tensor, all play a role in fusing radar data with visual data.

[0115] S5. Concatenate the 2D encoding information tensor CK and the 3D encoding information tensor CN after the encoder to obtain the overall encoding information tensor C. The specific steps are:

[0116] Create a (K+N)*m-dimensional tensor TKN (i.e., the overall coded information tensor C). The first K data of TKN are the coded information tensor CK after passing through r encoders, and the last N data are the coded information tensor CN after passing through r encoders.

[0117] S6. Perform a Decoder operation on the tensor C to obtain a decoded information tensor T;

[0118] Perform the Decoder operation on the tensor C. The specific steps are as follows:

[0119] S601. At the first Decoder, determine a (K+N)*m-dimensional starting bit vector cls, all values ​​of which are 0, and then input cls into the multi-head attention mechanism network to obtain a multi-head attention weight vector;

[0120] S602, then, the multi-head attention weight vector is subjected to residual and normalization processes to obtain an output vector A;

[0121] S603, input the tensor C and the vector A into the cross-head attention network to obtain the cross-head attention weight vector Ma; the specific process is as follows:

[0122] S6031. Determine the parameter vector of the hth cross attention network

[0123] S6032, then vector A and parameter vector Multiplying them together gives us the query vector Q A , respectively, the tensor C and the parameter vector Multiplying them gives us the key vector K T , value vector V T ;

[0124] S6033: According to the query vector Q A , key vector K T , value vector V T And the cross attention calculation formula is used to find Zi;

[0125] S6034, according to the Z and parameter matrix of multiple cross attention networks Find the cross multi-head attention weight matrix Ma;

[0126] S604, performing residual and normalization on the cross multi-head attention weight matrix Ma and the vector A to obtain a matrix Mb;

[0127] S605: Send Mb to the feedforward neural network Feed Forward, and then perform residual and normalization processes on the result to obtain the decoded information tensor T.

[0128] S7, the decoded information tensor T is passed through the MLP network (multi-layer perceptron) to obtain a target detection result;

[0129] Furthermore, the decoded information tensor T is used as the input of the next Decoder, and combined with the next encoded information tensor C in the cross-head attention network to calculate the next target.

[0130] The target detection method of the present invention combines the information of radar and vision sensors, and utilizes the depth perception capability of radar and the image features of vision to obtain a more comprehensive target description; the accuracy and robustness of target detection can be improved through fusion; the feature vector is mapped to a preset dimensional space through an extended dimensional network, and an encoding operation is performed. The scheme can extract and strengthen important feature information, and provide more powerful support for subsequent decoding and detection; combined with the cross-attention mechanism, it can more effectively highlight key features, strengthen the complementarity between radar and vision, and further improve the overall performance of target detection; the decoded information tensor is processed through the MLP operation to obtain the final target detection result, which helps to achieve accurate recognition of complex scenes.

[0131] The present invention also discloses a computer program product, including a computer program, which executes the steps of the above method when executed by a processor. The present invention further discloses a computer-readable storage medium, on which a computer program is stored, which executes the steps of the above method when executed by a processor. The present invention also discloses a target detection system based on radar and visual fusion, including an interconnected memory and a processor, on which a computer program is stored, which executes the steps of the above method when executed by a processor. The product, medium and system of the present invention correspond to the above method and also have the advantages described in the above method.

[0132] The present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by hardware related to computer program instructions. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned method embodiment when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. Computer-readable storage media include: any entity or device that can carry computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. The memory is used to store computer programs and / or modules, and the processor implements various functions by running or executing computer programs and / or modules stored in the memory, and calling data stored in the memory. The memory may include high-speed random access memory and may also include non-volatile memory, such as a hard disk, an internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0133] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A target detection method based on radar and vision fusion, characterized in that: Includes steps: S1, obtain image data and radar point cloud data; S2, sending the image data and radar point cloud data to the 2D detector and the 3D detector respectively, obtaining the 2D detection candidate boxes of K visual targets and the 3D detection candidate boxes of N radar targets, and representing them in the form of feature matrices; S3, mapping the feature matrices of the 2D detection candidate box and the 3D detection candidate box to a preset dimensional space through a dimension expansion network to obtain a 2D feature vector and a 3D feature vector after dimension expansion; S4, respectively performing encoding operations on the expanded 2D feature vector and the 3D feature vector to obtain a 2D encoding information tensor and a 3D encoding information tensor; S5, concatenating the 2D coding information tensor and the 3D coding information tensor to obtain an overall coding information tensor C; S6. Decoding the entire coded information tensor C to obtain a decoded information tensor T; S7. Perform MLP operation on the decoded information tensor T to obtain the target detection result.

2. The target detection method based on radar and vision fusion according to claim 1 is characterized in that: The specific process of step S3 is: For the feature matrix of the 2D detection candidate box, a fully connected neural network with k*m dimensions is passed to obtain a feature vector with K*m dimensions. For the feature matrix of the 3D detection candidate box, a fully connected neural network with n*m dimensions is passed to obtain a feature vector with N*m dimensions. The calculation formula of the fully connected neural network is: Tw+b; wherein T represents the feature matrix of the 2D or 3D detection candidate box, and w and b represent the parameter matrix.

3. The target detection method based on radar and vision fusion according to claim 1 is characterized in that: The specific process of step S4 is: S401, input the 2D feature vector and the 3D feature vector into the multi-head attention mechanism network to obtain multi-head attention weight vectors M1 and M2 respectively; S402, performing residual and normalization processes on the multi-head attention weight vectors M1 and M2, respectively, to obtain result vectors TK1 and TN1; S403, input the result vectors TK1 and TN1 into the cross-head attention mechanism network respectively, and obtain the cross-head attention weight vectors M3 and M4 respectively; S404, performing residual and normalization processes on the cross multi-head attention weight vectors M3 and M4, respectively, to obtain result vectors TK2 and TN2, respectively; S405, respectively send TK2 and TN2 to the feedforward neural network Feed Forward, and then perform residual and normalization processes on the results to obtain the encoded information tensors TK and TN after an Encoder, with dimensions of K*m and N*m respectively; S406: Use the obtained TK and TN as the input of the encoder respectively to perform the next encoder operation, so that after r encoders, the final 2D coding information tensor CK and 3D coding information tensor CN are obtained.

4. The target detection method based on radar and vision fusion according to claim 3 is characterized in that: The specific process of step S403 is: S4031. Determine the parameter vector of the hth cross attention network W k , W N ; S4032, connect TK1 and TN1 respectively Multiply to get the query vector Q k , Q N ,and Multiply to get the key vector K k , K N ,and Multiply to get the value vector V k 、V N ; S4033: Based on the query vector Q N , key vector K k , value vector V k And the cross attention calculation formula to find Z ki ; According to the query vector Q k , key vector K N , value vector V N And the cross attention calculation formula to find Z Ni ; S4034, according to Z ki , Z Ni With W K , W N Find the cross attention weight vectors M3 and M4.

5. The target detection method based on radar and vision fusion according to claim 4 is characterized in that: The cross attention calculation formula in step S4033 is: Among them, when calculating Zki, Q N represents the query vector calculated by TN1, K K represents the key vector calculated by TK1, V k represents the value vector calculated by TK1, K k T Denotes the key vector K K The transpose of k Represents vector Q N The number of columns, i.e. the vector dimension; Similarly, when calculating ZNi, Q K , K N 、V N Just substitute it into the formula.

6. The target detection method based on radar and vision fusion according to any one of claims 1 to 5, characterized in that: The specific process of step S6 is: S601. At the first Decoder, determine a (K+N)*m-dimensional starting bit vector cls, all values ​​of which are 0, and then input cls into the multi-head attention mechanism network to obtain a multi-head attention weight vector; S602, performing residual and normalization processes on the multi-head attention weight vector to obtain an output vector A; S603, input the tensor C and the vector A into the cross-head attention network to obtain the cross-head attention weight vector Ma; S604, performing residual and normalization on the cross multi-head attention weight matrix Ma and the vector A to obtain a matrix Mb; S605: Send Mb to the feedforward neural network Feed Forward, and then perform residual and normalization processes on the result to obtain the decoded information tensor T.

7. The target detection method based on radar and vision fusion according to claim 6 is characterized in that: The specific process of step S603 is: S6031. Determine the parameter vector of the hth cross attention network S6032. Add vector A and parameter vector Multiply to get the query vector Q A , respectively, the tensor C and the parameter vector Multiply to get the key vector K T , value vector V T ; S6033: According to the query vector Q A , key vector K T , value vector V T And the cross attention calculation formula is used to find Zi; S6034, according to the Z and parameter matrix of multiple cross attention networks Find the cross multi-head attention weight matrix Ma.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are performed.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program performs the steps of the method according to any one of claims 1 to 7.

10. A target detection system based on radar and vision fusion, comprising a memory and a processor connected to each other, wherein a computer program is stored in the memory, characterized in that: When the computer program is executed by a processor, the computer program performs the steps of the method according to any one of claims 1 to 7.

Images