Target detection and decision-making method and system based on multi-modal fusion and medium

Abstract

The invention provides a multi-modal fusion-based target detection and decision-making method and system and a medium, and the method comprises the steps: based on Faster-RCNN, employing a ResNet50 convolution layer to extract image features, employing FPN to achieve the multi-scale detection of the features, employing an RPN network layer to generate a candidate region, and finally generating a feature map through an ROI layer; based on a PointNet2 model, a sampling grouping strategy is adopted, candidate points are generated through a farthest point sampling method, then a local point cloud set is delimited by taking the candidate points as the center, the local point cloud set is input into a PointNet network, a local point cloud feature set is generated, and the point cloud feature set is input into a dimension degradation layer. According to the method, the correct perception ability of model learning is improved, the misjudgment of model detection is reduced, the purpose of enhancing the model is achieved from the data level, and a high-precision and high-accuracy three-dimensional target detection algorithm is supported.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a target detection and decision-making method, system and medium based on multimodal fusion. Background Art

[0002] The current research on 3D environment perception and reconstruction has a wide range of applications in autonomous navigation, augmented reality, indoor home furnishing, etc. 3D environment perception and reconstruction generally has two technical paths: sensors and vision, with 3D object detection methods based on point clouds and images as the main representative.

[0003] Compared with two-dimensional target detection, three-dimensional target detection adds depth perception of objects, changing from calibrating the position of objects in the image to calibrating the position of target objects in three-dimensional space. In two-dimensional target detection, feature recognition and ROI calibration are all based on RGB image vision, using traditional methods or deep learning methods. However, for three-dimensional targets, the depth distance information of objects is missing on RGB images, and the error of depth detection of objects using single-purpose RGB images is large. Instead, a binocular camera is needed to use the parallax of the two images to make up for the missing spatial dimension information. Although binocular cameras can provide higher-precision detection, the disadvantage that the camera is easily affected by light cannot be avoided.

[0004] A popular method in 3D target detection is to build a 3D target detection model based on point cloud data collected by LIDAR (laser radar). Although point cloud has the ability to perceive objects in 3D and can accurately calibrate the outline of the target object, the disorder and sparsity of point cloud determine that point cloud data lacks detailed texture information of objects and has inherent deficiencies in feature recognition. On the contrary, RGB images have rich texture information but lack spatial information.

[0005] Multimodal heterogeneous data requires the construction of corresponding modal representation models, mapping heterogeneous data to the same feature space, and realizing feature association through fusion algorithms, and finally completing the 3D target detection task based on comprehensive fusion features. The research directions of 3D target detection based on point cloud modality are mainly divided into four categories: based on original point cloud, based on voxels, and based on multiple views. The original point cloud-based method can retain the geometric features of the point cloud, such as the PointNet++ (CN110097556A) model, which retains geometric features through multi-scale acquisition and feature weighted splicing methods, and solves the problems of rotation invariance and disorder of point cloud data. The multi-view-based method, such as LaserNet, projects discrete point cloud data into a dense two-dimensional representation through a spherical projection, and each element point contains color and depth information, namely RGB-D; however, the projection of the point cloud to the Range view is a single shot, and there are problems such as occlusion and scale transformation. Feature information loss and deformation problems. Voxel-based methods, such as the PointPillars (CN114648698A) model, encode point clouds into sparse pseudo-images through top-view projection, which then degenerates into a 2D object detection task.

[0006] Research on image-based object detection is relatively mature, such as the YOLO series of algorithms, Faster-RCNN algorithm, etc., which have excellent performance in two-dimensional object detection. At the same time, due to the low hardware cost of monocular cameras and the great demand for application promotion, research on monocular three-dimensional object detection algorithms has also become a hot research direction, such as the FCOS3D (CN116052156A) model, which extracts multi-scale features through a convolutional network, constructs a feature pyramid network, and realizes the prediction of the 3D center point and size of the target.

[0007] Single modality has limitations, such as the lack of texture information in point clouds and the lack of spatial information in images. For example, LaserNet++ based on single-stage fusion establishes a mapping from image to point cloud, realizing feature-level fusion; BEVFusion (CN116051489A) based on multi-stage fusion uses KNN's neighboring points to back-project the point cloud coordinates on the point cloud BEV view, thereby establishing an associated mapping of the two modal features of point cloud and image on the BEV view, realizing the fusion of the semantic features of the image and the geometric features of the point cloud. However, the existing multi-modal-based 3D target detection model algorithms are all exploring and mining the advantageous features of modal data and performing associated fusion, and cannot effectively achieve feature enhancement and complementarity. Summary of the invention

[0008] In order to solve the deficiencies in the prior art, the present invention provides an automated testing method, system and medium for software and hardware to complete the identification test case, combining point cloud and RGB image to achieve feature complementarity of two modal data, and relatively complete data information can be obtained to support high-precision (calibration) and high-accuracy (recognition) three-dimensional target detection algorithms. The method is based on the skeleton model of Faster-RCNN to extract the semantic and texture features of the image; based on the PointNet++ skeleton model to extract the geometric features of the point cloud; for the extracted geometric, semantic and texture features, the multi-modal feature interaction of the high-order matrix is ​​realized through the fusion of bilinear pooling, and the local correlation relationship of the features is retained.

[0009] The first aspect of the present invention discloses a target detection and decision method based on multimodal fusion, comprising the following steps:

[0010] Step S101: designing image modality feature mapping based on the skeleton model of Faster-RCNN;

[0011] Step S102: designing point cloud modal feature mapping based on the skeleton model of PoinetNet2;

[0012] Step S103: The bilinear pooling method MLB based on the Hadamard product receives the feature map obtained in step S101 and step S102, maps the feature to a fusion representation space, and obtains a fusion feature;

[0013] Step S104: regularize the dimension of the fused features obtained in step S103 through the kernel function P; finally, in the MLP layer, perform high-order feature crossover on the fused features, and implement the target detection task through the CLS and BBox modules.

[0014] According to the method of the first aspect of the present invention, the image modality feature mapping process based on the skeleton model of Faster-RCNN in step S101 includes the following steps:

[0015] Step S1.1: Select Faster-RCNN as the backbone, introduce the FPN layer, and replace the original VGG16 part with ResNet50;

[0016] Step S1.2: Combine the image modality feature map replaced in step S1.1 and use RoiAlign to replace RoIPooling.

[0017] According to the method of the first aspect of the present invention, the process of designing point cloud modal feature mapping based on the skeleton model of PoinetNet2 in step S102 includes:

[0018] Step S2.1: Sampling and grouping, and then setting different radii for the sampling center points through a multi-scale combination method to generate point cloud features;

[0019] Step S2.2: Based on the point cloud features obtained in step S2.1, a dimension reduction layer DRL is introduced to match the image features and map the obtained features to a homogeneous feature representation space.

[0020] According to the method of the first aspect of the present invention, the bilinear pooling method MLB based on the Hadamard product in step S103 maps the features to the fusion representation space to achieve the high-order feature association fusion process of the features, including:

[0021] Step S3.1: Design a difference metric function S to measure the two modal features f of point cloud and image i ,f p The distribution difference reflects the difference in the correlation characteristics between the modalities. In the model training process, minimize S(f i ,f p ), the feature distributions of the two modal data will co-evolve and be projected into the same representation space;

[0022] Step S3.2: After normalizing the multiple features output by the modal representation model, use the feature outer product to measure their differences;

[0023] Step S3.3: Based on the outer product obtained in step 3.2, in three-dimensional space, let i, j, k be the unit vectors in the three-dimensional space, and the vector is a vector starting at (0,0,0) and ending at (x,y,z). Let Let vector By vector according to Definition, then the vector It is called vector The outer product of but:

[0024]

[0025] The above determinant reflects the contribution of each orthogonal component in the cross product. At the same time, the cross product marks the high-order statistics: area, volume, hypervolume, from line to surface, from surface to body;

[0026] Step S3.4: The multimodal fusion method based on Hadamard product retains the feature information of distinguishing semantics and is a generalized fusion operator with excellent performance; and introduces a low-rank decomposition strategy to control the scale of feature dimensions and alleviate the problem of high computational complexity of bilinear pooling;

[0027] Step S3.5: Design the fusion algorithm based on MLB, introduce nonlinear mapping functions σ, τ, where the Tanh activation function The activation function of the GeLu Gaussian error linear unit is:

[0028]

[0029] Increase the feature expression ability of fusion and solve the gradient disappearance problem; let the final bilinear pooling fusion function be:

[0030]

[0031] According to the method of the first aspect of the present invention, step S3.4 further includes: assuming that the fused multimodal feature is X={x1, x2, ..., x N},Y={y1,y2,y3,…,y M}, weight matrix W i is a kernel function with a bias coefficient of b. A kernel function can generate a fusion feature element f i , through L kernel functions, the fused feature f is obtained; based on the Hadamard product, the fused feature vector can be expressed as:

[0032]

[0033] By replacing W with low-order matrices U and V, the number of parameters can be reduced, and the weight matrix can be rewritten as U i ∈R N×d ,V i ∈R M×d , through the kernel function P to regularize the dimension of the fusion feature, P∈R d×p , where d≤(N,M), M is the dimension of feature X, and N is the dimension of feature Y, then:

[0034]

[0035] Where X, Y are input multi-modal features, W is the weight matrix, U, V are low-order matrices, and b is the bias coefficient;

[0036] The MLB algorithm is as follows: first, perform a linear transformation on the multi-mode features, calculate their Hadamard products, and finally add the bias b.

[0037] According to the method of the first aspect of the present invention, in step S104, the dimension of the fusion feature is regularized by the kernel function P; and finally the target detection task is realized by the CLS and BBox modules, which includes:

[0038] Step S4.1: Perform GeLu nonlinear transformation;

[0039] Step S4.2: Based on step 4.1, the dimension of the fused features is regularized by the kernel function P;

[0040] Step S4.3: At the MLP layer, high-order feature crossover is performed on the fused features, and the target detection task is achieved through the CLS and BBox modules.

[0041] According to the method of the first aspect of the present invention, the target detection task is realized through the CLS and BBox modules, including: integrating the classification and bounding box regression models of CLS and BBox to participate in the calculation of the loss function value of the training phase and the result output of the prediction phase.

[0042] According to the method of the first aspect of the present invention, the loss function is divided into three parts: Loss trImg ,Loss trPts ,Loss fusion , respectively represent the image loss function value, point cloud loss function value and fusion loss function value.

[0043] The second aspect of the present invention discloses a target detection and decision-making system based on multimodal fusion, including a computer device, which is used to execute the steps of the target detection and decision-making method based on multimodal fusion described in the first aspect.

[0044] The third aspect of the present invention discloses a computer-readable storage medium, on which a computer program is stored to implement the target detection and decision-making method based on multimodal fusion described in the first aspect.

[0045] In summary, the solution proposed in the present invention has the following technical effects: after aligning the multiple features output by the Faster RCNN skeleton model and the PointNet2 skeleton model, a difference measurement function is designed, namely a bilinear pooling function based on the Hadamard product, which projects the multi-modal features into the same representation space, and co-evolves and strengthens the associated features during the model optimization process. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 The image feature mapping network structure based on Faster-RCNN of the present invention;

[0048] Figure 2The point cloud feature mapping network structure of the present invention is based on PointNet2;

[0049] Figure 3 It is the bilinear pooling fusion model structure of the present invention;

[0050] Figure 4 The present invention is a flowchart of a target detection and decision-making method based on multimodal fusion. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0052] like Figure 4 FIG. 1 is a flow chart of a target detection and decision-making method based on multimodal fusion provided by the present invention, comprising the following steps:

[0053] Step S101: designing image modality feature mapping based on the skeleton model of Faster-RCNN;

[0054] Step S102: designing point cloud modal feature mapping based on the skeleton model of PoinetNet2;

[0055] Step S103: The bilinear pooling method MLB based on the Hadamard product receives the feature map obtained in step S101 and step S102, maps the feature to a fusion representation space, and obtains a fusion feature;

[0056] Step S104: regularize the dimension of the fused features obtained in step S103 through the kernel function P; finally, in the MLP layer, perform high-order feature crossover on the fused features, and implement the target detection task through the CLS and BBox modules.

[0057] like Figure 1 As shown, the image modality feature mapping process based on the skeleton model of Faster-RCNN in step S101 includes the following steps:

[0058] Step S1.1: Select Faster-RCNN as the backbone, introduce the FPN layer, and replace the original VGG16 part with ResNet50;

[0059] Step S1.2: Combine the image modality feature map replaced in step S1.1 and use RoiAlign to replace RoIPooling.

[0060] like Figure 2 As shown, the process of designing point cloud modal feature mapping based on the skeleton model of PoinetNet2 in step S102 includes:

[0061] Step S2.1: Sampling and grouping, and then setting different radii for the sampling center points through a multi-scale combination method to generate point cloud features;

[0062] Step S2.2: Based on the point cloud features obtained in step S2.1, a dimension reduction layer DRL is introduced to match the image features and map the obtained features to a homogeneous feature representation space.

[0063] like Figure 3 As shown, the bilinear pooling fusion model structure provided by the present invention. In step S103, the bilinear pooling method MLB based on the Hadamard product maps the features to the fusion representation space, and the high-order feature association fusion process of the features is realized, which includes:

[0064] Step S3.1: Design a difference metric function S to measure the two modal features f of point cloud and image i ,f p The distribution difference reflects the difference in the correlation characteristics between the modalities. In the model training process, minimize S(f i ,f p ), the feature distributions of the two modal data will co-evolve and be projected into the same representation space;

[0065] Step S3.2: After normalizing the multiple features output by the modal representation model, use the feature outer product to measure their differences;

[0066] Step S3.3: Based on the outer product obtained in step 3.2, in three-dimensional space, let the vector By vector according to Definition, then the vector It is called vector The outer product of set up but:

[0067]

[0068] The above determinant reflects the contribution of each orthogonal component in the cross product. At the same time, the cross product marks the high-order statistics: area, volume, hypervolume, from line to surface, from surface to body;

[0069] Step S3.4: The multimodal fusion method based on Hadamard product retains the feature information of distinguishing semantics and is an excellent generalized fusion operator; and introduces a low-rank decomposition strategy to control the scale of feature dimensions and alleviate the problem of high computational complexity of bilinear pooling;

[0070] Step S3.5: Design the fusion algorithm based on MLB, introduce nonlinear mapping functions σ, τ, where the Tanh activation function The activation function of the GeLu Gaussian error linear unit is:

[0071]

[0072] Increase the feature expression ability of fusion and solve the gradient disappearance problem; let the final bilinear pooling fusion function be:

[0073]

[0074] The step S3.4 also includes: assuming that the fused multi-mode feature is X = {x1, x2, ..., x N},Y={y1,y2,y3,…,y M}, weight matrix W i is a kernel function. A kernel function can generate a fusion feature element f i , through L kernel functions, the fused feature f is obtained; based on the Hadamard product, the fused feature vector can be expressed as:

[0075]

[0076] By replacing W with low-order matrices U and V, the number of parameters can be reduced, and the weight matrix can be rewritten as U i ∈R N×d ,V i ∈R M×d . The kernel function P is used to regularize the dimension of the fusion feature, P∈R d×p , where d≤(N,M), then:

[0077]

[0078] The MLB algorithm is as follows: first, perform a linear transformation on the multi-mode features, calculate their Hadamard products, and finally add the bias b.

[0079] In step S104, the dimension of the fusion feature is regularized by the kernel function P; finally, the target detection task is realized by the CLS and BBox modules. The process includes:

[0080] Step S4.1: Perform GeLu nonlinear transformation;

[0081] Step S4.2: Based on step 4.1, the dimension of the fused features is regularized by the kernel function P;

[0082] Step S4.3: At the MLP layer, high-order feature crossover is performed on the fused features, and the target detection task is implemented through the CLS and BBox modules. Implementing the target detection task through the CLS and BBox modules includes: integrating the classification and bounding box regression models of CLS and BBox to participate in the calculation of the loss function value in the training phase and the result output in the prediction phase.

[0083] The loss function of the model is divided into three parts: Loss trImg , Loss trPts , Loss fusion , respectively represent the image loss function value, point cloud loss function value and fusion loss function value. Among them, Loss trPts After the image data flows through the Share FC layer during the training phase, it flows to the classification output of the image classification regression model. and border regression The weighted sum of trPts After the point cloud data flows through PointNet2, it flows to the classification output of the point cloud classification regression and border regression The weighted sum of fusion Classification Loss output by the feature pooling fusion model cls And border regression Lossb box The overall loss function of the model is:

[0084] Loss total =w f ×Loss fusion +w i ×Loss trImg +w p ×Loss trPts

[0085] in:

[0086] Loss fusion =Loss seg ⊕Loss cls

[0087] Loss fusion =Loss seg ⊕Loss cls

[0088]

[0089] CLS in the model i , BBox i , SEGp , CLS p Some of them are only used to calculate the classification and regression loss function values ​​of the output images and point clouds in the training phase. As an auxiliary means of modality training, two single-modality models are pre-trained through these two branches. During the overall training, the role of modality features is enhanced and the fusion features are supplemented. The CLS, BBox classification and bounding box regression models of the fusion model DLP-Fusion participate in the calculation of the loss function value in the training phase and the result output in the prediction phase. The overall process of the algorithm is as follows: Algorithm 1.

[0090]

[0091] Experimental data set: The UN RGB-D data set is an RGBD data set for indoor scene understanding released by Princeton University. It includes two modal data types: point cloud and image. It is mainly used for scene classification tasks, semantic segmentation, object detection and other tasks. The data set contains 10,335 pairs of samples (5,285 of which are training set data) and 37 target object categories. This section predicts ten common categories (bed, table, sofa, chair, toilet, desk, dresser, night_stand, bookshelf, bathtub) based on SUN RGB-D data, and experimentally verifies the multimodal three-dimensional target detection algorithm based on bilinear pooling fusion established above. The experimental parameter settings of the DLP model are shown in Table 1:

[0092] Table 1 DLP model parameters

[0093]

[0094] As shown in Table 2, the mAP 0.25 indicator results of the multimodal fusion 3D object detection algorithm DLP based on bilinear pooling fusion designed in this section and the baseline model are compared in each category. It can be seen that compared with the baseline model PointNet2, the multimodal fusion algorithm DLP based on bilinear pooling in this section has improved 4.2mAP, and the indicator results of all categories are better than the baseline model, such as 7.2 on bookself, 2.2 on nightstand and 6.7 on dresser with similar appearance.

[0095] Compared with the baseline model PointNet2+Image(PN2+Img) of feature linear fusion (multi-feature splicing), the bilinear pooling fusion method has a more significant improvement, which to a certain extent shows the effectiveness of the correlation feature enhancement of the difference function S and the Hadamard product function prior calculation of high-order feature methods.

[0096] Table 2 Comparison of mAP (0.25) results of DLP algorithm on SUNRGB-D test set with the baseline model

[0097]

[0098] In Table 2, the test set is randomly divided into 1 / 4 of the whole sample, that is, 2025 sets. (Training set: test set: validation set = 2:1:1)

[0099] Table 3 shows the prediction examples of the bilinear pooling fusion 3D object detection model in this section. Overall, the prediction results of the two modalities of image and point cloud are highly accurate, and there is no significant difference in the position calibration, type recognition and human eye observation results of the object.

[0100] Table 3 Prediction examples of multimodal fusion 3D target detection model based on bilinear pooling fusion

[0101]

[0102]

[0103] For example 1, the fusion model can accurately identify the target category and perceive its position in space for two beds and two nightstands. Most of the nightstands hidden under the bed can be accurately identified and their size and position can be predicted. This shows that the image and point cloud modalities are complementary and enhanced. The size and position of the nightstand are accurately located through the point cloud features, and the category of the nightstand is accurately identified through the texture information of the image. For example 2, it can be seen that the model seems to have learned some symmetry rules. When there is a chair on the right side of the table, the table legs on the other side of the table are identified as chairs. At the same time, the learning of the desk model is not completely accurate, and the window sill is mistakenly identified as a desk. For example 3, the scene is cluttered and there are many cuboid-like interference objects. The model distinguishes the interference objects and accurately identifies the nearby bookshelf. However, for the bookshelf in the distance, the model has omissions, and due to occlusion and lack of point cloud data, the direction and size prediction of the upper right bookshelf are incorrect. For example 4, in a dark scene, the model was hardly affected by the addition of point cloud data, and accurately identified the sofa that was difficult to distinguish with the naked eye. For example 5, in a warm light scene (different from the color tone of daylight), the model was not affected by the change in light and was still able to accurately identify the toilet; perhaps the toilet and bathtub are strongly associated in the training set, and there is no bathtub next to the toilet, but the model mistakenly identified the bathtub.

[0104] Through analysis, on the one hand, due to the complementarity of features, the multimodal model not only improves the precision and accuracy, but also enhances the robustness of the model, which can make up for the defects of the single modality. For example, images are easily affected by lighting conditions and occlusion. Under the complementarity of point clouds, the accurate perception of space can effectively make up for these defects. On the other hand, some inspirations can be summarized: (1) For strongly associated objects in the training set, such as toilet-bathtub, table-chair, etc., the proportion of adversarial samples of associated objects should be increased (such as increasing the number of samples of associated objects with single targets), breaking this artificially introduced association law, so that the model can learn the correct perception ability and reduce misjudgment; (2) For objects in the scene that are misjudged due to high similarity, they are marked as others. When the model is iterated, similar adversarial samples of others are added to achieve the purpose of enhancing the model from the data level; (3) This idea of ​​analyzing the prediction effect of the samples after the model is deployed, analyzing the limitations of the model, and then adding corresponding adversarial samples can be widely generalized.

[0105] In addition, an embodiment of the present invention further provides a target detection and decision-making system based on multimodal fusion, including a computer device, which is used to execute the steps of the aforementioned target detection and decision-making method based on multimodal fusion.

[0106] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored to implement the aforementioned target detection and decision-making method based on multimodal fusion.

[0107] Please note that the technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. The above embodiments only express several implementation methods of the present application, and their descriptions are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the present application, several variations and improvements can be made, which all belong to the scope of protection of the present application. Therefore, the scope of protection of the patent in this application shall be based on the attached claims.

Claims

1. A target detection and decision-making method based on multimodal fusion, characterized in that: The steps include: Step S101: designing image modality feature mapping based on the skeleton model of Faster-RCNN; Step S102: designing point cloud modal feature mapping based on the skeleton model of PoinetNet2; Step S103: The bilinear pooling method MLB based on the Hadamard product receives the feature map obtained in step S101 and step S102, maps the feature to a fusion representation space, and obtains a fusion feature; Step S104: regularize the dimension of the fused features obtained in step S103 through the kernel function P; finally, in the MLP layer, perform high-order feature crossover on the fused features, and implement the target detection task through the CLS and BBox modules.

2. The method according to claim 1, characterized in that The image modality feature mapping process based on the skeleton model of Faster-RCNN in step S101 includes the following steps: Step S1.1: Select Faster-RCNN as the backbone, introduce the FPN layer, and replace the original VGG16 part with ResNet50; Step S1.2: Combine the image modality feature map replaced in step S1.1 and use RoiAlign to replace RoIPooling.

3. The method according to claim 2, characterized in that The process of designing point cloud modal feature mapping based on the skeleton model of PoinetNet2 in step S102 includes: Step S2.1: Sampling and grouping, and then setting different radii for the sampling center points through a multi-scale combination method to generate point cloud features; Step S2.2: Based on the point cloud features obtained in step S2.1, a dimension reduction layer DRL is introduced to match the image features and map the obtained features to a homogeneous feature representation space.

4. The method according to claim 3, characterized in that In step S103, the bilinear pooling method MLB based on the Hadamard product maps the features to the fusion representation space, and the high-order feature association fusion process of the features is realized, which includes: Step S3.1: Design a difference metric function S to measure the two modal features f of point cloud and image i ,f p The distribution difference reflects the difference in the correlation characteristics between the modalities. In the model training process, minimize S(f i ,f p ), the feature distributions of the two modal data will co-evolve and be projected into the same representation space; Step S3.2: After normalizing the multiple features output by the modal representation model, use the feature outer product to measure their differences; Step S3.3: Based on the outer product obtained in step 3.2, in three-dimensional space, let i, j, k be the unit vectors in the three-dimensional space, and the vector is a vector starting at (0,0,0) and ending at (x,y,z). Let Let vector By vector according to Definition, then the vector It is called vector The outer product of but: The above determinant reflects the contribution of each orthogonal component in the cross product. At the same time, the cross product marks the high-order statistics: area, volume, hypervolume, from line to surface, from surface to body; Step S3.4: The multimodal fusion method based on Hadamard product retains the feature information of distinguishing semantics and is an excellent generalized fusion operator; and introduces a low-rank decomposition strategy to control the scale of feature dimensions and alleviate the problem of high computational complexity of bilinear pooling; Step S3.5: Design the fusion algorithm based on MLB, introduce nonlinear mapping functions σ, τ, where the Tanh activation function The activation function of the GeLu Gaussian error linear unit is: Increase the fusion feature expression ability and solve the gradient vanishing problem; Assume that the fused multi-mode feature is X = {x1, x2, ..., x N },Y={y1,y2,y3,…,y M }, the kernel function used to project to the low-dimensional feature is U, V, and the projection matrix and bias coefficient of the final output result are set to P, b, and the final bilinear pooling fusion function is:

5. The method according to claim 3, characterized in that: The step S3.4 also includes: assuming that the fused multi-mode feature is X = {x1, x2, ..., x N },Y={y1,y2,y3,…,y M }, weight matrix W i is a kernel function with a bias coefficient of b. A kernel function can generate a fusion feature element f i , through L kernel functions, the fused feature f is obtained; based on the Hadamard product, the fused feature vector can be expressed as: By replacing W with low-order matrices U and V, the number of parameters can be reduced, and the weight matrix can be rewritten as U i ∈R U×d ,V i ∈R M ×d , through the kernel function P to regularize the dimension of the fusion feature, P∈R d×p , where d≤(N,M), M is the dimension of feature X, and N is the dimension of feature Y, then: Where X, Y are input multi-modal features, W is the weight matrix, U, V are low-order matrices, and b is the bias coefficient; The MLB algorithm is as follows: first, perform a linear transformation on the multi-mode features, calculate their Hadamard products, and finally add the bias b.

6. The method according to claim 4, characterized in that In step S104, the dimension of the fusion feature is regularized by the kernel function P; finally, the target detection task is realized by the CLS and BBox modules. The process includes: Step S4.1: Perform GeLu nonlinear transformation; Step S4.2: Based on step 4.1, the dimension of the fused features is regularized by the kernel function P; Step S4.3: At the MLP layer, high-order feature crossover is performed on the fused features, and the target detection task is achieved through the CLS and BBox modules.

7. The method according to claim 6, characterized in that The target detection task is realized by using the CLS and BBox modules, including: integrating the classification and bounding box regression models of CLS and BBox to participate in calculating the loss function value in the training phase and the result output in the prediction phase.

8. The method according to claim 7, characterized in that The loss function is divided into three parts: Loss trImg ,Loss trPts ,Loss fusion , respectively represent the image loss function value, point cloud loss function value and fusion loss function value.

9. A target detection and decision system based on multimodal fusion, comprising a computer device, characterized in that: The computer device is used to execute the steps of the target detection and decision-making method based on multimodal fusion as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program to implement the target detection and decision-making method based on multimodal fusion as described in any one of claims 1 to 8.

Images