3D open scene target detection method based on multi-modal information fusion

Abstract

According to the 3D open scene target detection method based on multi-modal information fusion, 3D point cloud data collected by a laser radar, visual image data collected by a visible light camera, and a text instruction which is input or generated based on a visual image and contains semantic information of a to-be-detected target are utilized; and feature extraction and feature fusion of the point cloud data, the image data and the semantic data are carried out respectively, and 3D open scene target detection is realized by using the fused features. According to the invention, through depth feature extraction, accurate alignment and fusion of the laser radar point cloud and the visual image and in combination with an open set target detection strategy, the problems of poor target detection effect and the like caused by limitation of single visual information and poor applicability of a multi-modal fusion scene are solved; according to the invention, accurate detection and positioning of multi-category unknown targets in an open scene are realized, and the accuracy and adaptability of 3D target detection in an open environment are significantly improved.

Description

Technical Field

[0001] This invention relates to a 3D open scene target detection method based on multimodal information fusion, belonging to the field of artificial intelligence technology. Background Technology

[0002] With the rapid development of autonomous driving, intelligent monitoring, and robot navigation, 3D object detection technology, as a core technology for environmental perception, has received widespread attention. Existing 3D object detection methods are mainly divided into two categories: those based on single sensors and those based on multi-sensor fusion. LiDAR can provide high-precision 3D point cloud information and has strong spatial geometric description capabilities, but it is limited by reflection intensity and point cloud sparsity, making it difficult to directly capture the semantic attributes of the target. Conversely, visual sensors (such as RGB cameras) can capture rich semantic and textural information, but lack direct depth information and are easily affected by changes in lighting and weakly textured environments, resulting in limited object detection performance, especially in scale estimation, positioning accuracy, and occlusion handling.

[0003] To address the aforementioned issues, multimodal fusion 3D target detection methods have been proposed both domestically and internationally in recent years. These methods leverage the complementary advantages of LiDAR and visual information to enhance detection performance [1,2]. However, existing multimodal fusion methods primarily focus on detecting known categories in closed scenes (such as fixed urban roads or enclosed areas), lacking the ability to identify unknown and diverse targets in open scenes, and thus failing to meet the general detection needs in complex real-world environments.

[0004] Existing 3D target detection methods based on visual images suffer from significant disadvantages in target scale estimation, localization accuracy, and occlusion handling due to their inability to accurately obtain structural information and depth of the target, and their susceptibility to factors such as ambient lighting. Furthermore, existing multimodal fusion technologies cannot be effectively extended to open scenes and cannot detect various unknown targets, thus limiting their practical applications. Summary of the Invention

[0005] The technical problem solved by this invention is to overcome the shortcomings of the prior art and propose a 3D open scene target detection method based on multimodal information fusion. This method solves the problems of poor target detection effect caused by the limitation of single visual information and poor applicability of multimodal fusion scene, and realizes accurate detection and localization of multi-category and unknown targets in open scenes.

[0006] The technical solution of this invention is: a 3D open scene target detection method based on multimodal information fusion, comprising:

[0007] 3D point cloud data is obtained by simultaneously acquiring data using lidar and visible light cameras. The visual image data I0 is combined and preprocessed to obtain the processed point cloud data. and visual image data I;

[0008] Processed point cloud data Feature extraction is performed on visual image data I, including LiDAR point cloud feature extraction, visual image feature extraction, and semantic feature extraction;

[0009] The extracted features are fused, and the fused features are used to complete the target detection.

[0010] The method utilizes LiDAR and a visible light camera to simultaneously acquire data and obtain 3D point cloud data. And visual image data I0, including:

[0011] Acquiring 3D point cloud data using LiDAR Recorded as Where p i =(x i ,y i ,z i ), where represents the three-dimensional coordinates of the i-th point, and N represents the number of points; the visual image acquired by the visible light camera is denoted as I0.

[0012] The preprocessing includes point cloud data preprocessing and image noise suppression preprocessing.

[0013] The point cloud data preprocessing utilizes outlier detection algorithms based on statistical analysis and density combination to remove outliers, specifically including:

[0014] Outlier detection based on statistical analysis: For any point p in point cloud data... i Calculate the average distance d from the point to its k nearest neighbors. i ;like Then the point is determined to be an outlier, where α is a set coefficient. σ is the global average distance. d Standard deviation;

[0015] Density-based outlier detection: For any point p in point cloud data... i Calculate the number n of points within a radius r. i , as a measure of local density; if n i Less than the set threshold n thresh Then point p i Consider it an outlier;

[0016] After unifying the outliers identified by statistical analysis and density detection, they are uniformly removed to obtain preprocessed point cloud data.

[0017] The visual image preprocessing employs a non-local mean denoising algorithm, which utilizes the similarity information present in the visual image to perform weighted average denoising by calculating the similarity between pixels. Specifically, it includes:

[0018] For each pixel in the visual image, find similar pixel blocks within the search window, and perform a weighted average of the pixels based on the similarity between the pixel blocks to remove noise. The processed visual image is denoted as I1.

[0019] Then, the visual image I1 is normalized using a linear normalization method, normalizing the image pixel values ​​to the [0,1] interval:

[0020]

[0021] Where I(x,y) represents the preprocessed visual image data, and I1(x,y) is the pixel value of the noise-suppressed visual image data at coordinates (x,y). and These are the minimum and maximum pixel values ​​in the visual image data after noise suppression, respectively.

[0022] The point cloud feature extraction of the lidar includes:

[0023] The point cloud data is divided into local regions, and the K-nearest neighbor algorithm based on Euclidean distance is used; for each point p i The k nearest neighbors are found using the K-nearest neighbor algorithm based on Euclidean distance, and the k nearest neighbors are used to find the k nearest neighbors and point p. i Based on this, local regions are constructed; points within each local region are assigned spatial information through location coding, the formula for which is:

[0024]

[0025] Where p i,x ,p i,y ,p i,z It is point p i The coordinate components, d model Here, j represents the dimension index; the input feature matrix is ​​constructed by concatenating the coordinate features of points with their positional codes.

[0026] F ori =Concat(p i PE (pi) )

[0027] Where F ori The input point cloud features are represented by Concat, which indicates a concatenation operation; these point cloud features serve as the initial input to the Transformer module.

[0028] The point cloud features are transformed linearly to obtain the query vector Q, key vector K, and value vector V, as shown in the formula:

[0029] Q = W q F orv K = W k F ori V = W v F ori ;

[0030] Among them W q W k W v The weight matrix is ​​learnable; then the attention score is calculated:

[0031]

[0032] Where d k It is the dimension of the key vector K; The attention score is represented by the value of softmax; the activation function is represented by softmax.

[0033] The output of multi-head attention is obtained by weighted summation:

[0034] F p =Concat(head1,…head) i ,…,head h W o

[0035] in, Q i =W q,i F ori ,K i =W k,i F ori V i =W v, i F ori W q,i W k,i W v,i W o F is a learnable weight matrix, where h is the number of heads; p The feature vectors extracted contain the global and local geometric relationships of the point cloud, which are the features of the LiDAR point cloud.

[0036] The visual image feature extraction employs an improved YOLOv7 network structure, including:

[0037] A cross-stage local network structure is introduced into the backbone network; the input visual image is I∈R C×H×WWhere C is the number of image channels, and H and W are the image spatial dimensions; the input visual image is divided into two parts I = [I1, I2] along the channel dimension and input into CSPNet. The CSPNet network structure formula is expressed as:

[0038] F CSP =F direct +Conv(F split )

[0039] Where F direct =I1 is a feature of direct connection, F split =I2 represents the features that need to undergo convolution after partitioning, Conv represents the convolution operation, and F CSP This represents the feature map output by the CSPNet network;

[0040] A spatial pyramid pooling-Fast layer is introduced into the neck network, with the input feature map being F. CSP SPPF obtains contextual features at different scales through parallel max-pooling operations:

[0041] F pool1 =MaxPool 5×5 (F CSP ),

[0042] F pool2 =MaxPool 9×9 (F CSP ),

[0043] F pool3 =MaxPool 13×13 (F CSP )

[0044] In the formula, MaxPool represents max pooling, F pool1 ,F pool2 ,F pool3 The output feature map F is obtained by concatenating the results of max pooling operations at different scales and then performing a convolution operation. i The formula is:

[0045] F i =Conv.Concat(F pool1 ,F pool2 ,F pool3 ) /

[0046] F i The final output is a feature vector containing visual image information, i.e., visual image features.

[0047] The semantic feature extraction is performed using the Text Encoder network in the BLIP pre-trained model, including:

[0048] For the input visual image, if a target to be detected is specified, the instruction containing the target is used as the input to the Text Encoder network; otherwise, if no target is specified, the image-text generation module in the BLIP pre-trained model automatically generates scene description text. This scene description text is then input into the Text Encoder for encoding. The Text Encoder first segments the text into tokens, converting them into word sequences, and adds positional encoding to preserve the text's order information. The word sequences are then processed through a multi-layer Transformer encoder, with each layer containing a self-attention mechanism and a feedforward neural network, capable of capturing the semantic structure and contextual relationships in the text. Finally, the text feature vector F is extracted. sem As a representation of semantic features, that is, semantic features.

[0049] The step of fusing the extracted features and using the fused features to complete target detection includes:

[0050] F p and visual image features F i The input is a multimodal feature fusion network, which employs a gated fusion mechanism. The fusion ratio of the two features is dynamically adjusted using a learnable gate vector g, resulting in the initially fused feature F. fusion1 :

[0051] F fusion1 =g⊙F p +(1-g)⊙F i ;

[0052] In the formula, ⊙ represents the Hadama product;

[0053] The features F after initial fusion fusion1 With semantic features F sem An attention-based matching method is used to calculate F. fusion1 With F sem Attention weights between Where softmax represents the activation function, the superscript T indicates the transpose operation, and d sem F represents sem The two features are weighted and fused using attention weights to obtain the final fused feature F. final :

[0054] F final =w⊙F fusion1 +(1-w)⊙F sem ;

[0055] The final fusion feature F finalThe input detection head consists of a classification branch and a regression branch; the classification branch uses the focus loss function Li. cls For category prediction, the regression branch uses the smoothed L1 loss function L. reg Perform 3D position and size regression; the total loss function is:

[0056] L = L cls +λL reg ;

[0057] The network parameters were optimized using the backpropagation algorithm. Data augmentation and learning rate adjustment strategies were employed during training to enhance the model's generalization ability and accelerate the training process. The result was a well-trained 3D open-scene object detection network model based on multimodal information. The input to this network model was the multimodal fusion feature F. final The output is the target detection result, including the target category and the coordinates of the center point, length and width of the corresponding target detection box.

[0058] The advantages of this invention compared to the prior art are:

[0059] (1) Information complementarity improves detection accuracy.

[0060] By deeply integrating the 3D geometric information and visual semantic information of LiDAR, the advantages of each modality are fully utilized, effectively overcoming the shortcomings of a single modality in terms of depth, semantics, or environmental adaptability, and achieving accurate positioning and recognition of targets in complex scenes.

[0061] (2) Support diverse target detection in open scenarios

[0062] By introducing an open set learning mechanism to expand the detection category boundary, it can identify unseen categories and unknown targets, thus solving the shortcomings of existing multimodal 3D target detection methods that are limited by closed scenes and category restrictions, and improving the system's versatility and robustness.

[0063] (3) Improve environmental adaptability and robustness

[0064] By fusing visual and lidar information, the dependence of single-modal detection on changes in lighting, occlusion, and weak texture environments is reduced, ensuring good detection performance even under complex lighting conditions and occlusion.

[0065] (4) The technical solution has a reasonable structure and is easy to implement.

[0066] It adopts a modular design, covering data preprocessing, feature extraction, alignment, fusion and open set detection. The algorithm flow is clear, and the fusion mechanism adopts an advanced attention mechanism, which balances detection effect and computational efficiency, making it suitable for real-time system deployment.

[0067] In summary, the 3D open scene target detection method proposed in this invention, through the extraction, precise alignment and fusion of depth features from LiDAR point clouds and visual images, combined with an open set target detection strategy, significantly improves the accuracy and adaptability of 3D target detection in open environments. It can meet the diverse practical application needs of autonomous driving, robot navigation and other applications, and has high innovation and practical value. Attached Figure Description

[0068] Figure 1 This is a flowchart of a 3D open scene target detection method based on multimodal information fusion. Detailed Implementation

[0069] like Figure 1 As shown, this invention relates to a 3D open scene target detection method based on multimodal information fusion, comprising:

[0070] 3D point cloud data is obtained by simultaneously acquiring data using lidar and visible light cameras. The visual image data I0 is combined and preprocessed to obtain the processed point cloud data. and visual image data I;

[0071] Processed point cloud data Feature extraction is performed on visual image data I, including LiDAR point cloud feature extraction, visual image feature extraction, and semantic feature extraction;

[0072] The extracted features are fused, and the fused features are used to complete the target detection.

[0073] The method utilizes LiDAR and a visible light camera to simultaneously acquire data and obtain 3D point cloud data. And visual image data I0, including:

[0074] Acquiring 3D point cloud data using LiDAR Recorded as Where p i =(x i ,y i ,z i ), where I represents the three-dimensional coordinates of the i-th point, and N represents the number of points; the visual image acquired by the visible light camera is denoted as I_0.

[0075] The preprocessing includes point cloud data preprocessing and image noise suppression preprocessing.

[0076] The point cloud data preprocessing utilizes outlier detection algorithms based on statistical analysis and density combination to remove outliers, specifically including:

[0077] Outlier detection based on statistical analysis: For any point p in point cloud data... i Calculate the average distance d from the point to its k nearest neighbors. i ;like Then the point is determined to be an outlier, where α is a set coefficient. σ is the global average distance. d Standard deviation;

[0078] Density-based outlier detection: For any point p in point cloud data... i Calculate the number n of points within a radius r. i , as a measure of local density; if n i Less than the set threshold n thresh Then point p i Consider it an outlier;

[0079] After unifying the outliers identified by statistical analysis and density detection, they are uniformly removed to obtain preprocessed point cloud data.

[0080] The visual image preprocessing employs a non-local mean denoising algorithm, which utilizes the similarity information present in the visual image to perform weighted average denoising by calculating the similarity between pixels. Specifically, it includes:

[0081] For each pixel in the visual image, find similar pixel blocks within the search window, and perform a weighted average of the pixels based on the similarity between the pixel blocks to remove noise. The processed visual image is denoted as I1.

[0082] Then, the visual image I1 is normalized using a linear normalization method, normalizing the image pixel values ​​to the [0,1] interval:

[0083]

[0084] Where I(x,y) represents the preprocessed visual image data, and I1(x,y) is the pixel value of the noise-suppressed visual image data at coordinates (x,y). and These are the minimum and maximum pixel values ​​in the visual image data after noise suppression, respectively.

[0085] The point cloud feature extraction of the lidar includes:

[0086] The point cloud data is divided into local regions, and the K-nearest neighbor algorithm based on Euclidean distance is used; for each point p i The k nearest neighbors are found using the K-nearest neighbor algorithm based on Euclidean distance, and the k nearest neighbors are used to find the k nearest neighbors and point p. iBased on this, local regions are constructed; points within each local region are assigned spatial information through location coding, the formula for which is:

[0087]

[0088] Where p i,x ,p i,y ,p i,z It is point p i The coordinate components, d model Here, j represents the dimension index; the input feature matrix is ​​constructed by concatenating the coordinate features of points with their positional codes.

[0089]

[0090] Where F ori The input point cloud features are represented by Concat, which indicates a concatenation operation; these point cloud features serve as the initial input to the Transformer module.

[0091] The point cloud features are transformed linearly to obtain the query vector Q, key vector K, and value vector V, as shown in the formula:

[0092] Q = W q F orv K = W k F ori V = W v F ori ;

[0093] Among them W q W k W v The weight matrix is ​​learnable; then the attention score is calculated:

[0094]

[0095] Where d k It is the dimension of the key vector K; The attention score is represented by the value of softmax; the activation function is represented by softmax.

[0096] The output of multi-head attention is obtained by weighted summation:

[0097] F p =Concat(head1,…head) i ,…,head h W o

[0098] in, Q i =W q,i F ori ,K i =Wk,i F ori V i =W v, i F ori W q,i W k,i W v,i W o F is a learnable weight matrix, where h is the number of heads; p The feature vectors extracted contain the global and local geometric relationships of the point cloud, which are the features of the LiDAR point cloud.

[0099] The visual image feature extraction employs an improved YOLOv7 network structure, including:

[0100] A cross-stage local network structure is introduced into the backbone network; the input visual image is I∈R C×H×W Where C is the number of image channels, and H and W are the image spatial dimensions; the input visual image is divided into two parts I = [I1, I2] along the channel dimension and input into CSPNet. The CSPNet network structure formula is expressed as:

[0101] F CSP =F direct +Conv(F split )

[0102] Where F direct =I1 is a feature of direct connection, F split =I2 represents the features that need to undergo convolution after partitioning, Conv represents the convolution operation, and F CSP This represents the feature map output by the CSPNet network;

[0103] A spatial pyramid pooling-Fast layer is introduced into the neck network, with the input feature map being F. CSP SPPF obtains contextual features at different scales through parallel max-pooling operations:

[0104] F pool1 =MaxPool 5×5 (F CSP ),

[0105] F pool2 =MaxPool 9×9 (F CSP ),

[0106] F pool3 =MaxPool 13×13 (F CSP )

[0107] In the formula, MaxPool represents max pooling, Fpool1 ,F pool2 ,F pool3 The output feature map F is obtained by concatenating the results of max pooling operations at different scales and then performing a convolution operation. i The formula is:

[0108] F i =Conv.Concat(F pool1 ,F pool2 ,F pool3 ) /

[0109] F i The final output is a feature vector containing visual image information, i.e., visual image features.

[0110] The semantic feature extraction is performed using the Text Encoder network in the BLIP pre-trained model, including:

[0111] For the input visual image, if a target to be detected is specified, the instruction containing the target is used as the input to the Text Encoder network; otherwise, if no target is specified, the image-text generation module in the BLIP pre-trained model automatically generates scene description text. This scene description text is then input into the Text Encoder for encoding. The Text Encoder first segments the text into tokens, converting them into word sequences, and adds positional encoding to preserve the text's order information. The word sequences are then processed through a multi-layer Transformer encoder, with each layer containing a self-attention mechanism and a feedforward neural network, capable of capturing the semantic structure and contextual relationships in the text. Finally, the text feature vector F is extracted. sem As a representation of semantic features, that is, semantic features.

[0112] The step of fusing the extracted features and using the fused features to complete target detection includes:

[0113] F p and visual image features F i The input is a multimodal feature fusion network, which employs a gated fusion mechanism. The fusion ratio of the two features is dynamically adjusted using a learnable gate vector g, resulting in the initially fused feature F. fusion1 :

[0114] F fusion1 =g⊙F p +(1-g)⊙F i ;

[0115] In the formula, ⊙ represents the Hadama product;

[0116] The features F after initial fusion fusion1 With semantic features F sem An attention-based matching method is used to calculate F. fusion1 With F sem Attention weights between Where softmax represents the activation function, the superscript T indicates the transpose operation, and d sem F represents sem The two features are weighted and fused using attention weights to obtain the final fused feature F. final :

[0117] F final =w⊙F fusion1 +(1-w)⊙F sem ;

[0118] The final fusion feature F final The input detection head consists of a classification branch and a regression branch; the classification branch uses the focus loss function Li. cls For category prediction, the regression branch uses the smoothed L1 loss function L. reg Perform 3D position and size regression; the total loss function is:

[0119] L = L cls +λL reg ;

[0120] The network parameters were optimized using the backpropagation algorithm. Data augmentation and learning rate adjustment strategies were employed during training to enhance the model's generalization ability and accelerate the training process. The result was a well-trained 3D open-scene object detection network model based on multimodal information. The input to this network model was the multimodal fusion feature F. final The output is the target detection result, including the target category and the coordinates of the center point, length and width of the corresponding target detection box.

[0121] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications to the technical solutions of the present invention based on the above-disclosed technical content without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall fall within the protection scope of the technical solutions of the present invention.

Claims

1. A 3D open scene target detection method based on multimodal information fusion, characterized in that, include: 3D point cloud data is obtained by simultaneously acquiring data using lidar and visible light cameras. The visual image data I0 is preprocessed to obtain the processed point cloud data. and visual image data I; Processed point cloud data Feature extraction is performed on visual image data I, including LiDAR point cloud feature extraction, visual image feature extraction, and semantic feature extraction; The extracted features are fused, and the fused features are used to complete the target detection.

2. The 3D open scene target detection method based on multimodal information fusion according to claim 1, characterized in that, The method utilizes LiDAR and a visible light camera to simultaneously acquire data and obtain 3D point cloud data. And visual image data I0, including: Acquiring 3D point cloud data using LiDAR Recorded as Where p i =(x i ,y i ,z i ), where represents the three-dimensional coordinates of the i-th point, and N represents the number of points; the visual image acquired by the visible light camera is denoted as I0.

3. The 3D open scene target detection method based on multimodal information fusion according to claim 2, characterized in that, The preprocessing includes point cloud data preprocessing and image noise suppression preprocessing.

4. The 3D open scene target detection method based on multimodal information fusion according to claim 3, characterized in that, The point cloud data preprocessing utilizes outlier detection algorithms based on statistical analysis and density combination to remove outliers, including: Outlier detection based on statistical analysis: For any point p in point cloud data... i Calculate the average distance d from the point to its k nearest neighbors. i ;like Then the point is determined to be an outlier, where α is a set coefficient. σ is the global average distance. d Standard deviation; Density-based outlier detection: For any point p in point cloud data... i Calculate the number n of points within a radius r. i , as a measure of local density; if n i Less than the set threshold n thresh Then point p i Consider it an outlier; After unifying the outliers identified by statistical analysis and density detection, they are uniformly removed to obtain preprocessed point cloud data.

5. The 3D open scene target detection method based on multimodal information fusion according to claim 3, characterized in that, The visual image preprocessing employs a non-local mean denoising algorithm, which utilizes similar structural information present in the visual image to perform weighted average denoising by calculating the similarity between pixels, including: For each pixel in the visual image, find similar pixel blocks within the search window, and perform a weighted average of the pixels based on the similarity between the pixel blocks to remove noise. The processed visual image is denoted as I1. Then, the visual image I1 is normalized using a linear normalization method, normalizing the image pixel values ​​to the [0,1] interval: Where I(x,y) represents the preprocessed visual image data, and I1(x,y) is the pixel value of the noise-suppressed visual image data at coordinates (x,y). and These are the minimum and maximum pixel values ​​in the visual image data after noise suppression, respectively.

6. The 3D open scene target detection method based on multimodal information fusion according to claim 3, characterized in that, The point cloud feature extraction of the lidar includes: The point cloud data is divided into local regions, and the K-nearest neighbor algorithm based on Euclidean distance is used; for each point p i The k nearest neighbors are found using the K-nearest neighbor algorithm based on Euclidean distance, and the k nearest neighbors are used to find the k nearest neighbors and point p. i Based on this, local regions are constructed; points within each local region are assigned spatial information through location coding, the formula for which is: where p i,x , p i,y , p i,z are the coordinate components of point p i , d model is the feature dimension, and j represents the dimension index; the input feature matrix is composed of the coordinate features of points and position encoding: Where F ori The input point cloud features are represented by Concat, which indicates a concatenation operation; these point cloud features serve as the initial input to the Transformer module. The point cloud features are transformed linearly to obtain the query vector Q, key vector K, and value vector V, as shown in the formula: Q=W q F orv ,K=W k F ori ,V=W v F ori ; Among them W q W k W v The weight matrix is ​​learnable; then the attention score is calculated: Where d k It is the dimension of the key vector K; The attention score is represented by the value of softmax; the activation function is represented by softmax. The output of multi-head attention is obtained by weighted summation: F p =Concat(head1,…head i ,…,head h )W o in, Q i =W q,i F ori ,K i =W k,i F ori V i =W v,i F ori W q,i W k,i W v,i W o F is a learnable weight matrix, where h is the number of heads; p The feature vectors extracted contain the global and local geometric relationships of the point cloud, which are the features of the LiDAR point cloud.

7. The 3D open scene target detection method based on multimodal information fusion according to claim 3, characterized in that, The visual image feature extraction employs an improved YOLOv7 network structure, including: A cross-stage local network structure is introduced into the backbone network; the input visual image is I∈R C×H×W Where C is the number of image channels, and H and W are the image spatial dimensions; the input visual image is divided into two parts I = [I1, I2] along the channel dimension and input into CSPNet. The CSPNet network structure formula is expressed as: F CSP =F direct +Conv(F split ) Where F direct =I1 is a feature of direct connection, F split =I2 represents the features that need to undergo convolution after partitioning, Conv represents the convolution operation, and F CSP This represents the feature map output by the CSPNet network. A spatial pyramid pooling-Fast layer is introduced into the neck network, with the input feature map being F. CSP SPPF obtains contextual features at different scales through parallel max-pooling operations: F pool1 =MaxPool 5×5 (F CSP ), F pool2 =MaxPool 9×9 (F CSP ), F pool3 =MaxPool 13×13 (F CSP ) In the formula, MaxPool represents max pooling, F pool1 ,F pool2 ,F pool3 The output feature map F is obtained by concatenating the results of max pooling operations at different scales and then performing a convolution operation. i The formula is: F i =Conv.Concat(F pool1 ,F pool2 ,F pool3 ) / F i The final output is a feature vector containing visual image information, i.e., visual image features.

8. The 3D open scene target detection method based on multimodal information fusion according to claim 3, characterized in that, The semantic feature extraction is performed using the Text Encoder network in the BLIP pre-trained model, including: For the input visual image, if a target to be detected is specified, the instruction containing the target is used as the input to the Text Encoder network; otherwise, if no target is specified, the image-text generation module in the BLIP pre-trained model automatically generates scene description text. This scene description text is then input into the Text Encoder for encoding. The Text Encoder first segments the text into tokens, converting them into word sequences, and adds positional encoding to preserve the text's order information. The word sequences are then processed through a multi-layer Transformer encoder, with each layer containing a self-attention mechanism and a feedforward neural network, capable of capturing the semantic structure and contextual relationships in the text. Finally, the text feature vector F is extracted. sem As a representation of semantic features, that is, semantic features.

9. A 3D open scene target detection method based on multimodal information fusion according to claim 8, characterized in that, The step of fusing the extracted features and using the fused features to complete target detection includes: F p and visual image features F i The input is a multimodal feature fusion network, which employs a gated fusion mechanism. The fusion ratio of the two features is dynamically adjusted using a learnable gate vector g, resulting in the initially fused feature F. fusion1 : F fusion1 =g⊙F p +(1-g)⊙F i ; In the formula, ⊙ represents the Hadamard product; The initial fusion of features F fusion1 With semantic features F sem An attention-based matching method is used to calculate F. fusion1 With F sem Attention weights between Where softmax represents the activation function, the superscript T indicates the transpose operation, and d sem F represents sem The two features are weighted and fused using attention weights to obtain the final fused feature F. final : F final =w⊙F fusion1 +(1-w)⊙F sem ; The final fusion feature F final The input detection head consists of a classification branch and a regression branch; the classification branch uses the focus loss function Li. cls For category prediction, the regression branch uses the smoothed L1 loss function L. reg Perform 3D position and size regression; the total loss function is: L=L cls +λL reg ; The network parameters were optimized using the backpropagation algorithm. Data augmentation and learning rate adjustment strategies were employed during training to enhance the model's generalization ability and accelerate the training process. The result was a well-trained 3D open-scene object detection network model based on multimodal information. The input to this network model was the multimodal fusion feature F. final The output is the target detection result, including the target category and the coordinates of the center point, length and width of the corresponding target detection box.

Images