Multimodal object detection method and system
By improving the YOLOv11 network model and introducing dual-stream backbone feature extraction and cross-modal fusion modules, the accuracy and stability issues of grape fruit detection in complex scenarios were resolved, achieving higher detection accuracy and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SOUTH CENTRAL UNIVERSITY FOR NATIONALITIES
- Filing Date
- 2026-04-27
- Publication Date
- 2026-07-24
AI Technical Summary
Existing technologies lack the accuracy and stability for grape fruit detection in complex scenarios, especially under conditions of changing lighting, foliage obstruction, and complex backgrounds, where traditional single-modal target detection methods struggle to meet robustness requirements.
An improved YOLOv11 network model is adopted, which introduces a dual-stream backbone feature extraction structure, a P3-level mid-term fusion strategy, a C3k2-AG module, and a cross-modal fusion module. Visible light and near-infrared images are processed separately through the dual-stream backbone network, and feature interaction and fusion are performed using the cross-modal fusion module to generate multi-scale feature maps for target detection.
It improves the detection accuracy and stability of grapes in complex scenarios and enhances detection performance under conditions of light variation and foliage obstruction.
Smart Images
Figure CN122454142A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image data processing technology, and in particular to a multimodal target detection method and system. Background Technology
[0002] Object detection is one of the core tasks in the field of computer vision, and it is widely used in scenarios such as agricultural automation and intelligent harvesting. Taking grape fruit detection as an example, factors such as changes in lighting, foliage occlusion, dense fruit distribution, and complex backgrounds in the natural environment pose serious challenges to detection accuracy and stability.
[0003] Traditional single-modal target detection methods based on visible light (RGB) images, while achieving significant progress, are highly susceptible to the influence of lighting conditions and color similarity. For example, in low light, backlighting, or when fruit and leaf colors are similar, RGB images provide limited discriminative features, leading to missed or false detections and failing to meet the robustness requirements of practical production. To overcome the limitations of single-modal methods, researchers have attempted to fuse multimodal data, such as visible light and near-infrared (NIR) images. NIR images provide stable structural contours and moisture information, naturally complementing RGB images. However, most existing multimodal fusion methods primarily employ simple feature integration strategies, such as direct concatenation at the channel level or element-wise addition. These approaches fail to explicitly model the dependencies and semantic correspondences between modalities, resulting in information redundancy and semantic misalignment in the fused features, failing to fully exploit the potential of complementary information.
[0004] Therefore, there is an urgent need for a multimodal target detection method that can improve the detection accuracy and stability of grapes in complex scenarios. Summary of the Invention
[0005] The main objective of this invention is to provide a multimodal target detection method and system, which aims to solve the technical problem of insufficient detection accuracy and stability of grape fruits in complex scenarios in existing technologies.
[0006] To achieve the above objectives, the present invention provides a multimodal target detection method, the method comprising the following steps: Acquire multimodal image data of the scene to be detected, wherein the multimodal image data includes a first modal image and a second modal image; The improved YOLOv11 network model is used to perform target detection on the multimodal image data to obtain target detection results. The improved YOLOv11 network model is a network model obtained by introducing a dual-stream backbone feature extraction structure, a P3-level mid-term fusion strategy, a C3k2-AG module, and a cross-modal fusion module into the YOLOv11 network model. Based on the target detection results, the category and location information of the target in the scene to be detected are determined.
[0007] Optionally, the step of using the improved YOLOv11 network model to perform target detection on the multimodal image data and obtain target detection results includes: The first modal image and the second modal image are respectively input into the dual-stream backbone network of the improved YOLOv11 network model to generate the first modal feature map and the second modal feature map; Extract the first feature map of the P3 level from the first modality feature map, and extract the second feature map of the P3 level from the second modality feature map; The first feature map and the second feature map are input into the cross-modal fusion module to generate a fused feature map; The fused feature map is input into the neck detection network for multi-scale feature aggregation to generate multi-scale aggregated features. The multi-scale aggregated features are input into multiple detection heads for target classification and bounding box regression to obtain target detection results.
[0008] Optionally, the step of inputting the first modality image and the second modality image into the two-stream backbone network of the improved YOLOv11 network model to generate the first modality feature map and the second modality feature map includes: The first modality image is input into the first backbone network, and features are extracted sequentially through the convolutional module, C3k2-AG module and spatial pyramid pooling layer in the first backbone network to generate a first modality feature map containing multiple resolutions. The second modality image is input into the second backbone network, and features are extracted sequentially through the convolutional module, C3k2-AG module and spatial pyramid pooling layer in the second backbone network to generate a second modality feature map containing multiple resolutions.
[0009] Optionally, the cross-modal fusion module is used to perform bidirectional cross-modal attention interaction on the first feature map and the second feature map to generate a fused feature map; The step of performing bidirectional cross-modal attention interaction on the first feature map and the second feature map to generate a fused feature map includes: The first feature map and the second feature map input to the cross-modal fusion module are pooled and serialized respectively to generate a first token sequence and a second token sequence; Using the first token sequence as the query and the second token sequence as the key and value, a first cross-modal attention interaction is performed to generate a first enhanced sequence; Using the second token sequence as the query and the first token sequence as the key and value, a second cross-modal attention interaction is performed to generate a second enhanced sequence; The first enhancement sequence and the second enhancement sequence are respectively restored to two-dimensional feature maps, and residual fusion is performed with the corresponding original modal feature maps to obtain the first fused feature and the second fused feature. The first fusion feature and the second fusion feature are spliced together, and channel compression is performed through a convolutional layer to obtain a fusion feature map.
[0010] Optionally, before the step of acquiring the multimodal image data of the scene to be detected, the method further includes: Obtain a multimodal target detection dataset, which includes paired visible light images and near-infrared images; The YOLOv11 network model is improved to obtain an initial YOLOv11 network model, which includes a dual-stream backbone feature extraction structure, a P3-level mid-term fusion strategy, a C3k2-AG module, and a cross-modal fusion module. The initial YOLOv11 network model is trained based on the multimodal object detection dataset to obtain the training results; Based on the training results, the model parameters of the initial YOLOv11 network model are optimized to obtain an improved YOLOv11 network model.
[0011] Optionally, the step of improving the YOLOv11 network model to obtain an initial YOLOv11 network model includes: The single-backbone feature extraction network of the YOLOv11 network model is extended into a first backbone network and a second backbone network set in parallel to obtain the first intermediate YOLOv11 network model. A mid-term fusion node is set at the P3 level of the feature pyramid of the first intermediate YOLOv11 network model to obtain the second intermediate YOLOv11 network model. Replace the C3k2 module in the second intermediate YOLOv11 network model with the C3k2-AG module to obtain the third intermediate YOLOv11 network model. A cross-modal fusion module is embedded at the feature fusion position of the third intermediate YOLOv11 network model to obtain the initial YOLOv11 network model.
[0012] Optionally, the C3k2-AG module is used to perform local-global feature enhancement and channel gating filtering on the input features to obtain the output features of the C3k2-AG module; The step of performing local-global feature enhancement and channel gating filtering on the input features to obtain the output features of the C3k2-AG module includes: The input features are split into first-branch features and second-branch features; After performing convolution and activation processing on the first branch features, local spatial correlation is enhanced through depthwise separable convolution to generate local enhanced features; The local enhanced features are modeled globally in terms of spatial and channel dimensions using an additive self-attention mechanism to generate attention-enhanced features. The attention enhancement features are adaptively filtered by a convolutional gating unit to generate gated enhancement features; The gating enhancement feature is spliced and fused with the second branch feature to obtain the output feature of the C3k2-AG module.
[0013] Optionally, the step of adaptively filtering the channel features of the attention enhancement features through a convolutional gating unit to generate gated enhancement features includes: The attention enhancement features are divided into candidate branch features and gated branch features; Spatial features are extracted from the candidate branch features using 1×1 convolution to obtain candidate features; The gated branch features are processed using deep convolution and the Sigmoid activation function to generate gated weights; The candidate features are multiplied by the gating weights to obtain gating features, and the gating features are residually fused with the attention enhancement features to obtain gating enhancement features.
[0014] Optionally, after the step of optimizing the model parameters of the initial YOLOv11 network model based on the training results to obtain an improved YOLOv11 network model, the method further includes: The improved YOLOv11 network model was evaluated using preset model evaluation metrics to obtain model evaluation results. Based on the model evaluation results, the improved YOLOv11 network model that passed the evaluation was determined; The improved YOLOv11 network model that passed the evaluation is accelerated by the inference optimization engine to generate an inference engine file.
[0015] Furthermore, to achieve the above objectives, the present invention also proposes a multimodal target detection system, the system comprising: The data acquisition module is used to acquire multimodal image data of the scene to be detected, wherein the multimodal image data includes a first modal image and a second modal image; The target detection module is used to perform target detection on the multimodal image data using the improved YOLOv11 network model to obtain target detection results. The improved YOLOv11 network model is a network model obtained by introducing a dual-stream backbone feature extraction structure, a P3-level mid-term fusion strategy, a C3k2-AG module, and a cross-modal fusion module into the YOLOv11 network model. The result determination module is used to determine the category information and location information of the target in the scene to be detected based on the target detection result.
[0016] Furthermore, to achieve the above objectives, the present invention also proposes a multimodal target detection device, the device comprising: a memory, a processor, and a multimodal target detection program stored in the memory and executable on the processor, the multimodal target detection program being configured to implement the steps of the multimodal target detection method described above.
[0017] Furthermore, to achieve the above objectives, the present invention also proposes a storage medium storing a multimodal target detection program, which, when executed by a processor, implements the steps of the multimodal target detection method described above.
[0018] This invention discloses a method for acquiring multimodal image data of a scene to be detected, including a first modality image and a second modality image. An improved YOLOv11 network model is used to perform target detection on the multimodal image data to obtain target detection results. The improved YOLOv11 network model is obtained by introducing a dual-stream backbone feature extraction structure, a P3-level mid-stage fusion strategy, a C3k2-AG module, and a cross-modal fusion module into the YOLOv11 network model. Based on the target detection results, the category and location information of the target in the scene to be detected are determined. Because this invention improves the YOLOv11 network model by introducing a dual-stream backbone feature extraction structure, a P3-level mid-stage fusion strategy, a C3k2-AG module, and a cross-modal fusion module, and then uses the improved YOLOv11 network model for target detection, compared to existing technologies, this invention improves the detection accuracy and stability of grapes in complex scenes. Attached Figure Description
[0019] Figure 1 This is a flowchart illustrating the first embodiment of the multimodal target detection method of the present invention; Figure 2 This is a schematic diagram of the overall model structure in the multimodal target detection method of the present invention; Figure 3 This is a flowchart illustrating the second embodiment of the multimodal target detection method of the present invention; Figure 4This is a comparative diagram of the mid-term fusion methods in the multimodal target detection method of the present invention; Figure 5 This is a schematic diagram of the Additive-CGLU structure in the multimodal target detection method of the present invention; Figure 6 This is a schematic diagram of the cross-modal fusion module structure in the multimodal target detection method of the present invention; Figure 7 This is a flowchart illustrating the third embodiment of the multimodal target detection method of the present invention; Figure 8 This is a structural block diagram of the first embodiment of the multimodal target detection system of the present invention; Figure 9 This is a schematic diagram of the structure of a multimodal target detection device in the hardware operating environment involved in the embodiments of the present invention.
[0020] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0021] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.
[0022] This invention provides a multimodal target detection method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the multimodal target detection method of the present invention.
[0023] In this embodiment, the multimodal target detection method includes steps S10 to S30: Step S10: Obtain multimodal image data of the scene to be detected, wherein the multimodal image data includes a first modal image and a second modal image.
[0024] It should be noted that the executing entity in this embodiment can be a computer server device with data processing, network communication, and program execution functions applied in agricultural visual inspection scenarios, such as a server, tablet computer, or personal computer, or an electronic device capable of performing the above functions (such as a multimodal target detection device). The following uses a system including a multimodal target detection device (hereinafter referred to as the system) as an example to illustrate this embodiment and the following embodiments.
[0025] Understandably, the scene to be detected can refer to a target area in an agricultural environment such as an orchard or greenhouse, for example, a vineyard. The first modal image can be a visible light (RGB) image, and the second modal image can be a near-infrared (NIR) image. Multimodal image data can be acquired simultaneously by an imaging device equipped with visible light and near-infrared sensors.
[0026] In specific implementations, after acquiring the multimodal image data of the scene to be detected, in order to ensure that the image quality meets the model input requirements, the multimodal image data can also be preprocessed. The preprocessing operations may include, but are not limited to: size normalization (e.g., uniformly scaling to 640×640 pixels), contrast enhancement, and noise reduction.
[0027] Step S20: Use the improved YOLOv11 network model to perform target detection on the multimodal image data and obtain the target detection result. The improved YOLOv11 network model is a network model obtained by introducing a dual-stream backbone feature extraction structure, a P3-level mid-term fusion strategy, a C3k2-AG module and a cross-modal fusion module into the YOLOv11 network model.
[0028] It should be noted that the dual-stream backbone feature extraction structure refers to two parallel feature extraction network branches, which are used to process the first modality image and the second modality image respectively, so as to preserve the differentiated feature expressions of each modality.
[0029] It should be understood that in multimodal object detection tasks, choosing an appropriate fusion level is one of the key factors affecting model performance and computational efficiency. Common multimodal fusion strategies mainly include three forms: early fusion, mid-stage fusion, and late-stage fusion, each with its unique advantages. Early fusion integrates data at the raw data level, capturing complementary information between different modes from the outset. Mid-stage fusion, performed after feature extraction, enhances feature representation. Late-stage fusion integrates the detection features for object detection after each modality has independently completed feature extraction.
[0030] Furthermore, while early fusion methods are simple to implement, the differences in imaging mechanisms and noise distribution among different modalities can easily introduce redundant information and modal interference, hindering the network's effective modeling of high-level semantic features. Later fusion, although reducing inter-modal interference, often fails to fully utilize the complementary advantages of multimodal features at mid-to-low levels, especially limiting its performance in small target and occluded scenarios. Mid-term fusion strategies, by introducing cross-modal interaction during feature extraction, can achieve effective semantic complementarity while maintaining the independence of each modal feature, representing a reasonable trade-off between performance and efficiency.
[0031] It should be noted that the above-mentioned P3-level mid-term fusion strategy refers to performing a single cross-modal fusion at the P3 level corresponding to 1 / 8 resolution in the feature pyramid network. This level has a good balance between spatial resolution and semantic abstraction.
[0032] Furthermore, the feature map corresponding to the P3 layer retains a high spatial resolution, which is beneficial for capturing the local boundary and structural information of small-scale, densely distributed targets such as grapes. At the same time, compared with shallow features, the P3 layer already contains a certain degree of semantic information, which helps the model distinguish between fruits and complex backgrounds such as branches and leaves, thereby improving the stability and robustness of detection.
[0033] It should be noted that the C3k2-AG module is an improved feature extraction unit based on the original C3k2 module of the YOLOv11 network model. It integrates additive self-attention mechanism and convolutional gating unit (CGLU) to enhance local-global feature modeling and channel selection capabilities.
[0034] Understandably, the introduction of the C3k2-AG module aims to alleviate the limitations of the original C3k2 module in simultaneously fusing global and local features in object detection tasks.
[0035] It should be noted that the Transformer Fusion Block (TFB) establishes an explicit bidirectional interaction relationship between visible light and near-infrared features through an iterative cross-attention mechanism, achieving adaptive alignment and information complementarity between modalities, which significantly enhances the model's efficiency in utilizing complementary information from multi-source data.
[0036] In a specific implementation, the step of using the improved YOLOv11 network model to perform target detection on the multimodal image data and obtain target detection results includes: inputting the first modality image and the second modality image into the dual-stream backbone network of the improved YOLOv11 network model respectively to generate a first modality feature map and a second modality feature map; extracting a first feature map at the P3 level from the first modality feature map and extracting a second feature map at the P3 level from the second modality feature map; inputting the first feature map and the second feature map into the cross-modal fusion module to generate a fused feature map; inputting the fused feature map into the detection neck network for multi-scale feature aggregation to generate multi-scale aggregated features; and inputting the multi-scale aggregated features into multiple detection heads for target classification and bounding box regression to obtain target detection results.
[0037] Understandably, in the original YOLOv11 network model, a single backbone branch can only extract features of one modality. To enable the detector to process both visible light and infrared image features simultaneously, this embodiment extends the backbone, designing a dual-stream feature extraction backbone (i.e., a dual-stream backbone network). For example, refer to... Figure 2 , Figure 2This is a schematic diagram of the overall model structure in the multimodal target detection method of the present invention. The overall model structure is divided into three parts: a dual-stream backbone network, a detection neck network, and a detection head. In the figure, the dual-stream backbone network includes two parallel network branches, a visible light branch and a near-infrared branch (i.e., the first backbone network and the second backbone network), which are composed of a convolutional module CBS (Conv-BN-SiLU), a feature extraction module C3k2-AG, and a spatial pyramid pooling layer SPPF.
[0038] Near-infrared and visible light images are input into the corresponding branches for single-modal feature extraction, and the process can be represented by formula (1).
[0039] (1) in, Let H, W, and C represent the feature maps of the i-th layer (i=3, 4, 5) in the visible light branch and the near-infrared branch, respectively. H, W, and C represent the height, width, and number of channels of the feature map. This indicates the input visible light image and near-infrared image. Represents the feature extraction function. and These are the parameters for the visible light branch and the near-infrared branch, respectively. In the feature extraction stage, multi-scale features are typically used to capture objects of different sizes.
[0040] It should be noted that the step of inputting the first modality image and the second modality image into the dual-stream backbone network of the improved YOLOv11 network model to generate the first modality feature map and the second modality feature map may include: inputting the first modality image into the first backbone network, and sequentially performing feature extraction through the convolutional module, C3k2-AG module and spatial pyramid pooling layer in the first backbone network to generate a first modality feature map containing multiple resolutions; inputting the second modality image into the second backbone network, and sequentially performing feature extraction through the convolutional module, C3k2-AG module and spatial pyramid pooling layer in the second backbone network to generate a second modality feature map containing multiple resolutions.
[0041] The CBS module performs multiple convolution operations on the input information, then batch normalizes it, and finally uses the SiLU activation function to obtain the output result. The C3k2-AG module in this embodiment integrates the core component ideas of ViT (Vision Transformer), improving the token mixer through convolution and optimizing the channel mixer through gating. This retains the global modeling advantages of the Transformer while maintaining the lightweight characteristics of CNNs, thus enabling the C3k2-AG module to capture features of images at different scales.
[0042] Meanwhile, an intermediate fusion method is selected, and the two modal features captured at the P3 level (i.e., the first feature map and the second feature map at the P3 level) are input into the cross-modal fusion module TFB to obtain the fused feature map. The cross-modal feature fusion is as shown in formula (2): (2) in, For the fusion features of the i-th layer, For feature fusion function, These are the parameters of the feature fusion function. In the TFB module, a dual-cross-attention feature fusion method is used to optimize the feature fusion function. Modeling is performed.
[0043] Finally, the fused feature map is input into the neck detection network for multi-scale feature aggregation. Then, three detection heads are used to detect images at three different scales and perform classification and regression to obtain the target detection results. As shown in formula (3): (3) in, Multi-scale feature aggregation function The head function is used for detection, responsible for classification and bounding box regression. For relevant parameters, These represent the classification confidence score and bounding box coordinates output by the detection head, respectively.
[0044] Step S30: Based on the target detection results, determine the category information and location information of the target in the scene to be detected.
[0045] It should be noted that the target detection result can refer to the specific information output by processing multimodal image data through the improved YOLOv11 network model, which describes the target in the image. This information can include at least the target's category and location. In the grape detection scenario, the target category could include green grapes, purple grapes, etc., and the location information is represented by a bounding box, which can precisely define the target's location and extent within the image.
[0046] This embodiment discloses acquiring multimodal image data of a scene to be detected, including a first modal image and a second modal image. An improved YOLOv11 network model is used to perform target detection on the multimodal image data to obtain target detection results. The improved YOLOv11 network model is obtained by introducing a dual-stream backbone feature extraction structure, a P3-level mid-stage fusion strategy, a C3k2-AG module, and a cross-modal fusion module into the YOLOv11 network model. Based on the target detection results, the category and location information of the target in the scene to be detected are determined. Because this embodiment improves the YOLOv11 network model by introducing a dual-stream backbone feature extraction structure, a P3-level mid-stage fusion strategy, a C3k2-AG module, and a cross-modal fusion module, and then uses the improved YOLOv11 network model for target detection, compared to existing technologies, this embodiment improves the detection accuracy and stability of grapes in complex scenes.
[0047] refer to Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the multimodal target detection method of the present invention.
[0048] Based on the first embodiment described above, in this embodiment, before step S10, steps S01 to S04 are further included: Step S01: Obtain a multimodal target detection dataset, which includes paired visible light images and near-infrared images.
[0049] Step S02: Improve the YOLOv11 network model to obtain an initial YOLOv11 network model, which includes a dual-stream backbone feature extraction structure, a P3-level mid-term fusion strategy, a C3k2-AG module, and a cross-modal fusion module.
[0050] Step S03: Train the initial YOLOv11 network model based on the multimodal object detection dataset to obtain the training results.
[0051] Step S04: Optimize the model parameters of the initial YOLOv11 network model based on the training results to obtain an improved YOLOv11 network model.
[0052] In a specific implementation, a publicly available multimodal grape fruit detection dataset can be used as a multimodal target detection dataset. This dataset consists of grape images from real orchard environments, including grapes in both cyan and purple colors, and covers major harvesting scenarios such as front lighting, backlighting, and motion blur.
[0053] It should be noted that the step of improving the YOLOv11 network model to obtain an initial YOLOv11 network model includes: expanding the single-backbone feature extraction network of the YOLOv11 network model into a first backbone network and a second backbone network set in parallel to obtain a first intermediate YOLOv11 network model; setting an intermediate fusion node at the P3 level of the feature pyramid of the first intermediate YOLOv11 network model to obtain a second intermediate YOLOv11 network model; replacing the C3k2 module in the second intermediate YOLOv11 network model with a C3k2-AG module to obtain a third intermediate YOLOv11 network model; and embedding a cross-modal fusion module at the feature fusion position of the third intermediate YOLOv11 network model to obtain the initial YOLOv11 network model.
[0054] It's important to explain that the P3 level corresponds to a 1 / 8 resolution feature map in the feature pyramid. This level strikes a good balance between spatial resolution and semantic abstraction. The P3 feature map retains high spatial resolution, which is beneficial for capturing the local boundary and structural information of small-scale, densely distributed targets like grapes. Simultaneously, compared to shallow features, the P3 level already contains a certain degree of semantic information, helping the model distinguish between fruits and complex backgrounds such as branches and leaves, thereby improving the stability and robustness of the detection.
[0055] For example, refer to Figure 4 , Figure 4 This is a schematic diagram comparing the mid-term fusion methods in the multimodal target detection method of the present invention. Figure 4 (a) indicates that midfusion occurs at multiple levels (P3, P4, P5), while Figure 4 (b) P3-midfusion represents a single cross-modal fusion performed at the P3 level of the feature pyramid (corresponding to 1 / 8 resolution). This embodiment proposes a single mid-fusion at the P3 level, where the fusion occurs at a specific P3 layer. After the backbone extracts feature maps from the visible light and near-infrared images, it passes them to the detection neck network. At the P3 layer, the feature maps from the two modalities are concatenated and processed by a trainable module. This method effectively utilizes P3 layer features, improving detection accuracy and performance while reducing model parameters and computational cost. Fusion at the P3 layer can balance spatial resolution and semantic information, avoiding early noise interference and later information loss.
[0056] It's important to explain that the C3k2-AG module is an improvement on the original C3k2 module, designed to alleviate the limitation of the original C3k2 module in simultaneously fusing global and local features. After processing by the CBS layer, the input features are split into two branches. The first branch features undergo local enhancement through the C3k-AG module, while the second branch retains global features and maintains efficiency through residual connections. In the C3k-AG module, the original BottleNeck module is replaced by the Additive-CGLU module.
[0057] It should be noted that the C3k2-AG module is used to perform local-global feature enhancement and channel gating filtering on the input features to obtain the output features of the C3k2-AG module.
[0058] Further, the step of performing local-global feature enhancement and channel gating filtering on the input features to obtain the output features of the C3k2-AG module may include: splitting the input features into first branch features and second branch features; performing convolution and activation processing on the first branch features, and then enhancing local spatial correlations through depthwise separable convolution to generate local enhanced features; performing global dependency modeling of the spatial and channel dimensions of the local enhanced features through an additive self-attention mechanism to generate attention-enhanced features; performing adaptive channel feature filtering on the attention-enhanced features through a convolutional gating unit to generate gated enhanced features; and concatenating and fusing the gated enhanced features with the second branch features to obtain the output features of the C3k2-AG module.
[0059] For example, refer to Figure 5 , Figure 5 This is a schematic diagram of the Additive-CGLU structure in the multimodal target detection method of this invention. For input features... In the Additive-CGLU module, a 1×1 convolution is first used to integrate the channel dimension. After BatchNorm and GELU activation, a depthwise separable convolution is used to enhance local spatial correlations. Then, a 1×1 convolution is used to restore the original number of channels, and the feature is fused with the input feature residuals to obtain the local enhanced features. Then, a token mixer is designed, constructing spatial attention and channel attention branches to model global dependencies. Unlike the self-attention mechanism of traditional transformers, additive similarity calculation is implemented through convolution operations, avoiding the high complexity of matrix multiplication. A 1x1 convolution maps the input features to query Q, key K, and value V. Q and K are then processed through spatial attention and channel attention operations to obtain spatial weights. and channel weight As shown in Formulas 4 and 5.
[0060] (4) (5) Here, P represents the pooling operation. After fusing the spatial and channel attention, similarity is calculated by addition instead of matrix multiplication. Attention weights are generated through 3×3 depthwise convolution, multiplied by V to obtain the output, and then the fused result is combined with intra-channel features through 3×3 group convolution to obtain attention-enhanced features. Ensure that the output dimension is consistent with the input.
[0061] Furthermore, a convolutional gated unit (CGLU) was designed to replace the traditional channel mixer (MLP), adaptively preserving important channel features through the CGLU. Attention-enhanced features processed by the attention mechanism are then processed... The system is divided into candidate branches and gated branches. Spatial features are extracted from the candidate branches using 1×1 convolutions to obtain candidate features. In the gated branch, local correlations are captured using 3×3 depthwise convolutions, and gate weights are generated using a Sigmoid function. The gated augmentation feature is obtained by multiplying the two branches and then performing residual fusion with the attention-enhanced feature. Important features are dynamically selected through convolutional gating units, thereby enhancing the nonlinear expressive power, as shown in Equations 6 and 7.
[0062] , (6) (7) Finally, the enhanced features are concatenated and output through the CBS layer. The C3k2-AG module enhances attention to key regions through convolutional additive attention and dynamically optimizes feature channels through the CGLU gating mechanism, enabling the network to focus more on key region features and improving the detection performance of grape fruits in complex backgrounds.
[0063] In a specific implementation, the step of adaptively filtering channel features of the attention enhancement features through a convolutional gating unit to generate gated enhancement features includes: dividing the attention enhancement features into candidate branch features and gated branch features; extracting spatial features from the candidate branch features using a 1×1 convolution to obtain candidate features; processing the gated branch features using a depthwise convolution and a sigmoid activation function to generate gate weights; multiplying the candidate features by the gate weights to obtain gated features; and performing residual fusion between the gated features and the attention enhancement features to obtain gated enhancement features.
[0064] It should be noted that the cross-modal fusion module is used to perform bidirectional cross-modal attention interaction on the first feature map and the second feature map to generate a fused feature map.
[0065] Further, the step of performing bidirectional cross-modal attention interaction on the first feature map and the second feature map to generate a fused feature map includes: pooling and serializing the first feature map and the second feature map input to the cross-modal fusion module respectively to generate a first token sequence and a second token sequence; performing a first cross-modal attention interaction using the first token sequence as a query and the second token sequence as a key and value to generate a first enhancement sequence; performing a second cross-modal attention interaction using the second token sequence as a query and the first token sequence as a key and value to generate a second enhancement sequence; restoring the first enhancement sequence and the second enhancement sequence to two-dimensional feature maps respectively, and performing residual fusion with the corresponding original modal feature maps to obtain a first fused feature and a second fused feature; concatenating the first fused feature and the second fused feature, and performing channel compression through a convolutional layer to obtain a fused feature map.
[0066] For example, refer to Figure 6 , Figure 6 This is a schematic diagram of the cross-modal fusion module structure in the multimodal target detection method of the present invention. The cross-modal fusion module (Transformer Fusion Block, TFB) projects the two-dimensional feature map into a token sequence, then uses a bidirectional cross-modal Transformer for attention interaction, and finally restores the interacted sequence to a two-dimensional feature map, and performs residual fusion with the original features. This preserves the local details of each modality while injecting global information from complementary modalities.
[0067] TFB extracts the first feature map from the bi-branch unimodal features. Second feature map Enhancement is performed separately using a cross-modal enhancement module (CEM), followed by cross-modal fusion. First, the extracted 2D feature maps are processed using average pooling and max pooling to obtain downsampled modal features. and The modal features are then flattened into a sequence of tokens, and positional embedding information is added to obtain the first token sequence. Second token sequence Next, we will enhance each of the two branches, taking the near-infrared branch as an example. We will use the first token sequence as the query and the second token sequence as the key and value, as shown in Formula 8: , , (8) in, , , This is the weight matrix.
[0068] A correlation matrix is constructed using dot product operations, and attention weights are obtained by normalization using the softmax function. These weights are then used to weight the Value of another modality to obtain cross-modal information. See Equations 9 to 12: (9) (10) (11) (12) The image is reprojected back to the original space through a nonlinear transformation and added to the input sequence via residual connections. This represents the output weight matrix before the FFN layer. Through processing with a residual structure and a feedforward neural network (FFN), and using learnable parameters to balance the saliency of the two modes, the final enhanced near-infrared branch token sequence is obtained. After enhancement processing, the visible light branch and near-infrared branch are upsampled back to their original spatial size, and residual fusion is performed with the original features. The fused features are then concatenated and compressed back to the original channels using a 1×1 convolution to obtain the fused feature map. Through the cross-modal fusion module TFB, the visible light branch and the near-infrared branch guide and complement each other, ultimately resulting in a more robust fusion characterization.
[0069] This embodiment discloses the acquisition of a multimodal object detection dataset, which includes paired visible light images and near-infrared images; improvement of the YOLOv11 network model to obtain an initial YOLOv11 network model, which includes a two-stream backbone feature extraction structure, a P3-level mid-stage fusion strategy, a C3k2-AG module, and a cross-modal fusion module; training of the initial YOLOv11 network model based on the multimodal object detection dataset to obtain training results; and optimization of the model parameters of the initial YOLOv11 network model based on the training results to obtain an improved YOLOv11 network model. Because this embodiment specifically improves the YOLOv11 network model and trains and optimizes the model, compared to existing technologies, this embodiment ensures the model's detection performance and generalization ability in complex agricultural scenarios.
[0070] refer to Figure 7 , Figure 7 This is a flowchart illustrating the third embodiment of the multimodal target detection method of the present invention.
[0071] Based on the above embodiments, in this embodiment, after step S30, steps S40 to S60 are further included: Step S40: Use preset model evaluation metrics to evaluate the improved YOLOv11 network model and obtain the model evaluation results.
[0072] Step S50: Based on the model evaluation results, determine the improved YOLOv11 network model that has passed the evaluation.
[0073] Step S60: The improved YOLOv11 network model that has passed the evaluation is accelerated by the inference optimization engine to generate an inference engine file.
[0074] It should be noted that the preset model evaluation metrics include: precision P, recall R, mean precision mAP, and floating-point operations GFLOPs.
[0075] It should be noted that ablation experiments were conducted on the multimodal grape dataset to verify the effectiveness of each improved module. Experimental results show that the model fully integrating the dual-stream backbone, P3-level mid-stage fusion, C3k2-AG module, and TFB module (i.e., the improved YOLOv11 network model) achieves 94.9% and 76.2% mAP50 and mAP50-95 metrics, respectively, representing improvements of 1.5% and 3.5% compared to the baseline model (i.e., the YOLOv11 network model).
[0076] It should be noted that the inference optimization engine is TensorRT. To meet the real-time requirements of actual deployment, this embodiment introduces an inference optimization engine to accelerate the model. The core mechanism of TensorRT mainly includes computation graph optimization, layer-level and operator-level fusion, mixed-precision inference, and hardware-aware optimization. First, the model trained and validated in PyTorch is exported to ONNX format to obtain an intermediate representation independent of the deep learning framework. After obtaining the ONNX model, TensorRT parses and constructs the computation graph, automatically reconstructing and optimizing the original computation graph by analyzing the network topology and operator types. Subsequently, TensorRT introduces a precision optimization strategy during the construction of the inference engine, converting some or all operators from FP32 precision to FP16 or INT8 precision. After completing graph optimization and precision configuration, TensorRT generates a high-efficiency inference engine strongly correlated with specific hardware and network structure, and serializes and saves it.
[0077] It should be noted that the average inference time per frame for the MAGT-YOLO model before acceleration (i.e., the improved YOLOv11 network model) was 24.69 ms, corresponding to an inference speed of 40.5 fps. After TensorRT acceleration, the average inference time was reduced to 3.28 ms, and the inference speed reached 304.84 fps, an improvement of approximately 7.5 times. Simultaneously, while significantly improving inference speed, TensorRT acceleration did not significantly impact the model's detection performance. The accelerated engine file (i.e., the inference engine file) achieved an mAP50 of 94.6% on the test set, a decrease of only about 0.3% compared to the unoptimized model, indicating that the accuracy loss is within an acceptable range. This demonstrates that TensorRT can achieve a good balance between speed and accuracy in mixed-precision inference and operator fusion. Through TensorRT optimization, the model's running efficiency is significantly improved while maintaining stable accuracy, which helps to promote the practical application of multimodal grape fruit detection algorithms.
[0078] This embodiment discloses a method for evaluating the improved YOLOv11 network model using preset model evaluation metrics to obtain model evaluation results; based on the model evaluation results, a qualified improved YOLOv11 network model is determined; and the qualified improved YOLOv11 network model is accelerated through an inference optimization engine to generate an inference engine file. Because this embodiment uses preset model evaluation metrics to evaluate the improved YOLOv11 network model and then accelerates the qualified improved YOLOv11 network model through an inference optimization engine, compared to existing technologies, this embodiment not only ensures the reliability and effectiveness of model deployment but also improves model inference efficiency.
[0079] Furthermore, this embodiment of the invention also proposes a storage medium storing a multimodal target detection program, which, when executed by a processor, implements the steps of the multimodal target detection method described above.
[0080] Reference Figure 8 , Figure 8 This is a structural block diagram of the first embodiment of the multimodal target detection system of the present invention.
[0081] like Figure 8 As shown, the multimodal target detection system proposed in this embodiment of the invention includes: a data acquisition module 801, a target detection module 802, and a result determination module 803.
[0082] The data acquisition module 801 is used to acquire multimodal image data of the scene to be detected, the multimodal image data including a first modal image and a second modal image.
[0083] The target detection module 802 is used to perform target detection on the multimodal image data using the improved YOLOv11 network model to obtain target detection results. The improved YOLOv11 network model is a network model obtained by introducing a dual-stream backbone feature extraction structure, a P3-level mid-term fusion strategy, a C3k2-AG module, and a cross-modal fusion module into the YOLOv11 network model.
[0084] The result determination module 803 is used to determine the category information and location information of the target in the scene to be detected based on the target detection result.
[0085] The target detection module 802 is further configured to input the first modality image and the second modality image into the dual-stream backbone network of the improved YOLOv11 network model, respectively, to generate a first modality feature map and a second modality feature map; extract a first feature map at the P3 level from the first modality feature map, and extract a second feature map at the P3 level from the second modality feature map; input the first feature map and the second feature map into the cross-modal fusion module to generate a fused feature map; input the fused feature map into the detection neck network for multi-scale feature aggregation to generate multi-scale aggregated features; and input the multi-scale aggregated features into multiple detection heads for target classification and bounding box regression to obtain target detection results.
[0086] The target detection module 802 is further configured to input the first modality image into a first backbone network, and sequentially extract features through a convolutional module, a C3k2-AG module, and a spatial pyramid pooling layer in the first backbone network to generate a first modality feature map containing multiple resolutions; and input the second modality image into a second backbone network, and sequentially extract features through a convolutional module, a C3k2-AG module, and a spatial pyramid pooling layer in the second backbone network to generate a second modality feature map containing multiple resolutions.
[0087] This system embodiment discloses the acquisition of multimodal image data of a scene to be detected, including a first modal image and a second modal image. An improved YOLOv11 network model is used to perform target detection on the multimodal image data to obtain target detection results. The improved YOLOv11 network model is obtained by introducing a dual-stream backbone feature extraction structure, a P3-level mid-stage fusion strategy, a C3k2-AG module, and a cross-modal fusion module into the YOLOv11 network model. Based on the target detection results, the category and location information of the target in the scene to be detected are determined. Because this system embodiment improves the YOLOv11 network model by introducing a dual-stream backbone feature extraction structure, a P3-level mid-stage fusion strategy, a C3k2-AG module, and a cross-modal fusion module, and then uses the improved YOLOv11 network model for target detection, compared to existing technologies, this system embodiment improves the detection accuracy and stability of grapes in complex scenes.
[0088] Based on the first embodiment of the multimodal target detection system of the present invention described above, a second embodiment of the multimodal target detection system of the present invention is proposed.
[0089] In this embodiment, the data acquisition module 801 is further configured to acquire a multimodal target detection dataset, which includes paired visible light images and near-infrared images; improve the YOLOv11 network model to obtain an initial YOLOv11 network model, which includes a two-stream backbone feature extraction structure, a P3-level mid-stage fusion strategy, a C3k2-AG module, and a cross-modal fusion module; train the initial YOLOv11 network model based on the multimodal target detection dataset to obtain training results; and optimize the model parameters of the initial YOLOv11 network model based on the training results to obtain an improved YOLOv11 network model.
[0090] The data acquisition module 801 is further configured to expand the single-backbone feature extraction network of the YOLOv11 network model into a first backbone network and a second backbone network set in parallel to obtain a first intermediate YOLOv11 network model; set an intermediate fusion node at the P3 level of the feature pyramid of the first intermediate YOLOv11 network model to obtain a second intermediate YOLOv11 network model; replace the C3k2 module in the second intermediate YOLOv11 network model with a C3k2-AG module to obtain a third intermediate YOLOv11 network model; and embed a cross-modal fusion module at the feature fusion position of the third intermediate YOLOv11 network model to obtain an initial YOLOv11 network model.
[0091] Other embodiments or specific implementations of the multimodal target detection system of the present invention can be referred to the above-described method embodiments, and will not be repeated here.
[0092] This application provides a multimodal target detection device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the multimodal target detection method in Embodiment 1 above.
[0093] The following is for reference. Figure 9 The diagram illustrates a structural schematic suitable for implementing the multimodal target detection device of the embodiments of this application. The multimodal target detection device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 9 The multimodal target detection device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.
[0094] like Figure 9As shown, the multimodal target detection device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the multimodal target detection device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the multimodal target detection device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows a multimodal target detection device with various systems, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems may be implemented alternatively.
[0095] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.
[0096] The multimodal target detection device provided in this application, employing the multimodal target detection method described in the above embodiments, can solve the technical problems of low efficiency in equipment fault diagnosis and low response speed and accuracy in equipment maintenance during after-sales service of industrial equipment in the prior art. Compared with the prior art, the beneficial effects of the multimodal target detection device provided in this application are the same as those of the multimodal target detection method provided in the above embodiments, and other technical features of this multimodal target detection device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.
[0097] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.
[0098] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
[0099] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.
[0100] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0101] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.
[0102] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.
Claims
1. A multimodal target detection method, characterized in that, The method includes: Acquire multimodal image data of the scene to be detected, wherein the multimodal image data includes a first modal image and a second modal image; The improved YOLOv11 network model is used to perform target detection on the multimodal image data to obtain target detection results. The improved YOLOv11 network model is a network model obtained by introducing a dual-stream backbone feature extraction structure, a P3-level mid-term fusion strategy, a C3k2-AG module, and a cross-modal fusion module into the YOLOv11 network model. Based on the target detection results, the category and location information of the target in the scene to be detected are determined.
2. The multimodal target detection method as described in claim 1, characterized in that, The step of using the improved YOLOv11 network model to perform target detection on the multimodal image data and obtain target detection results includes: The first modal image and the second modal image are respectively input into the dual-stream backbone network of the improved YOLOv11 network model to generate the first modal feature map and the second modal feature map; Extract the first feature map of the P3 level from the first modality feature map, and extract the second feature map of the P3 level from the second modality feature map; The first feature map and the second feature map are input into the cross-modal fusion module to generate a fused feature map; The fused feature map is input into the neck detection network for multi-scale feature aggregation to generate multi-scale aggregated features. The multi-scale aggregated features are input into multiple detection heads for target classification and bounding box regression to obtain target detection results.
3. The multimodal target detection method as described in claim 2, characterized in that, The step of inputting the first modality image and the second modality image into the dual-stream backbone network of the improved YOLOv11 network model to generate the first modality feature map and the second modality feature map includes: The first modality image is input into the first backbone network, and features are extracted sequentially through the convolutional module, C3k2-AG module and spatial pyramid pooling layer in the first backbone network to generate a first modality feature map containing multiple resolutions. The second modality image is input into the second backbone network, and features are extracted sequentially through the convolutional module, C3k2-AG module and spatial pyramid pooling layer in the second backbone network to generate a second modality feature map containing multiple resolutions.
4. The multimodal target detection method as described in claim 2, characterized in that, The cross-modal fusion module is used to perform bidirectional cross-modal attention interaction on the first feature map and the second feature map to generate a fused feature map; The step of performing bidirectional cross-modal attention interaction on the first feature map and the second feature map to generate a fused feature map includes: The first feature map and the second feature map input to the cross-modal fusion module are pooled and serialized respectively to generate a first token sequence and a second token sequence; Using the first token sequence as the query and the second token sequence as the key and value, a first cross-modal attention interaction is performed to generate a first enhanced sequence; Using the second token sequence as the query and the first token sequence as the key and value, a second cross-modal attention interaction is performed to generate a second enhanced sequence; The first enhancement sequence and the second enhancement sequence are respectively restored to two-dimensional feature maps, and residual fusion is performed with the corresponding original modal feature maps to obtain the first fused feature and the second fused feature. The first fusion feature and the second fusion feature are spliced together, and channel compression is performed through a convolutional layer to obtain a fusion feature map.
5. The multimodal target detection method as described in claim 1, characterized in that, Before the step of acquiring multimodal image data of the scene to be detected, the method further includes: Obtain a multimodal target detection dataset, which includes paired visible light images and near-infrared images; The YOLOv11 network model is improved to obtain an initial YOLOv11 network model, which includes a dual-stream backbone feature extraction structure, a P3-level mid-term fusion strategy, a C3k2-AG module, and a cross-modal fusion module. The initial YOLOv11 network model is trained based on the multimodal object detection dataset to obtain the training results; Based on the training results, the model parameters of the initial YOLOv11 network model are optimized to obtain an improved YOLOv11 network model.
6. The multimodal target detection method as described in claim 5, characterized in that, The steps for improving the YOLOv11 network model to obtain an initial YOLOv11 network model include: The single-backbone feature extraction network of the YOLOv11 network model is extended into a first backbone network and a second backbone network set in parallel to obtain the first intermediate YOLOv11 network model. A mid-term fusion node is set at the P3 level of the feature pyramid of the first intermediate YOLOv11 network model to obtain the second intermediate YOLOv11 network model. Replace the C3k2 module in the second intermediate YOLOv11 network model with the C3k2-AG module to obtain the third intermediate YOLOv11 network model. A cross-modal fusion module is embedded at the feature fusion position of the third intermediate YOLOv11 network model to obtain the initial YOLOv11 network model.
7. The multimodal target detection method as described in claim 5, characterized in that, The C3k2-AG module is used to perform local-global feature enhancement and channel gating filtering on the input features to obtain the output features of the C3k2-AG module; The step of performing local-global feature enhancement and channel gating filtering on the input features to obtain the output features of the C3k2-AG module includes: The input features are split into first-branch features and second-branch features; After performing convolution and activation processing on the first branch features, local spatial correlation is enhanced through depthwise separable convolution to generate local enhanced features; The local enhanced features are modeled globally in terms of spatial and channel dimensions using an additive self-attention mechanism to generate attention-enhanced features. The attention enhancement features are adaptively filtered by a convolutional gating unit to generate gated enhancement features; The gating enhancement feature is spliced and fused with the second branch feature to obtain the output feature of the C3k2-AG module.
8. The multimodal target detection method as described in claim 7, characterized in that, The step of adaptively filtering channel features of the attention enhancement features through convolutional gating units to generate gated enhancement features includes: The attention enhancement features are divided into candidate branch features and gated branch features; Spatial features are extracted from the candidate branch features using 1×1 convolution to obtain candidate features; The gated branch features are processed using deep convolution and the Sigmoid activation function to generate gated weights; The candidate features are multiplied by the gating weights to obtain gating features, and the gating features are residually fused with the attention enhancement features to obtain gating enhancement features.
9. The multimodal target detection method as described in claim 5, characterized in that, After the step of optimizing the model parameters of the initial YOLOv11 network model based on the training results to obtain an improved YOLOv11 network model, the method further includes: The improved YOLOv11 network model was evaluated using preset model evaluation metrics to obtain model evaluation results. Based on the model evaluation results, the improved YOLOv11 network model that passed the evaluation was determined; The improved YOLOv11 network model that passed the evaluation is accelerated by the inference optimization engine to generate an inference engine file.
10. A multimodal target detection system, characterized in that, The system includes: The data acquisition module is used to acquire multimodal image data of the scene to be detected, wherein the multimodal image data includes a first modal image and a second modal image; The target detection module is used to perform target detection on the multimodal image data using the improved YOLOv11 network model to obtain target detection results. The improved YOLOv11 network model is a network model obtained by introducing a dual-stream backbone feature extraction structure, a P3-level mid-term fusion strategy, a C3k2-AG module, and a cross-modal fusion module into the YOLOv11 network model. The result determination module is used to determine the category information and location information of the target in the scene to be detected based on the target detection result.