Image recognition method and system based on multi-modal data fusion

Abstract

The invention discloses an image recognition method and system based on multi-modal data fusion, and the method comprises the steps: synchronously collecting image data and non-image modal data for a target scene, carrying out the preprocessing of the image data and the non-image modal data, inputting the preprocessed data into a feature fusion network, and carrying out the recognition of the feature fusion network; layered visual features of image data are extracted through image branches of the feature fusion network, modal features of non-image modal data are extracted through modal branches, the extracted features are fused through an interaction layer, and a fusion feature vector containing cross-modal associated information is formed; and inputting the fusion feature vector into a classifier to identify information and confidence of a collection object in the target scene, and when the confidence is lower than a confidence threshold, performing feature extraction and recognition again to form collection-fusion-recognition-feedback closed-loop optimization. In this way, the internal relation and complementarity between different modal data are fully considered, so that the accuracy of image recognition is improved, and the adaptability of the recognition process is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to an image recognition method and system for multimodal data fusion. Background Art

[0002] The existing multimodal data fusion image recognition and analysis method has the following steps: (1) Multimodal data acquisition: Utilizing multiple data acquisition devices, such as image acquisition equipment and sensors, image data and other modal data related to the image are simultaneously acquired in the target recognition scenario. The image acquisition equipment is selected based on the requirements of different application scenarios, and the acquisition time is also determined based on the specific application scenario to ensure that the acquired data can fully reflect the characteristics of the target object.

[0003] (2) Multimodal data preprocessing: The collected image data and other modal data are preprocessed separately.

[0004] (3) Multimodal feature extraction and fusion: A deep learning model is used to extract features from preprocessed image data and other modal data. During feature extraction and fusion, corresponding weights are set based on the importance of different modal data. The weight values ​​are determined through multiple experiments and optimization to improve the effectiveness of the fused features.

[0005] (4) Accurate image recognition and analysis: The multimodal fusion feature vector is input into a pre-trained image recognition and classification model. This model is built based on classification algorithms such as support vector machines (SVMs) and random forests. The model is trained on a large number of labeled multimodal data samples. The classification model recognizes and analyzes the image based on the input fusion feature vector, determines the category, attributes, and other information of the target object in the image, and outputs accurate image recognition and analysis results. During the recognition and analysis process, the output results of the model are monitored in real time. When the confidence level of the recognition result is lower than the set threshold, it automatically returns to the multimodal data collection step and re-collects data for analysis to ensure the accuracy of the recognition result.

[0006] However, the above-mentioned existing accurate image recognition and analysis methods based on multimodal data fusion have some shortcomings: (1) Feature extraction model separation: In the feature extraction stage, independent models are used to extract features from data of different modalities. This approach makes the feature extraction processes of each modality independent of each other and does not fully consider the intrinsic connection and complementarity between different modal data.

[0007] (2) Simple fusion method: In the feature fusion step, fusion is achieved only through a simple “feature splicing + fixed weight” method. This fusion method is too simple and does not fully consider the differences and interactions between different modal features in the semantic space. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to provide an image recognition method and system for multimodal data fusion, which can improve the accuracy of image recognition and enhance the adaptability of the recognition process.

[0009] In order to solve the above technical problems, the technical solution adopted by the present invention is: A multimodal data fusion image recognition method comprises the following steps: Synchronously acquiring image data from an image acquisition device and non-image modality data from a non-image data acquisition device for a target scene, and preprocessing the image data and the non-image modality data; Inputting the preprocessed image data and the preprocessed non-image modality data into a feature fusion network, extracting hierarchical visual features of the preprocessed image data by the image branch of the feature fusion network, extracting modal features of the preprocessed non-image modality data by the modality branch of the feature fusion network, and fusing the hierarchical visual features and the modality features by the interaction layer of the feature fusion network to obtain a fused feature vector; The fused feature vector is input into a classifier to identify the information and confidence of the captured object in the target scene. When the confidence is lower than the confidence threshold, the steps of synchronously acquiring image data from the image acquisition device and acquiring non-image modality data from the non-image data acquisition device for the target scene are re-executed.

[0010] In order to solve the above technical problems, another technical solution adopted by the present invention is: A multimodal data fusion image recognition system includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the processor implements the various steps of the multimodal data fusion image recognition method described above.

[0011] The beneficial effects of the present invention are as follows: image data and non-image modal data are collected synchronously for the target scene, and the image data and non-image modal data are preprocessed, and the preprocessed data are input into the feature fusion network, the image branch of the feature fusion network extracts the hierarchical visual features of the image data, the modal branch extracts the modal features of the non-image modal data, and the interaction layer fuses the hierarchical visual features and modal features to form a fused feature vector containing cross-modal correlation information; the fused feature vector is input into the classifier to identify the information and confidence of the collected object in the target scene, and when the confidence is lower than the confidence threshold, the data backtracking mechanism is triggered to re-extract and identify features, forming a "collection-fusion-identification-feedback" closed-loop optimization. In this way, by fully considering the inherent connection and complementarity between different modal data, the accuracy of image recognition is improved, and the adaptability of the recognition process is enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a flowchart of an image recognition method for multimodal data fusion according to an embodiment of the present invention; Figure 2 Schematic diagram of an image recognition system for multimodal data fusion according to an embodiment of the present invention; Figure 3 This is a flowchart of collaborative multimodal data collection and preprocessing according to an embodiment of the present invention; Figure 4 This is a diagram of the network architecture for collaborative extraction and fusion of multimodal features according to an embodiment of the present invention; Figure 5 Schematic diagram of the cross-modal bidirectional attention mechanism of an embodiment of the present invention.

[0013] Description of labels: 1. An image recognition system with multimodal data fusion; 2. Memory; 3. Processor. DETAILED DESCRIPTION

[0014] To illustrate the technical content, achieved objectives and effects of the present invention in detail, the following description is given in conjunction with the embodiments and accompanying drawings.

[0015] Please refer to Figure 1 The embodiment of the present invention provides an image recognition method for multimodal data fusion, comprising the steps of: Synchronously acquiring image data from an image acquisition device and non-image modality data from a non-image data acquisition device for a target scene, and preprocessing the image data and the non-image modality data; Inputting the preprocessed image data and the preprocessed non-image modality data into a feature fusion network, extracting hierarchical visual features of the preprocessed image data by the image branch of the feature fusion network, extracting modal features of the preprocessed non-image modality data by the modality branch of the feature fusion network, and fusing the hierarchical visual features and the modality features by the interaction layer of the feature fusion network to obtain a fused feature vector; The fused feature vector is input into a classifier to identify the information and confidence of the captured object in the target scene. When the confidence is lower than the confidence threshold, the steps of synchronously acquiring image data from the image acquisition device and acquiring non-image modality data from the non-image data acquisition device for the target scene are re-executed.

[0016] From the above description, it can be seen that the beneficial effects of the present invention are: synchronously collecting image data and non-image modal data for the target scene, and preprocessing the image data and non-image modal data, and inputting the preprocessed data into the feature fusion network, the image branch of the feature fusion network extracts the hierarchical visual features of the image data, the modal branch extracts the modal features of the non-image modal data, and the interaction layer fuses the hierarchical visual features and modal features to form a fusion feature vector containing cross-modal correlation information; the fused feature vector is input into the classifier to identify the information and confidence of the collected object in the target scene, and when the confidence is lower than the confidence threshold, the data backtracking mechanism is triggered, and feature extraction and recognition are performed again to form a "collection-fusion-recognition-feedback" closed-loop optimization. In this way, by fully considering the inherent connection and complementarity between different modal data, the accuracy of image recognition is improved, and the adaptability of the recognition process is enhanced.

[0017] Furthermore, the interactive layer of the feature fusion network fuses the hierarchical visual features and the modal features, which includes: performing average pooling on the hierarchical visual features to generate a two-dimensional feature response map, extracting spatial dependency information of the two-dimensional feature response map using a convolution operation to output a spatial attention map, and adjusting the hierarchical visual features using the spatial attention map; Perform global average pooling on the modal features to obtain a statistical vector, input the statistical vector into a fully connected layer to obtain an attention weight, and use the attention weight to adjust the modal features.

[0018] From the above description, we can see that the spatial attention map is extracted by pooling and convolution of the hierarchical visual features, and the spatial attention map is used to adjust the hierarchical visual features. The modal features are pooled and fully connected layer calculations are performed to obtain attention weights. Using the attention weights to adjust the modal features can improve the saliency expression effect of the features, allowing the subsequent gated fusion unit to more accurately evaluate modal complementarity and perform weighted combination.

[0019] Furthermore, the interactive layer of the feature fusion network fuses the hierarchical visual features and the modal features, including: Calculating the association strength between the hierarchical visual features and the modality features based on an attention mechanism; The interactive layer of the feature fusion network fuses the hierarchical visual features and the modal features according to the association strength.

[0020] From the above description, it can be seen that by calculating the correlation strength between the hierarchical visual features and the modal features, the subsequent value vectors are weightedly fused to generate a fused feature vector containing explicit correlation between the modalities.

[0021] Furthermore, the interaction layer of the feature fusion network fuses the hierarchical visual features and the modal features according to the association strength, including: Obtaining a mapping weight of the hierarchical visual feature and a mapping weight of the modal feature according to the correlation strength between the hierarchical visual feature and the modal feature; According to the mapping weight of the hierarchical visual features W v and the mapping weights of the modal features W s , calculate the hierarchical visual features V and the modal characteristics S The fusion feature vector F :

[0022] g= σ ( W v V + W s S + b ) Where, b represents the bias vector, σ The function is used to compress the fusion weights to the interval [0,1], and g represents the gate vector.

[0023] From the above description, it can be seen that the gated fusion unit is introduced to realize the dynamic weighted fusion of image modality and other modal features, and the gated vector is applied to the input features, and weighted fusion is performed through Hadamard multiplication. This not only retains the independence of image and other modal information, but also realizes the detailed modeling of the contribution of each dimension of information. During the training process, the gated weights are automatically optimized through back propagation to ensure that the network adapts to diverse modal differences and data distribution.

[0024] Furthermore, the image branch of the feature fusion network extracts hierarchical visual features of the preprocessed image data, the modality branch of the feature fusion network extracts modal features of the preprocessed non-image modality data, and the interaction layer of the feature fusion network fuses the hierarchical visual features and the modality features, further comprising: Use joint loss function to optimize feature extraction parameters and fusion parameters; The joint loss function includes an image branch loss function, a modality branch loss function, a cross-modality contrast loss function, and a classification loss function of fused features.

[0025] From the above description, it can be seen that a multi-objective loss function is adopted, including image branch loss function, modality branch loss function, cross-modality contrast loss function and classification loss function of fusion features, in order to guide the synchronous optimization of each modality branch and fusion path during the training process.

[0026] Furthermore, preprocessing the image data and the non-image modality data includes: Mapping and aligning the non-image modality data according to pixel coordinates of the image data; performing coding analysis on the image data using a binary pattern noise detection method, identifying an abnormal variance region in the analysis result and determining a corresponding noise type, and removing noise from the image data according to the noise type; The interquartile range method is used to detect outliers in the aligned non-image modality data, and the outliers are repaired by interpolation.

[0027] From the above description, it can be seen that by mapping and aligning the non-image modality data according to the pixel coordinates of the image data, and then preprocessing the image data and non-image modality data, the effectiveness of feature collaborative modeling can be improved through the alignment accuracy in the fusion task.

[0028] Furthermore, the fused feature vector is input into a classifier to identify and obtain information and confidence of the captured object in the target scene, including: Inputting the fused feature vector into a classifier to obtain category probability distribution data and a modality contribution vector of the fused feature vector output by the classifier, wherein the modality contribution vector includes an image branch contribution and a modality branch contribution; The maximum category confidence of the identified acquisition object is obtained from the category probability distribution data.

[0029] Furthermore, the fused feature vector is input into a classifier to identify and obtain information and confidence of the collected object in the target scene, which then includes: If the maximum category confidence is lower than the confidence threshold, when the image branch contribution is less than the contribution threshold, the angle parameters of the image acquisition device are adjusted; when the modal branch contribution is less than the contribution threshold, the acquisition parameters of the non-image data acquisition device are adjusted.

[0030] From the above description, it can be seen that through the confidence-driven backtracking mechanism, the uncertainty of the results and the contribution evaluation of modal features can be identified, the acquisition strategy and model parameters can be adaptively adjusted, and the system's robustness to abnormal samples and boundary samples can be enhanced.

[0031] Furthermore, synchronously acquiring image data from an image acquisition device and non-image modality data from a non-image data acquisition device for a target scene includes: synchronously acquiring image data from an image acquisition device and non-image modality data from a plurality of non-image data acquisition devices for a target scene; The modal branch of the feature fusion network extracts modal features of the preprocessed non-image modal data, including: The feature fusion network loads corresponding multiple modal branches according to the non-image modal data obtained from multiple non-image data acquisition devices, and uses each modal branch to extract the modal features of the corresponding non-image modal data after preprocessing.

[0032] From the above description, it can be seen that by synchronously collecting multiple non-image modal data and dynamically loading the modal branches in the feature fusion network according to the multiple modal data, it is possible to dynamically access new modalities for image recognition without reconstructing the overall network structure, thereby improving the applicability of image recognition and enabling the image recognition process to be flexibly expanded.

[0033] Please refer to Figure 2 Another embodiment of the present invention provides an image recognition system for multimodal data fusion, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, each step of the above-mentioned image recognition method for multimodal data fusion is implemented.

[0034] The above-mentioned multimodal data fusion image recognition method and system of the present invention are suitable for improving the accuracy of image recognition and enhancing the adaptability of the recognition process. The following is an explanation through specific implementation methods: Please refer to Figure 1 , embodiment 1 of the present invention is: A multimodal data fusion image recognition method comprises the following steps: S1. Synchronously acquiring image data from an image acquisition device and non-image modality data from a non-image data acquisition device for a target scene, and preprocessing the image data and the modality data.

[0035] For details, please refer to Figure 3 , the alignment accuracy in the fusion task directly affects the effectiveness of feature collaborative modeling. In this embodiment, the modal data can be spectral data. For the registration of image data and spectral data, a calibration plate is first used to construct a mapping relationship between pixel coordinates and spectral sampling positions. The calibration plate usually combines a checkerboard pattern with a spectral calibration piece with known wavelength reflection characteristics. The projection position of the spectral sampling point in the image is obtained by solving the internal and external parameters of the camera to complete the fitting of the spatial mapping function. On this basis, the spectral data is distributed with the pixel coordinates of the two-dimensional image as the target, and the reflectance value of each band is mapped to the corresponding pixel position through bilinear interpolation, and finally a pixel-level multimodal data pair is formed ( I xy , S xy ),in( x , y ) are the image pixel coordinates, I xy is the RGB or grayscale intensity value of the point, S xy ∈ R b is the spectral vector corresponding to wavelength dimension b, R b This processing step ensures that the image and spectral information have a one-to-one correspondence at each pixel level, laying the foundation for subsequent collaborative feature extraction.

[0036] For time series modal data such as dynamic infrared thermal imaging, inter-frame registration becomes a challenge due to its time-varying physical properties and the influence of camera shake. To this end, the optical flow method is used to estimate the inter-frame motion vector field. The commonly used dense optical flow algorithm calculates the motion offset vector of each pixel between adjacent frames. d xy =( u xy , v xy ),in, u xy 、 v xyRepresenting horizontal and vertical displacement, the current frame's infrared image is aligned to the master frame's coordinate system through reverse mapping, and the motion estimation accuracy is optimized using the residual reconstruction error. The final registration error is controlled within ±1 pixel, ensuring the consistency of the spatial structure of infrared images and static images at different time steps, providing stable input for multi-frame fusion enhancement or temporal feature modeling.

[0037] In addition, the adaptive preprocessing strategy in this embodiment aims to improve the quality and distribution consistency of multi-source data and enhance the robustness of subsequent feature extraction and fusion. An integrated noise detection and adaptive denoising module is designed for the image data part. The local texture features of the image are encoded and analyzed through the local binary pattern noise detection algorithm, and the local variance abnormal area is identified and its noise type is determined. If the salt and pepper noise distribution is detected to be a point-like isolated feature, the median filter is preferentially enabled to perform nonlinear smoothing on the neighboring pixels to remove pulse interference, and then the non-local mean filter module is introduced; the Gaussian distribution noise area is weighted averaged based on the image block similarity, retaining the edge structure while achieving background smoothing, thereby forming a dynamic algorithm combination strategy for mixed noise scenes and performing multi-scale enhancement on the image.

[0038] The robust normalization mechanism is introduced into the preprocessing process of non-image modality data. First, the outliers are detected by the interquartile range method. The first quartile is defined as Q 1, the third quartile is Q 3. Calculate the interquartile range as IQR = Q 3- Q 1, all out of range [ Q 1-1.5× IQR , Q 3+1.5× IQR ] are considered as outliers and are uniformly interpolated, repaired or weakened to ensure that extreme values ​​have minimal impact on the overall statistical characteristics. After repair, the data is normalized to constrain its standard distribution to meet the range:

[0039] Where, μ represents the mean of the normalized data, σ The standard deviation ensures that the majority of the data is distributed within the standard range, preventing inter-modal scale differences from affecting network weight learning. In multispectral agricultural monitoring tasks, crop canopy images collected at dusk are affected by the superposition of salt-and-pepper and Gaussian noise. The noise detection module dynamically activates a combined denoising strategy. The red-edge spectral channel in the non-image portion exhibits jump anomalies in areas of strong reflection. After IQR threshold correction and normalization, the multimodal input response in the neural network is more balanced, improving the stability and accuracy of diseased area identification.

[0040] S2. Input the preprocessed image data and the preprocessed modal data into a feature fusion network, extract the hierarchical visual features of the preprocessed image data by the image branch of the input feature fusion network, extract the modal features of the preprocessed modal data by the modal branch of the input feature fusion network, and fuse the hierarchical visual features and the modal features by the interaction layer of the input feature fusion network to obtain a fused feature vector.

[0041] In this embodiment, please refer to Figure 4 The feature fusion network adopts a multi-branch architecture, with the image branch using a lightweight CNN network such as MobileNetV3 or a visual Transformer network such as Vision Transformer as the backbone structure. The convolution kernel size or the number of attention heads are dynamically and adaptively adjusted based on the resolution of the input image. For example, in industrial inspection scenarios, where the input image resolution is 1024×1024 pixels, the network uses a convolution kernel with a larger receptive field (7×7) or increases the number of attention heads to capture more complex structural information. In medical image recognition tasks, where the image resolution is 512×512 pixels, the network uses a smaller convolution kernel (3×3) and fewer attention heads to reduce computational burden while maintaining sensitivity to local structures.

[0042] Please refer to Figure 5 The modal branch constructs a specific structure according to the physical properties of the input data. The spectral data is processed by a one-dimensional convolutional neural network 1D-CNN. Multiple convolutional layers are constructed in the wavelength dimension to extract frequency features such as absorption peaks and peak widths, and the channel attention module is used to adjust the contribution of each band to downstream tasks. The depth data is processed by a point cloud structure-oriented neural network such as PointNet. The network first encodes the spatial coordinate features of each point using a multi-layer perceptron, and then aggregates local geometric information through maximum pooling, retaining the target boundary and shape structure to form global semantic features.

[0043] To enhance the interaction between modalities, a cross-modal bidirectional attention mechanism is introduced on the high-dimensional features output by each branch. Its core calculation formula is:

[0044] in, Indicates the image modality i Areas in T The query vector at time, k j ∈ R d Indicates the first j The key vector of the bands, R d expressd dimensional real vector space, k j’ Indicates the first j The key vector of band indices, attention weights α ij ∈[0,1] is used to measure the correlation strength between the two modalities at the corresponding position. The attention matrix generated according to the attention weight is further applied to the value vector for weighted fusion. After the attention matrix is ​​processed by linear transformation or full connection layer, the mapping weight of the corresponding hierarchical visual features is generated. W v and the mapping weights of modal features W s , to quantify the contribution of different modal features in the fusion, so as to generate a fusion feature vector containing explicit correlations between modalities.

[0045] For example, in the agricultural pest and disease identification scenario, a drone equipped with a hyperspectral sensor is used to obtain 1024×1024 resolution images and spectral data of 240 continuous bands. The image branch uses MobileNetV3 with an initial convolution kernel size of 5×5 and 64 channels. The spectral branch uses a 3-layer 1D-CNN network with convolution kernel sizes of 7, 5, and 3, and channels of 32, 64, and 128, respectively. The output feature dimensions of the two branches are unified to 256, and the interaction layer is fused through a bidirectional attention mechanism and then enters the fully connected layer to complete the recognition task.

[0046] The core of the cross-modal interaction layer is to introduce the gated fusion unit (GatedFusionUnit) to achieve dynamic weighted fusion of image modality and other modality features. Input features include image modality features and another modal feature , D v represents the dimension of the hierarchical visual features (i.e., image features) V, D s Represents the dimension of the modal feature S, which is fused and controlled by a fully connected network with shared weights. The network first calculates the gate vector g ∈ R D , D Represents the dimension of the fusion feature, which is used to control the proportion of the two modal features in the fusion process. The calculation formula is: g= σ ( W v V + W s S + b ) Where, and is the mapping weight between image features and modality features, b ∈ R D represents the bias vector, σ (·) is a Sigmoid function that compresses the fusion weight to the range of 0 to 1, achieving nonlinear control of modal importance. The gate vector acts on the input features, performs weighted fusion through Hadamard multiplication, and outputs the final fused features:

[0047] Where, Represents element-wise multiplication. This mechanism not only preserves the independence of images and other modal information, but also enables detailed modeling of the contribution of each dimension. During training, gating weights are automatically optimized through backpropagation, ensuring that the network adapts to diverse modal differences and data distributions.

[0048] For example, in the intelligent manufacturing quality inspection scenario, high-resolution industrial cameras and infrared thermal imagers are used to collect product surface images and temperature modal information. The image feature V comes from the output of the last layer of MobileNetV3, with a dimension of 256, and the temperature feature S comes from the output of 1D-CNN, with a dimension of 128. After being unified into 256 dimensions through linear mapping, they are sent to the gate control unit to generate a fusion gate. g ∈ R 256 ,The output fusion features are used in the defect classification model, which improves the recognition accuracy of early microcracks and temperature anomaly areas.

[0049] Furthermore, the joint training mechanism is a key link to improve the overall recognition performance. The feature fusion network in this embodiment adopts a multi-objective loss function design to guide the simultaneous optimization of each modality branch and fusion path during the training process. The overall loss function consists of four parts. The first is the classification loss of each of the image modality and the auxiliary modality. L v and L s , using the standard cross entropy form, corresponding to the difference between the classification results output by each branch and the true label, and improving their respective discriminative abilities through back propagation; followed by cross-modal contrast loss L contrast , which is used to constrain the semantic consistency of image features and other modal features in the embedding space. It adopts the form of a triplet loss function. Its core idea is to make the Euclidean distance of different modal features of the same object in the embedding space less than 0.5, and the feature distance of different objects greater than 1.5, thereby compressing similar samples and distancing heterogeneous samples. The formula is as follows:

[0050] Where, represents the image modality anchor feature, Represents positive samples of similar modal features, represents negative samples of heterogeneous modal features, α The boundary interval parameter is set to 1.0.

[0051] Then the classification loss of the fusion feature L f , using the focal loss function (FocalLoss), the core of which is to enhance the network's attention to difficult-to-classify samples by adjusting the loss weight coefficients of positive and negative samples, and is suitable for recognition tasks with imbalanced categories:

[0052] Where, γ represents the focus adjustment factor, α represents the balance factor, represents the predicted probability.

[0053] The final total loss function is: L =0.3 L v +0.3 L s +0.2 L contrast +0.2 L f The weight of each loss has been experimentally tuned to balance the independence of branches and the consistency of fusion, and to prevent a certain modality from causing a dominant shift in the overall training. The entire training process uses the AdamW optimizer, which has a dynamic learning rate adjustment and weight decay mechanism, which can effectively alleviate the problems of gradient explosion and overfitting. In the intelligent traffic scenario, high-definition cameras and lidars are used to collect vehicle images and spatial point cloud data. The image modality output is 512-dimensional, and the point cloud modality embedding is 256-dimensional. The embedding space distance is constrained by the triplet loss, and the focus loss sets the focus adjustment factor. γ =2.0, the ratio of positive and negative samples is about 1:4, and the final fusion classification accuracy is improved to 92.6% in a congested environment.

[0054] S3. Input the fused feature vector into the classifier to identify the information and confidence of the captured object in the target scene. When the confidence is lower than the threshold, re-execute the steps of synchronously acquiring image data from the image acquisition device and acquiring non-image modality data from the non-image data acquisition device for the target scene.

[0055] Specifically, the confidence-driven backtracking mechanism realizes adaptive adjustment of acquisition strategies and model parameters by evaluating the uncertainty of recognition results and the contribution of modal features, thereby enhancing the robustness of the system to abnormal samples and boundary samples.

[0056] Among them, the classifier output contains the category probability distribution p ( c i ), c i Represents the predicted probability of the i-th category and outputs the modal contribution vector w= [ w v , w s ], w v and w s Respectively represent the fusion weights of the image vision branch and the modal data branch (such as spectrum, depth, etc.), and satisfy the normalization constraint w v + w s =1.

[0057] When the maximum category confidence of the system recognition result max( p ( c i ))Below the set threshold θ =0.85, indicating that the current sample is at the edge of the classifier judgment. At this time, the main reason for the recognition failure is determined in combination with the modal contribution.

[0058] If the visual branch contribution is found w v If the value is less than 0.3, it indicates that the information provided by the image features is insufficient. The system will automatically send instructions to the camera, and the robotic arm will control the viewing angle to deflect ±15° to perform multi-angle image acquisition of the target area, compensate for the blind spots and occlusion of the image information, and improve the visual branch expression ability.

[0059] If the modal branch contribution w s If the value is less than 0.3, the system infers that the resolution or signal-to-noise ratio of the non-image mode is insufficient. In this case, the system triggers the dynamic adjustment of the sensor acquisition parameters, increasing the band sampling interval of the spectrometer from 10nm to 5nm, making the characteristic absorption peak clearer, thereby improving the ability to distinguish modal branches.

[0060] The backtracking mechanism builds a closed-loop control process between recognition confidence, modal contribution and acquisition configuration. In industrial inspection, when a weld image has a confidence of only 0.72 in the recognition stage and w v=0.21, the system automatically adjusts the camera angle for re-capturing, and re-integrates the recognition after completing the image update. The final confidence level is increased to 0.93, and the recognition result is stable and highly reliable.

[0061] The online fine-tuning mechanism for network parameters is used to quickly adapt to new sample distributions or modal feature shifts during real-time acquisition, ensuring the system's continued recognition accuracy in dynamically changing environments. Relying on a feedback optimization module, this mechanism uses the newly added sample sets after each retrospective resampling as the online training data source and employs a model distillation strategy to update the local parameters of the main network. The distillation process uses the historical prediction results of the main network as soft labels, minimizing losses by comparing the current predictions with historical knowledge. This approach avoids catastrophic forgetting caused by new samples while maintaining the original general feature extraction capabilities. During the online fine-tuning process, only the weight parameters in the last three fully connected layers and the fusion gated unit are updated, freezing the backbone structure of the image and modal feature extraction branches, significantly shortening training time and reducing system resource consumption. The entire fine-tuning process is completed on industrial-grade edge computing equipment, with training time controlled within five minutes, meeting the low-latency requirements of actual production lines. In the task of high-precision part surface defect detection, when the system's recognition confidence in some new samples decreases due to oil interference, the system quickly expands the sample set and performs online fine-tuning after collecting new perspective images and high-resolution spectral data through the backtracking mechanism. During the training process, the fusion gate weights are automatically adjusted to enhance the spectral branching effect. After the optimization is completed, the system's recognition accuracy for the same type of defects is improved from the original 87.6% to 94.1%, achieving rapid gain and adaptive recovery of model performance.

[0062] In this embodiment, the multi-branch network architecture can also adopt a modular design to support dynamic access to new modality types without the need to reconstruct the overall network structure. At the input end, the system presets an input interface compatible with mainstream deep learning deployment standards, and supports the use of TensorFlowServing or ONNXRuntime to receive new modality data streams, ensuring that each type of modality can be uniformly sent to the feature extraction subnetwork after normalization preprocessing. For newly connected modalities such as X-ray images and lidar point clouds, the system completes feature extraction by loading the corresponding branch structure. For example, the U-Net network is used to extract edge and tissue structure features for X-ray data, and PointNet++ is used to capture three-dimensional shape information for point cloud data. In the feature fusion stage, in order to ensure the dimensional consistency and semantic comparability between different modal features, a dimensional adaptation mechanism is introduced. This mechanism uses an extensible linear mapping matrix Perform dimension transformation on the new modal features, where D new is the feature dimension extracted from the new modality, D oldBy learning W, the new modal features can be projected into the fusion space for joint processing to achieve the unified dimension expected by the original fusion layer. The new modal features after dimension mapping and the original modal features are collaboratively learned in the interactive attention module and the gated fusion unit, thereby achieving a flexible combination of 2 to 5 modalities. In actual industrial defect detection scenarios, when the lidar point cloud is used to enhance structural boundary recognition, the dimension mapping matrix is ​​used by loading the 128-dimensional PointNet++ output. W ∈ R 256×128 After the conversion and input into the fusion layer, the overall inference delay of the system only increased by about 8%, but the model's recognition accuracy of tiny crack edges increased by more than 5%.

[0063] The second embodiment of the present invention is: Based on the first embodiment, the interaction layer of the feature fusion network is further limited to include an attention-guided feature enhancement module: In this embodiment, the feature enhancement module aims to improve the discriminative ability and semantic focus of pre-fusion features through adaptive weighting of salient regions and key modal channels.

[0064] Specifically, for image features , Representing image features V Dimensions ( H is the image height, W is the image width, D v is the number of channels), first perform an average pooling operation along the channel dimension to generate a two-dimensional feature response map M v ∈ R H×W , represents the average activation level of all channels at each pixel position, and then a convolution operation with a convolution kernel size of k×k is performed to extract spatial dependency information and output a spatial attention map A v ∈ R H×W , R H×W Represents the dimension of the spatial attention map ( H For height, W is the width), which is used to characterize the weight distribution of salient areas in the image.

[0065] For modal features , Represents the C-dimensional real vector space where the modal feature S is located. First, global average pooling is performed to obtain the statistical vector of the modal feature , and then input into a set of shared fully connected layers to generate attention weights , which represents the discrimination ability of each spectral channel in the current sample.

[0066] Finally, the attention map is applied to the original features by element-by-element multiplication, and the image branch is transformed into , the modal branch transformation is , where the symbol ⊙ represents weighting by position or channel. This feature enhancement mechanism improves the saliency of features, enabling subsequent gated fusion units to more accurately assess modality complementarity and perform weighted combinations.

[0067] For example, in the task of detecting crop diseases, there are spots on the edge of the leaf in the image, but the background interference is strong. The attention module will focus on the edge area. A v The weight is increased to 0.91, while the background area is compressed to 0.12. At the same time, the characteristic absorption peak band of the lesion in the spectral channel is A s The weight is increased by more than 0.85, and the fused feature expression is more concentrated on the key dimensions of the lesions, achieving highly robust recognition in complex backgrounds.

[0068] The third embodiment of the present invention is: In the intelligent agricultural pest monitoring scenario, pest identification is performed using the method of Embodiment 1 or 2. The specific steps are as follows: (1) Collaborative multimodal data acquisition and preprocessing: A drone equipped with a hyperspectral sensor was used as the acquisition device to fly over the farmland to collect data. To ensure that the multimodal data timestamp error was less than 5ms, a high-precision linkage trigger module was set up. During the acquisition process, the drone simultaneously acquired crop image data and corresponding spectral data.

[0069] (2) Image data preprocessing: For the collected images, a local binary pattern noise detection algorithm is first used to detect noise. Due to the complex farmland environment, the images may contain salt and pepper noise and Gaussian noise. If salt and pepper noise is detected, the median filter is used to remove the pulse interference first, and then the Gaussian noise area is processed using the non-local mean filter. While preserving the edge structure, the background is smoothed to complete the adaptive denoising. Then, a multi-scale enhancement algorithm is used to highlight the detailed features of the crops, making the disease spots and insect pest marks clearer.

[0070] (3) Spectral data preprocessing: For spectral data, temporal and spatial alignment is first performed. A calibration plate is used to construct a mapping relationship between pixel coordinates and spectral sampling positions. The projection position of the spectral sampling point in the image is solved by the camera's internal and external parameters. Then, bilinear interpolation is used to map the reflectance values ​​of each band to the corresponding pixel position. After that, the interquartile range method is used to detect outliers. Samples that are out of range are interpolated or weakened, and the data is normalized to the interval [0,1].

[0071] (4) End-to-end multimodal feature collaborative extraction and fusion: A parallel deep learning framework was constructed. MobileNetV3 was selected for the image branch because its lightweight design is suitable for the computational resource constraints of drones. Considering the image resolution of 1024×1024 pixels, the initial convolution kernel size was set to 5×5 and the number of channels was set to 64. The modality branch (spectral branch) used a 3-layer 1D-CNN network with convolution kernel sizes of 7, 5, and 3, respectively, and the number of channels of 32, 64, and 128, respectively.

[0072] (5) Feature extraction: The image branch extracts hierarchical visual features through residual convolution blocks. The shallow layer captures features such as leaf edges and shapes, and the deep layer obtains semantic regional features, such as the overall characteristics of lesions. The spectral branch uses the attention mechanism to screen key features and identify characteristic absorption peaks in spectral data. These absorption peaks are closely related to the status of crop diseases and insect pests.

[0073] (6) Cross-modal interaction fusion: The inter-modal dependency is calculated through the bidirectional attention mechanism of the cross-modal interaction layer. The attention weight is calculated based on the query vector of a certain region in the image modality and the key vector of a certain band in the spectral modality. This weight measures the strength of the association between the two modalities at the corresponding position. Then, through the gated fusion unit, the fusion ratio of the image modality and spectral modality features is dynamically adjusted according to the gate vector to generate a fused feature vector. The entire network adopts a joint loss function (including the classification loss of the image modality and spectral modality, the cross-modal contrast loss, and the classification loss of the fused features), and uses the AdamW optimizer for end-to-end training to simultaneously optimize the feature extraction parameters and fusion strategy.

[0074] (7) Accurate identification and dynamic feedback optimization: The fused feature vector is input into the improved SVM classifier to output the category (such as powdery mildew, aphid infestation, etc.), attribute (severity, etc.) and confidence level of crop pests and diseases.

[0075] (8) Confidence-driven backtracking: If the maximum category confidence of the recognition result is lower than the set threshold of 0.85, the data backtracking mechanism is triggered. Suppose that in a certain recognition, the confidence is only 0.78, and analysis shows that the spectral branch contribution is lower than 0.3, indicating that the spectral data provides insufficient information. The system automatically adjusts the band sampling interval of the spectrometer from 10nm to 5nm, re-collects the spectral data, and performs feature extraction and recognition.

[0076] (9) Online fine-tuning of network parameters: After each backtracking and supplementary sampling, the newly added sample set is used as the online training data source, and the model distillation strategy is used to update the local parameters of the main network. Only the weight parameters in the last three fully connected layers and the fusion gated unit are updated, and the backbone structure of the image and modal feature extraction branches is frozen. For example, when the system finds that new pest and disease samples cause the recognition accuracy to drop, the system backtracks to collect new data and performs online fine-tuning on industrial-grade edge computing equipment. The training time is controlled within five minutes, quickly adapting to the new pest and disease characteristics and improving the recognition accuracy.

[0077] In this intelligent agricultural pest and disease monitoring scenario, the above complete process can more accurately identify crop pests and diseases than traditional single-modal recognition methods, with the recognition accuracy increased by about 18%, effectively helping farmers to promptly detect and deal with pest and disease problems and reduce crop losses.

[0078] Please refer to Figure 2 , the fourth embodiment of the present invention is: A multimodal data fusion image recognition system 1 includes a memory 2, a processor 3, and a computer program stored in the memory 2 and executable on the processor 3. When the processor 3 executes the computer program, each step of a multimodal data fusion image recognition method according to any one of the first to third embodiments is implemented.

[0079] In summary, the present invention provides an image recognition method and system for multimodal data fusion, which realizes deep interaction and collaboration between different modal data through end-to-end multimodal feature collaborative extraction and fusion, fully explores the potential relationship between multimodal data, and eliminates the "modal gap" problem. In addition, the multimodal data collaborative acquisition and preprocessing strategy, as well as the dynamic feedback optimization mechanism, enable the model to automatically adjust the acquisition equipment parameters and network model parameters according to different application scenarios and data changes. At the same time, the precise identification and dynamic feedback optimization process, by real-time monitoring of the confidence of the recognition results, and performing data backtracking and re-identification when the confidence is low, effectively avoids misjudgments caused by data noise, incomplete features, etc.

[0080] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's description and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An image recognition method for multimodal data fusion, characterized in that: Including steps: Synchronously acquiring image data from an image acquisition device and non-image modality data from a non-image data acquisition device for a target scene, and preprocessing the image data and the non-image modality data; Inputting the preprocessed image data and the preprocessed non-image modality data into a feature fusion network, extracting hierarchical visual features of the preprocessed image data by the image branch of the feature fusion network, extracting modal features of the preprocessed non-image modality data by the modality branch of the feature fusion network, and fusing the hierarchical visual features and the modality features by the interaction layer of the feature fusion network to obtain a fused feature vector; The fused feature vector is input into a classifier to identify the information and confidence of the captured object in the target scene. When the confidence is lower than the confidence threshold, the steps of synchronously acquiring image data from the image acquisition device and acquiring non-image modality data from the non-image data acquisition device for the target scene are re-executed.

2. The image recognition method of multimodal data fusion according to claim 1, characterized in that: The interactive layer of the feature fusion network fuses the hierarchical visual features and the modal features, which previously includes: performing average pooling on the hierarchical visual features to generate a two-dimensional feature response map, extracting spatial dependency information of the two-dimensional feature response map using a convolution operation to output a spatial attention map, and adjusting the hierarchical visual features using the spatial attention map; Perform global average pooling on the modal features to obtain a statistical vector, input the statistical vector into a fully connected layer to obtain an attention weight, and use the attention weight to adjust the modal features.

3. The image recognition method of multimodal data fusion according to claim 1, characterized in that: The interactive layer of the feature fusion network fuses the hierarchical visual features and the modal features, including: Calculating the association strength between the hierarchical visual features and the modality features based on an attention mechanism; The interactive layer of the feature fusion network fuses the hierarchical visual features and the modal features according to the association strength.

4. The image recognition method of multimodal data fusion according to claim 3, characterized in that: The interactive layer of the feature fusion network fuses the hierarchical visual features and the modal features according to the association strength, including: Obtaining a mapping weight of the hierarchical visual feature and a mapping weight of the modal feature according to the correlation strength between the hierarchical visual feature and the modal feature; According to the mapping weight of the hierarchical visual features W v and the mapping weights of the modal features W s , calculate the hierarchical visual features V and the modal characteristics S The fusion feature vector F : g＝ σ ( W v V ＋ W s S ＋ b ) Where, b represents the bias vector, σ The function is used to compress the fusion weights to the interval [0,1], and g represents the gate vector.

5. The image recognition method of multimodal data fusion according to claim 1, characterized in that: The image branch of the feature fusion network extracts hierarchical visual features of the preprocessed image data, the modality branch of the feature fusion network extracts modal features of the preprocessed non-image modality data, and the interaction layer of the feature fusion network fuses the hierarchical visual features and the modality features, further comprising: Use joint loss function to optimize feature extraction parameters and fusion parameters; The joint loss function includes an image branch loss function, a modality branch loss function, a cross-modality contrast loss function, and a classification loss function of fused features.

6. The image recognition method of multimodal data fusion according to claim 1, characterized in that: Preprocessing the image data and the non-image modality data includes: Mapping and aligning the non-image modality data according to pixel coordinates of the image data; performing coding analysis on the image data using a binary pattern noise detection method, identifying an abnormal variance region in the analysis result and determining a corresponding noise type, and removing noise from the image data according to the noise type; The interquartile range method is used to detect outliers in the aligned non-image modality data, and the outliers are repaired by interpolation.

7. The image recognition method of multimodal data fusion according to claim 1, characterized in that: Inputting the fused feature vector into a classifier to identify and obtain information and confidence of the captured object in the target scene, including: Inputting the fused feature vector into a classifier to obtain category probability distribution data and a modality contribution vector of the fused feature vector output by the classifier, wherein the modality contribution vector includes an image branch contribution and a modality branch contribution; The maximum category confidence of the identified acquisition object is obtained from the category probability distribution data.

8. The image recognition method of multimodal data fusion according to claim 7, characterized in that: The fused feature vector is input into a classifier to identify and obtain information and confidence of the captured object in the target scene, which then includes: If the maximum category confidence is lower than the confidence threshold, when the image branch contribution is less than the contribution threshold, the angle parameters of the image acquisition device are adjusted; when the modal branch contribution is less than the contribution threshold, the acquisition parameters of the non-image data acquisition device are adjusted.

9. The image recognition method of multimodal data fusion according to claim 1, characterized in that: Synchronously acquiring image data from an image acquisition device and non-image modality data from a non-image data acquisition device for a target scene, including: synchronously acquiring image data from an image acquisition device and non-image modality data from a plurality of non-image data acquisition devices for a target scene; The modal branch of the feature fusion network extracts modal features of the preprocessed non-image modal data, including: The feature fusion network loads corresponding multiple modal branches according to the non-image modal data obtained from multiple non-image data acquisition devices, and uses each modal branch to extract the modal features of the corresponding non-image modal data after preprocessing.

10. An image recognition system for multimodal data fusion, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, each step of the image recognition method for multimodal data fusion according to any one of claims 1 to 9 is implemented.

Images