Multi-modal image recognition system and method based on deep learning
Through a multimodal image recognition system based on deep learning, the multi-branch CNN architecture and attention mechanism are used for feature fusion, which solves the data inconsistency and information fusion problems in multimodal image recognition and achieves efficient and accurate image recognition results.
Patent Information
- Application Number
- CN202510769831.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-07
- Publication Date
- 2025-10-28
AI Technical Summary
Most existing image recognition systems target single-modal images and have difficulty effectively processing and identifying multimodal images. They face challenges in data inconsistency and information fusion, and traditional methods find it difficult to efficiently process large amounts of multimodal image data.
A multimodal image recognition system based on deep learning is adopted, including data acquisition, preprocessing, deep learning model construction and training deployment. It uses a multi-branch CNN architecture and attention mechanism for feature fusion, combined with adaptive image enhancement and normalization processing, and adopts distributed training and adaptive learning rate adjustment to achieve accurate recognition of multimodal images.
It significantly improves the accuracy and robustness of multimodal image recognition, especially in complex scenarios, shortens the training cycle, and enhances the adaptability and generalization capabilities of the model.
Smart Images
Figure CN120852839A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image recognition technology, specifically a multimodal image recognition system and method based on deep learning. Background Technology
[0002] With the rapid development of computer technology, sensor technology, and communication technology, image data is experiencing explosive growth, and the importance of multimodal image data is becoming increasingly prominent in various fields. Multimodal images fuse image information from different sensors or imaging methods, providing more comprehensive and richer data. For example, in the medical field, CT images and MRI images provide detailed information on different human tissue structures, and their fusion can help doctors diagnose diseases more accurately. In the security field, the combination of visible light images and infrared images can achieve all-weather monitoring. However, most existing image recognition systems are designed for single-modal image recognition, and there are many problems with multimodal image recognition. For example, different modal images differ significantly in imaging principles, resolution, and grayscale range, leading to inconsistencies between data; multimodal image data is massive, making it difficult for traditional image processing methods to process and analyze efficiently; and information fusion between different modal images presents challenges. How to effectively extract and integrate key features from each modality is a key challenge for achieving accurate recognition. Therefore, this paper proposes a system and method that can efficiently process and accurately recognize multimodal images. Summary of the Invention
[0003] The purpose of this invention is to provide a deep learning-based multimodal image recognition system to solve the problems mentioned in the background art.
[0004] To achieve the above objectives, the present invention provides the following technical solution, comprising the following steps:
[0005] S1, Data Acquisition Module, is the front-end entry point of the entire system. During the acquisition process, it finely adjusts the parameters of the device according to the specific application scenario to ensure that the acquired image data has high quality and high availability.
[0006] S2, the data preprocessing module, improves image quality and unifies data format, preparing for subsequent deep learning model processing;
[0007] S3, the deep learning model building module, is responsible for learning the features of multimodal images and performing classification and recognition, and is the core part of the system;
[0008] S4, Model Training Module, is responsible for training the constructed deep learning model so that it can accurately recognize multimodal images;
[0009] S5, System Deployment and Application Module, deploys the trained deep learning model to real-world application scenarios to achieve fast and accurate recognition of new input multimodal images;
[0010] As a further preferred embodiment of this technical solution, step S1 includes the following steps:
[0011] S11 is equipped with a variety of image acquisition device interfaces, supporting high-resolution, multi-band and different imaging principle image inputs, ensuring that the acquired multimodal images have high complementarity in terms of spatial resolution, spectral resolution, etc., providing rich and reliable data sources for subsequent feature fusion;
[0012] S12, with its built-in automatic calibration and synchronization mechanism, can effectively eliminate image deviations caused by equipment differences and time delays, ensuring the spatiotemporal consistency of the acquired multimodal images and laying a good foundation for subsequent processing.
[0013] As a further preferred embodiment of this technical solution, step S2 includes the following steps:
[0014] S21. Image enhancement employs histogram equalization and adaptive filtering techniques. Histogram equalization enhances image contrast by stretching the grayscale histogram of the image, while adaptive filtering adjusts the filter parameters adaptively based on the local features of the image, which can remove noise while preserving the edge and detail information of the image.
[0015] S22. Normalization processing: Normalize the pixel values of different modal images to a uniform range of [0,1] or [-1,1]. Neural networks are sensitive to the scale of input data. Normalization can speed up the convergence of the model and improve training efficiency.
[0016] As a further preferred embodiment of this technical solution, the calculation formula for S22 is:
[0017] in, The x represents the original image pixel values, and max(x) and min(x) represent the maximum and minimum pixel values of the image, respectively. These are the normalized pixel values;
[0018] As a further preferred embodiment of this technical solution, step S3 includes the following steps:
[0019] S31: Multi-branch structure, each branch is dedicated to feature extraction for a specific modality of image. It adopts a CNN architecture with multiple convolutional and pooling layers. The convolutional layers extract features such as edges and textures of the image through convolutional kernels, while the pooling layers reduce the dimensionality and abstract the features.
[0020] S32: Feature fusion mechanism, which automatically focuses on key feature regions by calculating the attention weights of each modality feature at different scales, thereby improving the richness and accuracy of feature representation;
[0021] As a further preferred embodiment of this technical solution: the output features of the convolutional layer in S31 can be expressed as: Where I represents the input image, W is the convolution kernel weight, b is the bias term, and * indicates the convolution operation. For activation functions;
[0022] As a further preferred embodiment of this technical solution, step S4 includes the following steps:
[0023] S41: Dataset partitioning. The collected multimodal image dataset is divided into training set, validation set and test set according to a certain ratio, usually 7:2:1.
[0024] S42: Loss function selection. For classification tasks, the cross-entropy loss function is used, which can measure the difference between the model's predicted probability distribution and the true label distribution.
[0025] S43: Algorithm optimization. The Adam optimization algorithm is selected, which combines momentum and adaptive learning rate methods to effectively improve the convergence speed and stability of the model.
[0026] As a further preferred embodiment of this technical solution, the expression in S42 is: Where N represents the number of samples and C represents the number of categories. It is the true label (0 or 1) of the c-th class of the i-th sample. It is the probability predicted by the model that the i-th sample belongs to the c-th class;
[0027] The update rule for the Adam optimization algorithm in S43 is as follows:
[0028]
[0029] in and These are the first and second moments of the gradient, respectively; β1 and β2 are decay coefficients, typically taken as 0.9 and 0.999, respectively; n is the learning rate; ∈ is a constant used for numerical stabilization; θ t This represents the value of the model parameters at the t-th iteration. It is the gradient of the loss function;
[0030] As a further preferred embodiment of this technical solution, step S5 includes the following steps:
[0031] S51: Improve the efficiency of the model on real devices by using model pruning and quantization methods to optimize and compress the trained model;
[0032] S52: When a new multimodal image is input into the system, it is first enhanced and normalized by the data preprocessing module. Then, the processed image is input into the trained deep learning model. After feature extraction and fusion, the model outputs the image recognition result, including the category label and the corresponding confidence score.
[0033] The purpose of this invention is to provide a method for a multimodal image recognition system based on deep learning, so as to solve the problems mentioned in the background art.
[0034] To achieve the above objectives, the present invention provides the following technical solution, comprising the following steps:
[0035] A1: Acquire multimodal image data through various image acquisition devices;
[0036] A2: Perform enhancement and normalization processing on the acquired images;
[0037] A3: Select and build a deep learning model architecture suitable for multimodal image recognition;
[0038] A4: Prepare the training dataset, dividing it into training, validation, and test sets; determine the loss function and optimization algorithm; train the model using the training set, evaluate the model performance using the validation set, and adjust the model parameters based on the evaluation results until the model performance meets the expected requirements;
[0039] A5: Deploy the trained model into the application system to preprocess and recognize new multimodal images to obtain the final recognition results;
[0040] Compared with the prior art, the beneficial effects of the present invention are:
[0041] 1. This invention utilizes a multi-branch deep convolutional neural network and a multi-scale feature fusion method based on an attention mechanism to fully leverage the unique information and complementary advantages of each modality of image, deeply explore key features, and achieve accurate recognition of multimodal images in complex scenes. This significantly improves the accuracy of image recognition, especially under adverse conditions such as occlusion, deformation, and poor image quality, where its advantages are more obvious compared to traditional single-modality image recognition methods.
[0042] 2. The adaptive image enhancement algorithm and dynamic normalization method adopted in the data preprocessing module of this invention can effectively cope with the influence of different lighting conditions, imaging environment and other factors on image quality, enhance the model's adaptability to various complex conditions, and improve the robustness and stability of the system.
[0043] 3. This invention introduces a distributed training framework and an adaptive learning rate adjustment strategy, making full use of the hardware resources of large-scale computing clusters to achieve parallel processing of multimodal image data and rapid updating of model parameters, significantly shortening the model training cycle and improving R&D efficiency. At the same time, by combining data augmentation and regularization, it effectively prevents model overfitting and ensures that the model has good generalization ability when facing new data in practical applications. Attached Figure Description
[0044] Figure 1 The flowchart of the deep learning-based multimodal image recognition system and method of this invention is as follows. Figure 1 ;
[0045] Figure 2 The flowchart of the deep learning-based multimodal image recognition system and method of this invention is as follows. Figure 2 . Detailed Implementation
[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0047] Example 1:
[0048] See also Figures 1-2 As shown, this invention provides a multimodal image recognition system based on deep learning, comprising the following steps:
[0049] S1, Data Acquisition Module, is the front-end entry point of the entire system. During the acquisition process, it finely adjusts the parameters of the device according to the specific application scenario to ensure that the acquired image data has high quality and high availability.
[0050] S2, the data preprocessing module, improves image quality and unifies data format, preparing for subsequent deep learning model processing;
[0051] S3, the deep learning model building module, is responsible for learning the features of multimodal images and performing classification and recognition, and is the core part of the system;
[0052] S4, Model Training Module, is responsible for training the constructed deep learning model so that it can accurately recognize multimodal images;
[0053] S5, System Deployment and Application Module, deploys the trained deep learning model to real-world application scenarios to achieve fast and accurate recognition of new input multimodal images;
[0054] In this embodiment, specifically, S1 includes the following steps:
[0055] S11 is equipped with a variety of image acquisition device interfaces, supporting high-resolution, multi-band, and different imaging principle image inputs, ensuring that the acquired multimodal images have high complementarity in terms of spatial resolution and spectral resolution, providing rich and reliable data sources for subsequent feature fusion. It supports a variety of image acquisition devices, such as visible light cameras, infrared thermal imagers, ultrasound probes, CT scanners, MRI machines, etc., to acquire multimodal images including visible light images, infrared images, ultrasound images, CT images, MRI images, etc.
[0056] S12 features a built-in automatic calibration and synchronization mechanism, which can effectively eliminate image deviations caused by equipment differences and time delays, ensuring the consistency of acquired multimodal images in time and space, laying a good foundation for subsequent processing. For visible light cameras, the aperture, shutter speed, and ISO sensitivity can be adjusted to adapt to different lighting conditions and imaging requirements. For ultrasonic probes, appropriate frequencies and depths of focus can be set to obtain clear ultrasonic images.
[0057] In this embodiment, specifically, S2 includes the following steps:
[0058] S21. Image enhancement employs histogram equalization and adaptive filtering techniques. Histogram equalization enhances image contrast by stretching the grayscale histogram of the image, while adaptive filtering adjusts the filter parameters adaptively based on the local features of the image, which can remove noise while preserving the edge and detail information of the image.
[0059] S22. Normalization processing: Normalize the pixel values of different modal images to a uniform range of [0,1] or [-1,1]. Neural networks are sensitive to the scale of input data. Normalization can speed up the convergence of the model and improve training efficiency.
[0060] In this embodiment, the specific calculation formula for S22 is as follows:
[0061] in, The x represents the original image pixel values, and max(x) and min(x) represent the maximum and minimum pixel values of the image, respectively. These are the normalized pixel values;
[0062] In this embodiment, specifically, S3 includes the following steps:
[0063] S31: Multi-branch structure, each branch is dedicated to feature extraction for a specific modality of image. It adopts a CNN architecture with multiple convolutional and pooling layers. The convolutional layers extract features such as edges and textures of the image through convolutional kernels, while the pooling layers reduce the dimensionality and abstract the features.
[0064] S32: Feature fusion mechanism, which automatically focuses on key feature regions by calculating the attention weights of each modality feature at different scales, thereby improving the richness and accuracy of feature representation;
[0065] In this embodiment, specifically: the output features of the convolutional layer in S31 can be represented as: Where I represents the input image, W is the convolution kernel weight, b is the bias term, and * indicates the convolution operation. For activation functions;
[0066] In this embodiment, specifically, S4 includes the following steps:
[0067] S41: Dataset partitioning. The collected multimodal image dataset is divided into training set, validation set and test set according to a certain ratio, usually 7:2:1. The training set is used for updating the model's parameters, the validation set is used to evaluate the model's performance during the training process, thereby adjusting hyperparameters such as learning rate and regularization coefficient, and the test set is used to finally evaluate the model's generalization ability.
[0068] S42: Loss function selection. For classification tasks, the cross-entropy loss function is used, which can measure the difference between the model's predicted probability distribution and the true label distribution.
[0069] S43: Algorithm optimization. The Adam optimization algorithm is selected, which combines momentum and adaptive learning rate methods to effectively improve the convergence speed and stability of the model.
[0070] In this embodiment, specifically, the expression in S42 is: Where N represents the number of samples and C represents the number of categories. It is the true label (0 or 1) of the c-th class of the i-th sample. It is the probability predicted by the model that the i-th sample belongs to the c-th class;
[0071] The update rule for the Adam optimization algorithm in S43 is as follows:
[0072]
[0073] in and These are the first and second moments of the gradient, respectively; β1 and β2 are decay coefficients, typically taken as 0.9 and 0.999, respectively; n is the learning rate; ∈ is a constant used for numerical stabilization; θ t This represents the value of the model parameters at the t-th iteration. It is the gradient of the loss function;
[0074] In this embodiment, specifically, S5 includes the following steps:
[0075] S51: Improve the running efficiency of the model on actual devices. The trained model is optimized and compressed by model pruning and quantization. Model pruning reduces the size and computation of the model by removing unimportant weight connections. Quantization converts the floating-point weights in the model into integers or low-precision floating-point numbers, further reducing storage space and computational requirements.
[0076] S52: When a new multimodal image is input into the system, it is first enhanced and normalized by the data preprocessing module. Then, the processed image is input into the trained deep learning model. After feature extraction and fusion, the model outputs the image recognition result, including the category label and the corresponding confidence score.
[0077] Example 2:
[0078] See also Figures 1-2 As shown, this invention provides a method for a multimodal image recognition system based on deep learning, comprising the following steps:
[0079] A1: Acquire multimodal image data through various image acquisition devices;
[0080] A2: Perform enhancement and normalization processing on the acquired images;
[0081] A3: Select and build a deep learning model architecture suitable for multimodal image recognition;
[0082] A4: Prepare the training dataset, dividing it into training, validation, and test sets; determine the loss function and optimization algorithm; train the model using the training set, evaluate the model performance using the validation set, and adjust the model parameters based on the evaluation results until the model performance meets the expected requirements;
[0083] A5: Deploy the trained model into the application system to preprocess and recognize new multimodal images to obtain the final recognition results;
[0084] Working principle or structural principle: The data acquisition module acquires multimodal image data through various image acquisition devices, such as visible light images, infrared images, ultrasound images, CT images, MRI images, etc. These image data contain rich information about the target object under different imaging methods. The data preprocessing module preprocesses the acquired multimodal images. First, image enhancement is performed. Based on image characteristics and noise characteristics, enhancement parameters are automatically adjusted to improve image quality and highlight key features. Then, normalization processing is performed to unify the image data of different modalities to the same numerical range for easy subsequent processing. The deep learning model construction module constructs a multi-branch deep convolutional neural network architecture. Each branch performs feature extraction for a specific modality of image. During feature extraction, features such as edges and textures are extracted through convolutional layers and pooling layers. Finally, a multi-scale feature fusion method based on an attention mechanism is used to calculate... The system calculates the attention weights of each modality feature at different scales, automatically focuses on key feature regions, and fuses the weighted feature maps to form a comprehensive feature representation. The model training module is responsible for training the deep learning model, preparing the training dataset and dividing it into training, validation, and test sets. The cross-entropy loss function is used to measure the difference between the model's predicted probability distribution and the true label distribution. The Adam optimization algorithm is used to update the parameters. The model is trained using the training set and its performance is evaluated using the validation set. The model parameters are adjusted based on the evaluation results until the model performance meets the expected requirements. The system deployment and application module deploys the trained deep learning model to real-world application scenarios. After preprocessing the new input multimodal images, they are input into the trained model for recognition. The model outputs the image's category label and corresponding confidence score, achieving accurate recognition of multimodal images.
[0085] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A multimodal image recognition system based on deep learning, characterized in that: Includes the following steps: S1, Data Acquisition Module, is the front-end entry point of the entire system. During the acquisition process, it finely adjusts the parameters of the device according to the specific application scenario to ensure that the acquired image data has high quality and high availability. S2, the data preprocessing module, improves image quality and unifies data format, preparing for subsequent deep learning model processing; S3, the deep learning model building module, is responsible for learning the features of multimodal images and performing classification and recognition, and is the core part of the system; S4, Model Training Module, is responsible for training the constructed deep learning model so that it can accurately recognize multimodal images; S5, the system deployment and application module, deploys the trained deep learning model to real-world application scenarios, enabling rapid and accurate recognition of newly input multimodal images.
2. The deep learning-based multimodal image recognition system according to claim 1, characterized in that: S1 includes the following steps: S11 is equipped with a variety of image acquisition device interfaces, supporting high-resolution, multi-band and different imaging principle image inputs, ensuring that the acquired multimodal images have high complementarity in terms of spatial resolution, spectral resolution, etc., providing rich and reliable data sources for subsequent feature fusion; S12 features a built-in automatic calibration and synchronization mechanism, which can effectively eliminate image deviations caused by equipment differences and time delays, ensuring the spatiotemporal consistency of acquired multimodal images and laying a good foundation for subsequent processing.
3. The deep learning-based multimodal image recognition system according to claim 2, characterized in that: S2 includes the following steps: S21. Image enhancement employs histogram equalization and adaptive filtering techniques. Histogram equalization enhances image contrast by stretching the grayscale histogram of the image, while adaptive filtering adjusts the filter parameters adaptively based on the local features of the image, which can remove noise while preserving the edge and detail information of the image. S22. Normalization processing: Normalize the pixel values of different modal images to a uniform range of [0,1] or [-1,1]. Neural networks are sensitive to the scale of input data. Normalization can speed up the convergence of the model and improve training efficiency.
4. The deep learning-based multimodal image recognition system according to claim 3, characterized in that: The calculation formula for S22 is as follows: in, The x represents the original image pixel values, and max(x) and min(x) represent the maximum and minimum pixel values of the image, respectively. These are the normalized pixel values.
5. The deep learning-based multimodal image recognition system according to claim 4, characterized in that: S3 includes the following steps: S31: Multi-branch structure, each branch is dedicated to feature extraction for a specific modality of image. It adopts a CNN architecture with multiple convolutional and pooling layers. The convolutional layers extract features such as edges and textures of the image through convolutional kernels, while the pooling layers reduce the dimensionality and abstract the features. S32: Feature fusion mechanism, which automatically focuses on key feature regions by calculating the attention weights of each modality feature at different scales, thereby improving the richness and accuracy of feature representation.
6. The deep learning-based multimodal image recognition system according to claim 5, characterized in that: The output features of the convolutional layer in S31 can be expressed as follows: Where I represents the input image, W is the convolution kernel weight, b is the bias term, and * indicates the convolution operation. This is the activation function.
7. The deep learning-based multimodal image recognition system according to claim 6, characterized in that: S4 includes the following steps: S41: Dataset partitioning. The collected multimodal image dataset is divided into training set, validation set and test set according to a certain ratio, usually 7:2:
1. S42: Loss function selection. For classification tasks, the cross-entropy loss function is used, which can measure the difference between the model's predicted probability distribution and the true label distribution. S43: Algorithm optimization. The Adam optimization algorithm is selected, which combines momentum and adaptive learning rate methods, and can effectively improve the convergence speed and stability of the model.
8. The deep learning-based multimodal image recognition system according to claim 7, characterized in that: The expression in S42 is: Where N represents the number of samples and C represents the number of categories. It is the true label (0 or 1) of the c-th class of the i-th sample. It is the probability predicted by the model that the i-th sample belongs to the c-th class; The update rule for the Adam optimization algorithm in S43 is as follows: in and These are the first and second moments of the gradient, respectively; β1 and β2 are decay coefficients, typically taken as 0.9 and 0.999, respectively; n is the learning rate; ∈ is a constant used for numerical stabilization; θ t This represents the value of the model parameters at the t-th iteration. It is the gradient of the loss function.
9. The deep learning-based multimodal image recognition system according to claim 8, characterized in that: S5 includes the following steps: S51: Improve the efficiency of the model on real devices by using model pruning and quantization methods to optimize and compress the trained model; S52: When a new multimodal image is input into the system, it is first enhanced and normalized by the data preprocessing module. Then, the processed image is input into the trained deep learning model. After feature extraction and fusion, the model outputs the image recognition result, including the category label and the corresponding confidence score.
10. The method of a deep learning-based multimodal image recognition system according to any one of claims 1-9, characterized in that: Includes the following steps: A1: Acquire multimodal image data through various image acquisition devices; A2: Perform enhancement and normalization processing on the acquired images; A3: Select and build a deep learning model architecture suitable for multimodal image recognition; A4: Prepare the training dataset, dividing it into training, validation, and test sets; determine the loss function and optimization algorithm; train the model using the training set, evaluate the model performance using the validation set, and adjust the model parameters based on the evaluation results until the model performance meets the expected requirements; A5: Deploy the trained model into the application system to preprocess and recognize new multimodal images to obtain the final recognition result.