An image classification method based on early intervention strategy and multi-modal image

Abstract

This invention relates to an image classification method for multimodal images based on an early intervention strategy, comprising: an image acquisition step: acquiring input multimodal images; an early intervention step: taking the image input of each modality as the target modality in turn, and using the images of other modalities as reference modalities relative to the target modality, performing feature extraction on each reference modality to obtain cross-modal information relative to the target modality; a feature extraction step: performing feature extraction based on each target modality and the cross-modal information relative to each target modality to obtain cross-modal features corresponding to each target modality; a feature fusion step: performing feature fusion of all modalities based on the cross-modal features of each target modality to obtain fused features; and an image classification step: performing image classification processing based on the fused features.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to an image classification method for multimodal images based on an early intervention strategy. Background Technology

[0002] Multimodal imaging-based computer vision technology utilizes complementary information from different imaging devices (such as visible light, infrared, depth cameras, CT, MRI, and fundus cameras) to achieve accurate identification of target objects in complex scenes. For example, in the medical field, the combination of color fundus photography (CFP) and optical coherence tomography (OCT) is crucial for the diagnosis of fundus diseases.

[0003] Research has found that the current mainstream multimodal image feature extraction and fusion technology generally adopts the paradigm of "first single-modal extraction, then multimodal fusion". The routine process is as follows: (1) Use a visual encoder of a specific modality to extract the feature embedding of each modality independently; (2) After the feature extraction is completed, feature fusion is performed through the fusion module.

[0004] In existing technologies, the interaction of multimodal information lags behind the single-modal feature extraction process. Under this "lagging fusion" paradigm, the single-modal encoder cannot perceive information from other modalities when extracting features. This prevents complementary information from playing a corrective or guiding role in feature extraction, potentially leading to information loss and limiting the final classification accuracy. Summary of the Invention

[0005] To address the aforementioned problems, the present invention aims to propose an image classification method for multimodal images based on an early intervention strategy. This method introduces cross-modal information at the initial stage of feature extraction, avoiding the information loss problem caused by the inability of a single-modal encoder to perceive information from other modalities when extracting features under the traditional "lagging fusion" paradigm. This improves the accuracy of image classification.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] Firstly, this application provides an image classification method for multimodal images based on early intervention strategies, including: Image acquisition steps: Acquire multimodal image input; Early intervention steps: Take the image input of each modality as the target modality in turn, and use the images of other modalities as reference modalities relative to the target modality. Extract features from each reference modality to obtain cross-modal information relative to the target modality. Feature extraction steps: Based on each target modality and the cross-modal information relative to each target modality, feature extraction is performed to obtain the cross-modal features corresponding to each target modality; Feature fusion step: Based on the cross-modal features of each target modality, perform feature fusion of all modalities to obtain fused features; Image classification steps: Based on the fused features, perform image classification processing.

[0008] In one implementation, the method includes: setting a first feature extractor and a second feature extractor, wherein: The first feature extractor is used to extract features for each target modality and its cross-modal information in the feature extraction step to obtain cross-modal features; The second feature extractor is used in the early intervention step to extract features from a reference modality relative to the target modality to obtain intervention markers as cross-modal information.

[0009] In one implementation, the image acquisition step further includes a preprocessing step of convolving the image of each modality to obtain an initial image block embedding sequence for each modality.

[0010] In one implementation, the method further includes: setting an adapter module for mapping and integrating intervention markers of the reference modality in an early intervention step, generating an intervention marker sequence, and enabling the intervention marker sequence to be concatenated with the initial image patch embedding sequence of the target modality for feature extraction, thereby satisfying the compatibility of the target modality feature space.

[0011] In one implementation, the first feature extractor and the second feature extractor are visual feature encoders with the same structure.

[0012] In one implementation, during the feature fusion step, all cross-modal features are fused according to the weights of the cross-modal features of each target modality.

[0013] In one implementation, the classification result of each modality is obtained based on the cross-modal features of each target modality, and the weight of the cross-modal features of each target modality is determined based on the cross-modal features of each modality.

[0014] In one implementation, the image classification step involves obtaining the final classification result based on a weighted sum calculation, according to the classification results of each modality.

[0015] Secondly, a feature extraction and fusion method for multimodal images based on early intervention strategies is provided, including: Image acquisition steps: Acquire multimodal image input; Early intervention steps: Take the image input of each modality as the target modality in turn, and use the images of other modalities as reference modalities relative to the target modality. Extract features from each reference modality to obtain cross-modal information relative to the target modality. Feature extraction steps: Based on each target modality and the cross-modal information relative to each target modality, feature extraction is performed to obtain the cross-modal features corresponding to each target modality; Feature fusion step: Based on the cross-modal features of each target modality, feature fusion of all modalities is performed to obtain fused features.

[0016] Thirdly, a computer storage medium is provided, storing a computer program, which is executed by a processor to implement the methods described in the first and second aspects.

[0017] The present invention has the following advantages due to the adoption of the above technical solutions: The present invention avoids the problem of information loss caused by the inability of a single-modal encoder to perceive information from other modalities when extracting features under the traditional "lagging fusion" paradigm, thereby improving the accuracy of image classification. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a method in one embodiment of this application; Figure 2 This is a schematic diagram of the processing flow in an application example. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention are within the scope of protection of the present invention.

[0020] This application provides an image classification method for multimodal images based on early intervention strategies, including: Image acquisition steps: Acquire multimodal image input; Early intervention steps: Take the image input of each modality as the target modality in turn, and use the images of other modalities as reference modalities relative to the target modality. Extract features from each reference modality to obtain cross-modal information relative to the target modality. Feature extraction steps: Based on each target modality and the cross-modal information relative to each target modality, feature extraction is performed to obtain the cross-modal features corresponding to each target modality; Feature fusion step: Based on the cross-modal features of each target modality, perform feature fusion of all modalities to obtain fused features; Image classification steps: Based on the fused features, perform image classification processing.

[0021] The following is based on Figure 1 In a more detailed embodiment, the method of this application is described. And based on... Figure 2 The effect on a specific application is illustrated in a concrete application example.

[0022] Detailed Implementation Examples In the detailed embodiments of this application, it is assumed that the input image contains M modalities, while Figure 1 The diagram only illustrates the features of two modes; further details on other modes can be deduced from the accompanying drawings, which will not be provided here.

[0023] In this detailed embodiment, the images of each modality are preprocessed (including convolution, etc.) to obtain the initial image patch embedding sequence for each modality, and further feature extraction and fusion are performed based on this initial image patch embedding sequence.

[0024] Suppose the input contains images with M modalities. In the "early intervention" framework, each modality is sequentially designated as the target modality, denoted as t, while the remaining modalities serve as reference modalities, denoted as R = {1, 2, ..., M} / {t}. In this way, each modality alternates between the roles of target modality and reference modality during training.

[0025] In this detailed embodiment, a first feature extractor and a second feature extractor are provided. They may be the same or different. When the first and second feature extractors use network modules with the same structure, their network parameters may be the same or different.

[0026] The first feature extractor and the second feature extractor can be a visual feature encoder (VE), where the first feature extractor serves as the main VE and the second feature extractor serves as the auxiliary VE.

[0027] The primary VE is used to extract classification features when the modality is the target modality, while the auxiliary VE generates an intervention token ([INT] token) when the modality is the reference modality. This invention comprises three main steps: First, an intervention token is generated based on the reference modality; second, features are extracted from the target modality with the assistance of the intervention token; and finally, the results from all target modalities are fused. The specific steps are as follows: (1) Intervention Marker Generation. For each reference modality r (r belongs to R), we use the corresponding auxiliary VE to generate its intervention marker, that is, input the reference modality image into the auxiliary VE to obtain the corresponding feature vector. For multiple reference modalities, we collect the feature vectors output by all auxiliary VEs and combine them into a sequence. Since these markers will intervene in the early stage of target modality feature extraction, in order to improve their compatibility with the target modality feature space, we introduce an adapter module. The adapter uses a two-layer multilayer perceptron (MLP) to map and integrate the feature vectors of the reference modalities, and finally generate the intervention marker sequence.

[0028] (2) Target modality feature extraction with intervention markers. For the current target modality t, we use the corresponding master image velocimetry (VE) to extract the features required for classification. To fully utilize the intervention markers in the initial stage of feature extraction, we concatenate the intervention marker sequence with the initial image patch embedding sequence of the target modality to form a new input sequence. Subsequently, this concatenated sequence is fed into the master VE for feature extraction. In this way, the intervention markers can interact with the target modality features in the master VE, thereby continuously improving the feature extraction process of the target modality. Repeating the above process for each target modality yields a set of multimodal feature representations.

[0029] (3) Feature fusion. For each modality feature representation obtained in the previous step, a classification result is obtained. At the same time, a fully connected layer is used to calculate the weight of each classification result based on all feature representations. The final classification result is the weighted sum of the classification results of each modality.

[0030] The following example illustrates the effectiveness of this method.

[0031] Application Example 1 To verify the effectiveness of this invention, comparative experiments were conducted on three different datasets: MMC-AMD, Derm7pt, and MRNet. For the evaluation metric, we calculated the average precision (AP) for each class, and then averaged the AP across different classes to obtain the mAP as the evaluation metric. The experiments compared six current multimodal fusion methods: MM-MIL, SFuion, DynMM, RadDiag, CosCat, and MMRAD. For a fair comparison, all methods used the same feature extractor (Visual Foundation Model, VFM). The results are shown in Table 1. This invention outperforms current methods on all three datasets.

[0032] Table 1

[0033] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0034] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0035] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0036] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image classification method for multimodal images based on early intervention strategies, characterized in that, include: Image acquisition steps: Acquire multimodal image input; Early intervention steps: Take the image input of each modality as the target modality in turn, and use the images of other modalities as reference modalities relative to the target modality. Extract features from each reference modality to obtain cross-modal information relative to the target modality. Feature extraction steps: Based on each target modality and the cross-modal information relative to each target modality, feature extraction is performed to obtain the cross-modal features corresponding to each target modality; Feature fusion step: Based on the cross-modal features of each target modality, perform feature fusion of all modalities to obtain fused features; Image classification steps: Based on the fused features, perform image classification processing.

2. The method according to claim 1, characterized in that, include: Set up a first feature extractor and a second feature extractor, where: The first feature extractor is used to extract features for each target modality and its cross-modal information in the feature extraction step to obtain cross-modal features; The second feature extractor is used in the early intervention step to extract features from a reference modality relative to the target modality to obtain intervention markers as cross-modal information.

3. The method according to claim 2, characterized in that, The image acquisition step also includes preprocessing by convolving the image for each modality to obtain an initial image block embedding sequence for each modality.

4. The method according to claim 3, characterized in that, The method further includes: setting an adapter module for mapping and integrating intervention markers of the reference modality in an early intervention step, generating an intervention marker sequence, and enabling the intervention marker sequence to be concatenated with the initial image patch embedding sequence of the target modality for feature extraction, thereby satisfying the compatibility of the target modality feature space.

5. The method according to claim 2, characterized in that, The first and second feature extractors are visual feature encoders with the same structure.

6. The method according to claim 1, characterized in that, In the feature fusion step, all cross-modal features are fused according to the weights of the cross-modal features of each target modality.

7. The method according to claim 6, characterized in that, Based on the cross-modal features of each target modality, the classification result of each modality is obtained, and based on the cross-modal features of each modality, the weight of the cross-modal features of each target modality is determined.

8. The method according to claim 7, characterized in that, In the image classification step, the final classification result is obtained based on the classification result of each modality and the weighted sum calculation.

9. A method for feature extraction and fusion of multimodal images based on early intervention strategies, characterized in that, include: Image acquisition steps: Acquire multimodal image input; Early intervention steps: Take the image input of each modality as the target modality in turn, and use the images of other modalities as reference modalities relative to the target modality. Extract features from each reference modality to obtain cross-modal information relative to the target modality. Feature extraction steps: Based on each target modality and the cross-modal information relative to each target modality, feature extraction is performed to obtain the cross-modal features corresponding to each target modality; Feature fusion step: Based on the cross-modal features of each target modality, feature fusion of all modalities is performed to obtain fused features.

10. A computer storage medium, characterized in that, The device contains a computer program that is executed by a processor to implement the method described in any one of claims 1 to 8.

Images