Method, electronic device and computer readable storage medium for multi-modal alignment

Abstract

The application relates to a multi-modal alignment method, an electronic device and a computer readable storage medium, comprising the following steps: a first module processes first data to obtain a first feature vector; a second module processes second data to obtain a second feature vector; the first feature vector and the second feature vector have feature distribution differences; a semantic feature extraction module is used to process disease diagnosis label text to obtain a semantic feature vector; a semantic alignment module is used to align the semantic feature vector with the first feature vector and the second feature vector respectively to reduce the feature distribution differences in a shared semantic space; the application has the beneficial effect that the semantic feature vector is aligned with the first feature vector and the second feature vector respectively, semantic integration of cross-modal features is realized, and the modal heterogeneity problem caused by different data sources and structures is reduced or solved.

Description

Technical Field

[0001] This invention relates to the fields of computer science and biomedical engineering, specifically to methods for multimodal alignment, electronic devices, and computer-readable storage media. Background Technology

[0002] Cardiovascular disease is a major global public health threat, and early and accurate identification is crucial for prevention and control. Currently, multimodal diagnostic data such as electrocardiograms and electronic health records are widely used, but traditional methods mainly rely on unimodal analysis or simple multimodal fusion. Unimodal methods ignore individual patient differences and cross-group generalization ability, while existing fusion methods, although attempting to integrate multi-source data, neglect the characteristic heterogeneity between different modalities, resulting in poor fusion effects and limited detection accuracy and generalization ability. Summary of the Invention

[0003] To achieve the above objectives, the present invention provides the following technical solution: a multimodal alignment method to solve or reduce the problem of feature heterogeneity between different modalities in the background art; Includes the following steps: The first module processes the first data and obtains the first feature vector; The second module processes the second data to obtain the second feature vector; The first eigenvector and the second eigenvector have different feature distributions; The semantic feature extraction module processes the disease diagnosis label text to obtain semantic feature vectors; The semantic alignment module aligns the semantic feature vector with the first and second feature vectors respectively to reduce the difference in feature distribution in the shared semantic space. The aligned first and second feature vectors are merged to generate a merged feature vector; The fused feature vector is input into the classifier to obtain the detection result.

[0004] As a further improvement of the present invention, the semantic alignment module includes a contrastive loss function; The contrast loss function constrains the consistency between the first feature vector and the semantic feature vector, and between the second feature vector and the semantic feature vector, in the shared semantic space.

[0005] Improve the efficiency of fusing multiple feature vectors in the future.

[0006] As a further improvement of the present invention, the contrast loss function is a bidirectional contrast loss; The bidirectional comparison loss is calculated based on the first unidirectional loss and the second unidirectional loss; The first one-way loss uses the first feature vector as a benchmark to calculate its similarity relationship with the second feature vector; The second one-way loss uses the second feature vector as a benchmark to calculate its similarity relationship with the first feature vector.

[0007] As a further improvement of the present invention, the ViT visual encoder of the CLIP model is used as the backbone network; Bottleneck modules are inserted into the multi-layer backbone network to extract multi-scale first data features and obtain the first feature vector.

[0008] The multi-layer bottleneck module can output vectors at multiple scales, fully extracting the features of the vectors.

[0009] As a further improvement of the present invention, the semantic feature vector is aligned with the first feature vector of the intermediate level output by multiple bottleneck modules.

[0010] As a further improvement of the present invention, the second data is vectorized using word frequency statistics; The second feature vector is generated by processing a lightweight neural network that includes parallel linear subnetworks.

[0011] Lightweight networks can effectively prevent overfitting of tabular data.

[0012] As a further improvement of the present invention, the first data is electrocardiogram data and the second data is electronic health record data.

[0013] Align these two heterogeneous data sets to achieve full integration.

[0014] An electronic device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the multimodal alignment method as described above.

[0015] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multimodal alignment method as described above.

[0016] The beneficial effects of this invention are: by aligning the semantic feature vector with the first and second feature vectors respectively, semantic integration of cross-modal features is achieved, and features of different modalities are mapped into the same semantic space, thereby reducing or solving the modal heterogeneity problem caused by different data sources and structures. Attached Figure Description

[0017] Figure 1 This is the electrocardiogram (ECG) of the present invention.

[0018] Figure 2 This is a schematic diagram of the ECG feature extraction module of the present invention.

[0019] Figure 3This is a schematic diagram of the EHR feature extraction module of the present invention.

[0020] Figure 4 This is a schematic diagram of the semantic feature extraction module of the present invention.

[0021] Figure 5 This is a schematic diagram of the semantic guidance feature alignment module of the present invention.

[0022] Figure 6 This is a flowchart illustrating the fusion and classification module of the present invention.

[0023] Figure 7 This is a schematic diagram of the overall module flow of the present invention. Detailed Implementation

[0024] The following detailed implementation examples illustrate the specific implementation of the cardiovascular disease detection method based on semantically guided multimodal alignment of this invention. The steps are as follows: Step 1: ECG Feature Extraction: Step 1.1: Use CLIP's ViT visual encoder as the backbone network, with an input ECG image of size x3×h×w as shown below. Figure 1 As shown, the image is then divided into 225 non-overlapping 16×16 image blocks. These image blocks are then projected into a 1024-dimensional token embedding to form sequence F. 255×1024； Step 1.2: Divide the 24-layer Transformer into 3 stages (every 8 layers), inserting a bottleneck module after each stage to extract multi-level semantic features. The output of the three Transformer stages is represented as F. l , where l ∈ {1, 2, 3}, they pass through corresponding bottleneck networks. Each bottleneck first passes F through a linear layer. l The dimension is reduced to 768, resulting in an alignment vector I. l The second linear layer then restores the dimension to 1024 and adds the result to the original F by summing the residuals. l The resulting enhanced features The formula for enhancing features is as follows: ; in (.) (.), (.) represent the dimensionality reduction function, the ReLU activation function, and the recovery function, respectively. Step 1.3: Use CLIP's visual projection layer to remove the last bottleneck. The output is projected onto 768 dimensions to generate the final ECG feature vector FECG; the flowchart of the entire step 1 is as follows. Figure 2As shown; the FECG characteristic formula is as follows: ; Step 2: EHR Feature Extraction: Step 2.1: Use TF-IDF vectorization to process EHR records, and select high-frequency features to generate an l-dimensional vector z. in And input to an EHR encoder consisting of two linear layers; Step 2.2: Process z using a two-stage lightweight network; first, process z... in Projected onto a 400-dimensional space; the formula is as follows: ; in (.) is a linear transformation. (.) is the activation function. (.) indicates a drop operation. ∈ R l The second linear mapping layer consists of four parallel lightweight linear sublayers, each with input and output dimensions of 100 and 192, respectively. Vector z out Divided evenly into four segments Where l ∈ {1,2,3,4}, each segment is processed independently by one of the sub-layers. The corresponding output is represented as... The formula is as follows: ; Step 2.3: Concatenate the outputs of the four sub-layers to generate a 768-dimensional EHR feature F. EHR The flowchart for the entire process in step 2 is as follows: Figure 3 As shown; the formula is as follows: ; Where cat(.) represents the chaining operation. Step 3: Semantic Feature Extraction Step 3.1: Process the labeled template text using a pre-trained BERT encoder. The encoder's parameters are frozen during training. Embed class labels into predefined templates to construct cue phrases. Use the template "The sample of a {}patient" to generate cue phrases. Then convert these text cue phrases into a numerical tag index based on a fixed vocabulary. Step 3.2: The tag index is passed to the text encoder, outputting a 768-dimensional semantic feature vector F. prompt The flowchart for the entire process of step 3 is as follows: Figure 4 As shown; Step 4: Semantic-guided feature alignment: Step 4.1: Multi-scale ECG feature alignment: Align Fprompt Align with the intermediate features (I1, I2, I3) output by the three bottleneck modules; Step 4.2: EHR Feature Alignment: Constraint F EHR With F prompt Consistency in the shared semantic space; Step 4.3: Minimize the difference in feature distribution between modes using contrastive loss; the flowchart of the entire process in step 4 is as follows. Figure 5 As shown; Step 5: Integration and Classification: Step 5.1: F ECG Features and F EHR Features are concatenated into a fused feature vector F fusion The formula is as follows: ; Step 5.2: Input the data into the linear classifier to obtain the detection results for cardiovascular diseases; the flowchart of the entire process in step 5 is as follows. Figure 6 As shown; Step 6: Training and Optimization Step 6.1: Given two vectors v1 and v2 of the same dimension, their cosine similarity is calculated as v1·v2T, reflecting their angular proximity in the embedding space. To guide alignment, a contrastive loss function is used, as follows: ; in and Let represent the feature representation of the i-th sample from modes M1 and M2, where τ is the temperature parameter; Step 6.2: Take the average value of the two-way contrast loss to represent the overall alignment loss L2; ​​the formula is as follows: ; Step 6.3: Use the cross-entropy loss function to measure the deviation between the predicted probability and the true label, as shown in the following formula: ; Where y i ∈ {0, 1} represents the true label, p i This is the predicted probability of the positive class, where N is the number of samples. Step 6.4: Combine the EHR alignment loss, ECG multi-scale alignment loss, and classification cross-entropy loss to optimize the overall model parameters using gradient descent. The final integrated formula is as follows: ; ; + .

[0025] The above description is merely a preferred embodiment of the present invention, and the scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for multimodal alignment, characterized in that, Includes the following steps: The first module processes the first data and obtains the first feature vector; The second module processes the second data to obtain the second feature vector; The first eigenvector and the second eigenvector have different feature distributions; The semantic feature extraction module processes the disease diagnosis label text to obtain semantic feature vectors; The semantic alignment module aligns the semantic feature vector with the first and second feature vectors respectively to reduce the difference in feature distribution in the shared semantic space. The aligned first and second feature vectors are merged to generate a merged feature vector; The fused feature vector is input into the classifier to obtain the detection result.

2. The multimodal alignment method according to claim 1, characterized in that, The semantic alignment module includes a contrastive loss function; The contrast loss function constrains the consistency between the first feature vector and the semantic feature vector, and between the second feature vector and the semantic feature vector, in the shared semantic space.

3. The multimodal alignment method according to claim 2, characterized in that, The contrast loss function is a bidirectional contrast loss; The bidirectional comparison loss is calculated based on the first unidirectional loss and the second unidirectional loss; The first one-way loss uses the first feature vector as a benchmark to calculate its similarity relationship with the second feature vector; The second one-way loss uses the second feature vector as a benchmark to calculate its similarity relationship with the first feature vector.

4. The multimodal alignment method according to claim 1, characterized in that, The ViT visual encoder using the CLIP model is used as the backbone network; Bottleneck modules are inserted into the multi-layer backbone network to extract multi-scale first data features and obtain the first feature vector.

5. The multimodal alignment method according to claim 4, characterized in that, Align the semantic feature vectors with the first feature vectors of the intermediate level output by multiple bottleneck modules.

6. The multimodal alignment method according to claim 1, characterized in that, The second data is vectorized using word frequency statistics; The second feature vector is generated by processing a lightweight neural network that includes parallel linear subnetworks.

7. The multimodal alignment method according to claim 1, characterized in that, The first data is electrocardiogram data, and the second data is electronic health record data.

8. An electronic device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes a computer program, it implements the method of multimodal alignment as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of multimodal alignment as described in any one of claims 1-7.

Images