Multi-modal data fusion method and device based on cross attention mechanism

Abstract

The embodiment of the invention provides a multi-modal data fusion method and device based on a cross attention mechanism. The method comprises the following steps: acquiring biological information features and radiomics features of a target object; carrying out feature weighting and interactive processing on the biological information features and the radiomics features based on a cross attention mechanism, and generating the biological information features based on radiomics feature attention and the radiomics features based on the biological information feature attention; carrying out feature splicing on the four features to generate spliced features; and performing feature fusion on the spliced features based on a self-attention mechanism to generate fusion feature representation of the target object. The multi-modal data fusion method and device provided by the invention show higher flexibility and pertinence in the aspects of feature extraction and weighting.

Description

Technical Field

[0001] The present disclosure relates to the field of medical information technology, and in particular to a multimodal data fusion method and device based on a cross-attention mechanism. Background Art

[0002] In the field of medical imaging analysis, accurate disease prediction and treatment effect evaluation are crucial for clinical decision-making. To this end, the fusion of multimodal data has become a key research direction, because data from different modalities can provide complementary information and improve the accuracy of prediction and evaluation. However, existing fusion methods are often limited to simple combinations at the result level, such as voting or weighted averaging, which fail to fully utilize the interactive relationship between different modality features, resulting in poor prediction performance. In view of the above shortcomings, a new multimodal data fusion method is urgently needed to achieve higher accuracy and generalization ability in effect evaluation and state progression prediction. Summary of the invention

[0003] Some embodiments of the present disclosure provide a multimodal data fusion method and device based on a cross-attention mechanism, which can fully consider the complementarity and interaction between different modal features in the feature extraction and fusion process, and achieve higher accuracy and generalization ability.

[0004] The present disclosure provides a multimodal data fusion method based on a cross attention mechanism, comprising the following steps:

[0005] Get the biometric features of the target object bio And radiomics features rad ;

[0006] Based on the cross attention mechanism, the biological information feature bio And radiomics features rad Perform feature weighting and interaction processing to generate bioinformatic feature features based on radiomics feature attention rad2bio And the imaging omics feature feature based on bioinformatic feature attention bio2rad :

[0007] For the biometric feature feature bio 、Radiognomics features rad , bioinformatic features based on radiomics feature attention rad2bio , Imaging omics features based on bioinformatic feature attention bio2rad Perform feature splicing to generate splicing feature comb ;as well as

[0008] Based on the self-attention mechanism, the splicing feature comb Perform feature fusion to generate a fused feature representation of the target object.

[0009] Optionally, the biometric feature feature bio Including RNA signature, the radiomics signature effect rad Includes shape, texture, and / or intensity features extracted from medical images.

[0010] Optionally, the biometric feature feature is subjected to a cross-attention mechanism. bio And radiomics features rad Perform feature weighting and interaction processing to generate bioinformatic feature features based on radiomics feature attention rad2bio And the imaging omics feature feature based on bioinformatic feature attention bio2rad The step includes: bio And radiomics features rad Input is a criss-cross attention model, which is trained by a joint network based on a multi-layer perceptron and an attention mechanism.

[0011] Optionally, the cross-attention model performs spatial mapping of features of different modalities through a multi-layer perceptron, the dimensions of the bioinformatics features and imaging omics features obtained through spatial mapping are consistent, and the softmax function is used to form attention weights for the bioinformatics features and imaging omics features obtained through spatial mapping.

[0012] Optionally, the calculation formula for forming attention weights using the softmax function for the biological information features and radiomics features obtained by spatial mapping is:

[0013]

[0014] Among them, atten A2B is the weighted attention weight of feature A based on feature B, i is the weight of the i-th feature in feature B, and K is the dimension of feature B, where feature A is a biological information feature feature bio , the feature B is the radiomics feature feature rad .

[0015] Optionally, the self-attention mechanism is used to perform the splicing feature combIn the step of performing feature fusion and generating a fused feature representation of the target object, the importance of each element in the input feature sequence is dynamically weighted using a self-attention model.

[0016] Optionally, the step of dynamically weighting the importance of each element in the input feature sequence using the self-attention model includes: comb Perform linear transformation to generate queries, keys and values; calculate the similarity between the query and the key to determine the importance of each feature; normalize the similarity scores using the softmax function to obtain weights, and use the weights to perform weighted summation on the values ​​to generate weighted feature outputs; the weighted feature outputs are processed through a fully connected layer to generate a fused feature representation of the target object.

[0017] The present disclosure provides a multimodal data fusion device based on a cross attention mechanism, comprising:

[0018] Feature acquisition module, used to obtain the biometric features of the target object bio And radiomics features rad ;

[0019] A cross attention processing module is used to process the biological information feature based on the cross attention mechanism. bio And radiomics features rad Perform feature weighting and interaction processing to generate bioinformatic feature features based on radiomics feature attention rad2bio And the imaging omics feature feature based on bioinformatic feature attention bio2rad ;

[0020] Feature splicing module, used to combine the biometric features bio 、Radiognomics features rad , bioinformatic features based on radiomics feature attention rad2bio , Imaging omics features based on bioinformatic feature attention bio2rad Perform feature splicing to generate splicing feature comb ;as well as

[0021] The self-attention fusion module is used to fusion the spliced ​​features based on the self-attention mechanism. comb Perform feature fusion to generate a fused feature representation of the target object.

[0022] An embodiment of the present disclosure provides an electronic device, including:

[0023] Processor; and

[0024] A memory, configured to store executable instructions of the processor;

[0025] The processor is configured to perform the method by executing the executable instructions.

[0026] An embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method described above is implemented.

[0027] Compared with the prior art, the multimodal data fusion method and device based on the cross-attention mechanism provided by the embodiment of the present disclosure introduces a feature fusion framework based on the cross-attention mechanism, and performs end-to-end weighting of bioinformatics and imaging features to more finely model the complementary relationship between each modality of data. Through this multi-layer perceptron and cross-attention module, the model shows greater flexibility and pertinence in feature extraction and weighting, thereby overcoming the limitations of traditional methods and achieving higher accuracy and generalization in treatment effect evaluation and state progression prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0029] Figure 1 A flow chart of a multimodal data fusion method based on a cross-attention mechanism provided for some embodiments of the present disclosure;

[0030] Figure 2 An overall structural diagram of a network involved in a multimodal data fusion method based on a cross-attention mechanism provided in some embodiments of the present disclosure;

[0031] Figure 3 An algorithm flow chart of a weighted attention module involved in some embodiments of the present disclosure;

[0032] Figure 4 A schematic diagram of the structure of a multimodal data fusion device based on a cross-attention mechanism provided in some embodiments of the present disclosure;

[0033] Figure 5 A schematic diagram of the connection structure of electronic devices provided in some embodiments of the present disclosure. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the present disclosure will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0035] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments, and are not intended to limit the present disclosure. The singular forms "a", "said" and "the" used in the embodiments of the present disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two.

[0036] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0037] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a product or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such product or device. In the absence of more restrictions, the elements defined by the sentence "comprises a" do not exclude the presence of other identical elements in the product or device including the elements.

[0038] In related technologies, when dealing with feature fusion, multimodal fusion technology usually adopts a strategy of extracting features of each modality separately and then performing simple fusion in the final stage. This post-fusion method easily leads to the loss of important information between features and makes it difficult to capture the complex relationship between high-dimensional features. In addition, most methods lack an effective attention mechanism, making it difficult for the model to dynamically identify which features contribute more to the prediction when weighting features.

[0039] To this end, the present disclosure provides a multimodal data fusion method based on a cross attention mechanism, comprising the following steps:

[0040] S110, obtaining the biometric features of the target object bio And radiomics features rad ;

[0041] S120, based on the cross attention mechanism, the biological information feature bioAnd radiomics features rad Perform feature weighting and interaction processing to generate bioinformatic feature features based on radiomics feature attention rad2bio And the imaging omics feature feature based on bioinformatic feature attention bio2rad ;

[0042] S130, the biometric information feature bio 、Radiognomics features rad , bioinformatic features based on radiomics feature attention rad2bio , Imaging omics features based on bioinformatic feature attention bio2rad Perform feature splicing to generate splicing feature comb ;as well as

[0043] S140, based on the self-attention mechanism, the splicing feature feature comb Perform feature fusion to generate a fused feature representation of the target object.

[0044] The optional embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0045] Example 1

[0046] refer to Figure 1 and Figure 2 The embodiment of the present disclosure provides a multimodal data fusion method based on a cross-attention mechanism, which can be used for feature fusion and analysis of multimodal medical data, and is particularly suitable for fusion scenarios including bioinformatics features and imaging omics features to improve the accuracy of disease prediction and diagnosis. Specifically, the method includes the following steps:

[0047] S110, obtaining the biometric features of the target object bio And radiomics features rad .

[0048] In this step, the bioinformatics feature bio Bioinformatics features usually refer to molecular-level information related to the target object, which can include RNA expression data, gene mutation data, or protein expression levels. The above features can be obtained through high-throughput sequencing or other bioinformatics experiments. bio After preprocessing (such as standardization and normalization), it is converted into structured data form to facilitate subsequent processing.

[0049] Radiomics features rad(Image features) can be extracted from the medical imaging data of the target object (such as MRI, CT, ultrasound, etc.), including texture features, shape features, intensity features, etc. in the image. The above features can be automatically extracted from the image through image analysis tools (such as Pyradiomics) or deep learning models. rad After preprocessing (such as standardization and normalization), it is converted into structured data form to facilitate subsequent processing.

[0050] S120, based on the cross attention mechanism, the biological information feature bio And radiomics features rad Perform feature weighting and interaction processing to generate bioinformatic feature features based on radiomics feature attention rad2bio And the imaging omics feature feature based on bioinformatic feature attention bio2rad .

[0051] In this step, the cross attention model can be used to achieve weighting and interaction between features. bio And radiomics features rad Input the pre-trained cross-attention model to generate bioinformatic features based on radiomics feature attention rad2bio And the imaging omics feature feature based on bioinformatic feature attention bio2rad The cross attention model is obtained by training a joint network based on a multi-layer perceptron and an attention mechanism.

[0052] The cross-attention model uses a multi-layer perceptron (MLP) to spatially map features of different modalities, and the dimensions of the biological information features and imaging features obtained through spatial mapping are consistent. bio and radiomics features rad Dimension mapping is performed to make the feature dimensions of the two consistent. Then, the softmax function is used to form attention weights for the biological information features and imaging omics features obtained by spatial mapping.

[0053] As a specific example, see Figure 3 , with feature B (e.g. radiomics feature rad ) weighted feature A (e.g. bioinformatic feature bio ) as an example, the specific algorithm flow of the cross attention mechanism is as follows:

[0054] The features of different modalities are spatially mapped through a multi-layer perceptron (MLP): Feature B is first spatially mapped through a multi-layer perceptron (MLP) model to make its feature dimension the same as feature A so that they can interact and be weighted.

[0055] Generate attention weights: Use the Softmax function on the mapped features to generate attention weights. The specific calculation formula is as follows:

[0056]

[0057] Among them, atten A2B is the weighted attention weight of feature A based on feature B, i is the weight of the i-th feature in feature B, and K is the dimension of feature B.

[0058] Use the attention weight of feature A atten A2B Perform bitwise weighting on feature B to generate weighted features. The calculation formula is as follows:

[0059]

[0060] Among them, ⊙ represents the bitwise product, Indicates feature concatenation. Finally, the weighted feature A based on the attention mechanism of feature B is formed, which is recorded as feature B2A Similarly, the attention weight of feature A can be added to feature B in the same way to generate a weighted feature based on A, denoted as feature A2B .

[0061] S130, the biometric information feature bio 、Radiognomics features rad , bioinformatic features based on radiomics feature attention rad2bio , Imaging omics features based on bioinformatic feature attention bio2rad Perform feature splicing to generate splicing feature comb .

[0062] In this step, the original features of different modes and the weighted features generated by interaction are spliced ​​along a specific dimension to form a comprehensive feature vector feature comb .

[0063]

[0064] in, Represents feature concatenation. The concatenation process retains the complete information of each modal feature, while ensuring that the interactively weighted features are effectively utilized, providing a more comprehensive input for subsequent feature fusion.

[0065] S140, based on the self-attention mechanism, the splicing feature feature comb Perform feature fusion to generate a fused feature representation of the target object.

[0066] In this step, the self-attention mechanism is used to focus on the concatenated features. comb The self-attention mechanism can generate more representative fusion features of the target object by dynamically weighting the importance of each element in the input feature sequence.

[0067] In some embodiments, a self-attention model may be used to dynamically weight the importance of each element in the input feature sequence. Specifically, the following steps may be included:

[0068] Generation of query, key, and value: For each input element (i.e., concatenated feature comb ) performs a linear transformation to generate a query, key, and value. The above transformation can be achieved by training a learnable weight matrix W q , W k , W v The specific calculation formula is:

[0069] q=W q x, k = W k x, v = W v ·x

[0070] Here, x is the feature vector input to the self-attention layer.

[0071] Similarity score calculation: Self-attention determines the importance of each feature by calculating the similarity between query q and key k. Specifically, the similarity score is calculated using the following formula:

[0072]

[0073] Where Q is the query matrix, K T is the transpose of the key matrix, d k is the dimension of the key vector. The above operation helps stabilize the gradient during training and avoid the problem of too large or too small values ​​in similarity calculation.

[0074] Weight normalization and output: Use the softmax function to normalize the similarity score to obtain the weight, and then use the weight obtained above to weight the sum of the values ​​to generate the weighted feature output. Finally, the weighted combination of each input feature is output, and the formula is:

[0075]

[0076] Fully connected layer output: The weighted feature output is further processed by a fully connected layer to generate the final fused feature representation. The fully connected layer linearly combines the weighted features through the learned weights to further enhance the feature expression capability and generate the final feature representation for the task model.

[0077] Furthermore, the embodiment of the present application uses a cross entropy loss function to train the model to measure the difference between the model prediction value and the true label. The specific expression of the cross entropy loss function is as follows:

[0078]

[0079] Where N is the total number of training samples, yi is the true label of the i-th sample, and pi is the probability of the i-th sample predicted by the model. This loss function can effectively measure the prediction deviation in the classification task, ensuring that the model is continuously optimized during the training process and gradually approaches the true label.

[0080] In setting the learning rate, the embodiment of the present application adopts the minimum batch gradient descent method to update the network weights. By using the Softmax cross entropy loss function, the model can gradually optimize the network parameters to minimize the value of the loss function.

[0081] In order to improve the generalization ability and training efficiency of the model, the embodiment of the present application adopts the cosine decay learning rate algorithm. By dynamically adjusting the learning rate, the model can gradually reduce the learning rate during the training process, avoid the oscillation during the training process caused by the excessive learning rate, and ensure better convergence effect in the later stage of training. The adjustment formula of the learning rate is as follows:

[0082]

[0083] Among them, ηt is the learning rate of the tth round: is the minimum learning rate, which is set to 0 in this embodiment: is the maximum learning rate, which is set to 0.01 in this example: T cur is the current training round; T i is the number of iteration cycles, which is set to 50 in this embodiment.

[0084] Through the above training strategy, the accuracy and robustness of the multimodal data fusion method based on the cross-attention mechanism provided in the embodiment of the present application can be improved to provide a higher prediction accuracy.

[0085] The multimodal data fusion method based on the cross-attention mechanism provided by the embodiment of the present disclosure introduces an end-to-end attention mechanism, making feature fusion more flexible and targeted. The attention mechanism can help the model to dynamically weight and fuse features in the early stage, so that features that contribute more to the result prediction get higher weights. Through this mechanism, the model can learn the correlation and importance between different features, thereby improving the expressiveness and generalization ability of the model.

[0086] Compared with the prior art, the multimodal data fusion method and device based on the cross-attention mechanism provided by the embodiment of the present disclosure introduces a feature fusion framework based on the cross-attention mechanism, and performs end-to-end weighting of bioinformatics and imaging features to more finely model the complementary relationship between each modality of data. Through this multi-layer perceptron and cross-attention module, the model shows greater flexibility and pertinence in feature extraction and weighting, thereby overcoming the limitations of traditional methods and achieving higher accuracy and generalization in treatment effect evaluation and disease progression prediction.

[0087] Example 2

[0088] See also Figure 4 According to the specific implementation of the present disclosure, in the second aspect, the present disclosure provides a multimodal data fusion device 200 based on a cross-attention mechanism, including: a feature acquisition module 210, a cross-attention processing module 220, a feature splicing module 230, and a self-attention fusion module 240.

[0089] The feature acquisition module 210 is used to acquire the biological information feature of the target object. bio And radiomics features rad ;

[0090] The cross attention processing module 220 is used to process the biological information feature based on the cross attention mechanism. bio And radiomics features rad Perform feature weighting and interaction processing to generate bioinformatic feature features based on radiomics feature attention rad2bio And the imaging omics feature feature based on bioinformatic feature attention bio2rad ;

[0091] The feature splicing module 230 is used to combine the biometric information feature bio、Radiognomics features rad , bioinformatic features based on radiomics feature attention rad2bio , Imaging omics features based on bioinformatic feature attention bio2rad Perform feature splicing to generate splicing feature comb ;as well as

[0092] The self-attention fusion module 240 is used to fusion the splicing feature feature based on the self-attention mechanism. comb Perform feature fusion to generate a fused feature representation of the target object.

[0093] Example 3

[0094] like Figure 5 As shown, this embodiment provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method steps described in the above embodiment.

[0095] Example 4

[0096] An embodiment of the present disclosure provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions can execute the method steps described in the above embodiment.

[0097] Example 5

[0098] Reference below Figure 5 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, vehicle-mounted terminals, etc. and fixed terminals such as desktop computers, etc. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0099] like Figure 5As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. In RAM 403, various programs and data required for the operation of the electronic device are also stored. Processing device 401, ROM 402, and RAM 403 are connected to each other via a bus 405. An input / output (I / O) interface 405 is also connected to bus 405.

[0100] Typically, the following devices may be connected to the I / O interface 405: input devices 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 405 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 408 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 405. The communication devices 405 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0101] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 405, or installed from the storage device 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0102] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0103] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0104] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0105] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0106] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of the unit does not, in some cases, limit the unit itself.

[0107] Finally, it should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0108] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A multimodal data fusion method based on cross attention mechanism, characterized in that: The following steps are involved: Get the biometric features of the target object bio And radiomics features rad ; Based on the cross attention mechanism, the biological information feature bio And radiomics features rad Perform feature weighting and interaction processing to generate bioinformatic feature features based on radiomics feature attention rad2bio And the imaging omics feature feature based on bioinformatic feature attention bio2rad ; For the biometric feature feature bio 、Radiognomics features rad , bioinformatic features based on radiomics feature attention rad2bio , Imaging omics features based on bioinformatic feature attention bio2rad Perform feature splicing to generate splicing feature comb ; as well as Based on the self-attention mechanism, the splicing feature comb Perform feature fusion to generate a fused feature representation of the target object.

2. The multimodal data fusion method based on the cross attention mechanism according to claim 1 is characterized in that: The biometric feature bio Including RNA features, the radiomics features feature rad Includes shape, texture, and / or intensity features extracted from medical images.

3. The multimodal data fusion method based on the cross attention mechanism according to claim 1, characterized in that: The biological information feature is based on the cross attention mechanism bio And radiomics features rad Perform feature weighting and interaction processing to generate bioinformatic feature features based on radiomics feature attention rad2bio And the imaging omics feature feature based on bioinformatic feature attention bio2rad The steps include: The biometric feature bio And radiomics features rad Input is a criss-cross attention model, which is trained by a joint network based on a multi-layer perceptron and an attention mechanism.

4. The multimodal data fusion method based on the cross attention mechanism according to claim 3 is characterized in that: The cross-attention model uses a multi-layer perceptron to spatially map features of different modalities, and the dimensions of the bioinformatics features and imaging omics features obtained through spatial mapping are consistent, and the softmax function is used to form attention weights for the bioinformatics features and imaging omics features obtained through spatial mapping.

5. The multimodal data fusion method based on the cross attention mechanism according to claim 4 is characterized in that: The calculation formula for forming the attention weight using the softmax function for the biological information features and imaging omics features obtained by spatial mapping is: Among them, atten A2B is the weighted attention weight of feature A based on feature B, i is the weight of the i-th feature in feature B, and K is the dimension of feature B, where feature A is a biological information feature feature bio , the feature B is the radiomics feature feature rad .

6. The multimodal data fusion method based on the cross attention mechanism according to claim 1, characterized in that: The self-attention mechanism is used to analyze the splicing feature comb In the step of performing feature fusion and generating a fused feature representation of the target object, the importance of each element in the input feature sequence is dynamically weighted using a self-attention model.

7. The multimodal data fusion method based on the cross attention mechanism according to claim 6, characterized in that: The step of dynamically weighting the importance of each element in the input feature sequence using the self-attention model includes: For the splicing feature comb Perform linear transformation to generate queries, keys, and values; Calculate the similarity between the query and the key to determine the importance of each feature; The similarity scores are normalized using the softmax function to obtain weights, and the values ​​are weighted and summed using the weights to generate weighted feature outputs: The weighted feature output is processed through a fully connected layer to generate a fused feature representation of the target object.

8. A multimodal data fusion device based on a cross-attention mechanism, characterized in that: include: Feature acquisition module, used to obtain the biometric features of the target object bio And radiomics features rad ; A cross attention processing module is used to process the biological information feature based on the cross attention mechanism. bio And radiomics features rad Perform feature weighting and interaction processing to generate bioinformatic feature features based on radiomics feature attention rad2bio And the imaging omics feature feature based on bioinformatic feature attention bio2rad ; A feature splicing module is used to combine the biometric features bio 、Radiognomics features rad , bioinformatic features based on radiomics feature attention rad2bio , Imaging omics features based on bioinformatic feature attention bio2rad Perform feature splicing to generate splicing feature comb ; as well as The self-attention fusion module is used to fusion the spliced ​​features based on the self-attention mechanism. comb Perform feature fusion to generate a fused feature representation of the target object.

9. An electronic device, characterized in that: include: Processor: and A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images