Medical image and text conjoint analysis model based on multi-modal feature fusion

Abstract

The invention relates to the technical field of artificial intelligence, and provides a medical image and text conjoint analysis model based on multi-modal feature fusion, which adopts a text encoder to encode input medical text data, adopts an image encoder to encode input medical image data, and adopts a multi-modal feature fusion algorithm to encode the input medical image data. Performing feature fusion through matrix multiplication to obtain first data, and inputting the first data to a multi-head attention layer and a feedforward neural network layer for processing by adopting a multi-head attention mechanism module. And performing operation on the second data and the transposed image feature vector by adopting a feature selection module, and generating a probability value of a cross entropy loss function of a diagnosis result by adopting a classifier to enable the third data to pass through a full connection layer. The invention further discloses a construction method for quickly establishing the model, the structure of the model enables the model to adapt to different types of medical images and text data, key information in medical images and text reports can be integrated, deep interaction information can be fused, and the accuracy of the whole diagnosis model is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a medical image and text joint analysis model based on multimodal feature fusion. Background Art

[0002] Medical images and related text reports are often used in the medical diagnosis process to comprehensively judge the patient's condition. Medical images provide doctors with detailed information about the patient's internal structure and tissues, such as X-rays for detecting bone and lung abnormalities, and CT scans provide more detailed cross-sectional images. Text reports related to medical images are usually written by radiologists based on the imaging findings and other medical history information. These reports contain the radiologist's description of the imaging findings and diagnostic speculation. Doctors will consider these text reports and medical images together to help them make the most accurate diagnosis. Medical images and text reports are only part of the information source for doctors' diagnosis. Doctors need to combine multiple information such as the patient's medical history, physical examination, laboratory tests, and other imaging test results to make a final diagnosis and treatment plan. Multimodal feature fusion refers to combining data from different sources (such as text, images, and audio) to improve the performance of machine learning models. This fusion is achieved through different strategies, including feature-level fusion, decision-level fusion, hybrid-level fusion, and model-level fusion.

[0003] CN118247621A discloses a multimodal image fusion and processing method based on a multidimensional attention mechanism, comprising: obtaining a multimodal image sequence to be processed of a target object; predicting the key areas of the multimodal image sequence and the importance of images of different modalities through a pre-trained modality perception attention module to obtain a prediction result, wherein the key area refers to the visual area based on which the multimodal image sequence is parsed; performing weighted processing on the multimodal image sequence based on the prediction result to obtain an enhanced image sequence; using the enhanced image sequence as the input of a pre-trained feature extraction module, performing multimodal feature extraction and fusion processing on the enhanced image sequence to obtain feature information, and the feature information is used for downstream tasks implemented based on the multimodal image sequence.

[0004] CN117830227B discloses a method for staging oral cancer tumors based on pathology and CT multimodal models.The method comprises the following steps: S1. acquiring an oral multimodal dataset, wherein the oral multimodal dataset comprises an oral pathology dataset and an oral CT image dataset; S2. constructing a pathology feature extraction model based on FasterNet according to the oral pathology dataset, wherein FasterNet is pre-trained with a dataset of patients with different stages of oral squamous cell carcinoma; S3. constructing a CT feature extraction model according to the oral CT image dataset, wherein the CT feature extraction model is constructed using a 3DDenseNet based on an attention mechanism, wherein the 3DDenseNet based on the attention mechanism is output through four sequentially connected 3D dense connection blocks and an attention module, and then through a fully connected layer, wherein the The 3D dense connection block includes three parts: batch normalization, ReLU activation function, pooling and 3D convolution, wherein the size of the 3D convolution is 3×3×3; the attention module includes a channel attention module and a spatial attention module; S4. The features extracted by the pathological feature extraction model and the CT feature extraction model are fused, and a feature fusion and staging model based on a progressive feature pyramid is established: its input is CT tumor features and pathological tumor features, and at the same time, the CT tumor features and the pathological tumor features are respectively subjected to multimodal feature fusion through two end-to-end activation layers and convolution layers, and then the staging result is obtained through the activation layers and fully connected layers connected in sequence. The first activation layer of the CT tumor feature also The first activation layer of the pathological tumor feature is connected to the second convolution layer of the CT tumor feature, the first activation layer of the pathological tumor feature is also connected to the first convolution layer of the CT tumor feature, the first activation layer of the pathological tumor feature is also connected to the second convolution layer of the pathological tumor feature, and the second activation layer of the pathological tumor feature is also connected to the second convolution layer of the CT tumor feature; the CT tumor feature and the pathological tumor feature are each convolved once and then feature fused to obtain the shallow features of the tumor, which are used to make a preliminary judgment on the tumor condition; the shallow features are then fused with the pathological tumor features that have been convolved once and the CT tumor features that have been convolved once to obtain the deep features of the tumor, which are used to discover tumors with a certain infiltration depth; the deep features are then fused with the pathological tumor features that have been convolved twice to obtain the deep features of the tumor, which are used to discover tumors with a certain infiltration depth; the deep features are then fused with the pathological tumor features that have been convolved twice to obtain the deep features of the tumor, which are used to discover tumors with a certain infiltration depth. The accumulated pathological tumor features are fused to obtain the top-level features of the tumor, which are used to find tumors with unclear lymph node metastasis and low tumor density; in the final multimodal feature fusion stage of the progressive feature pyramid, the feature weights corresponding to each modality are assigned to emphasize the features of each modality and reduce the interference between different information, that is, y = αx + βx + γx, where x represents the feature vector from the nth layer to the lth layer, y represents the final multimodal feature vector, α, β, and γ represent the weights of the corresponding modalities, respectively, and are subject to the constraint of α + β + γ = 1; based on the feature fusion and staging model, a multimodal attention heat map of pathology and CT is generated; S6. Based on the multimodal attention heat map of pathology and CT, the final tumor staging is achieved.

[0005] In the existing technology, only deep learning-based image processing is used for medical image analysis, which results in the use and reliance on only a single modality of information and ignores the important semantics in the text information, thus leading to model limitations and ultimately resulting in problems such as reduced diagnostic accuracy. Summary of the invention

[0006] Long-term practice has found that medical diagnosis data has high dimensions and a high degree of internal coupling, and contains a lot of deep information. A single deep learning model for medical images can capture the microscopic potential information in medical images with high accuracy. However, it is difficult to integrate the key information in medical images and text reports, and it is difficult to integrate the deep interactive information between images and text, resulting in problems such as low accuracy of the entire diagnostic model.

[0007] In view of this, the present invention aims to propose a medical image and text joint analysis model based on multimodal feature fusion, including:

[0008] A text encoder, used to encode the input medical text data and generate text features;

[0009] An image encoder is used to encode the input medical image data and generate image features;

[0010] A feature fusion module, used for fusing input text features and image features through matrix multiplication to obtain first data;

[0011] A multi-head attention mechanism module, used for inputting the first data into a multi-head attention layer and a feedforward neural network layer for processing to obtain second data;

[0012] A feature selection module, used for performing a Hadamard product operation on the second data and the transposed image feature vector to obtain third data;

[0013] A classifier is used to pass the third data through a fully connected layer to generate a probability value of a cross entropy loss function of a diagnosis result.

[0014] Preferably, the number of the multi-head attention mechanism modules is 2 to 4.

[0015] Preferably, the text encoder comprises a pre-trained CXR-BERT model, and the image encoder comprises a pre-trained CheXNet-121 model.

[0016] Preferably, the second data is an Attention vector.

[0017] Preferably, the cross entropy loss function H in the classifier is,

[0018]

[0019] Where N is the number of samples in the training set, C is the number of categories; y jc Indicates whether the jth sample belongs to the cth category. If so, y jc =1, if not, then y jc =0, is the image in the jth sample in the training set, is the text in the jth sample in the training set, Calculate a probability distribution vector of length C, Represents the probability distribution vector value of the c-th category, that is, the probability that the j-th sample belongs to the c-th category.

[0020] Preferably, if the cross entropy loss function H obtains a preset value, the trained medical image and text joint analysis model based on multimodal feature fusion is output for the joint analysis of medical images and texts.

[0021] The present invention also provides a method for constructing a medical image and text joint analysis model based on multimodal feature fusion as described above, the construction method comprising:

[0022] Step S1, obtaining medical text data from historical data for encoding and generating text features;

[0023] Step S2, obtaining medical image data from historical data for encoding to generate image features;

[0024] Step S3, fusing the input text features and image features by matrix multiplication to obtain first data;

[0025] Step S4, inputting the first data into a multi-head attention layer and a feedforward neural network layer for processing to obtain second data;

[0026] Step S5, performing a Hadamard product operation on the second data and the transposed image feature vector to obtain third data;

[0027] Step S6: Pass the third data through a fully connected layer to generate a probability value of a cross entropy loss function of a diagnosis result.

[0028] Preferably, the construction method further includes, step S7, if the cross entropy loss function H obtains a preset value, outputting a trained medical image and text joint analysis model based on multimodal feature fusion.

[0029] The present invention provides an electronic device, at least one processor; and

[0030] a memory communicatively connected to the at least one processor; wherein,

[0031] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above-mentioned construction method.

[0032] The present invention provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute the construction method of the present application as described above.

[0033] The medical image and text joint analysis model based on multimodal feature fusion disclosed by the present invention uses a text encoder to encode the input medical text data to generate text features. The image encoder is used to encode the input medical image data to generate image features. The text features and image features input by the feature fusion module are used to perform feature fusion through matrix multiplication to obtain first data. The multi-head attention mechanism module is used to input the first data into the multi-head attention layer and the feedforward neural network layer for processing to obtain second data. The feature selection module is used to perform Hadamard product operation on the second data and the transposed image feature vector to obtain third data. The classifier is used to pass the third data through the fully connected layer to generate the probability value of the cross entropy loss function of the diagnosis result. The present invention also discloses a construction method of the medical image and text joint analysis model based on multimodal feature fusion as described above. The construction method can quickly establish a model, and the structure of the model enables it to adapt to different types of medical image and text data, and has good flexibility and scalability. It can also integrate the key information in the medical image and text report and fuse the deep interactive information between the image and the text, thereby improving the accuracy of the entire diagnosis model.

[0034] Other features and advantages of the present invention will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0036] Figure 1 A schematic diagram of a medical image and text joint analysis model based on multimodal feature fusion according to an embodiment of the present invention;

[0037] Figure 2 This is a comparison chart of F1-score of a medical image and text joint analysis model based on multimodal feature fusion according to an embodiment of the present invention;

[0038] Figure 3This is a cross entropy comparison chart of a medical image and text joint analysis model based on multimodal feature fusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0040] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0041] It should be noted that the terms "first", "second", "third", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so as to describe the embodiments of the present invention described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0042] In order to solve the technical problems existing in the prior art, a deep learning model for a single medical image can capture the microscopic deep potential information in the medical image with high accuracy. However, it is difficult to integrate the key information in the medical image and the text report and to fuse the deep interactive information between the image and the text, resulting in the low accuracy of the entire diagnosis model. The present invention provides a joint analysis model of medical images and texts based on multimodal feature fusion, such as Figure 1 As shown, a schematic diagram of a medical image and text joint analysis model based on multimodal feature fusion according to an embodiment of the present invention, the medical image and text joint analysis model based on multimodal feature fusion includes:

[0043] A text encoder, used to encode the input medical text data and generate text features;

[0044] An image encoder is used to encode the input medical image data and generate image features;

[0045] A feature fusion module, used for fusing input text features and image features through matrix multiplication to obtain first data;

[0046] A multi-head attention mechanism module, used for inputting the first data into a multi-head attention layer and a feedforward neural network layer for processing to obtain second data;

[0047] A feature selection module, used for performing a Hadamard product operation on the second data and the transposed image feature vector to obtain third data;

[0048] A classifier is used to pass the third data through a fully connected layer to generate a probability value of a cross entropy loss function of a diagnosis result.

[0049] The medical image and text joint analysis model based on multimodal feature fusion disclosed by the present invention uses a text encoder to encode the input medical text data to generate text features. The image encoder is used to encode the input medical image data to generate image features. The text features and image features input by the feature fusion module are used to perform feature fusion through matrix multiplication to obtain first data. The multi-head attention mechanism module is used to input the first data into the multi-head attention layer and the feedforward neural network layer for processing to obtain second data. The feature selection module is used to perform Hadamard product operation on the second data and the transposed image feature vector to obtain third data. The classifier is used to pass the third data through the fully connected layer to generate the probability value of the cross entropy loss function of the diagnosis result. The present invention also discloses a method for constructing the medical image and text joint analysis model based on multimodal feature fusion as described above. The structure of the model can adapt to different types of medical images and text data, and has good flexibility and scalability. It can also integrate the key information in the medical image and text report and fuse the deep interactive information between the image and the text, so as to improve the accuracy and comprehensive performance of the entire diagnosis model.

[0050] The medical image and text joint analysis model based on multimodal feature fusion disclosed in the present invention can process medical text and medical image data at the same time, extract text features and image features respectively through encoders, and make full use of information from two different sources, so that the model can obtain a more comprehensive perspective during diagnosis. Through the feature fusion module, the text features and image features are fused by matrix multiplication, which can capture the deep interactive relationship between text and images, enhance the expressiveness of features, and improve the performance of the model.

[0051] In order to extract high-level features in the data, a multi-head attention mechanism is used, which can simultaneously focus on different parts of the input features in different subspaces, thereby improving the richness and effectiveness of feature extraction. It helps the model to focus on more critical information in images and texts, thereby enhancing the understanding of complex data. In order to enhance the learning ability of the model, multiple Transformer blocks are used to capture high-order interactive information between images and texts. In the preferred embodiment of the present invention, the number of multi-head attention mechanism modules is 2 to 4. The feature selection module combines the second data with the transposed image feature vector through the Hadamard product operation to highlight key information and reduce the interference of redundant features. It helps to improve the robustness of the model and reduce the possibility of overfitting.

[0052] The pre-trained CXR-BERT model is used to encode the input medical text, such as a radiology diagnosis report, to generate a text feature representation. After pre-training on large-scale text data, high-quality text feature representation can be generated. If the input medical text is a radiology report, it is processed by the Sentence-BERT encoder to generate a text feature vector of fixed length. In the preferred case of the present invention, the text encoder includes a pre-trained CXR-BERT model, and the image encoder includes a pre-trained CheXNet-121 model. The image encoder uses the pre-trained CheXNet-121 model to encode the input medical image, such as a chest X-ray, to generate an image feature representation. The image encoder inputs a medical image, such as a chest X-ray, and undergoes multi-layer processing of a convolutional neural network to ultimately generate an image feature vector. This vector can be fused with the text feature vector in a subsequent feature fusion matrix. The text features and image features are input into the feature fusion matrix, and the fusion of the two modal features is achieved through matrix multiplication. The text features and image features are multiplied by the feature fusion matrix. The fused features contain both spatial information in the image and semantic information in the text. The fusion matrix is ​​updated by learnable parameters to ensure the effect of feature fusion.

[0053] In order to better generate through the multi-head attention mechanism, the most relevant information in the input features can be dynamically captured. In the preferred case of the present invention, the second data is an Attention vector. The Attention mechanism adjusts the weights of each feature dimension according to the importance, thereby highlighting the image features with more diagnostic value and suppressing irrelevant or noise features. This weighted feature extraction method ensures that the model can capture the text information and target tasks.

[0054] In order to pass the processed feature vector through the fully connected layer to generate the probability distribution of medical diagnosis, the probability distribution represents the model's predicted probability for different categories, such as different diseases. During the model training process, the cross entropy loss function H is used to minimize the difference between the predicted result and the true label. In the preferred case of the present invention, the cross entropy loss function H in the classifier is,

[0055]

[0056] Where N is the number of samples in the training set, C is the number of categories; y jc Indicates whether the jth sample belongs to the cth category. If so, y jc =1, if not, then y jc =0, is the image in the jth sample in the training set, is the text in the jth sample in the training set, Calculate a probability distribution vector of length C, represents the probability distribution vector value of the cth category, that is, the probability that the jth sample belongs to the cth category. Through computational experiments, such as Figure 2 As shown in Figure 2, the medical image and text joint analysis model based on multimodal feature fusion (Our's ExDenseMed Model) is significantly better than the simple eigenvector concatenation model (Simple Eigenvector Concatenation Model) in terms of F1-score. Figure 3 As shown in the figure, the medical image and text joint analysis model based on multimodal feature fusion also performs significantly better than the simple feature vector connection model in terms of cross entropy loss.

[0057] In order to obtain an efficient and stable joint analysis model of medical images and texts based on multimodal feature fusion, in the preferred case of the present invention, if the cross entropy loss function H obtains a preset value, the trained joint analysis model of medical images and texts based on multimodal feature fusion is output for the joint analysis of medical images and texts. When the cross entropy loss function H reaches the preset value, it means that the model has obtained good performance during the training process and can effectively perform a joint analysis of a given medical image and text. For example, an independent validation set can be used to evaluate the performance of the model, and indicators such as accuracy, recall, and F1-score can be calculated to confirm the effectiveness of the model. The stability and generalization ability of the model are further verified by the k-fold cross validation method.

[0058] The present invention also provides a method for constructing a medical image and text joint analysis model based on multimodal feature fusion as described above, the construction method comprising:

[0059] Step S1, obtaining medical text data from historical data for encoding and generating text features;

[0060] Step S2, obtaining medical image data from historical data for encoding to generate image features;

[0061] Step S3, fusing the input text features and image features by matrix multiplication to obtain first data;

[0062] Step S4, inputting the first data into a multi-head attention layer and a feedforward neural network layer for processing to obtain second data;

[0063] Step S5, performing a Hadamard product operation on the second data and the transposed image feature vector to obtain third data;

[0064] Step S6: Pass the third data through a fully connected layer to generate a probability value of a cross entropy loss function of a diagnosis result.

[0065] The construction method disclosed in the present invention extracts medical text data from historical data, processes the text data using appropriate coding technology, and generates corresponding text feature representations. Medical image data, such as radiographic images, CT images, etc., are extracted from historical data, and the image data is encoded using deep learning technology to generate corresponding image feature representations. For example, a convolutional neural network (CNN) is used to generate image features. The text features and image features generated in step S1 and step S2 are fused. The two features are combined by matrix multiplication to obtain first data, integrating text and image information. The first data is input into a multi-head attention layer to capture the complex relationship between features. After being processed by a feedforward neural network layer, the second data is obtained to further enhance the feature expression capability. The second data is subjected to a Hadamard product operation with the transposed image feature vector to combine the information of the two. This step generates third data, emphasizing the interactive influence between features. The third data is processed by a fully connected layer, and finally a probability value of a cross entropy loss function is generated, which is used to evaluate the output result of the model and obtain the probability distribution of medical diagnosis. By gradually extracting and fusing the features of medical text and images, a multimodal model is constructed to provide more accurate support for medical diagnosis. It fully considers the complementarity of text and image information and uses attention mechanisms and feature interactions to improve the performance and accuracy of diagnosis. Through the final fully connected layer output, the model can generate diagnostic probabilities for specific medical problems.

[0066] In order to improve the prediction accuracy of the medical image and text joint analysis model based on multimodal feature fusion, adapt to different medical image and text data, save time and improve work efficiency, in the preferred case of the present invention, the construction method also includes step S7, if the cross entropy loss function H obtains a preset value, output the trained medical image and text joint analysis model based on multimodal feature fusion.

[0067] The present invention provides an electronic device, at least one processor; and

[0068] a memory communicatively connected to the at least one processor; wherein,

[0069] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above-mentioned construction method.

[0070] The present invention provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute the construction method of the present application as described above.

[0071] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0072] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0073] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0074] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A medical image and text joint analysis model based on multimodal feature fusion, characterized in that: The medical image and text joint analysis model based on multimodal feature fusion includes: A text encoder, used to encode the input medical text data and generate text features; An image encoder is used to encode the input medical image data and generate image features; A feature fusion module, used for fusing input text features and image features through matrix multiplication to obtain first data; A multi-head attention mechanism module, used for inputting the first data into a multi-head attention layer and a feedforward neural network layer for processing to obtain second data; A feature selection module, used for performing a Hadamard product operation on the second data and the transposed image feature vector to obtain third data; A classifier is used to pass the third data through a fully connected layer to generate a probability value of a cross entropy loss function of a diagnosis result.

2. The medical image and text joint analysis model based on multimodal feature fusion according to claim 1 is characterized in that: The number of multi-head attention mechanism modules is 2 to 4.

3. The medical image and text joint analysis model based on multimodal feature fusion according to claim 1 is characterized in that: The text encoder includes a pre-trained CXR-BERT model, and the image encoder includes a pre-trained CheXNet-121 model.

4. The medical image and text joint analysis model based on multimodal feature fusion according to claim 1 is characterized in that: The second data is an Attention vector.

5. The medical image and text joint analysis model based on multimodal feature fusion according to any one of claims 1 to 4, characterized in that: The cross entropy loss function H in the classifier is, Where N is the number of samples in the training set, C is the number of categories; y jc Indicates whether the jth sample belongs to the cth category. If so, y jc =1, if not, then y jc =0, is the image in the jth sample in the training set, is the text in the jth sample in the training set, Calculate a probability distribution vector of length C, Represents the probability distribution vector value of the c-th category, that is, the probability that the j-th sample belongs to the c-th category.

6. The medical image and text joint analysis model based on multimodal feature fusion according to claim 5, characterized in that: If the cross entropy loss function H obtains the preset value, the trained medical image and text joint analysis model based on multimodal feature fusion is output for the joint analysis of medical images and texts.

7. A method for constructing a medical image and text joint analysis model based on multimodal feature fusion according to any one of claims 1 to 6, characterized in that: The construction method comprises: Step S1, obtaining medical text data from historical data for encoding and generating text features; Step S2, obtaining medical image data from historical data for encoding to generate image features; Step S3, fusing the input text features and image features by matrix multiplication to obtain first data; Step S4, inputting the first data into a multi-head attention layer and a feedforward neural network layer for processing to obtain second data; Step S5, performing a Hadamard product operation on the second data and the transposed image feature vector to obtain third data; Step S6: Pass the third data through a fully connected layer to generate a probability value of a cross entropy loss function of a diagnosis result.

8. The construction method according to claim 7, characterized in that: The construction method also includes, step S7, if the cross entropy loss function H obtains a preset value, outputting a trained medical image and text joint analysis model based on multimodal feature fusion.

9. An electronic device, characterized in that: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the construction method described in any one of claims 7 or 8.

10. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores instructions, and the instructions are used to enable the machine to execute the construction method as described in any one of claims 7 or 8 of the present invention.

Images