Medical image holographic dynamic typing diagnosis system and method based on multi-modal data fusion

Abstract

The invention discloses a medical image holographic dynamic typing diagnosis system and method based on multi-modal data fusion, and the system comprises a multi-modal data collection module, a data preprocessing module, a model recognition module, and a dynamic typing model. The method comprises the steps of constructing an encoder-decoder structure network based on multi-modal feature fusion, collecting medical images and related text descriptions, carrying out recognition through a pre-trained network, carrying out feature aggregation on different batches of recognition results, calculating a focus area change index, and carrying out multi-modal feature fusion. Holographic dynamic typing diagnosis based on multi-modal data fusion medical images is realized, a multi-modal fusion interaction mechanism is introduced based on an encoder-decoder structure network of multi-modal feature fusion, a corresponding relation between the medical images and text description cross-modal stability is established, semantic fusion and spatial representation are enhanced, and the diagnosis accuracy is improved. Therefore, the accuracy and reliability of dynamic typing diagnosis are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of multi-modal data fusion and computer vision, and in particular to a multi-modal data fusion based medical image holographic dynamic typing diagnosis system and method. BACKGROUND

[0002] With the rapid development of computer vision and deep learning, multi-modal data fusion based medical image intelligent analysis method has gradually become an important research direction of medical auxiliary diagnosis. Traditional medical image segmentation relies on single modal image for feature extraction and lesion identification, but such methods generally face the problems of high labeling cost, scarce labeled samples and limited feature expression, and it is difficult to fully depict complex lesion morphology and pathological semantic information. In order to make up for the lack of medical image data in the semantic level, researchers have begun to combine multi-modal fusion strategy of medical text description, which can supplement the expression ability of image features to a certain extent, so as to improve the accuracy of lesion identification and typing diagnosis.

[0003] However, how to efficiently utilize the complementary features of medical images and texts to realize the deep alignment and information interaction of cross-modal features is still a key problem. On the one hand, there are semantic distribution differences and unstable feature correlations between medical images and text descriptions, which makes it difficult for the model to establish a stable cross-modal correspondence. On the other hand, under the condition of small sample and weak labeling, the semantic fusion ability of the model is insufficient and the spatial representation is unstable, which easily causes lesion identification error and affects the accuracy and reliability of dynamic typing diagnosis. SUMMARY

[0004] In order to overcome the above-mentioned defects of the prior art, the present application provides a multi-modal data fusion based medical image holographic dynamic typing diagnosis system and method, which constructs an encoder-decoder structure network based on multi-modal feature fusion, introduces a multi-modal fusion interaction mechanism and a parallel attention mechanism to solve the problems proposed in the above background technology.

[0005] To achieve the above purpose, the present application provides the following technical scheme: A multi-modal data fusion based medical image holographic dynamic typing diagnosis system is provided, which comprises a multi-modal data acquisition module, a data preprocessing module, a model recognition module and a dynamic typing diagnosis module.

[0006] The multi-modal data acquisition module constructs a holographic data set containing an image and a text use data mapping mechanism, the data preprocessing module filters out text items irrelevant to the lesion area through a semantic matching algorithm and simultaneously uses a quality control mechanism to eliminate images or records that do not meet the labeling, the model recognition module uses a network based on the encoding-decoding result to identify the suspected lesion area in the medical image, and the dynamic typing diagnosis module aggregates the feature of the multiple image recognition results of the same patient to calculate the change index of the lesion area and generate the lesion dynamic typing result.

[0007] It should be noted that the specific types of lesion dynamic typing include progressive type, stable type and regression type.

[0008] A medical image holographic dynamic typing diagnosis method based on multi-modal data fusion is provided. Step 1: Construct an encoder-decoder structure network based on multi-modal feature fusion, use ResNet-50 as a visual encoder and Clip-Text as a text encoder to encode medical images and text descriptions, obtain medical image features and text descriptors, after preliminary fusion of medical image features and text descriptors, introduce a multi-modal fusion interaction mechanism to enhance the semantic correlation between medical image features and text features, perform cross-modal alignment in the feature space, and set a parallel attention mechanism in the decoding stage to recalibrate the features, and strengthen the response of the lesion-related feature channel and the spatial region.

[0009] The multi-modal fusion interaction mechanism includes a shallow fusion feature enhancement (SFFE) module and a deep fusion feature enhancement (DFFE) module, the shallow fusion feature enhancement (SFFE) module fuses the preliminary fusion features and shallow features to highlight the position information expression and edge detail response of the shallow features, and improves the edge positioning accuracy of the lesion area, the deep fusion feature enhancement (DFFE) module fuses the preliminary fusion features and deep features to highlight the rich semantic information in the deep features, and improves the identification ability and class discrimination performance of the lesion area, the shallow features include shallow features from the visual encoder branch and shallow features from the text encoder branch, the deep features include deep features from the visual encoder branch and deep features from the text encoder branch, and a cross-modal stable corresponding relationship is established.

[0010] It should be noted that the shallow fusion feature enhancement (SFFE) module is specifically: for the features from the encoder, shallow semantic features are obtained through joint global average pooling and global maximum pooling, and after adjusting the dimensions through 1x1 convolution, the preliminary fusion features are fused and weighted, and finally the residual connection is connected to the original shallow visual features, the fusion and weighting is a channel-by-channel weighting operation of the feature after adjusting the dimensions through 1x1 convolution and the weight of the preliminary fusion feature, and the weight of the preliminary fusion feature is obtained by global maximum pooling and sigmoid function on the preliminary fusion feature.

[0011] It should also be noted that the deep fusion feature enhancement (DFFE) module is specifically: for the features from the encoder, the channel is split into two channel size features, one of which is processed through global average pooling and the other is multiplied after adjusting the dimensions through convolution to obtain deep semantic features, and for the preliminary fusion features, after three-branch deep separable convolution of different scales, the addition operation is performed, the input of the addition operation also includes the input of the original preliminary fusion features, and finally the channel-by-channel multiplication is performed on the deep semantic features to perform feature weighting.

[0012] In addition, the decoding stage adopts a bottom-up multi-scale feature fusion strategy, processes four multi-modal fusion features of different scales as a decoder, and adjacent scale multi-modal features as input of parallel attention mechanism.

[0013] It should be noted that before entering the decoding stage, the four-layer output of the shallow fusion feature enhancement (SFFE) module and the four-layer output of the deep fusion feature enhancement (DFFE) module are added in the corresponding scale to realize the complementary fusion of spatial details and semantic information, and to provide multi-level interactive features for high-precision lesion segmentation and dynamic typing diagnosis.

[0014] Further, for small-scale input features, after upsampling, element-by-element addition operation is performed with large-scale input to obtain scale fusion features, and the fused features are strengthened through parallel branches.

[0015] Wherein, the first branch of the parallel branch pays attention to channel feature attention, and the second branch of the parallel branch pays attention to spatial feature attention. After the first branch of the parallel branch is respectively processed by global average pooling and global maximum pooling, two global channel descriptions are obtained. The two global channel descriptions are spliced along the channel dimension, and the result is added to the outputs of the two pooling layers. After passing through a Sigmoid activation function, channel attention weights are generated. Finally, the channel attention weights are multiplied with the original input of the first branch to realize feature recalibration through channel multiplication, highlight semantic channel features with high discriminability, and obtain the output comprehensive feature enhancement output of the parallel attention mechanism by element-wise addition of the outputs of the two branches. The output comprehensive feature enhancement output is used for subsequent mask prediction and typing analysis stages.

[0016] It should be noted that the parallel attention mechanism realizes adaptive enhancement of feature salient regions on the basis of multi-scale fusion, which significantly improves the network's ability to recognize fine-grained lesion regions in complex backgrounds.

[0017] Step 2: Pre-training the constructed network using a multi-modal medical image dataset to form a pre-trained model weight.

[0018] The multi-modal medical image dataset includes medical images and paired text descriptions.

[0019] In addition, during the pre-training process, a joint loss function is set, including a segmentation loss and a contrastive learning loss, which can comprehensively consider the pixel-level segmentation performance and cross-modal feature consistency.

[0020] It should be noted that the segmentation loss is used to supervise the coincidence degree of the model output mask and the real lesion region, force the model output to be consistent with the real mask, reduce missed detection or false detection, improve the spatial positioning accuracy, maintain the consistency of mask prediction under different modal images, enhance the robustness of the model, and provide a reliable spatial basis for subsequent lesion change calculation to support dynamic typing analysis.

[0021] It should also be noted that the contrastive learning loss is used to constrain the similarity of image features and their corresponding text features in the shared semantic space, improve the generalization ability of the model under the condition of data scarcity through cross-modal mutual supervision, and at the same time make different types of lesions form clear clusters in the embedding space, which is beneficial to dynamic typing.

[0022] Step 3: Collect new medical image data and related text description, and perform data preprocessing, and fuse the collected multi-modal data through the pre-trained network.

[0023] The fusion of the collected data through the trained network is specifically: multi-layer feature extraction of the input image is performed through a visual encoder to obtain four layers of visual initial features of different scales, semantic encoding of the medical text description is performed through a text encoder to obtain a text initial feature descriptor, the four layers of image initial features of different scales and the text initial feature descriptor are spliced in the channel dimension to obtain four layers of multi-modal initial features of different scales, a query vector Q from the visual encoder and a key K and a value V from the multi-modal initial feature representation are input into a Transformer module for preliminary multi-modal feature fusion to obtain preliminary fusion features, and the preliminary fusion features, the visual initial features and the multi-modal initial features are fused through a fusion interaction mechanism to obtain multi-modal interactive fusion features.

[0024] In addition, the data preprocessing includes image data preprocessing and text data preprocessing, the image data preprocessing includes normalization, noise suppression and size resampling, and the text data preprocessing includes text segmentation, stop word removal and medical entity recognition, the objects of the medical entity recognition include disease name, part and pathological feature.

[0025] Step 4: The four layers of multi-modal features after fusion are decoded through upsampling from bottom to top to generate a prediction mask, and the mask resolution is aligned with the original image through a spatial re-projection mechanism based on attention to obtain a lesion positioning result.

[0026] Step 5: According to the recognition results of different batches, feature aggregation is performed, a lesion area change index is calculated, and a dynamic typing result is generated.

[0027] The feature aggregation method includes time series aggregation and spatial consistency aggregation, the time series set stacks the feature vectors in the order of detection time, the spatial consistency aggregation calculates the mask overlap rate and the morphological change amount of the same region, and the time series and the spatial consistency change index are taken as input features, and a typing model based on clustering analysis or classification network is used to generate a dynamic typing diagnosis result of the lesion.

[0028] In addition, the calculation of the lesion area change index is used to quantify the lesion change feature, and the specific method is difference index calculation, and the calculated indexes include a lesion area change rate, a lesion density change rate and a texture complexity change index.

[0029] The multi-modal data fusion medical image holographic dynamic typing diagnosis system and method have the following technical effects and advantages: 1. By introducing a multi-modal fusion interaction mechanism in the constructed encoder-decoder structure network based on multi-modal feature fusion, a more efficient medical image and text semantic feature fusion is realized. For the clear position information of shallow features, a shallow fusion feature enhancement (SFFE) module is set to make the features pay more attention to the clear edge of the lesion, enhance the spatial representation ability, and for the rich semantic information of deep features, a deep fusion feature enhancement (DFFE) module is set to enhance the semantic fusion, jointly realize the multi-modal fusion interaction mechanism, improve the performance of the model in multi-modal fusion, establish the cross-modal correspondence between medical images and text descriptions, and improve the recognition accuracy of the lesion.

[0030] 2. The introduced parallel attention mechanism realizes multi-dimensional enhancement. On the one hand, it realizes feature recalibration to improve the expression ability of key features, and on the other hand, it pays more attention to local structural differences and effectively suppresses the interference of same-spectrum different objects. This module provides structured enhancement in the feature reconstruction and fusion stage, and provides more stable and discriminative feature support for subsequent mask prediction and dynamic typing diagnosis.

[0031] 3. By using the encoder-decoder structure network based on multi-modal feature fusion, a medical image holographic dynamic typing diagnosis based on multi-modal data fusion is realized. Joint segmentation loss and contrast learning loss are used to slow down the problems of fuzzy boundary, large size difference and class imbalance, enhance the semantic consistency of images and texts, align the image lesion features and text pathological descriptions in the shared semantic space, combine the multi-phase feature aggregation strategy, realize the dynamic evolution analysis and typing diagnosis of the lesion area, and effectively improve the diagnosis robustness of the model in complex multi-modal medical scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0033] Figure 1 The flow chart of the method for medical image holographic dynamic typing diagnosis based on multi-modal data fusion; Figure 2 The structure diagram of the encoder-decoder network based on multi-modal feature fusion; Figure 3 The structure diagram of the shallow fusion feature enhancement SFFE module in the multi-modal fusion interaction mechanism; Figure 4 The structure diagram of the deep fusion feature enhancement DFFE module in the multi-modal fusion interaction mechanism; Figure 5 The structure diagram of the parallel attention mechanism; Figure 6 A flow chart of a medical image holographic dynamic typing diagnosis system based on multi-modal data fusion is shown. DETAILED DESCRIPTION

[0034] To make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0035] Embodiment 1 Figure 1 A flow chart of a medical image holographic dynamic typing diagnosis method based on multi-modal data fusion proposed by the present application is shown, which includes: Step 1: combining Figure 2 As shown, an encoder-decoder structure network based on multi-modal feature fusion is constructed, ResNet-50 is used as a visual encoder and Clip-Text is used as a text encoder to encode medical images and text descriptions, to obtain medical image features and text descriptors. After preliminary fusion of the medical image features and the text descriptors, a multi-modal fusion interaction mechanism is introduced to enhance the semantic correlation between the medical image features and the text features, cross-modal alignment is performed in the feature space, and a parallel attention mechanism is set in the decoding stage to recalibrate the features, to strengthen the response of the lesion-related feature channels and the spatial regions.

[0036] In the specific implementation process, the text descriptors include the position of the lesion, the nature of the lesion and the severity of the lesion.

[0037] The multi-modal fusion interaction mechanism includes a shallow fusion feature enhancement (SFFE) module and a deep fusion feature enhancement (DFFE) module. The SFFE module fuses the preliminary fusion features and the shallow features to highlight the position information expression and edge detail response of the shallow features, to improve the edge positioning accuracy of the lesion area. The DFFE module fuses the preliminary fusion features and the deep features to highlight the rich semantic information in the deep features, to improve the recognition ability and class discrimination performance of the lesion area. The shallow features include shallow features from the visual encoder branch and shallow features from the text encoder branch. The deep features include deep features from the visual encoder branch and deep features from the text encoder branch, to establish a stable cross-modal correspondence.

[0038] It should be noted that the multi-modal fusion interaction mechanism Figure 3As shown, the shallow fusion feature enhancement (SFFE) module is specifically: for the features from the encoder, shallow semantic features are obtained through joint global average pooling and global maximum pooling, and after adjusting the dimensions through 1x1 convolution, the preliminary fusion features are fused and weighted, and finally the residual connection is connected to the original shallow visual features, the fusion and weighting is a channel-by-channel weighting operation of the features after adjusting the dimensions through 1x1 convolution and the weight of the preliminary fusion features, and the weight of the preliminary fusion features is obtained by global maximum pooling and Sigmoid function on the preliminary fusion features.

[0039] It should be further pointed out that, in combination Figure 4 As shown, the deep fusion feature enhancement (DFFE) module is specifically: for the features from the encoder, the features are split into two channel size features, one of which is subjected to global average pooling and the other is subjected to convolution dimension adjustment and then multiplied to obtain deep semantic features, and for the preliminary fusion features, after three-branch deep separable convolution of different scales, the addition operation is performed, the input of the addition operation also includes the input of the original preliminary fusion features, and finally the channel-by-channel multiplication is performed on the deep semantic features to perform feature weighting.

[0040] In addition, the decoding stage adopts a bottom-up multi-scale feature fusion strategy, processes four multi-modal fusion features of different scales as a decoder, and adjacent scale multi-modal features as input of a parallel attention mechanism.

[0041] It should be noted that before entering the decoding stage, the four-layer output of the shallow fusion feature enhancement (SFFE) module and the four-layer output of the deep fusion feature enhancement (DFFE) module are added to the corresponding scale to realize complementary fusion of spatial details and semantic information, and to provide multi-level interactive features for high-precision lesion segmentation and dynamic typing diagnosis.

[0042] Further, the set Figure 5 As shown, for small-scale input features, after upsampling, element-by-element addition operation is performed with large-scale input to obtain scale fusion features, and the fused features are enhanced through parallel branches.

[0043] Wherein, the first branch of the parallel branch focuses on channel feature attention, and the second branch of the parallel branch focuses on spatial feature attention. After the first branch of the parallel branch is respectively processed by global average pooling and global maximum pooling, two global channel descriptions are obtained. The two global channel descriptions are spliced along the channel dimension, and the result is added to the outputs of the two pooling layers. After passing through the Sigmoid activation function, the channel attention weight is generated. Finally, it is multiplied by the original input of the first branch. Feature recalibration is realized through channel multiplication, highlighting semantic channel features with high discriminability. After the second branch of the parallel branch is respectively processed by average pooling and maximum pooling, two global channel descriptions are obtained. The two global channel descriptions are element-wise added, and the result is spliced along the channel dimension with the outputs of the two pooling layers. After passing through the Sigmoid activation function, the spatial attention weight is generated. Finally, it is multiplied by the original input of the second branch. Compared with the first branch, the second branch pays more attention to local structural differences, effectively suppressing the interference of same-spectrum foreign objects. Finally, the outputs of the two branches are element-wise added to obtain the output comprehensive feature enhancement output of the parallel attention mechanism, which is used for subsequent mask prediction and typing analysis stage.

[0044] It should be noted that the parallel attention mechanism realizes adaptive enhancement of feature salient regions on the basis of multi-scale fusion, which significantly improves the network's ability to recognize fine-grained lesion regions in complex backgrounds.

[0045] Step 2: Pre-training the constructed network using a multi-modal medical image dataset to form a pre-trained model weight.

[0046] Wherein, the multi-modal medical image dataset includes medical images and paired text descriptions.

[0047] In the specific implementation process, the medical images include different types of medical scan images, including CT, MRI and ultrasound images, and the text descriptions include diagnosis information, lesion location, pathological features and doctor's annotation information corresponding to the images.

[0048] In addition, during the pre-training process, a joint loss function is set, including segmentation loss and contrastive learning loss, which can comprehensively consider the pixel-level segmentation performance and cross-modal feature consistency.

[0049] In the specific implementation process, the formed pre-trained model weight is used as the initial parameter of the multi-modal feature fusion network, and the trained model parameter file is saved for subsequent model inference and fine-tuning stage, which accelerates network convergence and improves task generalization ability.

[0050] It should be noted that the segmentation loss is used to supervise the coincidence degree of the model output mask and the real lesion area, force the model output to be consistent with the real mask, reduce missed detection or false detection, improve the spatial positioning accuracy, keep the mask prediction consistent under different modal images, enhance the robustness of the model, and provide reliable spatial basis for subsequent lesion change calculation to support dynamic typing analysis.

[0051] It should also be noted that the contrast learning loss is used to constrain the similarity of image features and their corresponding text features in the shared semantic space, improve the generalization ability of the model under the condition of data scarcity through cross-modal mutual supervision, and at the same time make different types of lesions form clear clustering in the embedding space, which is beneficial to dynamic typing.

[0052] In the specific implementation process, the segmentation loss selects the combination form of cross-entropy loss and Dice loss, and the contrast learning loss selects the CLIP type contrast loss, which minimizes the feature distance between matched samples and maximizes the feature difference between unmatched samples.

[0053] Step 3: Collect new medical image data and related text description, and perform data preprocessing, and fuse the collected multi-modal data through the pre-trained network.

[0054] Among them, the fusion of the collected data through the trained network is: the input image is subjected to multi-layer feature extraction through a visual encoder to obtain four layers of visual initial features of different scales, the medical text description is subjected to semantic encoding through a text encoder to obtain a text initial feature descriptor, the four layers of image initial features of different scales and the text initial feature descriptor are spliced in the channel dimension to obtain four layers of multi-modal initial features of different scales, the query vector Q from the visual encoder and the key K and value V from the multi-modal initial feature representation are input into the Transformer module for preliminary multi-modal feature fusion to obtain preliminary fusion features, and the preliminary fusion features, the visual initial features and the multi-modal initial features are subjected to fusion interaction mechanism to obtain multi-modal interactive fusion features.

[0055] In addition, the data preprocessing includes image data preprocessing and text data preprocessing, the image data preprocessing includes normalization, noise suppression and size resampling, and the text data preprocessing includes text segmentation, stop word removal and medical entity recognition, the objects of medical entity recognition include disease name, part and pathological feature.

[0056] In the specific implementation process, the stage of fusing and identifying the collected data belongs to the model inference stage, and the weight and parameter update are not performed, and the weight and parameter maintained in the pre-training stage are used.

[0057] Step 4: The fused four-layer multi-modal features are decoded by top-down upsampling to generate a prediction mask, and the mask resolution is aligned with the original image through a spatial attention-based re-projection mechanism to obtain the lesion positioning result.

[0058] In the specific implementation process, the attention-based re-projection is specifically to map the high response region in the feature layer back to the original pixel space, and through learning the response distribution of the feature to the pixel, the adaptive alignment of the prediction mask in the spatial position is realized.

[0059] Step 5: According to the recognition results of different batches, feature aggregation is performed to calculate the lesion area change index and generate a dynamic typing result.

[0060] The feature aggregation method includes time series aggregation and spatial consistency aggregation, the time series set stacks the feature vectors in the order of detection time, and the spatial consistency aggregation calculates the mask overlap rate and morphological change amount of the same region, and takes the time series and the spatial consistency change index as the input feature, and generates the dynamic typing diagnosis result of the lesion through the typing model based on clustering analysis or classification network.

[0061] In the specific implementation process, the time series set further constitutes a time series feature matrix to calculate the dynamic change trend of the feature vector in the time dimension, and further describe the evolution process of the lesion over time.

[0062] In addition, the calculation of the lesion area change index is used to quantify the lesion change characteristics, specifically a difference index calculation, and the calculated indexes include the lesion area change rate, the lesion density change rate and the texture complexity change index.

[0063] In the specific implementation process, the lesion area change rate is calculated by the difference between the current and the previous phase lesion area, and the proportion of the difference in the lesion area of the previous phase is the lesion area change rate, which represents the degree of lesion expansion or reduction, and the lesion density change rate is calculated by the difference between the current and the previous phase mask area, which reflects the change of tissue composition, and the texture complexity change index is calculated by the difference between different texture features in different phases, which quantifies the complexity change of the lesion internal structure, and different texture features can be a gray level co-occurrence matrix or a local binary pattern.

[0064] Figure 5 The process chart of the medical image holographic dynamic typing diagnosis system based on multi-modal data fusion proposed by the application is given, which includes: A multi-modal data acquisition module, a data preprocessing module, a model recognition module and a dynamic typing diagnosis module.

[0065] The multi-modal data acquisition module constructs a holographic data set containing an image and a text use data mapping mechanism, the data preprocessing module filters out text items irrelevant to the lesion area through a semantic matching algorithm, and a quality control mechanism eliminates images or records that do not meet the labeling, the model recognition module uses a network based on encoding-decoding results to identify suspected lesion areas in medical images, and the dynamic typing diagnosis module aggregates features of multiple image recognition results of the same patient to calculate a change index of the lesion area and generate a lesion dynamic typing result.

[0066] It should be noted that the specific types of lesion dynamic typing include progressive type, stable type and regression type.

[0067] The above only describes certain exemplary embodiments of the present application by way of illustration, and it is self-evident that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present application. Therefore, the above figures and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present application.

[0068] It should be noted that in this paper, if there are relationship terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the term "include" "contain" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or equipment including the element.

[0069] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any skilled person in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A multimodal data fusion-based medical image holographic dynamic subtyping diagnostic system, characterized in that, include: The multimodal data acquisition module constructs a holographic dataset containing images and text using a data mapping mechanism; The data preprocessing module uses a semantic matching algorithm to filter text entries that are irrelevant to the lesion area, while a quality control mechanism removes images or records that do not meet the annotation requirements. The model recognition module uses a network based on an encoder-decoder structure to identify suspected lesion areas in medical images; The dynamic classification and diagnosis module aggregates features from multiple image recognition results of the same patient, calculates the change indicators of the lesion area, and generates dynamic classification results of the lesion.

2. A medical image holographic dynamic classification and diagnostic method based on multimodal data fusion, characterized in that: include: Step 1: Construct an encoder-decoder network based on multimodal feature fusion. Use a visual encoder and a text encoder to encode medical images and text descriptions to obtain medical image features and text descriptors. After the medical image features and text descriptors are initially fused, a multimodal fusion interaction mechanism is introduced. At the same time, a parallel attention mechanism is set in the decoding stage to perform feature recalibration. Step 2: Pre-train the constructed network using a multimodal medical image dataset to form pre-trained model weights; Step 3: Collect new medical image data and related text descriptions, and perform data preprocessing. Then, fuse the collected multimodal data through a pre-trained network. Step 4: After the fused four-layer multimodal features are upsampled and decoded from bottom to top to generate a prediction mask, the mask resolution is aligned with the original image through an attention-based spatial reprojection mechanism to obtain the lesion localization result; Step 5: Based on the identification results of different batches, perform feature aggregation, calculate the lesion area change index, and generate dynamic classification results.

3. The method for dynamic classification and diagnosis of medical images based on multimodal data fusion according to claim 2, characterized in that, The multimodal fusion interaction mechanism includes a shallow fusion feature enhancement (SFFE) module and a deep fusion feature enhancement (DFFE) module. The shallow fusion feature enhancement (SFFE) module fuses preliminary fusion features and shallow features, and the deep fusion feature enhancement (DFFE) module fuses preliminary fusion features and deep features. The shallow features include shallow features from the visual encoder branch and shallow features from the text encoder branch, and the deep features include deep features from the visual encoder branch and deep features from the text encoder branch.

4. The method for holographic dynamic subtyping diagnosis of medical images based on multimodal data fusion according to claim 3, characterized in that, The Shallow Fusion Feature Enhancement (SFFE) module specifically works as follows: For features from the encoder, shallow semantic features are obtained through joint global average pooling and global max pooling. After adjusting the dimensions through 1×1 convolution, they are fused and weighted with the input preliminary fusion features. Finally, the residuals are connected to the original shallow visual features. The fusion weighting is a channel-wise weighting operation between the features adjusted by 1×1 convolution and the weights of the preliminary fusion features. The weights of the preliminary fusion features are obtained by applying global max pooling and the Sigmoid function to the preliminary fusion features.

5. The method for dynamic classification and diagnosis of medical images based on multimodal data fusion according to claim 3, characterized in that, The Deep Fusion Feature Enhancement (DFFE) module specifically works as follows: For features from the encoder, the channels are split into two features of the same size. One feature is multiplied by the other after global average pooling and dimensionality adjustment through convolution to obtain deep semantic features. For the preliminary fused features, the three branches are subjected to depthwise separable convolution at different scales and then added together. The input of the addition operation also includes the original preliminary fused features. Finally, the deep semantic features are multiplied channel by channel to perform feature weighting.

6. The method for dynamic classification and diagnosis of medical images based on multimodal data fusion according to claim 2, characterized in that, The decoding stage adopts a bottom-up multi-scale feature fusion strategy, processing four multi-modal fusion features of different scales as the decoder, and using multi-modal features of adjacent scales as input to the parallel attention mechanism. For small-scale input features, after upsampling, they are added element-wise with the large-scale input to obtain inter-scale fusion features. The fused features are then enhanced through parallel branches.

7. The method for dynamic classification and diagnosis of medical images based on multimodal data fusion according to claim 6, characterized in that, The first branch of the parallel branch obtains two global channel descriptions after global average pooling and global max pooling, respectively. The two global channel descriptions are concatenated along the channel dimension, and the result is added to the outputs of the two pooling layers. Channel attention weights are generated by passing through the Sigmoid activation function, and finally multiplied with the original input of the first branch. The second branch of the parallel branch obtains two global channel descriptions after average pooling and max pooling, respectively. The two global channel descriptions are added element-wise, and the result is concatenated along the channel dimension with the outputs of the two pooling layers. Spatial attention weights are generated by passing through the Sigmoid activation function, and finally multiplied with the original input of the second branch. Finally, the outputs of the two branches are added element-wise to obtain the output of the parallel attention mechanism.

8. The method for dynamic classification and diagnosis of medical images based on multimodal data fusion according to claim 2, characterized in that, Step 3, which involves fusing the collected data using the trained network, specifically involves: extracting multi-layer features from the input image using a visual encoder to obtain four layers of initial visual features at different scales; semantically encoding the medical text description using a text encoder to obtain initial text feature descriptors; concatenating the four layers of initial image features at different scales with the initial text feature descriptors along the channel dimension to obtain four layers of multimodal initial features at different scales; inputting the query vector Q from the visual encoder and the key K and value V from the multimodal initial feature representation into the Transformer module for preliminary multimodal feature fusion to obtain preliminary fused features; and finally, using a fusion interaction mechanism, combining the preliminary fused features, the initial visual features, and the multimodal initial features to obtain multimodal interactive fused features.

9. The method for dynamic classification and diagnosis of medical images based on multimodal data fusion according to claim 2, characterized in that, The feature aggregation in step 5 includes time series aggregation and spatial consistency aggregation. The time series set stacks feature vectors in the order of detection time. The spatial consistency aggregation calculates the mask overlap rate and morphological change amount of the same region. The time series and spatial consistency change indicators are used as input features. Through a classification model based on cluster analysis or classification network, dynamic classification diagnosis results of lesions are generated.

Images