Smart medical system based on multi-modal data fusion

Abstract

The invention discloses an intelligent medical system based on multi-modal data fusion, and the system comprises a multi-modal data feature extraction module which is used for extracting the internal information of medical image data Fimg, pathological section data Fpath, text case data Ftext and voice case data Faudio; the inter-modal relationship analysis module is used for obtaining a visual consistency feature Fvis and a first association intensity mark of image-pathology fusion, obtaining a language consistency feature Flang and a second association intensity mark of text-voice fusion, and obtaining a key association feature Fcross representing the association intensity of vision and language and a third association intensity mark; and the fusion module is unified to obtain final judgment data. Aiming at the common problem of image and pathology inconsistency in medical practice, the method not only records consistency information, but also systematically analyzes various aspects of inconsistency, and provides more comprehensive reference for clinical decision making.

Description

Technical Field

[0001] This invention relates to the field of smart healthcare, and more particularly to a smart healthcare system based on multimodal data fusion. Background Technology

[0002] The development of multimodal data fusion in smart healthcare is rooted in the inevitable trend of medical informatization evolution. From early paper-based medical records and single imaging examinations, to the widespread adoption of electronic medical record systems and image archiving systems in the digital healthcare era, and now to the full arrival of the multimodal data fusion era, this evolution reflects the historical trajectory of medical data development from isolated fragmentation to systematic integration. In clinical practice, the diagnosis of complex diseases requires multi-dimensional evidence support, the identification of rare diseases requires the integration of multiple examination results, and personalized treatment relies on comprehensive patient profiles. These urgent needs have jointly driven the development of multimodal fusion technology. Simultaneously, the rapid improvement in computing power, breakthroughs in deep learning algorithms, and the advancement of medical data standardization have provided a solid technological foundation for multimodal fusion.

[0003] Current medical data is abundant but fragmented. Existing systems typically handle multimodal data by feature concatenation or post-processing voting: directly concatenating feature vectors from different modalities and allowing the model to learn feature relationships during training; or training separate single-modal models and then voting to make decisions. However, this approach has the following problems: simple concatenation leads to feature duplication and interference from contradictory information, making it impossible to dynamically adjust the importance of each modality based on specific cases, and inconsistent evidence is smoothed out during the fusion process, causing potential risks to be overlooked. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a smart medical system based on multimodal data fusion.

[0005] The objective of this invention is achieved through the following technical solution: A first aspect of the present invention provides a smart healthcare system based on multimodal data fusion, applied to a backend server, comprising: The multimodal data feature extraction module is used to extract the internal information of medical image data F_img, pathological slide data F_path, text medical record data F_text, and audio medical record data F_audio, respectively, to obtain refined image features F_img', pathological features F_path', text features F_text', and audio features F_audio'; The intermodal relationship analysis module is used to fuse image features F_img' and pathological features F_path' to obtain the visual consistency feature F_vis and the first association strength label of image-pathology fusion; to fuse text features F_text' and speech features F_audio' to obtain the language consistency feature F_lang and the second association strength label of text-speech fusion; and to bridge the visual consistency feature F_vis and the language consistency feature F_lang to obtain the key association feature F_cross and the third association strength label representing the association strength between vision and language. The unified fusion module is used to perform unified fusion of image features F_img', pathological features F_path', text features F_text', speech features F_audio', visual consistency features F_vis, language consistency features F_lang, and key association features F_cross to obtain the final judgment data. Specifically, the first fusion weight of image features F_img', pathological features F_path', and visual consistency features F_vis is adjusted using a first association strength marker, the second fusion weight of text features F_text', speech features F_audio', and language consistency features F_lang is adjusted using a second association strength marker, and the first fusion weight, the second fusion weight, and the final judgment data are adjusted using a third association strength marker.

[0006] Furthermore, in the multimodal data feature extraction module: The medical image data F_img is processed by a three-dimensional self-attention network to obtain image features F_img' including the morphology, texture and density of the lesion area; The pathological slide data F_path is processed using the Visual Transformer (VIT) model to obtain pathological features F_path', which includes abnormal patterns in the tissue microenvironment. The text medical record data F_text is processed through a medical BERT model to obtain text features F_text' containing key information from the electronic medical record; The audio case data F_audio is processed by a temporal attention network model to obtain the diagnosis-related audio features F_audio' from the audio segments containing the patient's description of symptoms.

[0007] Furthermore, in the intermodal relationship analysis module, the fusion of image feature F_img' and pathological feature F_path' to obtain the visual consistency feature F_vis of image-pathology fusion and the first association strength marker includes: Image features F_img' and pathological features F_path' are input into a CNN semantic network containing four semantic heads to obtain image semantic img_semantic and pathological semantic path_semantic; the four semantic heads correspond to morphological features, density / texture features, diagnostic tendency and severity, respectively; The cosine similarity of the corresponding dimensions in the image semantic (img_semantic) and pathological semantic (path_semantic) representing morphological features, density / texture features, diagnostic tendency, and severity is calculated respectively to obtain the first consistency of morphological features, the second consistency of density / texture features, the third consistency of diagnostic tendency, and the fourth consistency of severity. The visual consistency feature F_vis is obtained by splicing, fusing and compressing the first consistency, second consistency, third consistency, fourth consistency, image semantic (img_semantic), and pathological semantic (path_semantic); the first association strength label is obtained by weighted averaging of the first consistency, second consistency, third consistency and fourth consistency.

[0008] Furthermore, in the intermodal relation analysis module, the fusion of text feature F_text' and speech feature F_audio' to obtain text-speech fusion language consistency feature F_lang and second association strength label includes: Text features F_text' include text symptom features text_symptoms and text emotion features text_emotion; speech features F_audio' include speech symptom features audio_symptoms and speech emotion features audio_emotion; The cosine similarity between text symptom features (text_symptoms) and audio symptom features (audio_symptoms) is calculated to obtain the fifth consistency of symptom description; the cosine similarity between text emotion features (text_emotion) and audio emotion features (audio_emotion) is calculated to obtain the sixth consistency of emotion expression; and the keyword matching degree between text feature F_text' and audio feature F_audio' is calculated to obtain the seventh consistency of keywords. The language consistency feature F_lang is obtained by concatenating, fusing, and compressing the fifth consistency, sixth consistency, seventh consistency, text symptom features, text emotion features, audio symptom features, and audio emotion features; the second association strength label is obtained by weighted averaging of the fifth consistency, sixth consistency, and seventh consistency.

[0009] Furthermore, the bridging of the visual consistency feature F_vis and the linguistic consistency feature F_lang to obtain the key association feature F_cross representing the association strength between vision and language, and a third association strength marker, includes: The first association value is obtained by calculating the cosine similarity of the diagnostic tendency part in the visual consistency feature F_vis and the concatenated part of the text symptom feature text_symptoms and the audio symptom feature audio_symptoms in the language consistency feature F_lang. The second association value is obtained by calculating the degree of matching between the severity component in the visual consistency feature F_vis and the text emotion feature text_emotion and the audio emotion feature audio_emotion representing the pain in the language consistency feature F_lang. The first association value, the second association value, the diagnostic tendency portion of the visual consistency feature F_vis, the concatenated portion of the text symptom features and the audio symptom features of the language consistency feature F_lang, the severity portion of the visual consistency feature F_vis, and the concatenated portion of the text emotion feature and the audio emotion feature of the language consistency feature F_lang representing pain are concatenated, fused, and compressed to obtain the key association feature F_cross; the lowest value between the first association value and the second association value is used as the third association strength label.

[0010] Furthermore, in the unified fusion module, the adjustment of the first fusion weights of image feature F_img', pathological feature F_path', and visual consistency feature F_vis in the unified fusion process using the first association strength marker includes: The image baseline weight of image feature F_img' is set to 0.3, the pathological baseline weight of pathological feature F_path' is set to 0.3, and the visual consistency feature baseline weight of visual consistency feature F_vis is set to 0.4. When the first correlation strength label is > 0.8, it means that the image feature F_img' and the pathological feature F_path' are highly consistent. At this time, the basic weight of the image and the basic weight of the pathology are both multiplied by 1.2, while the basic weight of the visual consistency feature is multiplied by 0.8. When the first correlation strength label is < 0.4, it means that the image feature F_img' and the pathological feature F_path' have low consistency. At this time, the basic weight of the image and the basic weight of the pathology are both multiplied by 0.8, while the basic weight of the visual consistency feature is multiplied by 1.2 to emphasize contradictory features. The processed image weights, pathological weights, and visual consistency feature weights are normalized to obtain the first fusion weight.

[0011] Furthermore, in the unified fusion module, the adjustment of the second fusion weights of text feature F_text', speech feature F_audio', and language consistency feature F_lang in the unified fusion process using the second association strength marker includes: The base weight of text feature F_text' is set to 0.3, the base weight of speech feature F_audio' is set to 0.3, and the base weight of language consistency feature F_lang is set to 0.4. When the second association strength label is > 0.8, it means that the text feature F_text' and the speech feature F_audio' are highly consistent. At this time, the basic weights of the text and speech features are multiplied by 1.2, while the basic weights of the language consistency features are multiplied by 0.8. When the second association strength label is < 0.4, it means that the text feature F_text' and the speech feature F_audio' have low consistency. At this time, the basic weights of text and speech are multiplied by 0.8, while the basic weight of language consistency feature is multiplied by 1.5 to emphasize contradictory features. The processed text weights, speech weights, and language consistency feature weights are normalized to obtain the second fusion weight.

[0012] Furthermore, in the unified fusion module, the adjustment of the first fusion weight, the second fusion weight, and the final judgment data using the third association strength marker includes: If the third association strength label is < 0.3, it indicates that the visual and linguistic association is weak. In this case, the interaction between the two sets of fusion weights is reduced, that is, the visual evidence dependence is increased and the linguistic evidence weight is reduced: the first fusion weight is increased by 1.5, and the second fusion weight is increased by 0.7. If the third association strength label is > 0.7, it indicates that vision and language are strongly associated. In this case, the two sets of fusion are promoted: the first fusion weight * 0.9 and the second fusion weight * 0.9.

[0013] Furthermore, in the unified fusion module, the unified fusion of image features F_img', pathological features F_path', text features F_text', speech features F_audio', visual consistency features F_vis, linguistic consistency features F_lang, and key association features F_cross includes: The visual group F_visual_group, which includes image features F_img', pathological features F_path', and visual consistency features F_vis, is fused using the first fusion weight; the language group F_language_group, which includes text features F_text', speech features F_audio', and language consistency features F_lang, is fused using the second fusion weight. The key association feature F_cross and the first attention F_visual_group are calculated using softmax, which represents the part of the visual group F_visual_group that the key association feature F_cross pays attention to; and the visual group F_visual_group and the first attention F_visual_group are multiplied to obtain the attention-weighted cross-group visual feature F_visual_selected. The key association feature F_cross and the second attention F_language group are calculated using softmax, which represents the part of the language group F_language group that the key association feature F_cross pays attention to; and the language group F_language group and the second attention F_language group are multiplied to obtain the attention-weighted cross-group language feature F_language_selected. When the third association strength label is > 0.5, the association is considered strong and deep fusion is performed. The weights of cross-group visual features F_visual_selected, cross-group language features F_language_selected, and key association features F_cross are set to 0.4, and then unified fusion is performed to obtain the fused feature F_fused. Otherwise, the association is considered weak and cautious fusion is performed. The weights of cross-group visual features F_visual_selected, cross-group language features F_language_selected, and key association features F_cross are set to 0.1, and then unified fusion is performed to obtain the fused feature F_fused.

[0014] Furthermore, in the unified fusion module, the final judgment data includes: The fused feature F_fused, the key association feature F_cross, the first association strength label, the second association strength label, and the third association strength label are concatenated to obtain the final feature representation final_representation; The final feature representation (final_representation) is input into the classification head to generate the judgment result.

[0015] The beneficial effects of this invention are: In this exemplary embodiment, a multimodal data feature extraction module is first used to refine intramodal features and extract key information. Then, an intermodal relationship analysis module is used to analyze intermodal relationships and establish pairwise associations. Finally, a unified fusion module generates a comprehensive judgment. Furthermore, through three association strength marking mechanisms, the fusion weights of each modality can be dynamically adjusted according to specific cases, achieving intelligent weight allocation. Addressing the common problem of inconsistencies between imaging and pathology in medical practice, this approach not only records consistency information but also systematically analyzes various aspects of inconsistency, providing a more comprehensive reference for clinical decision-making. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of a smart medical system based on multimodal data fusion provided in an exemplary embodiment of the present invention. Detailed Implementation

[0017] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0018] See Figure 1 , Figure 1 This diagram illustrates a smart healthcare system based on multimodal data fusion provided in an exemplary embodiment of the present invention, applied to a backend server, including: The multimodal data feature extraction module is used to extract the internal information of medical image data F_img, pathological slide data F_path, text medical record data F_text, and audio medical record data F_audio, respectively, to obtain refined image features F_img', pathological features F_path', text features F_text', and audio features F_audio'; The intermodal relationship analysis module is used to fuse image features F_img' and pathological features F_path' to obtain the visual consistency feature F_vis and the first association strength label of image-pathology fusion; to fuse text features F_text' and speech features F_audio' to obtain the language consistency feature F_lang and the second association strength label of text-speech fusion; and to bridge the visual consistency feature F_vis and the language consistency feature F_lang to obtain the key association feature F_cross and the third association strength label representing the association strength between vision and language. The unified fusion module is used to perform unified fusion of image features F_img', pathological features F_path', text features F_text', speech features F_audio', visual consistency features F_vis, language consistency features F_lang, and key association features F_cross to obtain the final judgment data. Specifically, the first fusion weight of image features F_img', pathological features F_path', and visual consistency features F_vis is adjusted using a first association strength marker, the second fusion weight of text features F_text', speech features F_audio', and language consistency features F_lang is adjusted using a second association strength marker, and the first fusion weight, the second fusion weight, and the final judgment data are adjusted using a third association strength marker.

[0019] Specifically, in this exemplary embodiment, the relationships between the evidence from each modality are first examined separately, and finally, all information is synthesized to form a diagnosis: First, in the multimodal data feature extraction module, intramodal feature refinement is performed. The four medical data (medical image data F_img, pathological slide data F_path, text case data F_text, and audio case data F_audio) modalities are respectively optimized internally and key information is extracted. The most diagnostically valuable features are extracted from the original data, while noise and redundant information are filtered out.

[0020] The relational analysis module then analyzes the inter-modal relationships between different modal features: (1) It spatially registers the anatomical features of the image with the microscopic tissue features of the pathology and analyzes the degree of consistency between the two; for example, if a CT scan shows a region of consolidation in the lungs and the pathology shows inflammatory exudate in the alveoli, the two are highly consistent in spatial location and tissue properties, which will produce a strong positive correlation feature; if the image suggests a malignant lesion while the pathology shows a benign change, a contradictory label feature will be produced. The first correlation strength label represents the correlation of the relationship. (2) It compares the text description in the electronic medical record with the actual voice content in the recording to verify the consistency between the two. In the subsequent preferred exemplary embodiment, it analyzes whether the severity of symptoms described in the text matches the emotional intensity expressed in the voice. For example, the text record says "mild cough", but the recording shows frequent coughing and wheezing. This inconsistency will be captured and encoded into the relational features. Similarly, the correlation is represented by the second correlation strength. (3) The cross-modal bridging feature, also known as the key association feature F_cross, is the most crucial relational feature. It directly analyzes the correlation between visual findings (imaging + pathology) and verbal descriptions (text + speech). This feature answers a core clinical question: Do the clinical symptoms described by the patient match the imaging findings? For example, if imaging reveals a space-occupying lesion in the liver, and the patient describes right upper quadrant pain and jaundice, the two are highly correlated; conversely, if imaging reveals significant abnormalities but the patient has no symptoms, the correlation is weak. In this case, the third association strength is used to represent the correlation.

[0021] Finally, the unified fusion module receives all outputs from the previous multimodal data feature extraction module and the intermodal relationship analysis module: four refined single-modal features (image feature F_img', pathology feature F_path', text feature F_text', and speech feature F_audio') and three relationship features (visual consistency feature F_vis, linguistic consistency feature F_lang, and key association feature F_cross). The strength of the image-pathology relationship feature determines the overall credibility of visual evidence: if the two are highly consistent, the weight of visual evidence is significantly increased (first fusion weight); if there is a contradiction, the weight of visual evidence is reduced, triggering contradiction analysis. Similarly, the text-speech relationship feature determines the credibility of semantic evidence (second fusion weight). The key association feature F_cross plays a crucial role as a decision confidence modulator in the third stage: when visual findings are highly correlated with linguistic descriptions, the system has high confidence in the diagnostic conclusion; when the correlation is weak, the system reduces the confidence, that is, it uses the third association strength label to adjust the first fusion weight, the second fusion weight, and the final judgment data.

[0022] In summary, this exemplary embodiment first utilizes a multimodal data feature extraction module to refine intramodal features and extract key information, then employs an intermodal relationship analysis module to establish pairwise associations, and finally generates a comprehensive judgment through a unified fusion module. Furthermore, through three association strength marking mechanisms, the fusion weights of each modality can be dynamically adjusted according to specific cases, achieving intelligent weight allocation. Addressing the common issue of inconsistencies between imaging and pathology in medical practice, it not only records consistency information but also systematically analyzes various aspects of inconsistency, providing a more comprehensive reference for clinical decision-making.

[0023] The following will describe in detail the preferred exemplary embodiments of each step: More preferably, in an exemplary embodiment, in the multimodal data feature extraction module: The medical image data F_img is processed by a three-dimensional self-attention network to obtain image features F_img' including the morphology, texture and density of the lesion area; The pathological slide data F_path is processed using the Visual Transformer (VIT) model to obtain pathological features F_path', which includes abnormal patterns in the tissue microenvironment. The text medical record data F_text is processed through a medical BERT model to obtain text features F_text' containing key information from the electronic medical record; The audio case data F_audio is processed by a temporal attention network model to obtain the diagnosis-related audio features F_audio' from the audio segments containing the patient's description of symptoms.

[0024] Specifically, in this exemplary embodiment, the medical image data F_img is processed through a three-dimensional self-attention network, focusing on the image features F_img' such as the morphology, texture, and density of the lesion region. In a preferred exemplary embodiment, the most likely abnormal anatomical regions in the image can be identified, such as lung nodules with abnormal density in a CT scan or brain white matter regions with abnormal signal in an MRI.

[0025] The pathological slide data F_path is processed using the Visual Transformer (VIT) model, which segments the high-resolution whole-slice image into small block sequences. In a preferred exemplary embodiment, abnormal patterns in the tissue microenvironment, such as the arrangement of malignant cells, the degree of interstitial reaction, and the state of angiogenesis, can be observed to obtain the pathological features F_path'.

[0026] The textual case data F_text undergoes deep processing by a medical BERT model to identify key information in the electronic medical record, thereby obtaining textual features F_text'. In a preferred exemplary implementation, the clinical value of different parts can be distinguished: the chief complaint and present illness receive the highest attention because these parts contain the most direct diagnostic clues; past medical history and family history receive moderate attention as background information; while routine descriptions and template content are appropriately suppressed.

[0027] The speech feature F_audio from the voice case data is extracted using a temporal attention network model, focusing on the speech segments when the patient describes symptoms. In a preferred exemplary implementation, the system not only analyzes the textual content of the speech but also analyzes paralinguistic features, such as the degree of tremor in the voice when describing pain, the frequency of rapid breathing, and anxiety in the tone of voice.

[0028] More preferably, in an exemplary embodiment, in the intermodal relationship analysis module, the step of fusing image feature F_img' and pathological feature F_path' to obtain image-pathology fusion visual consistency feature F_vis and a first association strength marker includes: Image features F_img' and pathological features F_path' are input into a CNN semantic network containing four semantic heads to obtain image semantic img_semantic and pathological semantic path_semantic; the four semantic heads correspond to morphological features, density / texture features, diagnostic tendency and severity, respectively; The cosine similarity of the corresponding dimensions in the image semantic (img_semantic) and pathological semantic (path_semantic) representing morphological features, density / texture features, diagnostic tendency, and severity is calculated respectively to obtain the first consistency of morphological features, the second consistency of density / texture features, the third consistency of diagnostic tendency, and the fourth consistency of severity. The visual consistency feature F_vis is obtained by splicing, fusing and compressing the first consistency, second consistency, third consistency, fourth consistency, image semantic (img_semantic), and pathological semantic (path_semantic); the first association strength label is obtained by weighted averaging of the first consistency, second consistency, third consistency and fourth consistency.

[0029] Specifically, in this exemplary embodiment, the relevant content of the fusion of two visual modal data in the intermodal relationship analysis module is disclosed, with image feature F_img' and pathological feature F_path' being 512 dimensions: (1) First, feature alignment and comparison are performed. The alignment is simplified by using a semantic rather than spatial approach, and semantic-level features are extracted for comparison: Image semantics `img_semantic` = CNN_semantic extraction(F_img') yields 128-dimensional semantic features; semantic features `path_semantic` = CNN_semantic extraction(F_path') yields 128-dimensional semantic features. The CNN semantic network includes four semantic heads, corresponding to morphological features, density / texture features, diagnostic tendency, and severity, respectively.

[0030] Then, four key consistency dimensions were calculated: First consistency = cosine similarity(img_semantic[0:32], path_semantic[0:32]), which corresponds to morphological feature consistency; Second consistency = cosine similarity(img_semantic[32:64], path_semantic[32:64]), which corresponds to density / texture feature consistency; Third consistency = cosine similarity(img_semantic[64:96], path_semantic[64:96]), which corresponds to diagnostic tendency consistency; Fourth consistency = cosine similarity(img_semantic[96:128], path_semantic[96:128]), which corresponds to severity consistency.

[0031] (2) Generate visual consistency feature F_vis and first association strength label: F_vis = concatenated([first consistency, second consistency, third consistency, fourth consistency, img_semantic, path_semantic]), and finally fused and compressed to obtain the visual consistency feature F_vis. Simultaneously, the first association strength label calculation (scalar value 0-1) = weighted average (first consistency, second consistency, third consistency, fourth consistency). In an optimal solution, the weights here are: diagnostic tendency consistency > morphological consistency > density consistency > severity consistency.

[0032] It should be noted that the role of the visual consistency feature F_vis (image-pathology consistency feature) is as follows: providing direct evidence: when the image and pathology are consistent, the F_vis feature value is strong; contradiction detection: when the two contradict each other, the F_vis feature value is weak (or contains a contradiction marker); the intensity of F_vis is used to adjust the final weight of the visual modality in the subsequent unified fusion module.

[0033] More preferably, in an exemplary embodiment, in the intermodal relation analysis module, the fusion of text feature F_text' and speech feature F_audio' to obtain text-speech fusion language consistency feature F_lang and a second association strength tag includes: Text features F_text' include text symptom features text_symptoms and text emotion features text_emotion; speech features F_audio' include speech symptom features audio_symptoms and speech emotion features audio_emotion; The cosine similarity between text symptom features (text_symptoms) and audio symptom features (audio_symptoms) is calculated to obtain the fifth consistency of symptom description; the cosine similarity between text emotion features (text_emotion) and audio emotion features (audio_emotion) is calculated to obtain the sixth consistency of emotion expression; and the keyword matching degree between text feature F_text' and audio feature F_audio' is calculated to obtain the seventh consistency of keywords. The language consistency feature F_lang is obtained by concatenating, fusing, and compressing the fifth consistency, sixth consistency, seventh consistency, text symptom features, text emotion features, audio symptom features, and audio emotion features; the second association strength label is obtained by weighted averaging of the fifth consistency, sixth consistency, and seventh consistency.

[0034] Specifically, in this exemplary embodiment, the relevant content of the fusion of two language modal data in the intermodal relationship analysis module is disclosed, with text feature F_text' and speech feature F_audio' as 512 dimensions: (1) First, perform temporal and semantic alignment: Text symptom features (text_symptoms = BERT_symptom extraction(F_text')) are 128-dimensional; text sentiment features (text_emotion = BERT_sentiment analysis(F_text')) are 64-dimensional. Similarly, speech feature parsing (audio_symptoms = speech symptom recognition(F_audio')) is 128-dimensional; speech sentiment features (audio_emotion = speech sentiment recognition(F_audio')) are 64-dimensional.

[0035] Then, three consistency dimensions are calculated: Fifth consistency = cosine similarity (text_symptoms, audio_symptoms), which corresponds to the consistency of symptom description; Sixth consistency = cosine similarity (text_emotion, audio_emotion), which corresponds to the consistency of emotional expression; Seventh consistency = keyword matching degree (F_text', F_audio'), which corresponds to keyword consistency.

[0036] (2) Generate language consistency feature F_lang and second association strength label: F_lang = concatenate([fifth consistency, sixth consistency, seventh consistency, text_symptoms, audio_symptoms, text_emotion, audio_emotion]); Second association strength marker = (Fifth consistency * 0.5 + Fifth consistency * 0.3 + Seventh consistency * 0.2).

[0037] It should be noted that the role of the language consistency feature F_lang (text-speech consistency feature) is as follows: symptom authenticity verification: consistency between text record and speech description → strong feature; sentiment consistency: matching the severity of text description with speech sentiment → strong feature; subsequent unified fusion module use: adjusting the final weight of language modalities with the strength of F_lang.

[0038] More preferably, in an exemplary embodiment, bridging the visual consistency feature F_vis and the linguistic consistency feature F_lang to obtain the key association feature F_cross representing the association strength between vision and language, and a third association strength marker, includes: The first association value is obtained by calculating the cosine similarity of the diagnostic tendency part in the visual consistency feature F_vis and the concatenated part of the text symptom feature text_symptoms and the audio symptom feature audio_symptoms in the language consistency feature F_lang. The second association value is obtained by calculating the degree of matching between the severity component in the visual consistency feature F_vis and the text emotion feature text_emotion and the audio emotion feature audio_emotion representing the pain in the language consistency feature F_lang. The first association value, the second association value, the diagnostic tendency portion of the visual consistency feature F_vis, the concatenated portion of the text symptom features and the audio symptom features of the language consistency feature F_lang, the severity portion of the visual consistency feature F_vis, and the concatenated portion of the text emotion feature and the audio emotion feature of the language consistency feature F_lang representing pain are concatenated, fused, and compressed to obtain the key association feature F_cross; the lowest value between the first association value and the second association value is used as the third association strength label.

[0039] Specifically, in this exemplary embodiment, visual-semantic bridging features in the intermodal relationship analysis module are disclosed: (1) First, perform a correlation strength analysis: Key association dimensions are extracted: Visual_Diagnosis = F_vis[50:100], assuming that this part of the visual consistency feature F_vis encodes diagnostic information; Language_Symptoms = F_lang[50:100], assuming that this part of the language consistency feature F_lang encodes symptom information. Then, the cosine similarity is calculated between the diagnostic tendency portion of the visual consistency feature F_vis (i.e., Visual_Diagnosis) and the concatenated portion of the text symptom features and audio symptom features in the language consistency feature F_lang (i.e., Language_Symptoms), yielding the first association value; the matching degree is calculated between the severity portion of the visual consistency feature F_vis and the concatenated portion of the text emotion feature and audio emotion feature in the language consistency feature F_lang representing pain, yielding the second association value.

[0040] (2) Generate key association features F_cross and third association strength labels: F_cross = concatenation([first association value, second association value, diagnostic tendency portion of visual consistency feature F_vis, concatenation of text symptoms and audio symptoms in language consistency feature F_lang, severity portion of visual consistency feature F_vis, concatenation of text emotion and audio emotion representing pain in language consistency feature F_lang).

[0041] Finally, the third association strength label = min(first association value, second association value), that is, take the weakest association as the overall strength.

[0042] More preferably, in an exemplary embodiment, a weight adjustment based on association strength markers in the unified fusion module is disclosed, firstly for the adjustment of the first fusion weight (image group): In the unified fusion module, the adjustment of the first fusion weights of image feature F_img', pathological feature F_path', and visual consistency feature F_vis in the unified fusion process using the first association strength markers includes: The image baseline weight of image feature F_img' is set to 0.3, the pathological baseline weight of pathological feature F_path' is set to 0.3, and the visual consistency feature baseline weight of visual consistency feature F_vis is set to 0.4. When the first correlation strength label is > 0.8, it means that the image feature F_img' and the pathological feature F_path' are highly consistent. At this time, the basic weight of the image and the basic weight of the pathology are both multiplied by 1.2, while the basic weight of the visual consistency feature is multiplied by 0.8. When the first correlation strength label is < 0.4, it means that the image feature F_img' and the pathological feature F_path' have low consistency. At this time, the basic weight of the image and the basic weight of the pathology are both multiplied by 0.8, while the basic weight of the visual consistency feature is multiplied by 1.2 to emphasize contradictory features. The processed image weights, pathological weights, and visual consistency feature weights are normalized to obtain the first fusion weight: i.e., first fusion weight = [image weight, pathological weight, visual consistency feature weight] / sum([image weight, pathological weight, visual consistency feature weight]).

[0043] More preferably, in an exemplary embodiment, a weight adjustment based on association strength markers in the unified fusion module is disclosed, here referred to as a second fusion weight adjustment (language group): In the unified fusion module, adjusting the second fusion weights of text feature F_text', speech feature F_audio', and language consistency feature F_lang in the unified fusion process using the second association strength markers includes: The base weight of text feature F_text' is set to 0.3, the base weight of speech feature F_audio' is set to 0.3, and the base weight of language consistency feature F_lang is set to 0.4. When the second association strength label is > 0.8, it means that the text feature F_text' and the speech feature F_audio' are highly consistent. At this time, the basic weights of the text and speech features are multiplied by 1.2, while the basic weights of the language consistency features are multiplied by 0.8. When the second association strength label is < 0.4, it means that the text feature F_text' and the speech feature F_audio' have low consistency. At this time, the basic weights of text and speech are multiplied by 0.8, while the basic weight of language consistency feature is multiplied by 1.5 to emphasize contradictory features. The processed text weights, speech weights, and language consistency feature weights are normalized to obtain the second fusion weight, which is: Second fusion weight = [text weight, speech weight, language consistency feature] / sum([text weight, speech weight, language consistency feature]).

[0044] More preferably, in an exemplary embodiment, in the unified fusion module, adjusting the first fusion weight, the second fusion weight, and the final judgment data using the third association strength marker includes: If the third association strength label is < 0.3, it indicates that the visual and linguistic association is weak. In this case, the interaction between the two sets of fusion weights is reduced, that is, the visual evidence dependence is increased and the linguistic evidence weight is reduced: the first fusion weight is increased by 1.5, and the second fusion weight is increased by 0.7. If the third association strength label is > 0.7, it indicates that vision and language are strongly associated. In this case, the two sets of fusion are promoted: the first fusion weight * 0.9 and the second fusion weight * 0.9.

[0045] The above three exemplary embodiments disclose weight adjustment based on association strength markers. The following exemplary embodiments will disclose the unified fusion process: More preferably, in an exemplary embodiment, a weight adjustment based on association strength labeling in the unified fusion module is disclosed: In the unified fusion module, the unified fusion of image features F_img', pathological features F_path', text features F_text', speech features F_audio', visual consistency features F_vis, linguistic consistency features F_lang, and key association features F_cross includes: The visual group F_visual_group, which includes image features F_img', pathological features F_path', and visual consistency features F_vis, is fused using the first fusion weight; the language group F_language_group, which includes text features F_text', speech features F_audio', and language consistency features F_lang, is fused using the second fusion weight. The key association feature F_cross and the first attention F_visual_group are calculated using softmax, which represents the part of the visual group F_visual_group that the key association feature F_cross pays attention to; and the visual group F_visual_group and the first attention F_visual_group are multiplied to obtain the attention-weighted cross-group visual feature F_visual_selected. The key association feature F_cross and the second attention F_language group are calculated using softmax, which represents the part of the language group F_language group that the key association feature F_cross pays attention to; and the language group F_language group and the second attention F_language group are multiplied to obtain the attention-weighted cross-group language feature F_language_selected. When the third association strength label is > 0.5, the association is considered strong and deep fusion is performed. The weights of cross-group visual features F_visual_selected, cross-group language features F_language_selected, and key association features F_cross are set to 0.4, and then unified fusion is performed to obtain the fused feature F_fused. Otherwise, the association is considered weak and cautious fusion is performed. The weights of cross-group visual features F_visual_selected, cross-group language features F_language_selected, and key association features F_cross are set to 0.1, and then unified fusion is performed to obtain the fused feature F_fused.

[0046] Specifically, in this exemplary embodiment, an implementation of the unified fusion module is disclosed: (1) First, there is integration within the group: Visual group fusion: Visual group F_visual_group = first fusion weight * F_img' + first fusion weight * F_path' + first fusion weight * F_vis; Language group fusion: Language group F_language_group = second fusion weight * F_text' + second fusion weight * F_audio' + second fusion weight * F_lang.

[0047] (2) Cross-group fusion (using F_cross as the guide): First, calculate the attention guided by F_cross: The first attention, attention_visual = softmax(F_cross · F_visual_group^T), determines which parts of the visual group F_cross focuses on; the second attention, attention_language = softmax(F_cross · F_language_group^T), determines which parts of the language group F_cross focuses on.

[0048] Then, attention-weighted cross-group features are calculated: Cross-group visual features F_visual_selected = attention_visual * F_visual_group Cross-group language features F_language_selected = attention_language * F_language_group Finally, cross-group fusion is performed: If the third association strength label > 0.5, the association is considered strong and deep fusion is performed, and the fusion feature F_fused = 0.4 * F_visual_selected + 0.4 * F_language_selected + 0.2 * F_cross; otherwise, the association is weak and fusion is performed cautiously, and the fusion feature F_fused = 0.6 * F_visual_selected + 0.3 * F_language_selected + 0.1 * F_cross.

[0049] More preferably, in an exemplary embodiment, in the unified fusion module, obtaining the final judgment data includes: The fused feature F_fused, the key association feature F_cross, the first association strength label, the second association strength label, and the third association strength label are concatenated to obtain the final feature representation final_representation; The final feature representation (final_representation) is input into the classification head to generate the judgment result. The judgment data is calculated as softmax(classification head(final_representation)).

[0050] The following content will show some examples: I. Normal Data Flow: (1) Patient data extracted by the multimodal data feature extraction module: Medical imaging data F_img: CT scan shows consolidation in the right lower lung; Pathological slide data F_path: inflammatory exudate in the alveoli; Text case data F_text: records "fever, cough, yellow sputum for 3 days"; Audio case data F_audio: Obvious coughing and rapid breathing in the recording; (2) Processing of the intermodal relationship analysis module: Visual consistency feature F_vis generation: high consistency between image and pathology → strong feature; Language consistency feature generation for F_lang: Text-speech consistency → strong feature; Key correlation feature F_cross generation: Lung consolidation (visual) + respiratory symptoms (semantic) → high correlation → strong feature; (3) Unified integration of modules: Input = [CT features, pathological features, text features, speech features, strong F_vis, strong F_lang, strong F_cross] Unified fusion module attention mechanism: Seeing strong visual consistency features (F_vis) → Increase the weight of CT and pathology. Seeing that the language consistency feature F_lang is strong → increase the weight of text and speech; Seeing the key correlation feature F_cross strong → confirms that this is a typical manifestation of pneumonia; Final fusion: High-confidence diagnosis of bacterial pneumonia.

[0051] II. Conflict Data Flow: (1) Patient data extracted by the multimodal data feature extraction module: Medical imaging data F_img: Lung nodules; Pathological slide data F_path: benign inflammatory changes; Text case data F_text: records "asymptomatic"; Voice case data F_audio: Normal breath sounds; (2) Processing of the intermodal relationship analysis module: Visual consistency feature F_vis generation: Image-pathology inconsistency → features contain contradictory markers; Language consistency feature F_lang generation: Text-speech consistency (both asymptomatic) → strong feature; Key correlation feature F_cross generation: Lung detection + asymptomatic → low correlation → weak feature; (3) Unified integration of modules: Inconsistent markers were detected for the visual consistency feature F_vis → Reduce visual modality weights. The key association feature F_cross was detected as weak → reduce the overall confidence. Output: "A pulmonary nodule was discovered incidentally; follow-up observation is recommended," with a moderate confidence level.

[0052] In yet another exemplary embodiment, process parameters can be displayed regardless of their differences, such as: { Final Diagnosis: Lung adenocarcinoma (early stage) Modal consistency analysis: { "Image-pathology consistency": 0.92, / / First correlation strength marker Text-to-speech consistency: 0.85, / / Second correlation strength marker "Clinical-examination correlation": 0.78, / / Third correlation strength marker Overall consistency: "Good" }, "Contribution of each modality": { "Image": 0.32, Pathology: 0.28, "Text": 0.18, "Audio": 0.12, "Fusion Feature": 0.10 }, Warnings and Recommendations: [ "The imaging and pathology results are highly consistent, leading to high diagnostic reliability." Recommendation: Regular follow-up and monitoring of tumor markers. ]} Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A smart healthcare system based on multimodal data fusion, applied to a backend server, characterized by: include: The multimodal data feature extraction module is used to extract the internal information of medical image data F_img, pathological slide data F_path, text medical record data F_text, and audio medical record data F_audio, respectively, to obtain refined image features F_img', pathological features F_path', text features F_text', and audio features F_audio'; The intermodal relationship analysis module is used to fuse image features F_img' and pathological features F_path' to obtain the visual consistency feature F_vis and the first association strength label of image-pathology fusion; to fuse text features F_text' and speech features F_audio' to obtain the language consistency feature F_lang and the second association strength label of text-speech fusion; and to bridge the visual consistency feature F_vis and the language consistency feature F_lang to obtain the key association feature F_cross and the third association strength label representing the association strength between vision and language. The unified fusion module is used to perform unified fusion of image features F_img', pathological features F_path', text features F_text', speech features F_audio', visual consistency features F_vis, language consistency features F_lang, and key association features F_cross to obtain the final judgment data. Specifically, the first fusion weight of image features F_img', pathological features F_path', and visual consistency features F_vis is adjusted using a first association strength marker, the second fusion weight of text features F_text', speech features F_audio', and language consistency features F_lang is adjusted using a second association strength marker, and the first fusion weight, the second fusion weight, and the final judgment data are adjusted using a third association strength marker.

2. The smart medical system based on multimodal data fusion according to claim 1, characterized in that: In the multimodal data feature extraction module: The medical image data F_img is processed by a three-dimensional self-attention network to obtain image features F_img' including the morphology, texture and density of the lesion area; The pathological slide data F_path is processed using the Visual Transformer (VIT) model to obtain pathological features F_path', which includes abnormal patterns in the tissue microenvironment. The text medical record data F_text is processed through a medical BERT model to obtain text features F_text' containing key information from the electronic medical record; The audio case data F_audio is processed by a temporal attention network model to obtain the diagnosis-related audio features F_audio' from the audio segments containing the patient's description of symptoms.

3. The smart medical system based on multimodal data fusion according to claim 2, characterized in that: In the intermodal relationship analysis module, the fusion of image feature F_img' and pathological feature F_path' to obtain the visual consistency feature F_vis of image-pathology fusion and the first association strength marker includes: Image features F_img' and pathological features F_path' are input into a CNN semantic network containing four semantic heads to obtain image semantic img_semantic and pathological semantic path_semantic; the four semantic heads correspond to morphological features, density / texture features, diagnostic tendency and severity, respectively; The cosine similarity of the corresponding dimensions in the image semantic (img_semantic) and pathological semantic (path_semantic) representing morphological features, density / texture features, diagnostic tendency, and severity is calculated respectively to obtain the first consistency of morphological features, the second consistency of density / texture features, the third consistency of diagnostic tendency, and the fourth consistency of severity. The visual consistency feature F_vis is obtained by splicing, fusing and compressing the first consistency, second consistency, third consistency, fourth consistency, image semantic (img_semantic), and pathological semantic (path_semantic); the first association strength label is obtained by weighted averaging of the first consistency, second consistency, third consistency and fourth consistency.

4. The smart medical system based on multimodal data fusion according to claim 3, characterized in that: In the intermodal relation analysis module, the fusion of text feature F_text' and speech feature F_audio' to obtain text-speech fusion language consistency feature F_lang and second association strength label includes: Text features F_text' include text symptom features text_symptoms and text emotion features text_emotion; speech features F_audio' include speech symptom features audio_symptoms and speech emotion features audio_emotion; The cosine similarity between text symptom features (text_symptoms) and audio symptom features (audio_symptoms) is calculated to obtain the fifth consistency of symptom description; the cosine similarity between text emotion features (text_emotion) and audio emotion features (audio_emotion) is calculated to obtain the sixth consistency of emotion expression; and the keyword matching degree between text feature F_text' and audio feature F_audio' is calculated to obtain the seventh consistency of keywords. The language consistency feature F_lang is obtained by concatenating, fusing, and compressing the fifth consistency, sixth consistency, seventh consistency, text symptom features, text emotion features, audio symptom features, and audio emotion features; the second association strength label is obtained by weighted averaging of the fifth consistency, sixth consistency, and seventh consistency.

5. The smart medical system based on multimodal data fusion according to claim 4, characterized in that: The process of bridging the visual consistency feature F_vis and the linguistic consistency feature F_lang to obtain the key association feature F_cross representing the strength of the association between vision and language, as well as a third association strength marker, includes: The first association value is obtained by calculating the cosine similarity of the diagnostic tendency part in the visual consistency feature F_vis and the concatenated part of the text symptom feature text_symptoms and the audio symptom feature audio_symptoms in the language consistency feature F_lang. The second association value is obtained by calculating the degree of matching between the severity component in the visual consistency feature F_vis and the text emotion feature text_emotion and the audio emotion feature audio_emotion representing the pain in the language consistency feature F_lang. The first association value, the second association value, the diagnostic tendency portion of the visual consistency feature F_vis, the concatenated portion of the text symptom features and the audio symptom features of the language consistency feature F_lang, the severity portion of the visual consistency feature F_vis, and the concatenated portion of the text emotion feature and the audio emotion feature of the language consistency feature F_lang representing pain are concatenated, fused, and compressed to obtain the key association feature F_cross; the lowest value between the first association value and the second association value is used as the third association strength label.

6. The smart medical system based on multimodal data fusion according to claim 5, characterized in that: In the unified fusion module, adjusting the first fusion weights of image feature F_img', pathological feature F_path', and visual consistency feature F_vis in the unified fusion process using the first association strength label includes: The image baseline weight of image feature F_img' is set to 0.3, the pathological baseline weight of pathological feature F_path' is set to 0.3, and the visual consistency feature baseline weight of visual consistency feature F_vis is set to 0.

4. When the first correlation strength label is > 0.8, it means that the image feature F_img' and the pathological feature F_path' are highly consistent. At this time, the basic weight of the image and the basic weight of the pathology are both multiplied by 1.2, while the basic weight of the visual consistency feature is multiplied by 0.

8. When the first correlation strength label is < 0.4, it means that the image feature F_img' and the pathological feature F_path' have low consistency. At this time, the basic weight of the image and the basic weight of the pathology are both multiplied by 0.8, while the basic weight of the visual consistency feature is multiplied by 1.2 to emphasize contradictory features. The processed image weights, pathological weights, and visual consistency feature weights are normalized to obtain the first fusion weight.

7. The smart medical system based on multimodal data fusion according to claim 6, characterized in that: In the unified fusion module, adjusting the second fusion weights of text feature F_text', speech feature F_audio', and language consistency feature F_lang in the unified fusion process using the second association strength marker includes: The base weight of text feature F_text' is set to 0.3, the base weight of speech feature F_audio' is set to 0.3, and the base weight of language consistency feature F_lang is set to 0.

4. When the second association strength label is > 0.8, it means that the text feature F_text' and the speech feature F_audio' are highly consistent. At this time, the basic weights of the text and speech features are multiplied by 1.2, while the basic weights of the language consistency features are multiplied by 0.

8. When the second association strength label is < 0.4, it means that the text feature F_text' and the speech feature F_audio' have low consistency. At this time, the basic weights of text and speech are multiplied by 0.8, while the basic weight of language consistency feature is multiplied by 1.5 to emphasize contradictory features. The processed text weights, speech weights, and language consistency feature weights are normalized to obtain the second fusion weight.

8. The smart medical system based on multimodal data fusion according to claim 7, characterized in that: In the unified fusion module, the adjustment of the first fusion weight, the second fusion weight, and the final judgment data using the third association strength marker includes: If the third association strength label is < 0.3, it indicates that the visual and linguistic association is weak. In this case, the interaction between the two sets of fusion weights is reduced, that is, the visual evidence dependence is increased and the linguistic evidence weight is reduced: the first fusion weight is increased by 1.5, and the second fusion weight is increased by 0.

7. If the third association strength label is > 0.7, it indicates that vision and language are strongly associated. In this case, the two sets of fusion are promoted: the first fusion weight * 0.9 and the second fusion weight * 0.

9.

9. The smart medical system based on multimodal data fusion according to claim 8, characterized in that: In the unified fusion module, the unified fusion of image features F_img', pathological features F_path', text features F_text', speech features F_audio', visual consistency features F_vis, language consistency features F_lang, and key association features F_cross includes: The visual group F_visual_group, which includes image features F_img', pathological features F_path', and visual consistency features F_vis, is fused using the first fusion weight; the language group F_language_group, which includes text features F_text', speech features F_audio', and language consistency features F_lang, is fused using the second fusion weight. The key association feature F_cross and the first attention F_visual_group are calculated using softmax, which represents the part of the visual group F_visual_group that the key association feature F_cross pays attention to; and the visual group F_visual_group and the first attention F_visual_group are multiplied to obtain the attention-weighted cross-group visual feature F_visual_selected. The key association feature F_cross and the second attention F_language group are calculated using softmax, which represents the part of the language group F_language group that the key association feature F_cross pays attention to; and the language group F_language group and the second attention F_language group are multiplied to obtain the attention-weighted cross-group language feature F_language_selected. When the third association strength label is > 0.5, the association is considered strong and deep fusion is performed. The weights of cross-group visual features F_visual_selected, cross-group language features F_language_selected, and key association features F_cross are set to 0.4, and then unified fusion is performed to obtain the fused feature F_fused. Otherwise, the association is considered weak and cautious fusion is performed. The weights of cross-group visual features F_visual_selected, cross-group language features F_language_selected, and key association features F_cross are set to 0.1, and then unified fusion is performed to obtain the fused feature F_fused.

10. The smart medical system based on multimodal data fusion according to claim 9, characterized in that: In the unified fusion module, the final judgment data includes: The fused feature F_fused, the key association feature F_cross, the first association strength label, the second association strength label, and the third association strength label are concatenated to obtain the final feature representation final_representation; The final feature representation (final_representation) is input into the classification head to generate the judgment result.

Images