A lightweight multi-modal data fusion and intelligent completion method and system for primary medical care
By employing a lightweight multimodal data fusion and intelligent completion method, and utilizing personalized OCR, contextual ASR, and knowledge graphs, the heterogeneity and low quality of primary healthcare data have been addressed. This has enabled efficient and reliable data conversion and completion, thereby improving the accuracy and efficiency of primary healthcare services.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI DIANJI UNIV
- Filing Date
- 2026-02-28
- Publication Date
- 2026-06-05
AI Technical Summary
The heterogeneity and low quality of primary healthcare data make it difficult for existing technologies to effectively handle handwritten medical records, oral accounts in mixed dialects, and POCT thermal paper outputs. Furthermore, the lack of targeted optimization means that models cannot adapt to changes in recording habits across different regions and among individual doctors, resulting in low recognition accuracy and unreliable completion results.
A lightweight multimodal data fusion method is adopted, which recognizes handwritten and speech data through personalized OCR and contextual ASR, combines knowledge graph and federated learning for entity alignment and completion, uses individual patient characteristics to calibrate candidate values, and dynamically optimizes the model to adapt to local changes.
It improves the accuracy of handwriting and speech recognition, generates high-quality, trustworthy structured knowledge, enhances the precision and efficiency of primary healthcare services, and ensures the system remains adaptable and effective in the long term.
Smart Images
Figure CN122153246A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical data technology, specifically to a lightweight multimodal data fusion and intelligent completion method and system for primary healthcare. Background Technology
[0002] Currently, the core challenge facing the digitization of primary healthcare data lies in the extreme heterogeneity and low quality of data sources. Existing technical solutions mostly focus on single modalities (such as plain text electronic medical records or independent imaging systems) or use general OCR / ASR / NLP models for processing, lacking targeted optimization for data formats unique to primary healthcare (such as handwritten medical records containing a large number of personal abbreviations and symbols, oral complaints mixed with dialects and professional terminology, and POCT thermal paper outputs with vastly different formats). In addition, existing data completion technologies mostly rely on training with complete datasets, which cannot effectively cope with the reality of high missing data and low consistency in primary healthcare data, and the models are usually static, making it difficult to adapt to the constantly changing recording habits of different regions and individual doctors. Summary of the Invention
[0003] This invention provides a lightweight multimodal data fusion and intelligent completion method and system for primary healthcare to solve the above-mentioned technical problems.
[0004] In a first aspect, the present invention provides a lightweight multimodal data fusion and intelligent completion method for primary healthcare, the method comprising:
[0005] Acquire doctors' handwritten and voice data;
[0006] The handwritten data is recognized using a preset OCR model to obtain the first text; and the speech data is recognized using a preset ASR model based on a medical domain terminology database to obtain the second text; wherein the preset OCR model is obtained by training based on the doctor's style latent vector.
[0007] Entity recognition is performed on the first text and the second text respectively, and the recognized first entity and second entity are spatiotemporally aligned. The aligned entity is used as a query graph, and the query graph is matched with a preset knowledge graph for subgraph matching.
[0008] When the matching result indicates that an entity is missing, a path connected to the query graph is obtained from the knowledge graph, and multiple candidate values for the missing entity are obtained based on the path. The multiple candidate values are calibrated using a preset federated model and combined with the patient's individual characteristics, and the calibration result is used as the completion data.
[0009] In some embodiments of the present invention, the step of recognizing the handwritten data using a preset OCR model to obtain the first text includes:
[0010] Extract writing stroke order pressure information or local features of handwriting images from the handwritten data;
[0011] A lightweight steganalysis network is used to learn new style latent vectors from the writing stroke pressure information or local features of the handwriting image;
[0012] The handwritten data is recognized using a preset OCR model to obtain a first recognition result, which includes multiple characters and their corresponding confidence levels.
[0013] If the confidence level is lower than the threshold, a replacement character is determined based on the context information of the target character and the new style latent vector. The target character is then modified to the replacement character to obtain the final first text, where the target character is a character with a confidence level lower than the threshold.
[0014] In some embodiments of the present invention, the preset ASR model includes a dialect feature matching layer and a feature extraction layer; the medical-based domain terminology database uses the preset ASR model to recognize the speech data and obtain second text, including:
[0015] Based on the medical domain terminology database, a tree-structured domain dictionary is constructed. The domain dictionary includes a root node and child nodes. The root node consists of standard medical terms, and the child nodes consist of colloquial expressions and phoneme mapping rules corresponding to various local dialects.
[0016] The acoustic features are extracted from the speech data using the feature extraction layer.
[0017] Based on the patient's current department information, the expected term vector is obtained by matching from the domain terminology database; the acoustic features and the expected term vector are input into the dialect feature matching layer to obtain the second text that matches the dialect.
[0018] In some embodiments of the present invention, the step of using a preset federated model to calibrate the multiple candidate values in conjunction with the patient's individual characteristics, and using the calibration result as supplementary data, includes:
[0019] The individual features and the multiple candidate values are input into the preset federated model to obtain the adjustment result, which includes the adjustment amount and selection probability of each candidate value.
[0020] The candidate values are adjusted based on the adjustment amount, and the adjusted values are weighted and aggregated using the selection probability. The aggregated result is then used as the calibration result.
[0021] In some embodiments of the present invention, the preset federation model is obtained in the following manner:
[0022] Obtain the model parameters of the teacher model from the central server;
[0023] Using the model parameters of the teacher model and local data, the local student model is trained to obtain the model parameters of the student model.
[0024] The model parameters of the student model are uploaded to the central server, so that the central server can aggregate the model parameters of the student models uploaded by various institutions, update the teacher model using the aggregated parameters, and use the updated teacher model as the federated model.
[0025] In some embodiments of the present invention, the method further includes:
[0026] Obtain the doctor's suggestions on the completed data, and use the suggestions and the completed data to train the local residual network;
[0027] The output of the trained residual network is fused with the completed data to obtain the final completed data.
[0028] In some embodiments of the present invention, the preset OCR model is obtained in the following manner:
[0029] Obtain handwritten samples from doctors;
[0030] A style latent vector is learned from the handwritten samples using an adversarial network, and the handwritten samples are enhanced based on the style latent vector;
[0031] The general OCR model was trained using enhanced handwritten samples to obtain a personalized OCR model for each doctor.
[0032] Secondly, the present invention also provides a lightweight multimodal data fusion and intelligent completion system for primary healthcare, the system comprising:
[0033] The data acquisition module is used to acquire doctors' handwritten and voice data;
[0034] The recognition module is used to recognize the handwritten data using a preset OCR model to obtain the first text; and to recognize the speech data using a preset ASR model based on a medical domain terminology library to obtain the second text; wherein the preset OCR model is obtained by training based on the doctor's style latent vector.
[0035] The matching module is used to perform entity recognition on the first text and the second text respectively, and to perform spatiotemporal alignment on the recognized first entity and the second entity, and to use the aligned entity as a query graph, and to perform subgraph matching on the query graph with a preset knowledge graph;
[0036] The completion module is used to query the knowledge graph to obtain the path connected to the query graph when the matching result indicates that the entity is missing, and obtain multiple candidate values for the missing entity based on the path; using a preset federated model, the multiple candidate values are calibrated in combination with the individual characteristics of the patient, and the calibration result is used as completion data.
[0037] In the lightweight multimodal data fusion and intelligent completion method and system for primary healthcare provided by this invention, adaptive perception technology (personalized OCR, contextual ASR) is used to complete the high-quality conversion from non-standard raw data to preliminary structured information, solving the problems of "inability to integrate" or "incorrect data." Secondly, knowledge-driven semantic technology (knowledge graph alignment and reasoning) integrates fragmented information into clinically meaningful patient profiles, solving the problems of "incomprehensibility" and "inability to connect information." Furthermore, through two-stage intelligent completion technology, medically reasonable and individualized information is generated for missing information, solving the problem of "incompleteness." Finally, through a dual-modal collaborative optimization framework, the entire system has the ability to continuously learn and evolve, ensuring the long-term effectiveness and adaptability of the solution and solving the problem of "becoming unusable after prolonged use." Therefore, the lightweight multimodal data fusion and intelligent completion method for primary healthcare provided by this invention can automatically transform low-quality, non-standard multi-dimensional data into high-quality, high-completeness, computable, and trustworthy structured knowledge in resource-constrained primary healthcare environments, ultimately achieving the core goal of improving the accuracy, efficiency, and data value of primary healthcare services. Attached Figure Description
[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0039] Figure 1 This is a flowchart illustrating the lightweight multimodal data fusion and intelligent completion method for primary healthcare provided in this embodiment of the invention. Detailed Implementation
[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0041] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.
[0042] "A and / or B" includes the following three combinations: A only, B only, and a combination of A and B.
[0043] The use of "applies to" or "configured to" in this invention implies an open and inclusive language, which does not exclude the applicability to or configuration to devices performing additional tasks or steps. Additionally, the use of "based on" implies openness and inclusivity, because processes, steps, calculations, or other actions "based on" one or more conditions or values may in practice be based on additional conditions or values beyond those conditions.
[0044] In this invention, the term "exemplary" is used to mean "serving as an example, illustration, or description." Any embodiment described as "exemplary" in this invention is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed herein.
[0045] Currently, general-purpose models exhibit low robustness to data noise specific to grassroots levels (such as paper creases, illegible handwriting, background interference, and colloquial expressions), resulting in a sharp drop in recognition accuracy in real-world scenarios. Furthermore, existing technologies can only achieve character or speech transcription, failing to deeply understand the semantic relationships between entities in medical records (such as the spatiotemporal correlation between symptoms and signs, and the correspondence between drugs and usage), thus unable to support high-quality structured data processing. Additionally, missing value handling often employs mean imputation or simple interpolation, lacking medical knowledge guidance, leading to low clinical reliability of the completed results. Moreover, the parameters of the model are fixed after system deployment, unable to continuously self-optimize with local data accumulation and changes in recording habits, resulting in long-term service performance degradation.
[0046] To address the aforementioned problems, embodiments of the present invention provide a lightweight multimodal data fusion and intelligent completion method and system for primary healthcare. The following description, in conjunction with the accompanying drawings, details the lightweight multimodal data fusion and intelligent completion method and system for primary healthcare provided by the embodiments of the present invention.
[0047] like Figure 1 As shown in the figure, this invention provides a lightweight multimodal data fusion and intelligent completion method for primary healthcare, which includes the following steps:
[0048] S101 acquires the doctor's handwritten data and voice data.
[0049] The handwritten data includes handwritten medical record images. These images can be uploaded by patients guided by an auxiliary system. The system provides intelligent guidance during the upload process to ensure that the image angle, clarity, and lighting conditions meet requirements. The preprocessing workflow for handwritten medical record images includes: tilt correction: tilt correction of the image based on the Hough transform algorithm; binarization processing: image processing using adaptive threshold binarization technology; noise removal: cleaning noise from the image using a noise removal method based on connected component analysis, etc.
[0050] Speech data was acquired using a directional microphone at a sampling rate of 16kHz and a quantization precision of 16 bits. The preprocessing workflow for the speech data included: speech activity detection: using the WebRTC VAD algorithm to detect speech activity and ensure the extraction of valid speech signals; noise reduction: using noise spectrum estimation and spectral subtraction for speech noise reduction processing.
[0051] In addition, POCT receipt images are also acquired. These images are captured under fixed lighting conditions using a customized fixture and a mobile phone camera, ensuring consistent and stable image quality. The preprocessing workflow for POCT receipt images includes: perspective transformation correction: perspective transformation correction is applied to the captured images; region of interest segmentation: regions of interest are identified in the image based on color clustering technology. It is understandable that POCT receipt image recognition can be achieved using a general OCR model, which will not be elaborated upon here.
[0052] S102, the handwritten data is recognized using a preset OCR model to obtain the first text; and the speech data is recognized using a preset ASR model based on a medical domain terminology database to obtain the second text.
[0053] The preset OCR model is obtained by training based on the doctor's style latent vectors.
[0054] S103, perform entity recognition on the first text and the second text respectively, and perform spatiotemporal alignment on the recognized first entity and second entity, use the aligned entity as a query graph, and perform subgraph matching on the query graph with a preset knowledge graph.
[0055] Schematic illustration: the first text output by the OCR model and the second text output by the ASR model are respectively processed by a shared, lightweight BiLSTM-CRF model for named entity recognition. The extracted entities are linked with the core layer standard nodes in the pre-defined knowledge graph to ensure that the recognized entities correspond to the standard entities in the knowledge graph.
[0056] In practical applications, a session ID is created for each medical visit to ensure that entities in different medical records can be distinguished. All extracted entities are timestamped, with audio data tagged by segment and text data by line, while the source modality (OCR or ASR) of the entities is also recorded. The rule-based engine groups entities that are close in time and semantically related. For example, the audio phrase "started having a headache yesterday" and the text phrase "physical examination: T 38.2℃" are associated with the same "onset event" node.
[0057] The aligned entity data is formalized into a query graph Q, and then the query graph Q is continuously matched with the preset knowledge graph for subgraph matching. The completion process is triggered only when it is found that an entity in the query graph Q (such as "blood routine") is missing an attribute that should normally be associated with in the knowledge graph (such as "white blood cell count").
[0058] S104, when the matching result indicates that an entity is missing, a path connected to the query graph is obtained from the knowledge graph, and multiple candidate values of the missing entity are obtained based on the path; the multiple candidate values are calibrated using a preset federated model and combined with the patient's individual characteristics, and the calibration result is used as the completion data.
[0059] The knowledge graph includes core entities such as diseases, symptoms, drugs, examinations, and signs extracted from authoritative medical guidelines, as well as stable logical relationships between them (such as "bacterial pneumonia", "may lead to", and "elevated white blood cell count").
[0060] In some examples, when missing data is detected (such as missing "complete blood count - neutrophil percentage"), the system completes the data through the following process:
[0061] The system uses the patient's current structured data (such as "Diagnosis: Acute tonsillitis", "Symptoms: Fever, Sore throat") as query conditions, searches for paths strongly associated with missing entities in the knowledge graph, and determines the common value ranges of the missing entities.
[0062] The system combines the medical logic of the static layer with the local experience of the dynamic layer to generate candidate completions. The medical logic of the static layer includes general medical knowledge (such as "acute infections are usually accompanied by elevated neutrophils"), while the experience of the dynamic layer is based on local patient data (such as "the percentage of neutrophils in patients with recent acute tonsillitis in this community is usually between 75% and 85%"). The dynamic layer is an evolving graph structure mined from actual clinical data from various primary care institutions through federated learning. This graph structure records statistical knowledge such as the co-occurrence frequency, temporal relationships, and efficacy correlations between entities in local practice (e.g., in this region, the probability of symptom A and examination B occurring simultaneously in elderly patients exceeds 80%).
[0063] Based on this information, the system generates one or more candidate hypotheses with confidence levels and medical interpretations.
[0064] The candidate hypothesis is fed as prior input into a regression / classification neural network trained using federated learning. This network integrates the patient's individual characteristics (such as age and medical history) with other completed examination results to output a more accurate completed numerical value or probability distribution.
[0065] The lightweight multimodal data fusion and intelligent completion method for primary healthcare provided in this invention successfully overcomes the limitations of "general recognition" and achieves "exclusive adaptation" through personalized learning and localized adaptation. Specifically, the system extracts and learns the latent style vectors of individual doctors to dynamically optimize the OCR model, thereby effectively improving the accuracy of handwritten character recognition. Simultaneously, by loading a localized terminology database and contextual information, modulating the ASR decoding process, and modeling and filtering major noise sources, the accuracy of speech recognition is significantly improved. This method increases the accuracy of character and speech recognition from 70% of the general model to over 95%, fundamentally ensuring the reliability of the original data conversion.
[0066] To further enhance data understanding and association capabilities, this invention introduces a medical knowledge graph as the core of understanding and association. Based on the identified entities, the system maps these entities to knowledge graph nodes and performs cross-modal and cross-field semantic alignment according to the relationships in the graph, forming a structured temporary patient subgraph. This process transforms the output data from discrete text into networked information rich in semantic relationships, providing a computable logical foundation for deep analysis and completion.
[0067] The completion process underwent two stages of optimization. The first stage involved generating medically logical candidate values and explanations using a knowledge graph. The second stage utilized a federated learning model to refine and calibrate the candidate values, taking into account individual patient characteristics. This approach ensures that the completion results not only meet medical standards but also allow for personalized adjustments based on each patient's individual circumstances, ultimately guaranteeing the interpretability and accuracy of the completion results.
[0068] Furthermore, this invention utilizes adaptive perception technologies (such as personalized OCR and contextual ASR) to achieve high-quality transformation from non-standard raw data to preliminary structured information, successfully solving the problems of "inability to access" or "inputting incorrect information." Through knowledge graph-driven semantic technology, the system integrates fragmented information into clinically meaningful patient profiles, resolving the issues of "incomprehensibility" and "inability to connect information."
[0069] During the completion process, the two-stage intelligent completion technology ensures the medical rationality and individual consistency of the completed data through knowledge reasoning and personalized adjustments based on patient characteristics. Finally, combined with a dual-mode collaborative optimization framework, the system achieves continuous learning and evolution capabilities, avoiding the problem of "becoming unusable over time" and ensuring the long-term effectiveness and adaptability of the solution.
[0070] This series of technological innovations ensures that low-quality, non-standard, multimodal data in primary healthcare environments can be automatically transformed into high-quality, highly complete, computable, and trustworthy structured knowledge. Ultimately, the system achieves its core goal of improving the accuracy, efficiency, and data value of primary healthcare services.
[0071] In some embodiments of the present invention, in daily use, the pressure information of a doctor's writing stroke order (via a pressure-sensitive digital input device) or local features of handwriting images are captured in real time. A lightweight steganalysis network dynamically updates the doctor's style latent vector. When ambiguity arises in the recognition, not only is error correction based on a context-based NLP model, but the updated latent vector is also used to perform "style calibration" on the last layer of features of the OCR model, thereby improving the recognition rate of abbreviations or cursive writing unique to a specific doctor. Specifically, the steps include the following:
[0072] Extract writing stroke order pressure information or local features of handwriting images from the handwritten data.
[0073] A new style latent vector is learned from the writing stroke pressure information or local features of the handwriting image using a lightweight steganalysis network.
[0074] The handwritten data is recognized using a preset OCR model to obtain a first recognition result, which includes multiple characters and their corresponding confidence levels.
[0075] If the confidence level is lower than the threshold, a replacement character is determined based on the context information of the target character and the new style latent vector. The target character is then modified to the replacement character to obtain the final first text, where the target character is a character with a confidence level lower than the threshold.
[0076] In some examples, when doctors are hired, the system requires them to write 50 sentences containing high-frequency medical words. By analyzing the handwritten data of these sentences, the system extracts the doctor's style latent vector Z, which can represent the doctor's unique writing style (including features such as stroke order and stroke pressure).
[0077] Using the style latent vector Z, the system controls a pre-trained handwriting generator to synthesize 2000 personalized character images. These synthesized images are then fine-tuned with a general OCR base model to obtain the doctor's initial personal OCR model M_d. This model is able to accurately recognize the doctor's personalized handwriting style.
[0078] Whenever a doctor writes an electronic prescription on a tablet, the system records the doctor's handwriting temporal coordinates and writing pressure information S in real time. Using a lightweight steganalysis network F, these temporal coordinates and pressure sequences are encoded into a style increment ΔZ. Subsequently, the system updates the doctor's style latent vector: Z' = Z + ηΔZ, where η is the learning rate. After each writing session, the system adds the newly written image and its recognition result (after doctor confirmation) as positive samples to a circular buffer for further training and model optimization.
[0079] When processing new medical record images, the system first uses the doctor's initial personal OCR model M_d to recognize the images. For characters with a confidence level below the threshold τ, the system performs error correction based on the following information:
[0080] Context information: The context of the target character is encoded into a context vector C using a lightweight BERT model;
[0081] Style latent vector: Combine the current doctor's style vector Z' to calculate the candidate character set of the target character, and select the character with the highest probability from the candidate set according to the joint probability P(char|C, Z') for replacement.
[0082] Using this method, the system can effectively replace low-confidence characters in the recognition results by utilizing contextual information and personalized style vectors, and finally generate the final first text.
[0083] In some embodiments of the present invention, the preset ASR model includes a dialect feature matching layer and a feature extraction layer; the medical-based domain terminology database uses the preset ASR model to recognize the speech data and obtain second text, including:
[0084] Based on the aforementioned medical domain terminology database, a tree-structured domain dictionary is constructed. This dictionary includes a root node and child nodes. The root node represents standard medical terms, while the child nodes represent colloquial expressions and phoneme mapping rules corresponding to various local dialects. For example, the dialect word "guo min" is mapped to the standard term "allergy" and its pinyin "guo min" using an acoustic model. It is understood that the domain terminology database allows administrators at primary healthcare institutions to dynamically add or delete entries based on local conditions.
[0085] Acoustic features are extracted from the speech data using the feature extraction layer.
[0086] Based on the patient's current department information, the expected term vector is obtained by matching from the domain terminology database. The acoustic features and the expected term vector are then input into the dialect feature matching layer to obtain the second text that matches the dialect. In other words, the dialect feature matching layer is used to modulate the acoustic features and the expected term vector, making the ASR model more inclined to output results that conform to both the acoustic signal and the expected dialect terminology in the current medical scenario during decoding.
[0087] In some examples, the ASR (Automatic Speech Recognition) model is specifically designed with a dialect feature matching layer and a feature extraction layer to adapt to the diverse dialects and colloquial expressions in primary healthcare environments. Specifically, the ASR model includes a feature extraction layer and a dialect feature matching layer.
[0088] The feature extraction layer is responsible for extracting acoustic features from the input speech data, such as spectral features and Mel-frequency cepstral coefficients (MFCC). These features capture key information in the speech signal and provide a foundation for subsequent recognition.
[0089] The dialect feature matching layer receives acoustic feature vectors from the feature extraction layer and matches them with expected term vectors from a medical terminology database. Through dialect feature matching, the ASR model can more accurately identify speech data containing dialect characteristics. Specifically, it modulates the acoustic features with the expected term vectors, making the decoding process more inclined to output results that conform to the current medical scenario and dialect characteristics.
[0090] The input to the dialect feature matching layer is the acoustic feature vector and the expected term vector (obtained from a medical terminology database based on the patient's department information); the output is the matched text result, i.e., the identified second text.
[0091] In some embodiments of the present invention, the step of using a preset federated model to calibrate the multiple candidate values in conjunction with the patient's individual characteristics, and using the calibration result as supplementary data, includes:
[0092] The individual features and the multiple candidate values are input into the preset federated model to obtain the adjustment result, which includes the adjustment amount and selection probability of each candidate value.
[0093] The candidate values are adjusted based on the adjustment amount, and the adjusted values are weighted and aggregated using the selection probability. The aggregated result is then used as the calibration result.
[0094] To illustrate, with the missing entity E as the core, the system performs one-hop or two-hop queries in the knowledge graph to find all paths connected to existing entities in the query graph.
[0095] For each path, the system calculates an association score based on the relationship type (such as causal relationship, accompanying relationship, inspection items, etc.) and the dynamic layer statistical weights.
[0096] Based on the scores, the Top-K paths are selected, and candidate values or value ranges related to the missing entity E are determined, forming a candidate set C_k = { (value_i, explanation_i, score_i)}, where value_i is the candidate value, that is, the specific value or category found in the knowledge graph on the path connected to the query graph that may be used to complete the missing entity; explanation_i is the medical explanation or basis for why the candidate value is a reasonable completion, usually based on medical knowledge or local clinical experience; score_i is the association score, which represents the association strength or reasonableness score between the candidate value and the query graph and its existing entities, and is used to measure the probability that the candidate value will become the correct completion.
[0097] The patient's feature vector (such as age, gender, existing symptoms, etc.) and the encoding vector of each candidate in the candidate set C_k are input into the refined network G trained by federated learning.
[0098] The network G outputs the adjustment δ_i for each candidate value and calculates the probability p_i of the final selection.
[0099] The final completion result is a weighted aggregate value, represented as: Σ(p_i * (value_i + δ_i)), with an explanation of the highest probability candidate.
[0100] In some embodiments of the present invention, the preset federation model is obtained in the following manner:
[0101] Obtain the model parameters of the teacher model from the central server.
[0102] Using the model parameters of the teacher model and local data, the local student model is trained to obtain the model parameters of the student model.
[0103] The model parameters of the student model are uploaded to the central server, so that the central server can aggregate the model parameters of the student models uploaded by various institutions, update the teacher model using the aggregated parameters, and use the updated teacher model as the federated model.
[0104] In some examples, through a federated learning mechanism, participating institutions use local data to train models and update model parameters and transfer knowledge through teacher models on a central server. The specific process is as follows:
[0105] Each participating institution trains a complete "student model" S_A locally using its own private data. This model is trained individually based on the institution's local data, effectively adapting to the characteristics of local data.
[0106] Periodically, a stronger “teacher model” T (trained on a large amount of desensitized public data), maintained by a central server, distributes its current encrypted model parameters to various institutions via a secure multi-party computation protocol.
[0107] Each participating institution (such as institution A) trains its own student model S_A locally based on the encrypted parameters of the received teacher model T and local data. By combining the student model S_A with the local data, the student model S_A is gradually optimized, thereby obtaining a new set of model parameters.
[0108] During local training, the teacher model T generates softened labels (soft probability distributions) by making predictions on local data. These labels describe the teacher model's predictions on the local data. The softened labels are transmitted via a secure protocol to ensure that the original data does not leave the local machine.
[0109] Organization A uses softened labels generated by the teacher model and local data to train its own student model S_A, minimizing the KL divergence between the output of student model S_A and the softened labels, thereby optimizing the student model.
[0110] After training its local student model S_A, Institution A only uploads encrypted slices of the model's gradients to the central server. This method ensures data privacy and shares only updated model parameters, not the original data.
[0111] After receiving the model parameters of the student models uploaded by each institution, the central server aggregates them and updates the global "student model" S_global. The updated student model S_global serves as the "teacher model" T' for the next round and is then distributed to the participating institutions for further training. Understandably, in practical applications, participating institutions use the teacher model T' (i.e., the updated federated model) obtained from the central server for tasks such as data completion.
[0112] The lightweight multimodal data fusion and intelligent completion method for primary healthcare provided in this invention can solve the problems of data silos and commonalities by enabling participating institutions' models to continuously absorb collective experience across institutions through a federated learning mechanism, while protecting privacy. The knowledge graph of the dynamic experience layer is updated weekly through federated learning, ensuring that the system can quickly reflect local disease trends and changes in treatment habits.
[0113] In some embodiments of the present invention, the method further includes:
[0114] Obtain the doctor's suggestions on the completed data, and use the suggestions and the completed data to train the local residual network.
[0115] The output of the trained residual network is fused with the completed data to obtain the final completed data.
[0116] In some examples, by obtaining doctors' suggestions on the completed data and combining this feedback, a local personalized residual network is trained to optimize the final completed data. The specific process is as follows:
[0117] On each terminal device, a lightweight, personalized residual network is maintained on top of the main model. This network specifically learns the variance (i.e., residuals) of the local, doctor- or device-specific data distribution. All updates apply only to this residual network and do not affect the main model. This approach reduces the computational and storage overhead required for personalized updates and enables rapid deployment and rollback.
[0118] Doctors confirm or modify the system-generated completion suggestions, forming a feedback pair. This feedback pair includes input features (such as contextual information about the missing data) and the doctor's expected amount of correction to the completed data (i.e., the adjustment of the doctor's suggestions).
[0119] The collected feedback data is used to train a local residual network R. This network consists of 3 fully connected layers with very few parameters, focusing on personalized adjustments for local data.
[0120] During training, the residual network will refine and adjust the completed data based on the doctor's feedback.
[0121] The trained residual network R is fused with the output of the main model (i.e., the federated model). The new inference process is: final output = main model output + R (input features).
[0122] The residual network R is used to further refine the output of the main model to adapt to the characteristics of the local data. Specifically, feedback from doctors on the completed data (including confirmations or modification suggestions) is first collected to form feedback pairs. The local residual network R, consisting of a small number of fully connected layers, is then trained using these feedback pairs, focusing on learning personalized adjustments for the local data.
[0123] During the inference phase, the federated model first generates preliminary completed data. The residual network R receives input features related to the completed data (such as contextual information about the missing data) and outputs adjustments. The final completed data is obtained by adding the output of the federated model to the output of the residual network R.
[0124] Residual Network R refines the output of the main model by learning the unique writing or diagnostic habits of local doctors. Input features include not only feedback but also contextual information related to the completed data, enabling the residual network to gain a more comprehensive understanding of the data context and make more accurate adjustments.
[0125] In this way, the residual network R ensures that the final completed data better matches the doctor's expectations and the patient's actual situation, improving the accuracy and personalization of the completed data.
[0126] The lightweight multimodal data fusion and intelligent completion method for primary healthcare provided in this invention enables terminal models to quickly learn the unique writing or diagnostic habits of local doctors with minimal overhead, thereby achieving personalized adaptation. It possesses the characteristics of becoming more accurate and user-friendly with continued use, maintaining high precision and user engagement over the long term, and solving the problem of static model degradation. Through federated knowledge distillation, the system improves model performance by nearly 90% compared to centralized training, while ensuring that data remains locally available. Residual learning technology allows personalized adaptation to be completed on a regular tablet within 24 hours, without cloud intervention.
[0127] In some embodiments of the present invention, the preset OCR model is obtained in the following manner:
[0128] Obtain handwritten samples from doctors.
[0129] A style latent vector is learned from the handwritten samples using an adversarial network, and the handwritten samples are then augmented based on the style latent vector.
[0130] The general OCR model was trained using enhanced handwritten samples to obtain a personalized OCR model for each doctor.
[0131] In some examples, a small number of handwritten samples are collected for each doctor. A lightweight generative adversarial network is used to learn the style latent vector of the doctor's handwriting from the handwritten samples. By fine-tuning this style latent vector, enhanced training samples covering high-frequency medical characters in the doctor's handwriting style can be synthesized in batches for quickly initializing a personalized OCR model for each doctor.
[0132] In summary, the lightweight multimodal data fusion and intelligent completion method for primary healthcare provided by this invention features a lightweight design for OCR, ASR, atlas, and completion models, ensuring smooth operation on ordinary terminals at the primary care level. Knowledge transfer is achieved through encrypted exchange of model parameters and gradients, ensuring that the original data remains locally stored and is not leaked. This design significantly reduces the hardware threshold and cost for deployment at the primary care level, while fully complying with the compliance requirements of the Data Security Law and the Personal Information Protection Law for medical data processing, effectively eliminating privacy concerns regarding data sharing between institutions.
[0133] In some embodiments of the present invention, the present invention also provides a lightweight multimodal data fusion and intelligent completion system for primary healthcare, the system comprising:
[0134] The data acquisition module is used to acquire doctors' handwritten and voice data;
[0135] The recognition module is used to recognize the handwritten data using a preset OCR model to obtain the first text; and to recognize the speech data using a preset ASR model based on a medical domain terminology library to obtain the second text; wherein the preset OCR model is obtained by training based on the doctor's style latent vector.
[0136] The matching module is used to perform entity recognition on the first text and the second text respectively, and to perform spatiotemporal alignment on the recognized first entity and the second entity, and to use the aligned entity as a query graph, and to perform subgraph matching on the query graph with a preset knowledge graph;
[0137] The completion module is used to query the knowledge graph to obtain the path connected to the query graph when the matching result indicates that the entity is missing, and obtain multiple candidate values for the missing entity based on the path; using a preset federated model, the multiple candidate values are calibrated in combination with the individual characteristics of the patient, and the calibration result is used as completion data.
[0138] The lightweight multimodal data fusion and intelligent completion system for primary healthcare provided in this embodiment corresponds to the aforementioned lightweight multimodal data fusion and intelligent completion method for primary healthcare, and will not be elaborated further here.
[0139] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0140] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.
[0141] The foregoing has provided a detailed description of a lightweight multimodal data fusion and intelligent completion method and system for primary healthcare provided by embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A lightweight multimodal data fusion and intelligent completion method for primary healthcare, characterized in that, The method includes: Acquire doctors' handwritten and voice data; The handwritten data is recognized using a preset OCR model to obtain the first text; and the speech data is recognized using a preset ASR model based on a medical domain terminology database to obtain the second text; wherein the preset OCR model is obtained by training based on the doctor's style latent vector. Entity recognition is performed on the first text and the second text respectively, and the recognized first entity and second entity are spatiotemporally aligned. The aligned entity is used as a query graph, and the query graph is matched with a preset knowledge graph for subgraph matching. When the matching result indicates that an entity is missing, a path connected to the query graph is obtained from the knowledge graph, and multiple candidate values for the missing entity are obtained based on the path. The multiple candidate values are calibrated using a preset federated model and combined with the patient's individual characteristics, and the calibration result is used as the completion data.
2. The lightweight multimodal data fusion and intelligent completion method for primary healthcare as described in claim 1, characterized in that, The step of using a preset OCR model to recognize the handwritten data and obtain the first text includes: Extract writing stroke order pressure information or local features of handwriting images from the handwritten data; A lightweight steganalysis network is used to learn new style latent vectors from the writing stroke pressure information or local features of the handwriting image; The handwritten data is recognized using a preset OCR model to obtain a first recognition result, which includes multiple characters and their corresponding confidence levels. If the confidence level is lower than the threshold, a replacement character is determined based on the context information of the target character and the new style latent vector. The target character is then modified to the replacement character to obtain the final first text, where the target character is a character with a confidence level lower than the threshold.
3. The lightweight multimodal data fusion and intelligent completion method for primary healthcare as described in claim 1, characterized in that, The preset ASR model includes a dialect feature matching layer and a feature extraction layer; the medical-based domain terminology database uses the preset ASR model to recognize the speech data and obtain second text, including: Based on the medical domain terminology database, a tree-structured domain dictionary is constructed. The domain dictionary includes a root node and child nodes. The root node consists of standard medical terms, and the child nodes consist of colloquial expressions and phoneme mapping rules corresponding to various local dialects. The acoustic features are extracted from the speech data using the feature extraction layer. Based on the patient's current department information, the expected term vector is obtained by matching from the domain terminology database; the acoustic features and the expected term vector are input into the dialect feature matching layer to obtain the second text that matches the dialect.
4. The lightweight multimodal data fusion and intelligent completion method for primary healthcare as described in claim 1, characterized in that, The process of using a pre-defined federated model, combined with the patient's individual characteristics, to calibrate the multiple candidate values, and using the calibration results as supplementary data, includes: The individual features and the multiple candidate values are input into the preset federated model to obtain the adjustment result, which includes the adjustment amount and selection probability of each candidate value. The candidate values are adjusted based on the adjustment amount, and the adjusted values are weighted and aggregated using the selection probability. The aggregated result is then used as the calibration result.
5. The lightweight multimodal data fusion and intelligent completion method for primary healthcare as described in claim 1, characterized in that, The preset federation model is obtained in the following way: Obtain the model parameters of the teacher model from the central server; Using the model parameters of the teacher model and local data, the local student model is trained to obtain the model parameters of the student model. The model parameters of the student model are uploaded to the central server, so that the central server can aggregate the model parameters of the student models uploaded by various institutions, update the teacher model using the aggregated parameters, and use the updated teacher model as the federated model.
6. The lightweight multimodal data fusion and intelligent completion method for primary healthcare as described in claim 5, characterized in that, The method further includes: Obtain the doctor's suggestions on the completed data, and use the suggestions and the completed data to train the local residual network; The output of the trained residual network is fused with the completed data to obtain the final completed data.
7. The lightweight multimodal data fusion and intelligent completion method for primary healthcare as described in claim 1, characterized in that, The preset OCR model is obtained in the following way: Obtain handwritten samples from doctors; A style latent vector is learned from the handwritten samples using an adversarial network, and the handwritten samples are enhanced based on the style latent vector; The general OCR model was trained using enhanced handwritten samples to obtain a personalized OCR model for each doctor.
8. A lightweight multimodal data fusion and intelligent completion system for primary healthcare, characterized in that, The system includes: The data acquisition module is used to acquire doctors' handwritten and voice data; The recognition module is used to recognize the handwritten data using a preset OCR model to obtain the first text; and to recognize the speech data using a preset ASR model based on a medical domain terminology library to obtain the second text; wherein the preset OCR model is obtained by training based on the doctor's style latent vector. The matching module is used to perform entity recognition on the first text and the second text respectively, and to perform spatiotemporal alignment on the recognized first entity and the second entity, and to use the aligned entity as a query graph, and to perform subgraph matching on the query graph with a preset knowledge graph; The completion module is used to query the knowledge graph to obtain the path connected to the query graph when the matching result indicates that the entity is missing, and obtain multiple candidate values for the missing entity based on the path; using a preset federated model, the multiple candidate values are calibrated in combination with the individual characteristics of the patient, and the calibration result is used as completion data.