Multi-modal doctor-patient interaction clinical assistance method and system based on privacy computing

Abstract

The application discloses a multi-modal doctor-patient interaction clinical assistance method and system based on privacy calculation, and relates to the technical field of privacy calculation. The method comprises the following steps: performing cross-modal semantic association on a holographic record of diagnosis and treatment, and constructing a structured diagnosis and treatment data unit; inputting the structured diagnosis and treatment data unit into a clinical reasoning model, capturing the association and reasoning through an attention mechanism, and outputting an auxiliary diagnosis prompt; using the feedback of medical staff on the prompt as a supervision signal to drive local incremental training of the model to optimize parameters; using a privacy calculation protocol to encrypt and desensitize the parameter update amount and then send the parameter update amount to a federal learning coordination node; the coordination node aggregates multiple institutional update amounts to generate a federal aggregation model, and distributes the federal aggregation model to each participating institution for deployment and update. The technical problems of data loss in the doctor-patient interaction and clinical reasoning process, untraceable clinical thinking and difficulty in realizing multi-institutional collaborative modeling under data privacy constraints in the existing medical information system are solved, and the technical effects of improving the real-time performance and accuracy of clinical assistance are achieved.

Description

Technical Field

[0001] This invention relates to the field of privacy computing technology, specifically to a multimodal doctor-patient interaction clinical assistance method and system based on privacy computing. Background Technology

[0002] Medical institutions have made significant progress in the digitization and structuring of objective examination data such as imaging, laboratory tests, and pathology. Electronic medical record systems, medical image storage and transmission systems, and laboratory information systems are widely used in clinical practice. These systems, centered on outcome data, provide fundamental support for the recording and retrieval of diagnostic conclusions. However, in stark contrast to the highly digitized examination data, the most clinically valuable doctor-patient interaction data and the doctor's clinical reasoning process have long remained in a low-structured or even unstructured state. Patients' original descriptions of symptoms, doctors' questioning paths, the thought process behind differential diagnoses, and the decision-making logic behind examination selections are often recorded only in brief free text or post-hoc summaries, making it difficult to fully and accurately reflect the diagnosis and treatment process. This "results-oriented, process-neglecting" information model leads to the compression or loss of a large amount of original diagnostic clues and clinical cognitive information during the recording stage, making clinical thinking untraceable, communication quality unquantifiable, and medical quality control and medical teaching highly dependent on post-hoc inference. At the same time, medical data is inherently highly sensitive and privacy-sensitive. Due to legal and ethical requirements and data barriers between institutions, original diagnostic and treatment data is difficult to share directly between different medical institutions. This limits the amount of data available to a single institution, hindering the widespread application of high-quality clinical artificial intelligence. Summary of the Invention

[0003] This application provides a multimodal doctor-patient interaction clinical assistance method and system based on privacy computing. It solves the technical problems of missing data in the doctor-patient interaction and clinical reasoning process in the existing medical information system, untraceable clinical thinking, and difficulty in achieving multi-institutional collaborative modeling under data privacy constraints. It achieves the technical effect of improving the real-time performance and accuracy of clinical assistance by realizing the structured expression of doctor-patient multimodal interaction data, real-time decision-making assistance of clinical reasoning models, and continuous joint optimization across medical institutions without disclosing the original diagnosis and treatment data.

[0004] The first aspect of this application provides a multimodal doctor-patient interaction clinical assistance method based on privacy computing, the method comprising: A structured diagnostic data unit is constructed by performing cross-modal semantic feature association on the holographic record of the diagnosis and treatment scenario. This structured diagnostic data unit is then input into a clinical reasoning model deployed in the first participating institution. After cross-modal association capture based on an attention mechanism, clinical task reasoning is performed, and auxiliary diagnostic prompts are output. Feedback from medical staff to these auxiliary diagnostic prompts is used as a supervisory signal, combined with the structured diagnostic data unit, to drive the clinical reasoning model to optimize model parameters based on local incremental training. The first participating institution uses a privacy computing protocol to encrypt and de-identify the model parameter updates of the clinical reasoning model before sending them to the federated learning coordination node. The federated learning coordination node receives and aggregates multiple model parameter updates from multiple participating institutions, performs secure aggregation computation, and generates a federated aggregation model. The federated learning coordination node then distributes the federated aggregation model to the multiple participating institutions for model deployment and updates.

[0005] A second aspect of this application provides a multimodal doctor-patient interaction clinical assistance system based on privacy computing, the system comprising: Feature Association Component: Performs cross-modal semantic feature association on the holographic record of the diagnosis and treatment scenario to construct a structured diagnosis and treatment data unit; Clinical Task Reasoning Component: Inputs the structured diagnosis and treatment data unit into the clinical reasoning model deployed in the first participating institution, performs clinical task reasoning after capturing cross-modal associations based on an attention mechanism, and outputs auxiliary diagnostic prompts; Model Parameter Optimization Component: Uses the feedback operations of medical staff on the auxiliary diagnostic prompts as a supervision signal, and combines the structured diagnosis and treatment data unit to drive the clinical reasoning model to optimize model parameters based on local incremental training; Encryption and Desensitization Component: The first participating institution uses a privacy computing protocol to encrypt and desensitize the model parameter updates of the clinical reasoning model before sending them to the federated learning coordination node; Secure Aggregation Component: The federated learning coordination node receives and aggregates multiple model parameter updates from multiple participating institutions, performs secure aggregation calculations, and generates a federated aggregated model; Model Deployment Component: The federated learning coordination node distributes the federated aggregated model to the multiple participating institutions for model deployment and updates.

[0006] One or more technical solutions provided in this application have at least the following technical effects or advantages: First, multimodal semantic association analysis is performed on the holographic medical records generated from doctor-patient interactions, integrating information such as voice, text, and behavior into unified structured medical data. Then, this structured data is input into a clinical reasoning model deployed locally at participating institutions. An attention mechanism is used to uncover the relationships between different modalities, completing clinical reasoning and generating auxiliary diagnostic suggestions. Subsequently, feedback from doctors regarding the adoption, correction, or rejection of these suggestions is used as a supervisory signal to perform local incremental optimization of the model in conjunction with corresponding medical data. After local training, participating institutions encrypt and anonymize the model parameter updates according to a privacy computing protocol, uploading only the parameter update information to the federated learning coordination node. The federated learning coordination node then securely aggregates parameter updates from multiple institutions to form a unified federated model. Finally, the updated federated model is distributed to all participating institutions, enabling continuous iteration and collaborative optimization of the model. Attached Figure Description

[0007] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0008] Figure 1 A schematic diagram of the process of a multimodal doctor-patient interaction clinical assistance method based on privacy computing provided in this application embodiment.

[0009] Figure 2 A schematic diagram of the structure of a multimodal doctor-patient interaction clinical support system based on privacy computing provided in this application embodiment.

[0010] Figure labeling: Feature association component 11, Clinical task reasoning component 12, Model parameter optimization component 13, Encryption and desensitization component 14, Security aggregation component 15, Model deployment component 16. Detailed Implementation

[0011] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0012] Example 1, as Figure 1 As shown, this application provides a clinical assistance method for multimodal doctor-patient interaction based on privacy computing, wherein the method includes: Cross-modal semantic feature association is performed on the holographic record of the diagnosis and treatment scenario to construct structured diagnosis and treatment data units.

[0013] In this embodiment, the system uses an integrated perception engine to perform holographic acquisition of the doctor-patient interaction process, obtaining a holographic record of the diagnosis and treatment scenario, including dialogue voice stream, text stream, and behavioral event stream. Subsequently, the system performs timestamp standardization and alignment of the three types of stream data in the holographic record of the diagnosis and treatment scenario using a unified clock as a reference. Then, it performs text conversion and medical entity recognition on the aligned voice stream, extracts medical entities, and further performs relation extraction and semantic embedding on the text stream, transforming entities and their relations into structured semantic representations. At the same time, the behavioral event stream is encoded, representing each operation event as a temporal action sequence containing event type, object identifier, parameter value, and occurrence time, which is used to reflect the doctor's decision-making actions and information retrieval paths at different stages. Subsequently, the medical entity recognition results of the dialogue speech stream, the structured semantic representation of the text stream, and the temporal action sequence of the behavioral event stream are vectorized and mapped respectively. A cross-modal attention mechanism is introduced in this vector space to calculate the dynamic association weights between elements of different modalities, generating structured diagnosis and treatment data units that can be directly used for subsequent clinical reasoning and training. Each structured diagnosis and treatment data unit contains an event-level timestamp, event type label, entity and attribute fields, cross-modal fusion feature vector and its attention weight, so as to ensure that the diagnosis and treatment process information retains the original context, temporal relationship and traceability after structuring, thereby realizing the cross-modal semantic feature association and structured expression of the holographic record of the diagnosis and treatment scene.

[0014] Furthermore, the holographic recording of the diagnosis and treatment scene is collected by an integrated perception engine, which includes a voice recording module, a text extraction module, and a behavior capture module. The holographic recording of the diagnosis and treatment scene includes the voice stream recorded by the voice recording module, the text stream captured by the text extraction module, and the behavior event stream collected by the behavior capture module.

[0015] Preferably, in actual diagnosis and treatment, an integrated perception engine is deployed in the examination room environment and on the doctor's work terminal to continuously and synchronously collect data on the entire process of doctor-patient interaction, forming a holographic record of the diagnosis and treatment scenario. This integrated perception engine consists of a voice recording module, a text extraction module, and a behavior capture module working together, and operating in coordination with a unified time reference. The voice recording module uses directional or array microphones placed in the examination room to collect real-time dialogue between the doctor and patient during the consultation, follow-up questions, explanations, and communication processes, generating continuous voice stream data and adding a high-precision timestamp to each voice segment to reflect the true sequence and rhythm of the doctor-patient interaction; the text extraction module... Deployed at the doctor's workstation or interfaced with electronic medical record systems and examination and testing systems, the system automatically captures text information displayed, entered, or retrieved during the diagnosis and treatment process. This text stream includes at least the chief complaint and medical history, previous medical records, examination and testing results, and edited medical orders, and simultaneously records the time nodes of text generation or retrieval. The behavior capture module listens to and records human-computer interaction events on the doctor's terminal, collecting the flow of operational behavior events generated by the doctor during the diagnosis and treatment process, such as mouse clicks, keyboard input, page switching, function calls, examination requests, and medical order issuance. It records the corresponding event type, operation object identifier, and occurrence time for each operation event. During the collection process, the voice stream, text stream, and behavior event stream are marked and cached within the integrated perception engine according to a unified timestamp system, thereby ensuring that the data of different modalities are alignable and correlated. Through the above methods, the system can fully acquire a holographic record of the diagnosis and treatment scenario, which includes the language communication between doctors and patients, information display, and doctors' decision-making behavior, without changing the doctor's original workflow. This provides a sufficient, continuous, and authentic data foundation for subsequent cross-modal semantic feature association and the construction of structured diagnosis and treatment data units.

[0016] Furthermore, the method involves performing cross-modal semantic feature association on the holographic record of the diagnosis and treatment scenario to construct structured diagnosis and treatment data units, and includes: The speech stream is converted into text using ASR to obtain speech-to-text transcription. Acoustic features are extracted from the speech stream, and medical entity recognition is performed on the speech-to-text transcription to obtain medical entity recognition results. Medical entity recognition, relation extraction, and semantic embedding are progressively performed on the text stream to output a structured semantic representation. Behavioral feature encoding is performed on the behavioral event stream to obtain a temporal action sequence. The medical entity recognition results, structured semantic representation, and temporal action sequence are mapped to a shared semantic vector space, and cross-modal semantic feature association is performed to construct the structured diagnostic data unit.

[0017] Optionally, after acquiring a holographic record of the diagnosis and treatment scenario, the system first preprocesses the speech stream acquired by the speech recording module, including noise reduction, echo cancellation, and endpoint detection. The continuous speech is then segmented into several speech segments. For each speech segment, the system calls the Automatic Speech Recognition (ASR) model to convert the speech signal into corresponding speech-to-text. Each transcribed text is appended with its start and end timestamps from the original speech stream. Simultaneously, the system extracts acoustic features from the speech segments. These acoustic features include at least fundamental frequency features, speech rate features, pause duration, volume variation, and intonation variation, which are used to characterize expressions of emphasis, denial, hesitation, or uncertainty during the speaking process. After obtaining the speech-to-text and its corresponding acoustic features, the system performs word segmentation, part-of-speech tagging, and syntactic segmentation on the speech-to-text. The results are then input into a Named Entity Recognition (NER) model pre-trained on a medical corpus to annotate medical-related entities in the text. These medical entities include symptom names, descriptions of signs, anatomical locations, durations, frequencies, triggers, relieving factors, past illnesses, medication names, adverse reactions, and diagnostic names. During entity recognition, the system uses acoustic features as auxiliary features to distinguish between affirmative, negative, or interrogative expressions, thereby generating medical entity recognition results with semantic labels, timestamps, and pragmatic attributes.

[0018] Subsequently, the text stream captured by the text extraction module is processed according to its generation time sequence. During this process, the same entity category system and method as the speech stream are used to perform medical entity recognition on the text stream, thereby uniformly labeling the medical entities appearing in the text. After entity recognition is completed, relation extraction is performed on the text stream. Specifically, sentences or semantic segments are used as the basic processing units to segment and semantically divide the text stream. Within each processing unit, based on the medical entity recognition results, entity combinations within the same semantic segment and adjacent semantic segments are enumerated to generate candidate entity pairs. These candidate entity pairs must satisfy at least one of the following constraints: first, the two entities appear in the same sentence; second, the two entities appear in adjacent sentences, and the distance between the sentences does not exceed a preset threshold; third, at least one of the two entities is a symptom, disease, examination item, or drug entity. After generating candidate entity pairs, a medical rule constraint layer is used to screen the candidate entity pairs for legality. This rule constraint layer consists of a pre-configured medical relation rule base. The rules are defined in the form of "entity type combination + triggering condition" to constrain the types of relationships that can be established.

[0019] For example, when the entity type of a candidate entity pair is [symptom, anatomical location], and the two have positional modification, referential, or dependency relationships in the text, it is marked as a "symptom-location" candidate relationship; when the entity type of a candidate entity pair is [symptom, time description], and the time description appears after the symptom description or is connected by time trigger words such as continuous, recurrent, or intermittent, it is marked as a "symptom-duration" candidate relationship; when the entity type of a candidate entity pair is [disease, examination item], and the text contains trigger words such as "suggested examination," "complete examination," or "further steps," it is marked as a "disease-examination item" candidate relationship; when the entity type of a candidate entity pair is [symptom, anatomical location], and the two have positional modification, referential, or dependency relationships in the text, it is marked as a "symptom-location" candidate relationship; when the entity type of a candidate entity pair is [symptom, time description], and the time description appears after the symptom description or is connected by time trigger words such as continuous, recurrent, or intermittent, it is marked as a "symptom-duration" candidate relationship; when the entity type of a candidate entity pair is [disease, examination item], and the time description appears after the symptom description or is connected by time trigger words such as continuous, recurrent, or intermittent, it is marked as a "disease-examination item" candidate relationship; when the entity type of a candidate entity pair is [symptom, anatomical location ... When the type is [inspection item, inspection result], and the inspection result appears as a supplementary description of the inspection item in the syntactic structure, it is marked as a candidate relationship of "inspection item-result"; when the entity type of the candidate entity pair is [drug, disease], and there are trigger words such as "used for", "treatment", "applicable to", it is marked as a candidate relationship of "drug-indication"; when the entity type of the candidate entity pair is [drug, adverse reaction], and there are trigger words such as "occurs", "causes", "adverse reaction is", it is marked as a candidate relationship of "drug-adverse reaction"; candidate entity pairs that do not meet any of the rule constraints will be directly filtered to avoid generating relationships that do not conform to medical logic.

[0020] For candidate entity pairs filtered through the rule constraint layer, the system invokes a deep learning relation extraction model to determine the relation type. This deep learning relation extraction model includes at least a text encoding layer, an entity tagging and feature fusion layer, and a relation classification layer. In the text encoding layer, the original text is segmented into words or sub-words, and the resulting word sequence is mapped to corresponding word vector representations. These word vector sequences are then input into a context encoding network composed of stacked multi-layer encoding units. This encoding network includes at least several layers of self-attention computation units and feedforward neural network units. In each encoding unit, a self-attention mechanism is used to calculate global dependencies between words to capture long-distance semantic associations. The self-attention output is then fed into the feedforward neural network for nonlinear transformation, and the encoding result is output through residual connections and layer normalization operations. Through multi-layer stacking, the context encoding network aggregates semantic information within and across sentences layer by layer, ultimately outputting a context semantic representation sequence corresponding one-to-one with each word position.

[0021] In the entity labeling and feature fusion layer, the term intervals corresponding to candidate entities are located in the context semantic representation sequence based on the start and end position indices of the entities. Then, a pooling operation is performed on the context semantic representation within the interval, including at least average pooling or max pooling, to obtain an entity context vector that represents the overall semantics of the entity. Next, entity type embedding vectors are introduced for each candidate entity. These vectors are obtained from the entity type identifier through embedding mapping and represent the medical semantic category to which the entity belongs. The entity context vector and the corresponding entity type embedding vector are concatenated to form a first entity feature vector and a second entity feature vector, respectively. These two entity feature vectors are then concatenated in a predetermined order to form a joint feature representation of the entity pair used for relation determination. Through the above pooling and concatenation operations, the model explicitly introduces entity location and entity type information while preserving context semantic information, thereby improving the accuracy and stability of relation determination.

[0022] In the relation classification layer, the joint feature representation of entity pairs is input into a fully connected neural network for feature mapping. This fully connected neural network includes at least one or more linear transformation layers, each followed by a non-linear activation function to achieve feature space mapping and compression. In the multi-layer fully connected structure, the preceding layers extract higher-order combined features, and the subsequent layers map to a predefined relation type space. In the last layer, a normalization function processes the output to obtain the probability distribution of candidate entity pairs across different relation types, representing the confidence level of the entity pair belonging to different relation types. When using a multi-class classification approach, the normalization function is the Softmax function, used to output the probability distribution of mutually exclusive relation types; when using a multi-label classification approach, the normalization function is the Sigmoid function, used to output the probability of multiple relation types being true simultaneously.

[0023] With the combined effect of the rule constraint layer and the deep learning relation extraction model, the relation extraction results between entities in the text stream can be obtained. Each relation includes at least a relation type identifier, an associated entity identifier, the entity's position index in the text, and a relation confidence score, which are used to construct subsequent structured semantic representations and participate in cross-modal semantic feature association. After completing entity and relation extraction, the system organizes medical entities and their relations into structured semantic units, and maps these structured semantic units into low-dimensional dense vectors through a semantic embedding model, thus obtaining the structured semantic representation on the text side.

[0024] Next, the behavioral event stream collected by the behavior capture module is encoded with behavioral features. Specifically, for each behavioral event, its event type, target, parameter information, and occurrence time are analyzed and encoded into a behavioral feature vector. These vectors are then serialized chronologically to form a temporal action sequence reflecting the doctor's operational path, information retrieval order, and decision-making actions. After constructing the features for the three modalities of speech, text, and behavior, the system maps the medical entity recognition results, structured semantic representations, and temporal action sequences to a unified shared semantic vector space. Within this shared semantic vector space, a cross-modal attention mechanism is introduced to calculate the correlation weights between speech entity features, text semantic features, and behavioral action features, constructing a cross-modal association topology. Based on this topology, different modal features are then weighted and fused to generate a cross-modal joint feature representation sequence, while simultaneously retaining the association weight information between each modality as interpretable data. Finally, the system uses diagnosis and treatment events as the basic encapsulation unit, and uniformly encapsulates cross-modal joint feature representation sequences, lists of associated medical entities, entity relationship information, behavioral action sequences, and corresponding time range identifiers to form structured diagnosis and treatment data units. This enables the dispersed multimodal information in the diagnosis and treatment process to be uniformly expressed at the semantic and temporal levels, providing standardized input for the reasoning and training of subsequent clinical reasoning models.

[0025] Furthermore, the method involves performing cross-modal semantic feature association on the holographic record of the diagnosis and treatment scenario to construct structured diagnosis and treatment data units, and includes: In the shared semantic vector space, a cross-modal attention mechanism is used to calculate the dynamic association weights among the medical entity recognition results, structured semantic representations, and temporal action sequences, constructing a cross-modal association topology. Based on the cross-modal association topology, cross-modal feature weighted fusion is performed on the temporal association context of the medical entity recognition results, structured semantic representations, and temporal action sequences to output a cross-modal joint feature representation sequence. The original modal association weight sequence of the cross-modal joint feature representation sequence is extracted from the cross-modal association topology. Based on the cross-modal joint feature representation sequence, a list of associated medical entities is selected from the medical entity recognition results. Using medical events as the basic unit, the list of associated medical entities, the original modal association weight sequence, and the cross-modal joint feature representation sequence are structurally encapsulated to output the structured medical data unit.

[0026] Optionally, after constructing and mapping the features of the three modalities (speech, text, and behavior) to a shared semantic vector space, the system acquires the entity feature vector sequence corresponding to the medical entity recognition result, the structured semantic representation sequence of the text stream, and the temporal action feature vector sequence of the behavior stream within the shared semantic vector space. The system uses a time-aligned window or a diagnosis / treatment event window as constraints to jointly model features of different modalities within the same time frame. Specifically, the system constructs a query vector, a key vector, and a numerical vector for each modal feature sequence and introduces a cross-modal attention mechanism to calculate the correlation between different modal features. This attention mechanism obtains dynamic association weights between different modal features by weighted normalization of the similarity between the query vector and the key vector, which characterizes the degree of mutual influence between speech, text, and behavior features in the same diagnosis / treatment context. Subsequently, using the dynamic association weights as edge weights and each modal feature node as a vertex, a cross-modal association topology is constructed to explicitly represent the semantic and temporal relationships of multimodal information. After constructing the cross-modal association topology, the system performs cross-modal feature weighted fusion on the medical entity recognition results, structured semantic representations, and temporal action sequences based on the topological structure. Specifically, according to the association weights determined in the cross-modal association topology, different modal feature vectors located within the same diagnostic event or time window are weighted and combined, thereby fusing multimodal information into a unified semantic representation. Through this weighted fusion operation, a cross-modal joint feature representation sequence is generated. This sequence, while maintaining the original temporal structure, comprehensively reflects the diagnostic semantic information jointly expressed by speech expression, text semantics, and doctor's actions. Subsequently, from the cross-modal association topology, according to the temporal order of the cross-modal joint feature representation sequence, the original modal association weights corresponding to each joint feature are extracted and organized into an original modal association weight sequence. This original modal association weight sequence is used to characterize the contribution ratio of each modality's information to the formation of the joint feature at different diagnostic stages and serves as an important basis for subsequent model inference and result interpretation. Then, the similarity between each medical entity and its corresponding cross-modal joint feature representation is calculated. Medical entities exceeding a preset threshold are marked as associated medical entities. These associated medical entities are sorted according to their order of appearance in the diagnosis and treatment process, forming an associated medical entity list. This list reflects key information such as symptoms, signs, diseases, examination items, or medications that are of primary concern in the current diagnosis and treatment event. Finally, the system uses the diagnosis and treatment event as the basic encapsulation unit. For each diagnosis and treatment event, the associated medical entity list, the corresponding cross-modal joint feature representation sequence fragment, the original modal association weight sequence, and the event start and end time identifiers are uniformly encapsulated to generate structured diagnosis and treatment data units. This ensures the integrity, traceability, and interpretability of the diagnosis and treatment process information after structuring, thereby improving the accuracy and stability of clinical auxiliary reasoning and providing a reliable data foundation for diagnosis and treatment process review, quality control, and continuous model optimization.

[0027] The structured diagnostic data unit is input into the clinical reasoning model deployed in the first participating institution. After cross-modal association capture based on the attention mechanism, clinical task reasoning is performed, and auxiliary diagnostic prompts are output.

[0028] In one embodiment, after constructing the structured diagnostic data unit, the system sends the structured diagnostic data unit as standardized input to the clinical reasoning model deployed on the local server of the first participating institution to generate auxiliary diagnostic prompts in real time during the consultation or treatment process. Specifically, the system first decapsulates the structured diagnostic data unit, reading the list of associated medical entities, the original modal association weight sequence, and the cross-modal joint feature representation sequence. Subsequently, the list of associated medical entities is vectorized, mapping the category, attributes, and occurrence time information of each entity to entity feature vectors, and the cross-modal joint feature representation sequence is serialized input encoding. Then, the clinical reasoning model uses the cross-modal joint feature representation sequence as the query vector, the entity feature vector sequence and / or key temporal action features as key and value vectors, and combines them with the original modal association weight sequence to calculate the association strength between different modal evidence, thereby capturing the dynamic correspondence between "patient statement - textual evidence - doctor's operation" and generating evidence-enhanced contextual representations. After obtaining the context representation, the model inputs it into the diagnostic inference head, examination suggestion inference head, and risk warning inference head respectively to perform parallel inference, forming auxiliary diagnostic prompts that include preliminary inference results, examination item suggestions, risk warnings, and related medical entities. This auxiliary diagnostic prompt is then presented to the doctor's terminal interface in an interactive manner in real time, so that medical staff can obtain immediate decision support without changing the existing diagnosis and treatment process, thereby reducing the probability of diagnostic omissions and unreasonable examinations.

[0029] Furthermore, the structured diagnostic data unit is input into a clinical reasoning model deployed in the first participating institution. After cross-modal association capture based on an attention mechanism, clinical task reasoning is performed, and auxiliary diagnostic prompts are output. The method includes: The structured diagnostic data unit is decapsulated to obtain a list of associated medical entities, a sequence of original modal association weights, and a sequence of cross-modal joint feature representations. The list of associated medical entities is encoded into entity feature vectors, the sequence of cross-modal joint feature representations is used as the main feature input, and the sequence of original modal association weights is used as a prior attention reference. These are all input into the clinical reasoning model to perform clinical task reasoning, and the model outputs preliminary inference results, examination item suggestions, and risk warnings. The preliminary inference results, examination item suggestions, and risk warnings are fused, and the list of associated medical entities is used as evidence to generate the auxiliary diagnostic prompt information.

[0030] Preferably, the system first reads the pre-encapsulated event-level index information from the structured diagnostic data unit and parses out the corresponding list of associated medical entities, the original modal association weight sequence, and the cross-modal joint feature representation sequence according to the event sequence. Then, the system performs entity feature encoding on the list of associated medical entities; that is, it extracts the entity type features, entity attribute features, and temporal features for each medical entity and converts these features into numerical vector representations through embedding mapping. Entity type features are used to characterize differences in categories such as symptoms, diseases, examination items, or drugs; entity attribute features are used to characterize degree, duration, frequency, etc.; and temporal features are used to characterize the position of the entity in the diagnostic process. The system combines multiple entity feature vectors within the same diagnostic event to form an entity feature vector sequence, which is used as evidence input in the reasoning process. After completing entity encoding, the cross-modal joint feature representation sequence is fed into the backbone network of the clinical reasoning model as the main feature input to represent the overall context information of the current diagnostic scenario. Simultaneously, the entity feature vector sequence is input into the model for explicit modeling of key medical entities. During the attention calculation process within the model, the system introduces the original modal association weight sequence as a priori attention reference. This allows the model to assign differentiated attention weights to different pieces of evidence based on existing cross-modal association strengths when calculating attention allocation between subject features and entity features, thereby prioritizing the use of highly relevant diagnostic evidence. Under the combined effect of these multi-source features, the clinical reasoning model jointly models subject features and entity evidence, and executes clinical task reasoning in multiple task reasoning heads to obtain preliminary inference results, examination item suggestions, and risk warnings. Finally, the system performs result fusion processing on the preliminary inference results, examination item suggestions, and risk warnings. That is, it integrates the three types of reasoning results according to a preset splicing strategy, mapping and binding each reasoning conclusion to its associated medical entity evidence, generating a structured output containing "reasoning conclusion - evidence entity - association weights." Based on this, the system formats and semantically organizes the fusion results to form auxiliary diagnostic prompts for doctors, ensuring that the prompts not only provide diagnostic suggestions but also clearly explain the corresponding key evidence sources and diagnostic basis.

[0031] Furthermore, the method also includes: The original modality association weight sequence is used as an attention bias term and introduced into the self-attention module of the clinical reasoning model. Under the modulation of the attention bias term, the self-attention module performs global context modeling on the subject feature input, generates a deep semantic feature sequence, and then calculates the association between the deep semantic feature sequence and the entity feature vector through a cross-attention mechanism to generate an entity-enhanced feature representation. The entity-enhanced feature representation is input in parallel to the differential diagnosis reasoning head, examination suggestion reasoning head, and risk warning reasoning head at the output of the self-attention module, and clinical task reasoning is performed in parallel to output the preliminary inference result, examination item suggestions, and risk warning.

[0032] Optionally, the original modal association weight sequence parsed from the structured diagnostic data unit is first introduced into the self-attention module of the clinical reasoning model as an attention bias term in the attention weight calculation. Specifically, in the self-attention module, the main feature input is first mapped into a query vector, a key vector, and a value vector. After calculating the similarity score between the query vector and the key vector, the system maps the original modal association weight sequence into a bias matrix with the same dimension as the similarity score matrix according to the alignment method of the diagnostic events. This bias matrix is ​​then weighted and superimposed with the similarity score. In this way, the original modal association weight sequence does not directly replace the attention result learned by the model, but rather modulates the attention distribution as a prior constraint, enabling the self-attention module to highlight information from highly relevant modalities or key diagnostic stages during global modeling. After introducing an attention bias term, the self-attention module performs global context modeling on the main feature input. Specifically, based on the bias-modulated attention weights, it performs a weighted summation of the value vectors to obtain an attention output that integrates multimodal evidence and temporal context. This attention output is processed through residual connections, normalization, and a feedforward network to form a deep semantic feature sequence. This deep semantic feature sequence maintains the order of diagnostic events in the temporal dimension and comprehensively reflects the overall diagnostic context composed of patient statements, textual evidence, and doctor's operational behavior in the semantic dimension. After obtaining the deep semantic feature sequence, the system further performs entity-centric cross-attention calculation. In this process, the entity feature vector corresponding to each medical entity in the associated medical entity list is used as the query vector, and the deep semantic feature sequence is used as the key vector and value vector. The similarity between the entity query vector and the context key vector is calculated. This similarity, after normalization, is used to weighted convergence of the context value vectors, thereby generating an entity context aggregation vector for each medical entity. Subsequently, this entity context aggregation vector is concatenated with the entity's own attribute feature vector to form the final entity-enhanced feature representation. Through this process, each entity-enhanced feature representation is centered on the entity and explicitly includes the most relevant contextual information in the current diagnostic and treatment context. Then, the system inputs the entity-enhanced feature representations in parallel into multiple task inference heads located at the output of the self-attention module, such as differential diagnosis inference heads, examination suggestion inference heads, and risk warning inference heads. Each task inference head consists of an independent feature map and task output. Based on the shared entity-enhanced feature representation, each inference head independently executes the corresponding clinical task inference.

[0033] In the differential diagnosis inference head, the input entity augmentation feature representation is organized into an entity feature sequence. The system maps the entity augmentation features to query vectors and key vectors respectively, calculates the relevance weights between entities, and uses these weights to perform a weighted summation of the entity features, thereby obtaining an aggregated feature vector that represents the overall diagnostic evidence at the current stage of diagnosis. After completing the entity feature aggregation, the aggregated feature vector is input into a feature mapping unit for nonlinear transformation. This feature mapping unit consists of at least one linear transformation layer, with each linear transformation layer followed by a nonlinear activation function, used to combine and model the entity evidence, extracting high-order feature representations for diagnostic discrimination. Next, the transformed features are input into a diagnostic classification unit, which consists of multiple fully connected layers, used to map the features to a predefined diagnostic category space. By normalizing the classification output, the predicted probability corresponding to each candidate diagnosis is obtained, thereby outputting the preliminary inference result and its confidence level.

[0034] In the examination suggestion inference head, the entity augmentation feature representation is input to the feature alignment unit and projected onto the examination semantic space through a linear mapping. This projection represents the coverage of current diagnostic evidence across different potential diagnostic directions. This linear mapping is achieved through matrix transformation while maintaining consistent feature dimensions. Next, the system processes the current evidence features through a difference modeling unit. Specifically, it compares the mapped entity augmentation features with a preset examination requirement feature template. This comparison can be achieved through feature difference or feature concatenation followed by linear transformation, characterizing the difference between the current evidence and the ideal diagnostic evidence, thus forming an evidence gap feature representation. After obtaining the evidence gap feature representation, it is input to the examination scoring unit, which consists of multiple fully connected layers and is used to calculate a recommendation score for each candidate examination item. The system sorts the examination items based on the calculated recommendation scores, obtaining recommended examination items and their priority ranking results, which serve as examination item suggestions.

[0035] In the risk warning inference head, the entity-enhanced feature representation is input into the anomaly feature extraction unit for anomaly enhancement processing. This anomaly feature extraction unit is implemented through linear transformation combined with a nonlinear activation function, used to amplify the dimensional changes in entity features related to high-risk states. Next, the anomaly-enhanced features are input into the risk level discrimination unit, which is also composed of multiple fully connected layers. This unit maps the anomaly features to a predefined risk level space, and then outputs the corresponding risk level and risk warning information through normalization and threshold determination. Through the aforementioned attention bias modulation, self-attention global modeling, entity-centric cross-attention enhancement, and collaborative execution of multi-task parallel inference, the clinical inference model can organize diagnostic evidence around key medical entities, highlighting core evidence nodes while ensuring global contextual consistency, thereby improving the accuracy, stability, and interpretability of multi-task inference results.

[0036] The feedback from medical staff to the auxiliary diagnostic prompts is used as a supervisory signal, and combined with the structured diagnostic data unit, it drives the clinical reasoning model to optimize model parameters based on local incremental training.

[0037] In one embodiment, after the clinical inference model completes an auxiliary diagnostic prompt output and presents it on the doctor's interactive terminal, the system uses the interactive feedback operation of medical staff on the auxiliary diagnostic prompt as a supervision signal to perform local incremental training and parameter optimization on the model. During this process, a corresponding supervision signal is set according to the type of feedback operation. For example, when the feedback operation is an acceptance-type feedback, the auxiliary diagnostic prompt is used as a positive supervision signal to construct positive training samples; when the feedback operation is an ignore / rejection-type feedback, the auxiliary diagnostic prompt is used as a negative supervision signal to construct training samples. Subsequently, the system performs incremental training on the clinical inference model in the local computing environment of the first participating institution. During the training process, parameter updates are performed only for a limited number of rounds on newly constructed training samples to avoid excessive perturbation to the existing model's capabilities, thereby achieving stable local incremental optimization. Through the above methods, the model can continuously absorb the professional judgment and experience of medical staff in real diagnosis and treatment scenarios, so that the model parameters can be gradually adjusted in a direction that conforms to the actual clinical decision-making habits. In this way, without relying on additional manual annotation, the model can continuously improve its clinical reasoning ability and personalize its adaptation. At the same time, it provides a high-quality local training foundation for subsequent cross-institutional model collaborative updates based on privacy computing.

[0038] Furthermore, the method utilizes the feedback from medical staff to the auxiliary diagnostic prompts as a supervisory signal, combined with the structured diagnostic data unit, to drive the clinical reasoning model to optimize model parameters based on local incremental training. After displaying the auxiliary diagnostic prompts on the doctor's interactive terminal, the feedback operations of medical staff are captured. If the feedback operation is an adoption-type feedback, the auxiliary diagnostic prompts are used as positive supervision signals and paired with the structured diagnostic data units to form training samples. If the feedback operation is a correction-editing-type feedback, the corrected diagnostic prompts are used as positive supervision signals and paired with the structured diagnostic data units to form training samples. Using the training samples, the model parameters of the clinical reasoning model are optimized based on local incremental training.

[0039] Preferably, after the clinical reasoning model generates auxiliary diagnostic prompts, the system displays these prompts in real-time on the doctor's interactive terminal in a visual manner. Each prompt is configured with corresponding interactive controls to allow medical staff to confirm, modify, or supplement the prompt results. While displaying the prompts, the system monitors and records the doctor's interactive terminal's operation events to capture feedback from medical staff regarding the auxiliary diagnostic prompts. Each feedback operation is automatically appended with a timestamp and a corresponding diagnostic event identifier. When the feedback operation is identified as an acceptance-type feedback, it indicates that the medical staff approves of the overall or key conclusions of the auxiliary diagnostic prompts provided by the system. In this case, the auxiliary diagnostic prompts are directly used as positive supervision signals and paired with the structured diagnostic data units used to generate the prompts to construct corresponding training samples. The structured diagnostic data units serve as model inputs, and the auxiliary diagnostic prompts serve as target output labels. When the feedback operation is identified as a correction / editing-type feedback, it indicates that the medical staff has adjusted the diagnostic conclusions, examination suggestions, or risk warnings based on the prompts. After capturing the corrected content, the system parses the revised diagnostic prompts from medical staff into structured supervision labels, using these labels as positive supervision signals. These labels are then paired with corresponding structured clinical data units to form corrected training samples, ensuring that the training samples reflect the actual clinical judgments of medical staff. After constructing the training samples, the system organizes and caches them. Each training sample includes at least one structured clinical data unit as input, and positive or corrected diagnostic prompts as the supervision target. The system records the source feedback type, corresponding clinical event identifier, and generation time for each training sample for subsequent training scheduling and sample management.

[0040] Subsequently, the local incremental training process is initiated. The system re-inputs the structured diagnostic data units into the clinical inference model. The model generates predicted outputs following the same forward computation path as the inference phase. The model's predicted outputs are then compared with the supervision signals in the corresponding training samples to calculate the training loss function. This training loss function includes at least one or more of the following: diagnostic result loss, examination suggestion loss, and risk warning loss. It measures the degree of deviation between the model output and the feedback from medical staff. After loss calculation, gradient calculations are performed on the model parameters based on the backpropagation mechanism, and parameter update operations are executed. To achieve incremental training rather than full retraining, the system selectively adjusts the model parameters during the parameter update phase. That is, gradient updates are only made available for parameters relevant to the current task's inference, such as inference head parameters and high-level parameters related to entity enhancement features, while parameters of the underlying general feature extraction layer or those that have already converged are frozen, thus avoiding significant fluctuations in the overall model performance. During training, the system performs only a limited number of parameter updates for each batch of training samples and records the magnitude of model parameter changes after each update. When the magnitude of parameter changes falls below a preset threshold or the maximum number of training rounds is reached, the current incremental training process is automatically terminated. By limiting the size of the training samples, the number of training epochs, and the range of updated parameters, the model can gradually absorb feedback from healthcare professionals without causing abnormal deviations in its behavior due to single or small amounts of feedback. After completing local incremental training, the updated model parameters are loaded into the clinical inference model for subsequent inference tasks. Simultaneously, this incremental training process is marked and archived to distinguish it from subsequent parameter updates uploaded via a privacy-preserving computation mechanism. Through this approach, the clinical inference model can continuously optimize itself using professional feedback from healthcare professionals in real-world clinical scenarios without relying on centralized data aggregation, achieving smooth performance improvements and personalized adaptation.

[0041] Furthermore, if the feedback operation is an ignore rejection type feedback, then the auxiliary diagnostic prompt information is used as a negative supervision signal and paired with the structured diagnostic data unit to form a training sample.

[0042] Optionally, when the system detects that the medical staff's feedback on the auxiliary diagnostic prompts is a rejection response, it indicates that the medical staff did not adopt the diagnostic conclusions, examination suggestions, or risk warnings provided by the system in the current treatment scenario. For this type of feedback, the system extracts the structured clinical data units used to generate the auxiliary diagnostic prompts as input to the training samples. Simultaneously, the auxiliary diagnostic prompts themselves are marked as negative supervision signals to indicate that the prompt results do not conform to the medical staff's clinical judgment in the current treatment context. Subsequently, the structured clinical data units are paired with the negative supervision signals to construct negative training samples, which are then stored in the local training buffer. In subsequent local incremental training, the negative training samples are used to impose constraints on the clinical reasoning model. By applying penalty or inhibition terms to rejected diagnostic conclusions, examination suggestions, or risk warnings during the loss calculation stage, the model reduces the probability of outputting such results under similar input conditions. Through the above methods, the system can effectively transform medical staff's explicit rejection of diagnosis and treatment into usable training signals, avoiding the model from repeatedly outputting prompts that do not conform to clinical reality. This ensures the stability of the model while gradually improving the accuracy and practicality of the clinical reasoning model.

[0043] The first participating institution uses a privacy computing protocol to encrypt and de-identify the model parameter updates of the clinical reasoning model before sending them to the federated learning coordination node.

[0044] In one embodiment, after completing local incremental training based on feedback from medical staff and obtaining parameter update results for the clinical inference model, the first participating institution does not directly share any original clinical data or complete model parameters externally. Instead, it encrypts and desensitizes the model parameter updates according to a preset privacy computing protocol. This ensures that the parameter updates of a single participating institution cannot be restored or used to infer the characteristics of the original clinical data without decryption and aggregation, thus guaranteeing the privacy and security of parameter-level information during transmission and use. After encryption and desensitization, the system encapsulates the model parameter updates into a privacy data packet and sends it to the federated learning coordination node through a secure communication channel. This privacy data packet only contains parameter update information processed by the privacy computing protocol and necessary identification information, and does not contain any original data content that can identify specific patients, doctors, or treatment processes. Through this method, the first participating institution achieves secure sharing of model knowledge while ensuring that local clinical data does not leave the domain, providing a compliant and reliable data foundation for the subsequent multi-institution parameter aggregation and joint model updates performed by the federated learning coordination node.

[0045] Furthermore, the first participating institution uses a privacy computing protocol to encrypt and de-identify the model parameter updates of the clinical inference model before sending them to the federated learning coordination node. The method includes: Obtain the model parameter update amount of the clinical reasoning model; after encrypting and desensitizing the model parameter update amount using a privacy computing protocol, encapsulate the privacy data packet; and send the privacy data packet to the federated learning coordination node through a secure communication channel.

[0046] Optionally, after the first participating institution completes local incremental training, the system saves a snapshot of the model parameters before the start of this round of incremental training locally. After training is completed, it reads the updated model parameters and calculates the corresponding model parameter update by comparing the two parameter states. This model parameter update can be represented in parameter difference or gradient form to reflect the adjustment information generated by local training on the model. To facilitate subsequent processing, the system structures and unifies the format of the parameter update to meet the parameter dimensions and index specifications required for federated learning aggregation. After obtaining the model parameter update, the system calls the privacy computing protocol to perform encryption and desensitization processing on the model parameter update. Specifically, the model parameter update is first encoded and encapsulated, and then an encryption step is introduced to encrypt and transform the parameter update, making it impossible to read directly before decryption or participation in secure aggregation calculation. At the same time, combined with preset desensitization strategies, the parameter update is randomly perturbed, masked, or injected with noise to reduce the risk of reverse analysis or association with the original training data for a single parameter update. After completing the encryption and anonymization process, the system encapsulates the processing results along with necessary metadata, such as model version identifiers and parameter index information, to form a privacy data packet. The system then sends this privacy data packet to the federated learning coordination node via a secure communication channel. This secure communication channel employs an encrypted transmission mechanism to prevent data theft or tampering during transmission. During transmission, the system only transmits the privacy data packet processed by the privacy computation protocol, without transmitting any original diagnostic data, training samples, or plaintext model parameters. Through these steps, the first participating institution, while ensuring local data security and privacy compliance, achieves a reliable upload of model parameter updates, providing reliable input for the federated learning coordination node to subsequently perform secure aggregation computation and joint model updates.

[0047] The federated learning coordination node receives and aggregates multiple model parameter updates from multiple participating institutions, performs secure aggregation computation, and generates a federated aggregation model.

[0048] In one embodiment, the federated learning coordinating node first receives privacy data packets from multiple participating institutions via a secure communication channel. For each received privacy data packet, the coordinating node verifies its integrity and legitimacy, including verifying the source identifier, model version identifier, and parameter structure consistency of the data packet, to ensure that the parameter updates participating in the aggregation all correspond to the same round of federated training tasks. Data packets that pass verification are temporarily stored in the aggregation buffer, while data packets that fail verification are discarded or marked as abnormal and do not participate in subsequent aggregation calculations. After completing data reception and verification, the federated learning coordinating node initiates the secure aggregation calculation process. Specifically, without decrypting or restoring the parameter updates of individual participating institutions in plaintext, the coordinating node performs aggregation operations on the encrypted and de-identified parameter updates uploaded by each participating institution according to the privacy computing protocol. This aggregation operation includes at least a weighted summation of the update amounts at corresponding parameter positions, where the weights can be determined based on the number of participating institutions, sample size, or a preset strategy. In this way, the coordinating node only obtains the overall statistical results of the parameter updates of multiple participating institutions and cannot infer the local training information of any single institution. After completing the secure aggregation computation, the federated learning coordinating node merges the updated parameters with the parameters from the previous round of federated model calculations to obtain an updated set of model parameters, thus forming a federated aggregated model. This model comprehensively incorporates model optimization information trained on local data by multiple participating institutions, while maintaining the invisibility of each institution's original clinical data and local model parameters. Through this process, the federated learning coordinating node achieves secure aggregation and joint updating of cross-institutional model parameters without directly accessing any original clinical data or plaintext parameters from a single institution, providing a reliable foundation for subsequently distributing a unified and optimized federated model to all participating institutions.

[0049] The federated learning coordination node distributes the federated aggregation model to the multiple participating institutions for model deployment and updates.

[0050] In one embodiment, after generating the federated aggregation model, the federated learning coordination node performs version encapsulation and consistency marking on the generated model. During this process, the coordination node assigns a unique model version identifier to the model and records the corresponding federated training rounds, participating institution set, and parameter structure information to ensure that each participating institution can correctly identify the model update source and scope of application. Subsequently, the coordination node standardizes and encapsulates the federated aggregation model parameters to form a model update package that can be distributed remotely. After model encapsulation, the coordination node sends the model update package to each participating institution via a secure communication channel. This update package contains only the federated model parameters and necessary version and verification information, without any local parameter updates or original clinical data from any single participating institution, thus ensuring data security and privacy compliance during cross-institutional model distribution. Upon receiving the model update package, each participating institution first verifies the model version identifier and integrity verification information to confirm the compatibility of the federated aggregation model with its currently deployed model. After successful verification, the participating institution loads the federated aggregation model into its local clinical inference model deployment environment to replace or update the original model parameters. During model updates, the system can employ smooth updates or phased switching to avoid interrupting ongoing clinical inference tasks. After completing model deployment and updates, participating institutions will use the updated clinical inference model in subsequent diagnosis and treatment scenarios, and continue to perform local incremental training based on newly generated structured diagnosis and treatment data and feedback from medical staff. Through this approach, the federated aggregation model can maintain synchronous evolution among participating institutions, enabling cross-institutional sharing of model capability improvements, while ensuring that each institution's local diagnosis and treatment data remains locally and is not leaked externally.

[0051] In summary, the embodiments of this application have at least the following technical effects: First, cross-modal semantic feature association is performed on the holographic record of the diagnosis and treatment scenario to construct a structured diagnosis and treatment data unit. Then, the structured diagnosis and treatment data unit is input into the clinical reasoning model deployed in the first participating institution. After cross-modal association capture based on an attention mechanism, clinical task reasoning is performed, and auxiliary diagnostic prompts are output. Next, the feedback operations of medical staff to the auxiliary diagnostic prompts are used as supervisory signals, combined with the structured diagnosis and treatment data unit to drive the clinical reasoning model to optimize model parameters based on local incremental training. Then, the first participating institution uses a privacy computing protocol to encrypt and de-identify the model parameter updates of the clinical reasoning model before sending them to the federated learning coordination node. Then, the federated learning coordination node receives and aggregates multiple model parameter updates from multiple participating institutions, performs secure aggregation computation, and generates a federated aggregated model. Finally, the federated learning coordination node distributes the federated aggregated model to the multiple participating institutions for model deployment and updates. It solves the technical problems of missing data in the doctor-patient interaction and clinical reasoning process in the existing medical information system, the lack of traceability of clinical thinking, and the difficulty in achieving multi-institutional collaborative modeling under data privacy constraints. It achieves the technical effect of improving the real-time and accuracy of clinical assistance by realizing the structured expression of multimodal doctor-patient interaction data, real-time decision-making assistance of clinical reasoning models, and continuous joint optimization across medical institutions without disclosing the original diagnosis and treatment data.

[0052] Example 2, based on the same inventive concept as the privacy-based computing-based multimodal doctor-patient interaction clinical assistance method in the foregoing examples, such as... Figure 2 As shown, this application provides a multimodal doctor-patient interaction clinical assistance system based on privacy computing, wherein the system includes: Feature Association Component 11: Performs cross-modal semantic feature association on the holographic record of the diagnosis and treatment scenario to construct a structured diagnosis and treatment data unit; Clinical Task Reasoning Component 12: Inputs the structured diagnosis and treatment data unit into the clinical reasoning model deployed in the first participating institution, performs clinical task reasoning after capturing cross-modal association based on the attention mechanism, and outputs auxiliary diagnostic prompts; Model Parameter Optimization Component 13: Uses the feedback operation of medical staff on the auxiliary diagnostic prompts as a supervision signal, and drives the clinical reasoning model to perform model parameter optimization based on local incremental training in combination with the structured diagnosis and treatment data unit; Encryption and Desensitization Component 14: The first participating institution uses a privacy computing protocol to encrypt and desensitize the model parameter update of the clinical reasoning model before sending it to the federated learning coordination node; Secure Aggregation Component 15: The federated learning coordination node receives and aggregates multiple model parameter updates from multiple participating institutions, performs secure aggregation calculation, and generates a federated aggregation model; Model Deployment Component 16: The federated learning coordination node distributes the federated aggregation model to the multiple participating institutions for model deployment and updates.

[0053] Furthermore, the feature association component 11 is used to perform the following method: The diagnostic and treatment scene is holographically recorded by an integrated perception engine, which includes a voice recording module, a text extraction module, and a behavior capture module. The holographic recording of the diagnostic and treatment scene includes the voice stream recorded by the voice recording module, the text stream captured by the text extraction module, and the behavior event stream captured by the behavior capture module.

[0054] Furthermore, the feature association component 11 is used to perform the following method: The speech stream is converted into text using ASR to obtain speech-to-text transcription. Acoustic features are extracted from the speech stream, and medical entity recognition is performed on the speech-to-text transcription to obtain medical entity recognition results. Medical entity recognition, relation extraction, and semantic embedding are progressively performed on the text stream to output a structured semantic representation. Behavioral feature encoding is performed on the behavioral event stream to obtain a temporal action sequence. The medical entity recognition results, structured semantic representation, and temporal action sequence are mapped to a shared semantic vector space, and cross-modal semantic feature association is performed to construct the structured diagnostic data unit.

[0055] Furthermore, the feature association component 11 is used to perform the following method: In the shared semantic vector space, a cross-modal attention mechanism is used to calculate the dynamic association weights among the medical entity recognition results, structured semantic representations, and temporal action sequences, constructing a cross-modal association topology. Based on the cross-modal association topology, cross-modal feature weighted fusion is performed on the temporal association context of the medical entity recognition results, structured semantic representations, and temporal action sequences to output a cross-modal joint feature representation sequence. The original modal association weight sequence of the cross-modal joint feature representation sequence is extracted from the cross-modal association topology. Based on the cross-modal joint feature representation sequence, a list of associated medical entities is selected from the medical entity recognition results. Using medical events as the basic unit, the list of associated medical entities, the original modal association weight sequence, and the cross-modal joint feature representation sequence are structurally encapsulated to output the structured medical data unit.

[0056] Furthermore, the clinical task reasoning component 12 is used to perform the following methods: The structured diagnostic data unit is decapsulated to obtain a list of associated medical entities, a sequence of original modal association weights, and a sequence of cross-modal joint feature representations. The list of associated medical entities is encoded into entity feature vectors, the sequence of cross-modal joint feature representations is used as the main feature input, and the sequence of original modal association weights is used as a prior attention reference. These are all input into the clinical reasoning model to perform clinical task reasoning, and the model outputs preliminary inference results, examination item suggestions, and risk warnings. The preliminary inference results, examination item suggestions, and risk warnings are fused, and the list of associated medical entities is used as evidence to generate the auxiliary diagnostic prompt information.

[0057] Furthermore, the clinical task reasoning component 12 is used to perform the following methods: The original modality association weight sequence is used as an attention bias term and introduced into the self-attention module of the clinical reasoning model. Under the modulation of the attention bias term, the self-attention module performs global context modeling on the subject feature input, generates a deep semantic feature sequence, and then calculates the association between the deep semantic feature sequence and the entity feature vector through a cross-attention mechanism to generate an entity-enhanced feature representation. The entity-enhanced feature representation is input in parallel to the differential diagnosis reasoning head, examination suggestion reasoning head, and risk warning reasoning head at the output of the self-attention module, and clinical task reasoning is performed in parallel to output the preliminary inference result, examination item suggestions, and risk warning.

[0058] Furthermore, the model parameter optimization component 13 is used to perform the following method: After displaying the auxiliary diagnostic prompts on the doctor's interactive terminal, the feedback operations of medical staff are captured. If the feedback operation is an adoption-type feedback, the auxiliary diagnostic prompts are used as positive supervision signals and paired with the structured diagnostic data units to form training samples. If the feedback operation is a correction-editing-type feedback, the corrected diagnostic prompts are used as positive supervision signals and paired with the structured diagnostic data units to form training samples. Using the training samples, the model parameters of the clinical reasoning model are optimized based on local incremental training.

[0059] Furthermore, the model parameter optimization component 13 is used to perform the following method: If the feedback operation is an ignore rejection type feedback, then the auxiliary diagnostic prompt information is used as a negative supervision signal and paired with the structured diagnosis and treatment data unit to form a training sample.

[0060] Furthermore, the encryption and desensitization component 14 is used to perform the following method: Obtain the model parameter update amount of the clinical reasoning model; after encrypting and desensitizing the model parameter update amount using a privacy computing protocol, encapsulate the privacy data packet; and send the privacy data packet to the federated learning coordination node through a secure communication channel.

[0061] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A multimodal doctor-patient interaction clinical assistance method based on privacy computing, characterized in that, The method includes: Perform cross-modal semantic feature association on the holographic record of the diagnosis and treatment scenario to construct structured diagnosis and treatment data units; The structured diagnostic data unit is input into the clinical reasoning model deployed in the first participating institution. After cross-modal association capture based on the attention mechanism, clinical task reasoning is performed and auxiliary diagnostic prompts are output. The feedback from medical staff to the auxiliary diagnostic prompts is used as a supervisory signal, and combined with the structured diagnostic data unit, it drives the clinical reasoning model to optimize the model parameters based on local incremental training. The first participating institution uses a privacy computing protocol to encrypt and de-identify the model parameter update of the clinical reasoning model before sending it to the federated learning coordination node; The federated learning coordination node receives and aggregates multiple model parameter updates from multiple participating institutions, performs secure aggregation calculations, and generates a federated aggregation model. The federated learning coordination node distributes the federated aggregation model to the multiple participating institutions for model deployment and updates. The diagnostic and treatment scene is recorded holographically by an integrated perception engine, which includes a voice recording module, a text extraction module, and a behavior capture module. The holographic record of the diagnostic and treatment scene includes the voice stream recorded by the voice recording module, the text stream captured by the text extraction module, and the behavior event stream captured by the behavior capture module. The method involves performing cross-modal semantic feature association on holographic records of diagnostic and treatment scenarios to construct structured diagnostic and treatment data units. The speech stream was converted into text using ASR (Automatic Speech Retrieval System) to obtain speech-transcribed text. After extracting acoustic features from the speech stream, medical entity recognition is performed in conjunction with the speech-transcribed text to obtain the medical entity recognition result. By progressively performing medical entity recognition, relation extraction, and semantic embedding on the text stream, a structured semantic representation is output. The behavioral event stream is encoded with behavioral features to obtain a temporal action sequence; The medical entity recognition results, structured semantic representations, and temporal action sequences are mapped to a shared semantic vector space, cross-modal semantic feature association is performed, and the structured diagnosis and treatment data unit is constructed. The method involves performing cross-modal semantic feature association on holographic records of diagnostic and treatment scenarios to construct structured diagnostic and treatment data units. In the shared semantic vector space, a cross-modal attention mechanism is used to calculate the dynamic association weights between the medical entity recognition results, structured semantic representations, and temporal action sequences, thereby constructing a cross-modal association topology; Based on the cross-modal association topology, the medical entity recognition results, structured semantic representations, and temporal association contexts of the temporal action sequences are subjected to cross-modal feature weighted fusion to output a cross-modal joint feature representation sequence; Extract the original modal association weight sequence from the cross-modal association topology to obtain the cross-modal joint feature representation sequence; Based on the cross-modal joint feature representation sequence, a list of associated medical entities is selected from the medical entity recognition results; Using medical events as the basic unit, the associated medical entity list, the original modal association weight sequence, and the cross-modal joint feature representation sequence are structurally encapsulated to output the structured medical data unit; The structured diagnostic data unit is input into a clinical reasoning model deployed in the first participating institution. After cross-modal association capture based on an attention mechanism, clinical task reasoning is performed, and auxiliary diagnostic prompts are output. The method includes: Decapsulate the structured diagnostic data unit to obtain a list of associated medical entities, a sequence of original modal association weights, and a sequence of cross-modal joint feature representations; The list of associated medical entities is encoded into entity feature vectors, the cross-modal joint feature representation sequence is used as the main feature input, and the original modal association weight sequence is used as the prior attention reference. These are all input into the clinical reasoning model to perform clinical task reasoning and output preliminary inference results, examination item suggestions, and risk warnings. By integrating the preliminary inference results, suggested examination items, and risk warnings, and using the list of related medical entities as evidence, the auxiliary diagnostic prompt information is generated; The method further includes: The original modality association weight sequence is used as an attention bias term and introduced into the self-attention module of the clinical reasoning model; The self-attention module, under the modulation of the attention bias term, performs global context modeling on the subject feature input, generates a deep semantic feature sequence, and then calculates the correlation between the deep semantic feature sequence and the entity feature vector through a cross-attention mechanism to generate an entity-enhanced feature representation. The entity-enhanced feature representation is input into the differential diagnosis inference head, examination suggestion inference head, and risk warning inference head at the output of the self-attention module, and clinical task inference is performed in parallel to output the preliminary inference results, examination item suggestions, and risk warnings.

2. The multimodal doctor-patient interaction clinical assistance method based on privacy computing as described in claim 1, characterized in that, The method uses the feedback from medical staff to the auxiliary diagnostic prompts as a supervisory signal, and combines it with the structured diagnostic data unit to drive the clinical reasoning model to optimize model parameters based on local incremental training. After displaying the auxiliary diagnostic prompts on the doctor's interactive terminal, the feedback from medical staff is captured. If the feedback operation is an adoption-type feedback, then the auxiliary diagnostic prompt information is used as a positive supervision signal and paired with the structured diagnosis and treatment data unit to form a training sample; If the feedback operation is a correction and editing type feedback, then the corrected diagnostic prompt information is used as a positive supervision signal and paired with the structured diagnosis and treatment data unit to form a training sample; Using the training samples, the clinical reasoning model is optimized based on local incremental training.

3. The multimodal doctor-patient interaction clinical assistance method based on privacy computing as described in claim 2, characterized in that, If the feedback operation is an ignore rejection type feedback, then the auxiliary diagnostic prompt information is used as a negative supervision signal and paired with the structured diagnosis and treatment data unit to form a training sample.

4. The multimodal doctor-patient interaction clinical assistance method based on privacy computing as described in claim 1, characterized in that, The first participating institution uses a privacy computing protocol to encrypt and de-identify the model parameter updates of the clinical inference model before sending them to the federated learning coordination node. The method includes: Obtain the update amount of the model parameters of the clinical reasoning model; After encrypting and desensitizing the model parameter update data using a privacy computing protocol, a privacy data packet is encapsulated. The privacy data packet is sent to the federated learning coordination node via a secure communication channel.

5. A multimodal doctor-patient interaction clinical assistance system based on privacy computing, characterized in that, The system is used to implement the privacy-preserving computation-based multimodal doctor-patient interaction clinical assistance method according to any one of claims 1-4, the system comprising: Feature association component: Performs cross-modal semantic feature association on the holographic record of the diagnosis and treatment scenario to construct structured diagnosis and treatment data units; Clinical task reasoning component: The structured diagnosis and treatment data unit is input into the clinical reasoning model deployed in the first participating institution. After cross-modal association capture based on the attention mechanism, clinical task reasoning is performed and auxiliary diagnostic prompts are output. Model parameter optimization component: The feedback operation of medical staff to the auxiliary diagnostic prompts is used as a supervision signal, and combined with the structured diagnosis and treatment data unit, it drives the clinical reasoning model to optimize the model parameters based on local incremental training; Encryption and desensitization component: The first participating institution uses a privacy computing protocol to encrypt and desensitize the model parameter update of the clinical reasoning model before sending it to the federated learning coordination node; Secure aggregation component: The federated learning coordination node receives and aggregates multiple model parameter updates from multiple participating institutions, performs secure aggregation calculations, and generates a federated aggregation model; Model deployment component: The federated learning coordination node distributes the federated aggregated model to the multiple participating institutions for model deployment and updates.

Images