Intelligent structured medical record generation method and system based on multi-modal doctor-patient interaction

Abstract

The application provides a kind of intelligent structured medical record generation method and system based on multimodal doctor-patient interaction, which comprises: real-time acquisition of dialogue voice and transcription into text sequence, while recognizing visual attention entity by listening to mouse operation in electronic medical record system.Based on the history of visual attention entity and text sequence, a logical demonstration track is constructed, and an implicit reward function is derived from it using a reverse reinforcement learning algorithm. Use the function to calculate the action reward value of each combination of visual attention entity and dialogue text, select the highest value combination as the optimal alignment strategy to determine the timing causal relationship. Map the entity and text to the medical knowledge graph, extract the shortest semantic path as the implicit clinical reasoning chain, and generate structured electronic medical record. The application deduces the diagnosis and treatment decision logic from the doctor's multimodal behavior through reverse reinforcement learning, solving the technical problem that traditional methods cannot establish the internal causal relationship between the doctor's visual attention focus and spoken content.

Description

Technical Field

[0001] This invention belongs to the field of electronic medical record generation technology, specifically relating to an intelligent structured medical record generation method and system based on multimodal doctor-patient interaction. Background Technology

[0002] In current healthcare IT practices, to improve clinical efficiency, technological means are commonly used to collect multimodal data in diagnostic and treatment interactions, aiming to assist or automate the generation of electronic medical records. However, existing technologies have general limitations in comprehensively utilizing multimodal raw information flows to generate structured medical records. Specifically, current methods process and present information flows from different modalities in isolation, failing to reveal the inherent logical connections and causal orientations between these concurrent or adjacent information points. This results in information records generated by existing technologies lacking deep contextual coherence and failing to accurately reflect the complete cognitive process of physicians making corresponding decisions based on specific information inputs during diagnosis and treatment, thus affecting the medical logical rigor of the final generated medical records. Summary of the Invention

[0003] This invention provides an intelligent structured medical record generation method and system based on multimodal doctor-patient interaction to solve the above-mentioned technical problems.

[0004] In a first aspect, the present invention provides an intelligent structured medical record generation method based on multimodal doctor-patient interaction, the method comprising the following steps:

[0005] The doctor-patient dialogue voice is collected in real time during the doctor-patient interaction process, and the dialogue text generated by the transcribed doctor-patient dialogue voice is mapped to the semantic vector space to obtain the dialogue text sequence.

[0006] During doctor-patient interaction, the screen coordinates of the mouse pointer are monitored in real time when the doctor performs mouse operations in the electronic medical record system. The interface content of the user interface control below the screen coordinates is obtained as candidate data by calling the user interface tree interface of the electronic medical record system.

[0007] A context vector is constructed based on the current context information of the doctor-patient interaction process. The first semantic feature containing candidate data and the second semantic feature after occluding candidate data are calculated based on the context vector. When the semantic offset between the first semantic feature and the second semantic feature exceeds the preset salience threshold, the candidate data is marked as a visually significant entity.

[0008] Based on the correlation features between visually focused entities and dialogue text sequences, and combined with the collected historical focus entity sequences and historical text sequences, a logical demonstration trajectory is generated.

[0009] Based on the logical demonstration trajectory, the implicit reward function representing the diagnosis and treatment decision logic is derived by using the inverse reinforcement learning algorithm.

[0010] The action reward value of each candidate combination of visual attention entity and dialogue text sequence is calculated based on the implicit reward function. The combination with the highest action reward value is selected as the optimal alignment strategy to align the temporal causal relationship between visual attention entity and dialogue text sequence.

[0011] The aligned visual attention entities and dialogue text sequences are mapped to the first and second nodes in a pre-defined medical knowledge graph, respectively. A path search algorithm is then executed in the medical knowledge graph to obtain the shortest semantic path connecting the first and second nodes as an implicit clinical reasoning chain.

[0012] By combining visually focused entities, dialogue text sequences, and implicit clinical reasoning chains, structured electronic medical record texts are generated after doctor-patient interactions.

[0013] Optionally, the step of constructing a context vector based on the current context information of the doctor-patient interaction process, and calculating a first semantic feature containing candidate data and a second semantic feature after masking the candidate data based on the context vector, and marking the candidate data as a visually significant entity when the semantic offset between the first semantic feature and the second semantic feature exceeds a preset saliency threshold, includes the following steps:

[0014] A context vector is constructed based on the current context information of the doctor-patient interaction process. Global semantic features are extracted from the context vector by an encoder. The text embedding vector of the candidate data is concatenated with the global semantic features to generate the first semantic feature containing the candidate data.

[0015] Construct a zero-mask vector with the same dimension as the candidate data, and use the zero-mask vector to replace the candidate data positions in the context vector to form a masking sequence;

[0016] The masking sequence is input into the encoder. After being processed by the multi-layer self-attention mechanism in the encoder, the hidden state vector of the output layer is extracted to generate the second semantic feature after masking candidate data.

[0017] Calculate the Euclidean distance and cosine similarity between the first semantic feature and the second semantic feature in their respective semantic vector spaces, and calculate the weighted sum of the Euclidean distance and cosine similarity to obtain the semantic offset.

[0018] If the semantic offset exceeds the preset saliency threshold, the candidate data will be marked as a visually relevant entity.

[0019] Optionally, the construction of a context vector based on the current context information of the doctor-patient interaction process, and the extraction of global semantic features from the context vector by the encoder, includes the following steps:

[0020] Collect current context information of the doctor-patient interaction process, including electronic medical record text fragments, patient's medical history summary, and dialogue text within a preset time window;

[0021] The current context information is converted into a word sequence by a word segmentation tool and then mapped into a context vector. Different positional encoding and modality type encoding are added to each context vector according to different source attributes.

[0022] All context vectors are input into the encoder based on the Transformer architecture, and the vector representation of the CLS special terms output by the last layer of the encoder is extracted as global semantic features.

[0023] Optionally, the implicit reward function representing the diagnostic and treatment decision logic, derived by using a reverse reinforcement learning algorithm based on the logical demonstration trajectory, includes the following steps:

[0024] Define the state space as the set of semantic categories of visually concerned entities, and define the action space as the set of intent categories of the dialogue text sequence;

[0025] The feature extraction function is used to calculate the combined association features between each pair of states in the state space and actions in the action space. The combined association features include at least semantic similarity features, time interval decay features, and medical atlas path distance features.

[0026] Initialize the weight parameters of the implicit reward function, extract the trajectory association features of the logical demonstration trajectory using the feature extraction function, and calculate the expected empirical features of the logical demonstration trajectory by combining the trajectory association features and the weight parameters.

[0027] The maximum entropy principle is applied to maximize the likelihood probability of the logical demonstration trajectory under the implicit reward function, while constraining the probability distribution entropy of the unobserved trajectory under the implicit reward function.

[0028] The gradient ascent algorithm is used to iteratively update the weight parameters until the expected policy features derived from the updated weight parameters converge to the expected empirical features, and the implicit reward function after convergence is determined.

[0029] Optionally, the step of calculating the action reward value of each candidate combination of the visually concerned entity and the dialogue text sequence based on the implicit reward function, and selecting the combination with the highest action reward value as the optimal alignment strategy to align the temporal causal relationship between the visually concerned entity and the dialogue text sequence includes the following steps:

[0030] The system acquires newly generated visually focused entities as the current state and multiple dialogue text sequences within a preset time window as a set of candidate actions.

[0031] The feature extraction function is called to calculate the real-time feature vector of the current state and each combination of candidate actions in the candidate action set;

[0032] The real-time feature vector is input into the implicit reward function, and the non-normalized reward value is obtained by calculating the linear weighted sum of the weight parameters and the real-time feature vector.

[0033] A state transition probability model is introduced, and the cumulative expected reward for each candidate action in the current state is calculated based on the reward value. After normalizing all the cumulative expected rewards, an alignment probability distribution is generated. The target candidate action with the highest alignment probability is selected and paired with the current state. Temporal causal association labels are generated for the dialogue text sequence and visual attention entity corresponding to the target candidate action and the current state, respectively.

[0034] Optionally, the step of mapping the aligned visual attention entities and dialogue text sequences to a first node and a second node in a preset medical knowledge graph, respectively, and executing a path search algorithm in the medical knowledge graph to obtain the shortest semantic path connecting the first node and the second node as an implicit clinical reasoning chain includes the following steps:

[0035] Entity linking technology is used to map visually focused entities and dialogue text sequences to their corresponding unique entity IDs in the medical knowledge graph, which are then identified as the first and second nodes.

[0036] Perform a forward breadth-first search starting from the first node, and simultaneously perform a backward breadth-first search starting from the second node;

[0037] When a common intersection node is detected between the forward breadth-first search path and the backward breadth-first search path, the search is stopped and a candidate reasoning path consisting of the first node, the common intersection node, and the second node is extracted.

[0038] Invalid paths containing non-clinical logical relationships are filtered out based on the relation type attribute of each edge in the candidate reasoning path;

[0039] Calculate the sum of the weights of all edges in the remaining valid candidate reasoning paths, and select the valid candidate reasoning path with the smallest sum of weights as the most semantic path.

[0040] The node names and edge relationship names in the shortest semantic path are filled into a preset natural language generation template to generate an implicit clinical reasoning chain.

[0041] Optionally, filtering out invalid paths containing non-clinical logical relationships based on the relation type attribute of each edge in the candidate inference path includes the following steps:

[0042] A whitelist of clinical logical relationships is predefined, which includes at least causal relationships, concurrent relationships, treatment relationships, and contraindication relationships.

[0043] For any candidate reasoning path, traverse each connecting edge of the candidate reasoning path and read the relation type label of the connecting edge in the medical knowledge graph;

[0044] If the relation type label is not in the clinical logical relation whitelist, the candidate reasoning path will be filtered out.

[0045] Optionally, the step of generating the structured electronic medical record text after doctor-patient interaction by combining visually focused entities, dialogue text sequences, and implicit clinical reasoning chains includes the following steps:

[0046] Structured cue words are generated by combining visually focused entities, dialogue text sequences, and implicit clinical reasoning chains;

[0047] The structured prompts are input into a pre-defined large language model, and the text stream output by the large language model is parsed into structured electronic medical record text after doctor-patient interaction.

[0048] In a second aspect, the present invention also provides an intelligent structured medical record generation system based on multimodal doctor-patient interaction, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the intelligent structured medical record generation method based on multimodal doctor-patient interaction as described in any of the first aspects.

[0049] Thirdly, the present invention also provides a computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform an intelligent structured medical record generation method based on multimodal doctor-patient interaction according to any one of the first aspects.

[0050] The beneficial effects of this invention are:

[0051] This invention establishes a technical framework for identifying and quantifying the inherent logical connections between specific user interface attention behaviors and synchronous doctor-patient dialogue text during doctor-patient interactions through structured extraction and analysis. Based on an inverse reinforcement learning algorithm model, this invention learns and derives a quantitative evaluation function representing the inherent laws of clinical diagnosis and treatment decisions from historical multimodal interaction behavior sequences. Based on this evaluation function, the temporal causal relationship alignment of real-time acquired visual attention entities and dialogue text content is optimized. Furthermore, by mapping the aligned information entities to a pre-defined medical knowledge graph and performing semantic path analysis, a traceable clinical reasoning basis is constructed for the generated medical record text. In summary, this invention overcomes the technical shortcomings of existing technologies that lack contextual coherence and logical rigor in medical record content due to the inability to establish inherent logical connections between multimodal data, significantly improving the accuracy and interpretability of the generated structured electronic medical records and greatly reducing the time doctors spend modifying electronic medical records after generation. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating one embodiment of the intelligent structured medical record generation method based on multimodal doctor-patient interaction in this application.

[0053] Figure 2 This is a schematic diagram of the process topology of an intelligent structured medical record generation method based on multimodal doctor-patient interaction in one embodiment of this application.

[0054] Figure 3 This is a schematic diagram of the flow topology of the structured electronic medical record generation process in one embodiment of this application. Detailed Implementation

[0055] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0056] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and that the objects distinguished by "first," "second," etc., are of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0057] Figure 1 This is a flowchart illustrating an intelligent structured medical record generation method based on multimodal doctor-patient interaction in one embodiment. It should be understood that, although... Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but may be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps. (See reference...) Figure 1 and Figure 2 The intelligent structured medical record generation method based on multimodal doctor-patient interaction disclosed in this invention specifically includes the following steps:

[0058] S101. During the doctor-patient interaction process, the doctor-patient dialogue voice is collected in real time, and the dialogue text generated by the transcribed doctor-patient dialogue voice is mapped to the semantic vector space to obtain the dialogue text sequence.

[0059] The process involves deploying a high-sensitivity omnidirectional microphone array in the consultation room environment to capture the sound signals of doctors and patients during free conversation. Beamforming technology is used to suppress ambient background noise and enhance the speaker's voice gain in the main direction. An automatic speech recognition engine extracts acoustic features from the denoised doctor-patient dialogue, discretizing the continuous analog audio signal into a phoneme sequence, and then decoding it into the original dialogue record in natural language text form. To enable the computer to understand and process this text data, a pre-trained language model, such as BERT or its medical variant ClinicalBERT, is introduced to map discrete words or characters into a high-dimensional continuous real-valued vector space. In this process, the language model uses the self-attention mechanism of deep neural networks to capture long-distance dependencies between words. Each generated vector not only contains the word's meaning but also its syntactic and semantic information in the current context. The final dialogue text sequence consists of a series of semantic vectors arranged chronologically, which accurately represent key information in doctor-patient communication, such as symptom descriptions, medical history inquiries, and diagnostic suggestions.

[0060] S102. During doctor-patient interaction, monitor the screen coordinates of the mouse pointer when the doctor performs mouse operations in the electronic medical record system in real time, and obtain the interface content of the user interface controls below the screen coordinates as candidate data by calling the user interface tree interface of the electronic medical record system.

[0061] While the doctor operates the electronic medical record system, a background process listens for hardware interrupt signals from the mouse input device in real time through low-level hook functions or accessibility APIs provided by the operating system, using high-frequency polling. When a mouse movement or click event is detected, the absolute position coordinates of the current cursor in the monitor's screen coordinate system are immediately captured. Subsequently, the program traverses the UI component hierarchy of the currently active window by calling the user interface tree interface exposed by the electronic medical record application. This traversal process uses a hit-test algorithm, recursively searching from the root node until the lowest-level leaf node control whose bounding box covers the coordinates (X, Y). Once the target control is locked, its attribute information is read, including but not limited to control type, label name, and the text value it contains. This unstructured or semi-structured data extracted from the user interface is defined as candidate data. This step maps the physical two-dimensional coordinates of the mouse to logical medical semantic objects, thereby understanding the underlying intent of the doctor's gaze focus. For example, when the doctor hovers the mouse over the checkbox for "hypertension history," not only is the mouse position recorded, but the medical concept of "hypertension" is also extracted. The extracted candidate data was then standardized to remove format specifiers and converted into plain text strings, preparing for subsequent semantic analysis and realizing the initial transformation of doctors' visual attention into data.

[0062] S103. Construct a context vector based on the current context information of the doctor-patient interaction process. Calculate the first semantic feature when candidate data is included and the second semantic feature after candidate data is masked based on the context vector. When the semantic offset between the first semantic feature and the second semantic feature exceeds a preset salience threshold, mark the candidate data as a visually relevant entity.

[0063] The process of constructing the context vector aims to simulate the human mechanism for judging the importance of information during cognition. First, multimodal information before and after the current interaction moment is collected, including the transcribed dialogue text fragments and the text fields already entered on the electronic medical record interface. These heterogeneous data are concatenated and encoded into a unified high-dimensional context vector. To quantify the importance of a specific screen content to the current diagnostic context, a masking-contrast technique is employed. First, complete information containing the embedded representation of candidate data is input into a semantic encoder to calculate the first semantic feature, which represents the doctor's complete cognitive state when seeing the data. Then, a zero-mask vector or a specific masking marker, with the exact same dimension as the candidate data, is artificially constructed to replace the vector fragments in the original context vector corresponding to the candidate data position, forming a masking sequence. This masking sequence is then input into the same encoder to obtain the second semantic feature, which represents the cognitive state when the specific visual information is missing. The difference between the two is calculated as the semantic offset. If the semantic offset exceeds a preset saliency threshold, it indicates that the candidate data is crucial and indispensable for understanding the current context, and therefore it is identified as a visually significant entity.

[0064] S104. Based on the correlation features between visually focused entities and dialogue text sequences, and combined with the collected historical focus entity sequences and historical text sequences, a logical demonstration trajectory is generated.

[0065] The first step involves extracting deep correlation features between visually focused entities and the dialogue text sequence. These features include not only semantic similarity but also temporal synchronicity—the time difference between when the doctor looks at a particular data point and when they discuss the related topic. Using a sliding time window algorithm, the visually focused entity captured at each moment is paired with neighboring dialogue text fragments. This is combined with the collected historical focus entity sequence (representing the doctor's previous visual path) and historical text sequence (representing the previous dialogue context) to construct a temporal tuple sequence containing states and actions. In this sequence, each element represents the doctor's cognitive state at a specific moment and their subsequent decision-making behavior. This trajectory implicitly represents the causal logical chain from acquiring information (reviewing medical records, consulting) to forming a judgment (making a diagnosis, prescribing medication). These multimodal data are rigorously sorted by timestamps and serialized to form a logically demonstrative trajectory that can be understood by machine learning algorithms. This trajectory is formally represented as a set of state-action pairs.

[0066] S105. Based on the logical demonstration trajectory, the implicit reward function representing the diagnosis and treatment decision logic is derived by using the inverse reinforcement learning algorithm.

[0067] The core objective of the inverse reinforcement learning algorithm is not to directly learn the doctor's behavior, but to deduce the implicit reward function governing these behaviors. This is based on the fundamental assumption that every diagnostic and treatment action taken by the doctor aims to maximize some potential clinical benefit (such as diagnostic accuracy or treatment effectiveness). In this scheme, a maximum entropy inverse reinforcement learning framework is used, with logical demonstration trajectories serving as expert examples. The algorithm attempts to find a reward function that maximizes the probability of the expert trajectory occurring while maintaining the maximum entropy (i.e., the fewest assumptions) for unobserved trajectories. By defining a feature extraction function, states and actions are mapped to feature vectors, and the implicit reward function is parameterized as a linear combination of weight vectors and feature vectors. During training, the weights are iteratively updated so that the expected features generated by the strategy based on the current reward function gradually approximate the expected features of the expert demonstration trajectory. When the two converge, the resulting implicit reward function quantifies the rationality of taking specific diagnostic and treatment actions under specific medical conditions.

[0068] S106. Calculate the action reward value of each candidate combination of visual attention entity and dialogue text sequence based on implicit reward function, and select the combination with the highest action reward value as the optimal alignment strategy to align the temporal causal relationship between visual attention entity and dialogue text sequence.

[0069] In this step, after obtaining the implicit reward function representing the logic of diagnostic and treatment decisions, the function is used to address the temporal misalignment between visual attention and verbal expression. In actual clinical practice, doctors may see abnormal values ​​on lab reports first, and only after a few seconds of consideration do they ask the patient about related symptoms. This time lag often leads to inaccurate direct temporal alignment. Therefore, this step adopts a value-based alignment strategy. For each visually attentive entity, multiple dialogue text fragments within its temporal neighborhood are selected as candidate combinations. Each combination is input into the trained implicit reward function to calculate its action reward value. This reward value reflects the clinically logical rationality of associating the visual entity with the text fragment. This essentially uses learned expert logic to correct temporal biases and confirm the causal relationship that "the doctor looks at this data because he wants to ask this question." In this way, a precise temporal causal relationship is established between visually attentive entities and the dialogue text sequence.

[0070] S107. Map the aligned visual attention entities and dialogue text sequences to the first and second nodes in the preset medical knowledge graph, respectively. Execute a path search algorithm in the medical knowledge graph to obtain the shortest semantic path connecting the first and second nodes as the implicit clinical reasoning chain.

[0071] First, entity linking technology is used to map aligned visually relevant entities and keywords in the dialogue text sequence to standard concept nodes in a pre-defined medical knowledge graph, determining the first node (e.g., "elevated white blood cell count") and the second node (e.g., "bacterial infection"). The medical knowledge graph is a vast semantic network containing entities such as diseases, symptoms, drugs, and examinations, along with their interrelationships. Next, pathfinding algorithms, such as bidirectional breadth-first search or A / B search, are executed within the graph. The algorithm finds the shortest semantic path connecting the first and second nodes. This path is not necessarily the shortest in terms of physical distance, but rather the path with the smallest sum of semantic weights, representing the most direct deduction process in medical logic. The searched path is the implicit clinical reasoning chain, demonstrating how doctors deduce the final diagnostic conclusion from observed phenomena, filling the logical gaps missing in the multimodal interaction records.

[0072] S108. Combine visual attention entities, dialogue text sequences, and implicit clinical reasoning chains to generate structured electronic medical record text after doctor-patient interaction.

[0073] The process involves extracting high-confidence visually relevant entities (representing objective findings), semantically aligned dialogue text sequences (representing subjective requests and consultation information), and mined implicit clinical reasoning chains (representing diagnostic basis and logic). These are then hierarchically organized according to the standard paragraph structure of electronic medical records. The structured data is then input into a large language model fine-tuned from a medical corpus. This model utilizes its language organization capabilities to refine the fragmented entities and logical chains into a coherent and professional medical narrative. In this way, the generated electronic medical record retains the rich details of multimodal interaction while possessing a rigorous medical logical structure, ultimately outputting a complete structured electronic medical record text.

[0074] In one implementation, obtaining the interface content of the user interface controls below the screen coordinates as candidate data by calling the user interface tree interface of the electronic medical record system includes the following steps:

[0075] Before calling the user interface tree interface, monitor the rendering state properties of the user interface controls below the screen coordinates in real time.

[0076] If the rendering state property is marked as loading or placeholder, a dynamic waiting mechanism based on visual changes is initiated until the pixel redraw event of the user interface control is detected to stop and the rendering state property changes to ready.

[0077] Detect whether the content of the modified user interface controls contains privacy-de-identifying mask features;

[0078] If the data contains privacy-de-identifying mask features, a click command is simulated to activate the temporary plaintext display logic of the electronic medical record system, and the interface content at the moment the plaintext display window pops up is captured. The interface content that is ready to be rendered and has no de-identification mask is extracted as candidate data.

[0079] In this implementation, before calling the user interface tree interface of the electronic medical record system to obtain the content of the control below the screen coordinates, it must be ensured that the control has completed data loading and graphics rendering; otherwise, null values ​​or incorrect intermediate state data may be obtained. The top-level UI element object at the current mouse pointer coordinates (x, y) is periodically queried through the operating system's underlying Accessibility API (such as UIAutomation under Windows or Accessibility under macOS) or through a hook program directly injected into the electronic medical record system process. After obtaining the object, the "rendering state" field in its internal property list is read (e.g., the `IsOffscreen` property is inverted). This property is an enumeration value indicating that the control is currently in a state such as "uninitialized," "loading," "rendering," or "ready." For electronic medical record systems built using Web technology stacks, the `readyState` property of the element can be queried through the DOM tree interface provided by the browser kernel, or by checking key CSS class names (such as `loading-spinner`). If the detected state property indicates that the control has not yet fully displayed its content, the subsequent data extraction process is paused to avoid reading dirty data due to asynchronous loading.

[0080] To address the common UI lag issues in electronic medical record systems caused by network latency or large data loading, simple state attribute checks may be insufficient to handle all situations. Therefore, a dynamic waiting mechanism based on visual changes is further employed. This mechanism allocates a buffer in memory to continuously capture screenshots of the target control area. By comparing the pixel matrix differences between adjacent frames, it determines whether the interface is still undergoing visual changes. The image matrix of the current frame is then calculated. Image matrix of the previous frame Difference between The calculation formula is: , where I(i,j) represents the pixel grayscale value at coordinates (i,j) in the image. A very small pixel change threshold is set. (For example, 0 or a positive number close to 0). If This indicates that the control area is still undergoing pixel redrawing (e.g., during a loading animation rotation, or text being rendered line by line). At this point, the program enters a waiting loop, continuously monitoring the next frame. Only when K consecutive frames (e.g., 5 frames) are reached... Furthermore, the control is only considered stable when the aforementioned rendering state attribute changes to "ready state". This dual confirmation mechanism combines low-level attribute checking with high-level visual perception, effectively solving the data synchronization problem in asynchronous loading scenarios and ensuring that the extracted content is the complete information ultimately presented to the doctor.

[0081] Due to the sensitivity of medical data, many electronic medical record systems anonymize sensitive fields such as patient names, often by using asterisks (…). The extracted text string can be partially or entirely replaced by specific mask characters. Therefore, regular expression matching techniques can be used to scan the initial extracted text content. A privacy-de-identifying mask feature library is defined, containing common de-identification patterns. Then, a pattern matching algorithm is used to check whether the text contains consecutive mask characters or substrings that conform to specific de-identification rules. Let the extracted text string be... The desensitization feature set is If it exists Make p match If the substring in the string is selected, it is determined that the content of the control contains privacy masking features. In this case, if the substring is directly extracted... Using this data as training data or as the basis for generating medical records will result in the loss of crucial information, severely impacting the performance of subsequent models. Once the content of a control is detected to contain privacy-de-identifying mask features, proactive interactive methods must be employed to activate the system's temporary plaintext display logic.

[0082] Specifically, when a user clicks or hovers over a masked field, the system either pops up a pop-up containing the complete information or temporarily replaces the asterisk with plaintext. Using the operating system's input simulation interface (such as `SendInput` or `Robot` classes), a simulated left-click command or mouse hover event is sent to the center coordinates of the target control. Immediately afterward, the program initiates a high-frequency screen capture mode, listening for newly added window handles or redraw events of existing control areas at millisecond intervals. When a new pop-up window or a change in control text is detected, the aforementioned privacy feature detection is performed again. Once it is detected that the new interface content no longer contains the mask (i.e., becomes plaintext), the text content at that instant is immediately captured and saved as candidate data. This process needs to be completed within a very short time window to avoid affecting the doctor's normal operating experience and is executed silently in the background.

[0083] In one implementation, a context vector is constructed based on the current context information of the doctor-patient interaction process. A first semantic feature containing candidate data and a second semantic feature after masking the candidate data are calculated based on the context vector. When the semantic offset between the first and second semantic features exceeds a preset salience threshold, the candidate data is marked as a visually significant entity, including the following steps:

[0084] A context vector is constructed based on the current context information of the doctor-patient interaction process. Global semantic features are extracted from the context vector by an encoder. The text embedding vector of the candidate data is concatenated with the global semantic features to generate the first semantic feature containing the candidate data.

[0085] Construct a zero-mask vector with the same dimension as the candidate data, and use the zero-mask vector to replace the candidate data positions in the context vector to form a masking sequence;

[0086] The masking sequence is input into the encoder. After being processed by the multi-layer self-attention mechanism in the encoder, the hidden state vector of the output layer is extracted to generate the second semantic feature after masking candidate data.

[0087] Calculate the Euclidean distance and cosine similarity between the first semantic feature and the second semantic feature in their respective semantic vector spaces, and calculate the weighted sum of the Euclidean distance and cosine similarity to obtain the semantic offset.

[0088] If the semantic offset exceeds the preset saliency threshold, the candidate data will be marked as a visually relevant entity.

[0089] In this embodiment, the primary task in constructing the context-aware module of the multimodal doctor-patient interaction system is to fuse the discretely distributed heterogeneous data of the current moment into a unified, computer-understandable representation. This process begins with the real-time capture of comprehensive information from the doctor-patient interaction scenario, encompassing transcribed texts of doctor-patient dialogues that have already occurred within a preset time window, historical text fragments of doctors' operations on the electronic medical record interface, and basic electronic file summaries of patients. To enable these textual information sources with different structures to be computed on the same dimension, a pre-trained language model based on the Transformer architecture is used as the basic encoder. First, a word segmentation tool is used to convert all the above textual information into a standardized sequence of tokens, and corresponding positional encodings are added to preserve temporal information. This sequence is then input into the encoder, which uses its internal multi-layer self-attention mechanism to capture long-distance dependencies, outputting a vector that can represent the overall semantic state of the current diagnosis and treatment scenario, i.e., global semantic features. Meanwhile, for the screen candidate data where the doctor's mouse is currently hovering or selected, detected by the system, it is independently mapped into text embedding vectors through a shared-weight embedding layer. To simulate the complete cognitive state a doctor would form while viewing specific data, it is necessary to deeply integrate the candidate data with the global context. This can be achieved by embedding text vectors into global semantic features through vector concatenation or feature-weighted fusion, generating a first semantic feature that includes the candidate data. This feature can be mathematically represented as ,in Representative feature fusion function.

[0090] In the inference process of deep neural networks, the completeness of the input data directly determines the semantic distribution of the output features. To simulate the hypothetical scenario of "how a doctor would understand the condition if they hadn't seen this data," a state of missing information needs to be artificially constructed. Specifically, a zero-mask vector with the same dimension as the aforementioned text embedding vector is first created, with all elements initialized to zero, or represented using a vector corresponding to a specific [MASK] tag in the model vocabulary. Then, in the constructed context input sequence containing all information, the word position index corresponding to the candidate data is precisely located. The original candidate data vector at that position is replaced with the constructed zero-mask vector, thus forming a masked sequence. This operation erases the semantic information of the target data while maintaining the total length and temporal structure of the input sequence, creating a local information gap at the information theory level. Next, the constructed masked sequence is input into a deep learning encoder, which contains multiple stacked self-attention modules that dynamically aggregate information by calculating the mutual attention between words in the sequence. When the masked sequence enters the model, because the candidate data positions are replaced by zero masks, the model cannot obtain effective semantic support when calculating the attention weights of that position with other contextual information. This forces the attention flow that should have been focused on that data to be redistributed to other parts of the sequence (such as historical dialogue or past medical history). After layer-by-layer nonlinear transformation and feature extraction in the deep encoder network, the hidden state vector is extracted from the output of the last layer of the model, or a special classification word vector representing the semantics of the entire sentence is extracted as the second semantic feature after masking the candidate data. The generation process of this feature can be formally described as... ,in This represents the forward propagation mapping function of the encoder.

[0091] After obtaining the first and second semantic features, a composite metric strategy combining geometric distance and directional similarity is adopted. Euclidean distance sensitively reflects the magnitude of the absolute positional shift of a vector in space, while cosine similarity excels at measuring the semantic consistency of vectors; the two complement each other. First, the Euclidean norm distance between the two feature vectors is calculated, and then the cosine value of the angle between them is calculated. To unify these two indicators with different dimensions into a comprehensive indicator, the semantic offset G is defined as a weighted combination of the two. The specific calculation formula is as follows: ,in It is a preset balancing weight coefficient used to adjust the proportion of distance and direction metrics in the total offset. This represents the L2 norm of the vector. Mathematical operations transform the abstract concept of "attention" into a concrete numerical value. A large G value indicates that occluding the data has caused significant oscillations in the semantic space; conversely, a small G value indicates that the data has minimal impact on the current context. This composite calculation method effectively overcomes the measurement failure problem that may occur with single indicators in high-dimensional sparse spaces, ensuring that the offset truly reflects the semantic value of the candidate data.

[0092] Finally, binarization is performed based on the calculated semantic offset to complete the final labeling of visually significant entities. An empirical or statistically derived saliency threshold is set. The calculated semantic offset G is compared with this threshold. If... If the first semantic feature and the second semantic feature are clearly defined, a significant semantic gap is determined, meaning that the existence of candidate data greatly reshapes the current diagnostic and treatment context. In clinical logic, this is interpreted as the data point playing a decisive role in the doctor's understanding of the condition and making judgments, thus constituting a high-value diagnostic and treatment clue. Therefore, the system formally marks the candidate data as a visually relevant entity and incorporates it into the subsequent medical record generation logic flow. Conversely, if... If the addition of such data does not cause significant changes in the semantic space, it is considered to be part of routine browsing or background information and is therefore removed.

[0093] In one implementation, a context vector is constructed based on the current context information of the doctor-patient interaction process, and the global semantic features extracted from the context vector by the encoder include the following steps:

[0094] Collect current context information of the doctor-patient interaction process, including electronic medical record text fragments, patient's medical history summary, and dialogue text within a preset time window;

[0095] The current context information is converted into a word sequence by a word segmentation tool and then mapped into a context vector. Different positional encoding and modality type encoding are added to each context vector according to different source attributes.

[0096] All context vectors are input into the encoder based on the Transformer architecture, and the vector representation of the CLS special terms output by the last layer of the encoder is extracted as global semantic features.

[0097] In this implementation, the first step is to comprehensively collect and integrate diverse diagnostic and treatment information at the current moment. Specifically, this collection process is not a simple data capture, but rather involves extracting key content from different data sources through a multi-threaded real-time monitoring mechanism. First, a text extraction interface is used to capture text fragments in the doctor's current operating area on the electronic medical record system interface in real time, representing the doctor's current visual focus and operational intent. Second, a backend database query interface is used to retrieve historical medical history summaries from the patient's file, including static background information such as previously diagnosed diseases, surgical history, and allergy history, providing a long-term reference benchmark for the current diagnosis. Finally, the buffer of the speech recognition module is synchronously invoked to extract the doctor-patient dialogue text record within a preset time window before the current moment, reflecting the immediate communication content and symptom descriptions of both parties. These three types of unstructured text data, from different sources and with varying timeliness, are cleaned and standardized, removing irrelevant formatting symbols and redundant whitespace. Then, they are concatenated according to the logical order of "past history - current operation - real-time dialogue" to form a complete current context information stream.

[0098] Deep learning-based word segmentation tools (such as WordPiece or Byte-Pair Encoding algorithms) are used to divide the concatenated current context information into the smallest semantic units, i.e., word sequence. To address the semantic confusion that may arise from mixing multi-source data, while mapping word elements to basic word embedding vectors, not only is a positional encoding representing the relative or absolute position of each word element in the sequence added to preserve word order information, but a specific modality type encoding is also superimposed on each word element based on its source attributes (past history, current operation, dialogue text). Let the basic embedding vector of the i-th word element be... The position encoding vector is The modality type encoding vector is The final generated context vector It is the element-wise addition of these three: This encoding mechanism allows the same word to have completely different vector representations depending on whether it appears in the past or in the current conversation, thus enabling the model to easily distinguish the semantic boundary between "past illness" and "current symptoms".

[0099] After vectorization, the sequence of all context vectors is input into a deep neural network encoder based on the Transformer architecture to extract highly abstract semantic features. This encoder consists of multiple stacked Transformer Encoder Blocks, each containing a multi-head self-attention mechanism and a feedforward neural network. As information flows through each layer of the network, the self-attention mechanism calculates the association weights between any two lexical units in the sequence, allowing the model to dynamically focus on the most crucial information segments for understanding the current context, regardless of their distance within the sequence. Specifically, a special categorical lexical unit (labeled [CLS]) is added at the beginning of the input sequence. This lexical unit does not represent any specific word meaning but is specifically designed to aggregate global semantic information for the entire sequence. Through deep interaction and feature fusion across multiple network layers, the vector at the [CLS] position gradually absorbs the semantic essence of all other lexical units in the sequence. Finally, the vector corresponding to the [CLS] position in the output of the last layer of the encoder is extracted as the global semantic feature.

[0100] In one implementation, deriving the implicit reward function representing the diagnostic decision-making logic based on the logical demonstration trajectory and using an inverse reinforcement learning algorithm includes the following steps:

[0101] Define the state space as the set of semantic categories of visually concerned entities, and define the action space as the set of intent categories of the dialogue text sequence;

[0102] The feature extraction function is used to calculate the combined association features between each pair of states in the state space and actions in the action space. The combined association features include at least semantic similarity features, time interval decay features, and medical atlas path distance features.

[0103] Initialize the weight parameters of the implicit reward function, extract the trajectory association features of the logical demonstration trajectory using the feature extraction function, and calculate the expected empirical features of the logical demonstration trajectory by combining the trajectory association features and the weight parameters.

[0104] The maximum entropy principle is applied to maximize the likelihood probability of the logical demonstration trajectory under the implicit reward function, while constraining the probability distribution entropy of the unobserved trajectory under the implicit reward function.

[0105] The gradient ascent algorithm is used to iteratively update the weight parameters until the expected policy features derived from the updated weight parameters converge to the expected empirical features, and the implicit reward function after convergence is determined.

[0106] In this implementation, the diagnosis and treatment process is discretized and modeled, abstracted as a Markov decision process in reinforcement learning. Within this framework, the state space is defined as the set of semantic categories of visually relevant entities. Specifically, for each visually relevant entity (e.g., "white blood cell count," "history of hypertension"), a medical entity recognition model is used to classify it into a predefined semantic category (e.g., "examination indicators," "disease diagnosis," "drug name," etc.), and all possible categories constitute the state space S. Correspondingly, the action space is defined as the set of intent categories of the dialogue text sequence. Natural language understanding technology is used to analyze each verbal response from the doctor, identifying the underlying intent (e.g., "asking about symptoms," "explaining the condition," "issuing medical orders," etc.), and all possible intent categories constitute the action space A. This definition simplifies doctor-patient interaction into a series of decision sequences where "after seeing a certain type of information (state), a certain communication intent (action) is generated." Through this formal definition, irregular multimodal data streams are transformed into structured state-action pairs.

[0107] After determining the current state and the set of candidate actions, the next task is to transform this abstract textual data into computable mathematical features. The system calls a pre-built multi-dimensional feature extraction function. Regarding the current state With each action in the candidate action set Perform pairwise calculations. The feature extraction function is designed as a vector function with three components: The first component The semantic similarity feature is calculated using the following formula: ,here This represents the text embedding vector extracted using the BERT model; the second component... The time interval decay characteristic is calculated using the following formula: ,in and These are timestamps for the state and the action, respectively. The preset time decay constant (e.g., 0.5); the third component The path distance feature of the medical atlas is calculated using the following formula: Where dist(s,a) represents the shortest path length connecting state entity nodes and action intent nodes in the preset medical knowledge graph (set to infinity if no path exists). This is calculated for each candidate combination using a specific function. Generate a real-valued feature vector containing semantic, temporal, and logical information. Through batch processing, all potential matching relationships in the candidate set are transformed into a unified vector space representation.

[0108] The core of inverse reinforcement learning algorithms lies in deriving an implicit reward function that can explain expert behavior. This function is modeled as a linear combination of feature vectors, i.e. , where w is the weight parameter to be solved. First, the weight parameter w is randomly initialized. Then, the aforementioned feature extraction function is used to process the collected logical demonstration trajectories (i.e., a series of real state-action pairs) from the actual doctor's diagnosis and treatment process. For each trajectory Calculate its trajectory association features, which are the sum of the combined association features of all state-action pairs in the trajectory, i.e. Based on this, the average eigenvalue of all observed expert demonstration trajectories is calculated, which is called the empirical characteristic expectation. , where N is the total number of trajectories. This metric represents the average feature level exhibited by experienced doctors in real-world clinical scenarios. It serves as the target for algorithm optimization. The subsequent learning process essentially involves finding a set of weights w such that the expected features generated by the machine-generated strategy under these weights can infinitely approximate this empirical feature expectation, thereby achieving a mathematical replication of the doctor's diagnostic style.

[0109] To find the most objective and unbiased reward function from an infinite number of possible interpretations, the maximum entropy principle is introduced as an optimization criterion. This principle advocates selecting the model with the maximum probability distribution entropy, provided that known constraints (i.e., matching the expected empirical features) are met, thereby avoiding unreasonable assumptions about unobserved data. Specifically, in this approach, it involves finding a probability distribution... This maximizes the likelihood probability of the observed logical demonstration trajectory under this distribution, while also maximizing the entropy H(P) of the distribution itself. Mathematically, this means that the probability of the trajectory occurring is exponentially proportional to its total reward, i.e. ,in It represents the cumulative reward for the entire trajectory. The process of solving the above optimization problem employs the gradient ascent algorithm, aiming to find the optimal weight parameters. In each iteration, an optimal strategy is derived using the current weight parameter w. The strategy is then used to generate a large number of sampled trajectories in a simulation environment. The average eigenvalue of these sampled trajectories, i.e., the expected value of the strategy features, is calculated. Subsequently, the expected value of the empirical features is calculated. With strategy feature expectation The difference between them is the gradient of the objective function with respect to the weight parameters. Using this gradient according to the formula Update the weight parameters, where The learning rate is used. This update process continuously adjusts the weights, causing the trajectory features generated by the model to gradually converge with the expert trajectory features. The above steps are repeated until the gradient value approaches zero, meaning the policy feature expectation converges to the empirical feature expectation. The weight parameters are then determined. This is the optimal solution. Substituting it into R(s,a) will yield the converged implicit reward function.

[0110] In one implementation, the action reward value of each candidate combination of visually concerned entities and dialogue text sequences is calculated based on an implicit reward function, and the combination with the highest action reward value is selected as the optimal alignment strategy to align the temporal causal relationship between visually concerned entities and dialogue text sequences. This includes the following steps:

[0111] The system acquires newly generated visually focused entities as the current state and multiple dialogue text sequences within a preset time window as a set of candidate actions.

[0112] The feature extraction function is called to calculate the real-time feature vector of the current state and each combination of candidate actions in the candidate action set;

[0113] The real-time feature vector is input into the implicit reward function, and the non-normalized reward value is obtained by calculating the linear weighted sum of the weight parameters and the real-time feature vector.

[0114] A state transition probability model is introduced, and the cumulative expected reward for each candidate action in the current state is calculated based on the reward value. After normalizing all the cumulative expected rewards, an alignment probability distribution is generated. The target candidate action with the highest alignment probability is selected and paired with the current state. Temporal causal association labels are generated for the dialogue text sequence and visual attention entity corresponding to the target candidate action and the current state, respectively.

[0115] In this embodiment, when the system detects a new visually interested entity, it locks it into the current state. Because there is an uncertain time lag between a doctor's thinking and expression (they may be reading and speaking simultaneously, or they may read, think for a few seconds, and then speak), it is not possible to simply match the speech at the current moment. The system automatically backtracks and extracts all independent dialogue text fragments within a preset time window before and after the current moment, forming a candidate action set. This set includes not only the doctor's words but also the patient's responses, as the doctor's visual attention is often used to verify the patient's descriptions. After determining the current state and the candidate action set, the feature extraction function is then called. Regarding the current state With each action in the candidate action set One-to-one pairing calculations are performed. The feature extraction function not only analyzes the semantic similarity of the two texts (e.g., the semantic distance between "white blood cells" and "inflammation"), but also calculates the distance decay factor between them on the time axis and the logical distance in the medical knowledge graph. Through these multi-dimensional calculations, a pairing is formed for each candidate combination. Generate a high-dimensional real-time feature vector Through batch processing, all potential matching relationships in the candidate set are transformed into a unified vector space representation.

[0116] After obtaining the real-time feature vectors of each candidate combination, the implicit reward function trained during the inverse reinforcement learning phase is used for value evaluation. The core of the implicit reward function is a set of optimized weight parameters. This set of parameters characterizes the diagnostic and treatment logic preferences of experienced physicians. Each real-time feature vector... Inputting this function will perform a linear weighted summation operation, that is, calculate the dot product. The calculated results This is a non-normalized scalar value, known as the immediate reward value. This value intuitively reflects the degree of fit between the candidate action and the current visual state: the higher the value, the more the combination conforms to the conventional logic of clinical diagnosis; the lower the value, the more likely the two are merely a coincidence in time and have no substantial connection. To make the final matching decision, a long-term perspective and a probabilistic model are also needed.

[0117] Specifically, a state transition probability model P(s'|s,a) is introduced to estimate the likelihood of transitioning to the next state after taking a certain action, and the cumulative expected return is calculated by combining the ideas of the Bellman equation. ,in Let V(s') be the discount factor, and V(s') be the value of the subsequent state. This step considers the impact of actions on future interactions. Subsequently, the cumulative expected reward of all actions in the candidate set is Softmax normalized to generate an aligned probability distribution. This distribution clearly shows the probability that each candidate action is a correct match. Based on the maximum likelihood principle, the system selects the candidate action with the highest probability value. As the target action, it is compared with the current state. The pairing is then performed. Finally, a unique temporal causal relationship label is assigned to this pair of "visual entities-dialogue texts," clearly identifying the logical chain of "because A was seen, B was said."

[0118] In one implementation, the aligned visual attention entities and dialogue text sequences are mapped to a first node and a second node in a preset medical knowledge graph, respectively. A path search algorithm is then executed within the medical knowledge graph to obtain the shortest semantic path connecting the first node and the second node as an implicit clinical reasoning chain. This includes the following steps:

[0119] Entity linking technology is used to map visually focused entities and dialogue text sequences to their corresponding unique entity IDs in the medical knowledge graph, which are then identified as the first and second nodes.

[0120] Perform a forward breadth-first search starting from the first node, and simultaneously perform a backward breadth-first search starting from the second node;

[0121] When a common intersection node is detected between the forward breadth-first search path and the backward breadth-first search path, the search is stopped and a candidate reasoning path consisting of the first node, the common intersection node, and the second node is extracted.

[0122] Invalid paths containing non-clinical logical relationships are filtered out based on the relation type attribute of each edge in the candidate reasoning path;

[0123] Calculate the sum of the weights of all edges in the remaining valid candidate reasoning paths, and select the valid candidate reasoning path with the smallest sum of weights as the most semantic path.

[0124] The node names and edge relationship names in the shortest semantic path are filled into a preset natural language generation template to generate an implicit clinical reasoning chain.

[0125] In this implementation, a medical knowledge graph serves as the underlying knowledge base, consisting of a massive number of entity nodes (such as diseases, symptoms, and drugs) and semantic relationship edges connecting these nodes (such as "cause" and "treatment"). Each entity has a unique universal identifier within the graph. Visually relevant entities originate from phrases captured from the screen, while dialogue text sequences contain colloquial expressions; both exhibit polysemy, synonymy, and non-standardization. Therefore, simple string matching is insufficient. First, natural language processing techniques are used to standardize and clean the input text, removing noise characters and performing lemmatization. Next, a deep semantic encoder based on BERT or BioBERT is employed to transform the standardized text into a high-dimensional query vector. Simultaneously, all entity nodes in the medical knowledge graph are traversed, and their standard names and aliases are also transformed into graph node vectors. The semantic similarity between the query vector and the graph node vector is calculated, typically using cosine similarity as the metric. For each input entity, the graph node with the highest similarity exceeding a pre-set confidence threshold is selected as the target anchor point. Through this process, the "visual attention entity" is identified as the starting point (first node) in the graph, and the core intent in the "dialogue text sequence" is identified as the ending point (second node).

[0126] After determining the starting and ending points, it is necessary to find the potential logical connections between them in the medical knowledge graph network. Since medical knowledge graphs typically contain millions of nodes and hundreds of millions of edges, using traditional one-way depth-first search or breadth-first search would cause the search space to explode exponentially with the path length, leading to computational timeouts or memory overflows. Therefore, this solution employs a bidirectional breadth-first search algorithm to optimize the path discovery process. This algorithm simultaneously initiates two search processes: the forward search process starts from the first node and expands outward layer by layer along the directed edges; the backward search process starts from the second node and backtracks inward layer by layer along the opposite direction of the directed edges. In each search iteration, the algorithm maintains two boundary queues, recording the nodes to be visited at the current level, and uses a hash table to record visited nodes and their parent node paths. Each time a new node is expanded, it is checked in real-time whether the node already exists in the visited node set of the other node. Once the same node is detected at the boundaries of the forward and backward searches (i.e., the common intersection node), the search immediately stops. At this point, the algorithm uses backtracking pointers to extract the first half of the path from the first node to the common intersection, and the second half of the path from the common intersection to the second node, and then splices the two together to form a complete connected path. This bidirectional search strategy greatly reduces the search radius and the number of nodes visited, and can reduce the time complexity exponentially while ensuring that a connected path is found, thus successfully extracting candidate reasoning paths that include the starting point, intermediate nodes, and the ending point.

[0127] While the original connected paths found in the knowledge graph may be topologically valid, they may contain numerous meaningless or non-causal relationships in medical logic. For example, paths might be connected by relationships such as "belongs to a category" or "synonym," which, while taxonomically correct, fail to explain causal reasoning in the diagnostic and treatment process (e.g., "A causes B" or "C treats D"). Therefore, rigorous clinical logic filtering is necessary for the extracted candidate reasoning paths. This step relies on a predefined whitelist of clinical logic relationships, which only includes edge attributes with clear clinical reasoning value, such as "cause," "concurrency," "diagnostic basis," "treatment," and "contraindication." Each edge in the candidate path is traversed, and its relation type attribute defined in the knowledge graph metadata is read. If any edge in the path has a relation type that does not belong to the whitelist, the entire path is deemed invalid reasoning, lacking clinical explanatory power, and is completely removed from the candidate set. Only when all links in the path conform to clinical logic specifications is the path retained. After logical filtering, multiple valid candidate reasoning paths may still exist connecting the start and end points. To simulate the "shortest path principle" or "maximum probability principle" in doctors' thinking—that is, doctors tend to use the most direct and obvious explanation—it is necessary to quantitatively evaluate the remaining paths to select the optimal solution. Therefore, a path weighting calculation mechanism is introduced.

[0128] Specifically, in the medical knowledge graph construction phase, each edge is assigned a weight value, which reflects the tightness of the semantic association or the strength of the causal relationship between two nodes (the smaller the weight, the tighter the association and the shorter the distance). Weights can be set based on the co-occurrence frequency of medical literature or expert scores. The total weight of valid candidate reasoning paths is calculated, which is the sum of the weights of all edges on the path. Let path P contain an edge set. The weight of each edge is Then the total path weight The system iterates through all valid candidate paths, compares their total weight values, and selects the correct path. The shortest path is designated as the shortest semantic path. "Shortest" doesn't simply mean the fewest nodes, but rather the least semantic resistance and the smoothest logical chain. Finally, the selected shortest semantic path is parsed, and the node names and edge relationship names are extracted sequentially. Starting from the origin, along the path direction, the current node name and the next hop node name are filled into the corresponding edge relationship template slots. If the path contains multiple hops, conjunctions (such as "furthermore," "incidentally," and "and") are used to string together the short sentences generated by the multiple templates into a coherent declarative sentence. For example, the path "elevated white blood cell count -> indication -> bacterial infection -> leading to -> fever" is transformed into "The examination showed elevated white blood cell count, indicating a bacterial infection, which in turn led to fever."

[0129] In one implementation, filtering out invalid paths containing non-clinical logical relationships based on the relation type attribute of each edge in the candidate inference path includes the following steps:

[0130] A whitelist of clinical logical relationships is predefined, which includes at least causal relationships, concurrent relationships, treatment relationships, and contraindication relationships.

[0131] For any candidate reasoning path, traverse each connecting edge of the candidate reasoning path and read the relation type label of the connecting edge in the medical knowledge graph;

[0132] If the relation type label is not in the clinical logical relation whitelist, the candidate reasoning path will be filtered out.

[0133] In this implementation, a medical knowledge graph is typically a complex network containing a massive number of entities and relationships. This network includes not only medical logical relationships with causal reasoning value but also a large number of non-logical descriptive relationships (such as "alias," "department," "medical insurance code," "discoverer," etc.). Incorporating these non-logical relationships into the reasoning path would result in disorganized and unprofessional medical records. Therefore, it is necessary to predefine a set containing all allowed relationship types for clinical logical deduction, i.e., a whitelist of clinical logical relationships. This whitelist should at least cover the following core medical logical relationships: "cause," used to describe the necessary connection between etiology and pathological outcome; "concurrence," used to describe disease states that frequently occur simultaneously; "treatment," used to connect disease with medication or surgical plans; "contraindication," used to warn of restrictions on medication or procedures under specific conditions; and "manifestation," used to associate disease with clinical symptoms or signs.

[0134] Candidate reasoning paths typically consist of a series of entity nodes and edges connecting these nodes. To verify whether the path possesses reasonable clinical interpretability, a traversal program needs to be initiated to sequentially visit each connecting edge along the path direction. For the currently visited connecting edge, the query interface of the graph database is invoked to read the attribute information of that edge stored in the medical knowledge graph metadata, especially the relation type label. This label clearly identifies the semantic relationship represented by the edge. After obtaining the relation type label of the current connecting edge, it is compared with a pre-loaded whitelist of clinical logic relations for set membership testing. If, during the traversal of the path, it is found that the relation type label of any edge is not within the whitelist, the entire candidate reasoning path P is immediately determined to be an invalid path, meaning that non-clinical logic impurities have been mixed into the path, compromising the purity and causal coherence of the reasoning chain.

[0135] In one implementation, generating structured electronic medical record text after doctor-patient interaction by combining visually focused entities, dialogue text sequences, and implicit clinical reasoning chains includes the following steps:

[0136] Structured cue words are generated by combining visually focused entities, dialogue text sequences, and implicit clinical reasoning chains;

[0137] The structured prompts are input into a pre-defined large language model, and the text stream output by the large language model is parsed into structured electronic medical record text after doctor-patient interaction.

[0138] In this embodiment, refer to Figure 3 After completing the deep analysis and logical reasoning of multimodal data, the first step is to construct a hierarchical prompting word project. Specifically, a standardized JSON or XML template is defined as the skeleton of the structured prompting words. This template is divided into functional blocks such as "Chief Complaint," "Present Illness," "Physical Examination," "Auxiliary Examinations," "Diagnosis," and "Treatment Opinions" based on the standard paragraphs of electronic medical records. High-confidence visual attention entities are mapped to the "Auxiliary Examinations" or "Patient History" blocks as objective evidence support; the aligned dialogue text sequence is cleaned and filled into the "Chief Complaint" and "Present Illness" blocks as subjective descriptive material; the mined implicit clinical reasoning chain is converted into natural language descriptions and filled into the "Diagnostic Basis" or "Procedure Records" blocks to demonstrate the decision-making logic. In addition, a role setting instruction is added to the head of the prompting words, explicitly requiring the model to play the role of a senior practicing physician and follow specific medical terminology and medical record writing formats. In this way, the originally messy multimodal perception data is transformed into a logically rigorous, semantically rich, and strongly constrained structured context.

[0139] After constructing the structured prompts, they are fed as input data into a pre-defined large language model to generate the final electronic medical record text. This large language model is typically a specialized model that has undergone supervised fine-tuning and human feedback reinforcement learning on massive medical corpora (such as medical textbooks, guidelines, and anonymized medical records), possessing extremely strong medical text generation and inductive capabilities. After inputting the structured context into the model, it employs an autoregressive decoding strategy to predict and output the text stream word by word. To address the stability issues of long text generation, bundle search or kernel sampling strategies can be used to balance accuracy and diversity. The model output is not the final static document, but a continuous text stream. The system needs to monitor and parse this text stream in real time. Using regular expressions or a specific parser, predefined paragraph separators or markers in the text stream are identified, and the continuous text stream is segmented and filled back into the corresponding fields of the electronic medical record system. For example, when the marker "

Diagnosis

[0140] The present invention also discloses an intelligent structured medical record generation system based on multimodal doctor-patient interaction, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the intelligent structured medical record generation method based on multimodal doctor-patient interaction as described above.

[0141] The processor can be a central processing unit (CPU). Of course, depending on the actual use, it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc., and this application does not limit it.

[0142] The memory can be an internal storage unit of a computer device, such as a hard disk or RAM, or an external storage device, such as a plug-in hard disk, smart memory card (SMC), secure digital card (SD), or flash memory card (FC) provided on the computer device. Furthermore, the memory can be a combination of internal storage units and external storage devices of a computer device. The memory is used to store computer programs and other programs and data required by the computer device. The memory can also be used to temporarily store data that has been output or will be output. This application does not limit this.

[0143] The present invention also discloses a computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to be configured to perform the intelligent structured medical record generation method based on multimodal doctor-patient interaction described in any of the above embodiments.

[0144] The computer program can be stored in a machine-readable medium. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or certain middleware. The machine-readable medium includes any entity or device capable of carrying computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the machine-readable medium includes, but is not limited to, the above-mentioned components.

[0145] The intelligent structured medical record generation method based on multimodal doctor-patient interaction described in the above embodiments is stored in the computer-readable storage medium and loaded and executed on the processor to facilitate the storage and application of the above method.

[0146] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of protection of this application is limited to these examples; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of one or more embodiments of this application as described above, which are not provided in detail for the sake of brevity.

[0147] One or more embodiments in this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of this application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments in this application should be included within the protection scope of this application.

Claims

1. A method for generating intelligent structured medical records based on multimodal doctor-patient interaction, characterized in that, Includes the following steps: The doctor-patient dialogue voice is collected in real time during the doctor-patient interaction process, and the dialogue text generated by the transcribed doctor-patient dialogue voice is mapped to the semantic vector space to obtain the dialogue text sequence. During doctor-patient interaction, the screen coordinates of the mouse pointer are monitored in real time when the doctor performs mouse operations in the electronic medical record system. The interface content of the user interface control below the screen coordinates is obtained as candidate data by calling the user interface tree interface of the electronic medical record system. A context vector is constructed based on the current context information of the doctor-patient interaction process. The first semantic feature containing candidate data and the second semantic feature after occluding candidate data are calculated based on the context vector. When the semantic offset between the first semantic feature and the second semantic feature exceeds the preset salience threshold, the candidate data is marked as a visually significant entity. Based on the correlation features between visually focused entities and dialogue text sequences, and combined with the collected historical focused entity sequences and historical text sequences, a logical demonstration trajectory is generated. The formal representation of the logical demonstration trajectory is a set of state-action pairs. Define the state space as the set of semantic categories of visually concerned entities, and define the action space as the set of intent categories of the dialogue text sequence; The feature extraction function is used to calculate the combined association features between each pair of states in the state space and actions in the action space. The combined association features include at least semantic similarity features, time interval decay features, and medical atlas path distance features. Initialize the weight parameters of the implicit reward function, extract the trajectory association features of the logical demonstration trajectory using the feature extraction function, and calculate the expected empirical features of the logical demonstration trajectory by combining the trajectory association features and the weight parameters. The maximum entropy principle is applied to maximize the likelihood probability of the logical demonstration trajectory under the implicit reward function, while constraining the probability distribution entropy of the unobserved trajectory under the implicit reward function. The gradient ascent algorithm is used to iteratively update the weight parameters until the policy feature expectation derived from the updated weight parameters converges to the empirical feature expectation, and the implicit reward function after convergence is determined. The action reward value of each candidate combination of visual attention entity and dialogue text sequence is calculated based on the implicit reward function. The combination with the highest action reward value is selected as the optimal alignment strategy to align the temporal causal relationship between visual attention entity and dialogue text sequence. The aligned visual attention entities and dialogue text sequences are mapped to the first and second nodes in a pre-defined medical knowledge graph, respectively. A path search algorithm is then executed in the medical knowledge graph to obtain the shortest semantic path connecting the first and second nodes as an implicit clinical reasoning chain. By combining visually focused entities, dialogue text sequences, and implicit clinical reasoning chains, structured electronic medical record texts are generated after doctor-patient interactions.

2. The intelligent structured medical record generation method based on multimodal doctor-patient interaction according to claim 1, characterized in that, The process of constructing a context vector based on the current context information of the doctor-patient interaction process, calculating a first semantic feature containing candidate data and a second semantic feature after masking the candidate data based on the context vector, and marking the candidate data as a visually significant entity when the semantic offset between the first semantic feature and the second semantic feature exceeds a preset salience threshold includes the following steps: A context vector is constructed based on the current context information of the doctor-patient interaction process. Global semantic features are extracted from the context vector by an encoder. The text embedding vector of the candidate data is concatenated with the global semantic features to generate the first semantic feature containing the candidate data. Construct a zero-mask vector with the same dimension as the candidate data, and use the zero-mask vector to replace the candidate data positions in the context vector to form a masking sequence; The masking sequence is input into the encoder. After being processed by the multi-layer self-attention mechanism in the encoder, the hidden state vector of the output layer is extracted to generate the second semantic feature after masking candidate data. Calculate the Euclidean distance and cosine similarity between the first semantic feature and the second semantic feature in their respective semantic vector spaces, and calculate the weighted sum of the Euclidean distance and cosine similarity to obtain the semantic offset. If the semantic offset exceeds the preset saliency threshold, the candidate data will be marked as a visually relevant entity.

3. The intelligent structured medical record generation method based on multimodal doctor-patient interaction according to claim 2, characterized in that, The process of constructing a context vector based on the current context information of the doctor-patient interaction process and extracting global semantic features from the context vector through an encoder includes the following steps: Collect current context information of the doctor-patient interaction process, including electronic medical record text fragments, patient's medical history summary, and dialogue text within a preset time window; The current context information is converted into a word sequence by a word segmentation tool and then mapped into a context vector. Different positional encoding and modality type encoding are added to each context vector according to different source attributes. All context vectors are input into the encoder based on the Transformer architecture, and the vector representation of the CLS special terms output by the last layer of the encoder is extracted as global semantic features.

4. The intelligent structured medical record generation method based on multimodal doctor-patient interaction according to claim 1, characterized in that, The step of calculating the action reward value of each candidate combination of visually concerned entities and dialogue text sequences based on the implicit reward function, and selecting the combination with the highest action reward value as the optimal alignment strategy to align the temporal causal relationship between visually concerned entities and dialogue text sequences includes the following steps: The system acquires newly generated visually focused entities as the current state and multiple dialogue text sequences within a preset time window as a set of candidate actions. The feature extraction function is called to calculate the real-time feature vector of the current state and each candidate action combination in the candidate action set; The real-time feature vector is input into the implicit reward function, and the non-normalized reward value is obtained by calculating the linear weighted sum of the weight parameters and the real-time feature vector. A state transition probability model is introduced, and the cumulative expected reward for each candidate action in the current state is calculated based on the reward value. After normalizing all the cumulative expected rewards, an alignment probability distribution is generated. The target candidate action with the highest alignment probability is selected and paired with the current state. Temporal causal association labels are generated for the dialogue text sequence and visual attention entity corresponding to the target candidate action and the current state, respectively.

5. The intelligent structured medical record generation method based on multimodal doctor-patient interaction according to claim 1, characterized in that, The step of mapping the aligned visual attention entities and dialogue text sequences to a first node and a second node in a preset medical knowledge graph, respectively, and then executing a path search algorithm in the medical knowledge graph to obtain the shortest semantic path connecting the first node and the second node as an implicit clinical reasoning chain includes the following steps: Entity linking technology is used to map visually focused entities and dialogue text sequences to their corresponding unique entity IDs in the medical knowledge graph, which are then identified as the first and second nodes. Perform a forward breadth-first search starting from the first node, and simultaneously perform a backward breadth-first search starting from the second node; When a common intersection node is detected between the forward breadth-first search path and the backward breadth-first search path, the search is stopped and a candidate reasoning path consisting of the first node, the common intersection node, and the second node is extracted. Invalid paths containing non-clinical logical relationships are filtered out based on the relation type attribute of each edge in the candidate reasoning path; Calculate the sum of the weights of all edges in the remaining valid candidate reasoning paths, and select the valid candidate reasoning path with the smallest sum of weights as the most semantic path. The node names and edge relationship names in the shortest semantic path are filled into a preset natural language generation template to generate an implicit clinical reasoning chain.

6. The intelligent structured medical record generation method based on multimodal doctor-patient interaction according to claim 5, characterized in that, The step of filtering out invalid paths containing non-clinical logical relationships based on the relation type attribute of each edge in the candidate inference path includes the following steps: A whitelist of clinical logical relationships is predefined, which includes at least causal relationships, concurrent relationships, treatment relationships, and contraindication relationships. For any candidate reasoning path, traverse each connecting edge of the candidate reasoning path and read the relation type label of the connecting edge in the medical knowledge graph; If the relation type label is not in the clinical logical relation whitelist, the candidate reasoning path will be filtered out.

7. The intelligent structured medical record generation method based on multimodal doctor-patient interaction according to claim 1, characterized in that, The process of generating structured electronic medical record text after doctor-patient interaction by combining visually focused entities, dialogue text sequences, and implicit clinical reasoning chains includes the following steps: Structured cue words are generated by combining visually focused entities, dialogue text sequences, and implicit clinical reasoning chains; The structured prompts are input into a pre-defined large language model, and the text stream output by the large language model is parsed into structured electronic medical record text after doctor-patient interaction.

8. An intelligent structured medical record generation system based on multimodal doctor-patient interaction, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes a computer program, it implements the intelligent structured medical record generation method based on multimodal doctor-patient interaction as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing instructions thereon, characterized in that, When executed by the processor, the instruction causes the processor to be configured to perform the intelligent structured medical record generation method based on multimodal doctor-patient interaction according to any one of claims 1 to 7.

Images