Pathogenesis record generation method and system based on mobile terminal and multi-modal data

Abstract

The invention provides a disease course record generation method and system based on a mobile terminal and multi-modal data, and the method comprises the following steps: firstly constructing a directional collection channel and a ward round session associated with a specific patient through the signal interaction between the mobile terminal and a sickbed; in the session, multi-modal data streams with timestamps, such as voice, images and operation instructions, are obtained in real time, and audio semantic anchor points and visual semantic anchor points are extracted from the multi-modal data streams. And constructing an image-text semantic unit by calculating and associating the audio and video anchor points of which the semantic similarity exceeds a threshold. And taking the unit as a current observation state, and verifying the legality of the state transition through a finite-state machine model. And based on the image-text semantic units passing the verification, template slot filling and term standardization are carried out, and a structured disease course record data packet is generated and written back to the HIS system. The method has the effect of improving the working efficiency and quality of clinical documents.

Description

Technical Field

[0001] This invention belongs to the field of medical data processing technology, specifically relating to a method and system for generating medical records based on mobile terminals and multimodal data. Background Technology

[0002] With the widespread adoption of mobile information technology in the medical field, doctors using mobile devices (such as smartphones or tablets) for ward rounds has become the new norm in clinical work. During ward rounds, doctors generate a massive amount of multimodal, real-time information about patients, including conversations, visual observations of wounds or vital signs, interpretation of monitor data, and verbal or tactile instructions. This information forms the most original and authentic data source for medical records. Currently, the final transformation of this information into structured medical records in electronic medical records systems mainly relies on doctors manually transcribing and entering data after ward rounds, using their memory and scattered notes. This process is not only time-consuming and labor-intensive, but also highly susceptible to distortion and incompleteness in medical records due to memory bias, information omissions, or transcription errors. Summary of the Invention

[0003] This invention provides a method and system for generating medical records based on mobile terminals and multimodal data to solve the above-mentioned technical problems.

[0004] In a first aspect, the present invention provides a method for generating medical records based on mobile terminals and multimodal data, the method comprising the following steps: Based on the signal interaction characteristics between the doctor's mobile terminal and the target patient's bed area, a directional acquisition channel including spatial fencing and acoustic filtering is constructed, and a ward round session associated with the target patient is instantiated. During ward rounds, multiple types of discrete ward round record data streams are acquired through multiple parallel input interfaces of the mobile terminal, and each ward round record data stream is assigned an absolute timestamp and type label to form a ward round data sequence. The ward round record data stream includes voice data stream, image data stream, and operation instruction data stream. Feature extraction was performed on the ward round data sequence to generate audio semantic anchors containing medical entity information and visual semantic anchors containing visual subject features, respectively. Calculate the semantic similarity between audio semantic anchors and visual semantic anchors, and perform non-linear association across time dimension between audio semantic anchors and visual semantic anchors whose semantic similarity exceeds a preset similarity threshold to construct text-image semantic units; The system acquires the historical treatment status of the target patient and loads the corresponding standard clinical pathway status map. It maps the text and image semantic units to the current observation status and verifies the legality of the transition between the current observation status and the historical treatment status through a finite state machine model. Based on the validated image and text semantic units, and according to the preset medical record template, slot filling and medical terminology standardization are performed to generate a structured medical record data package and write it back to the hospital information management system.

[0005] Optionally, the step of constructing a directional acquisition channel including spatial fencing and acoustic filtering based on the signal interaction characteristics between the doctor's mobile terminal and the target patient's bed area, and instantiating the ward round session associated with the target patient, includes the following steps: The signal strength received by the doctor's mobile terminal when it receives a wireless beacon broadcast in the area where the target patient's bed is located is analyzed, and the physical distance between the mobile terminal and the bed area is calculated based on the log-normal distribution propagation model. When the physical distance between the mobile terminal and the bed area is less than the preset near-field interaction threshold, the mobile terminal is triggered to activate the multi-microphone array and lock the unique identifier of the target patient carried by the wireless beacon to instantiate the ward round session associated with the target patient. Calculate the time difference of arrival of the sound source signals received by the multi-microphone array, and estimate the azimuth angle of the main sound source relative to the mobile terminal by combining the spatial attitude parameters of the mobile terminal. The weighted delay coefficient of the received signal of each microphone in the multi-microphone array is calculated based on the azimuth angle of the sound source, and a pickup beam pointing to the bed area is formed by using a delay summation beamforming algorithm. Based on the spatial response characteristics of the pickup beam, background noise signals outside the pickup beam range are suppressed, and a uniquely identified independent directional acquisition channel is established in the mobile terminal. The system continuously monitors the rate of change in received signal strength and terminates the ward round session and releases the directional acquisition channel when the received signal strength falls below a preset disconnection threshold.

[0006] Optionally, the step of continuously monitoring the rate of change of the received signal strength and terminating the ward round session and releasing the directional acquisition channel when the received signal strength is lower than a preset disconnection threshold includes the following steps: During the ward round session, the strength of the continuously sampled received signal is smoothed using the Kalman filter algorithm; Calculate the time derivative of the received signal strength after smoothing to determine the movement trend of the mobile terminal relative to the bed area; When the mobile trend shows that the mobile terminal is moving away from the bedside area and the received signal strength is consistently lower than the preset disconnection threshold, a timer is started to confirm the departure. If the timer reaches the preset time limit and the received signal strength is always lower than the disconnection threshold, it is determined that the mobile terminal has left the effective interaction range, triggering a session termination command, marking all data cached in the directional acquisition channel as session end state, and disabling the mobile terminal's multi-microphone array from occupying the directional acquisition channel; Generate a session integrity check code and append the session integrity check code to the end of the data packet for the room check session.

[0007] Optionally, the step of extracting features from the ward round data sequence to generate audio semantic anchors containing medical entity information and visual semantic anchors containing visual subject features includes the following steps: The speech data stream was separated from the ward round data sequence, and the speech data stream was converted into a natural language text sequence using an automatic speech recognition model; The natural language text sequence is input into a pre-trained medical entity recognition model to extract a set of keywords containing medical entities, and the set of keywords is vectorized into audio semantic anchors; Image data streams were separated from the ward round data sequence, and image enhancement and normalization preprocessing were performed on each frame of the image data stream. The preprocessed image data stream is input into a pre-trained medical feature recognition model to identify target images with medical features in the image data stream and extract medical visual feature vectors of the regions of interest in the target images. The medical visual feature vectors are dimensionality reduced and image classification labels are added to generate visual semantic anchors corresponding to the target image.

[0008] Optionally, the step of calculating the semantic similarity between audio semantic anchors and visual semantic anchors, and then performing a non-linear cross-time association between audio semantic anchors and visual semantic anchors whose semantic similarity exceeds a preset similarity threshold to construct a text-image semantic unit includes the following steps: A cross-modal semantic feature space is constructed by combining all audio semantic anchors and visual semantic anchors; Calculate the cosine similarity between each audio semantic anchor vector and each visual semantic anchor vector in the cross-modal semantic feature space to form a many-to-many semantic similarity matrix; Traverse the semantic similarity matrix and filter out candidate anchor pairs whose cosine similarity values ​​exceed a preset association threshold; Ignoring the original timestamp order in the ward round data sequence, the maximum weight matching algorithm is applied to all candidate anchor pairs to determine the optimal pairing relationship between audio semantic anchors and visual semantic anchors. Based on the optimal pairing relationship, the text segment corresponding to the successfully paired audio semantic anchor point is bound to the original image corresponding to the visual semantic anchor point, and encapsulated into a graphic semantic unit with independent logical meaning. For the remaining audio semantic anchors or remaining visual semantic anchors that fail to match, secondary weak association merging is performed by selecting the matched text-image semantic units with the closest time interval based on the original timestamps of the remaining audio semantic anchors or remaining visual semantic anchors.

[0009] Optionally, the step of obtaining the target patient's historical treatment status and loading the corresponding standard clinical pathway status map, mapping the textual semantic units to the current observed status, and verifying the legality of the transition between the current observed status and the historical treatment status using a finite state machine model includes the following steps: By querying the hospital information management system, the postoperative days and the medical record data of the previous time point of the target patient are obtained, and the historical treatment status vector of the target patient is obtained by parsing. Based on the admission diagnosis code of the target patient, the corresponding standard clinical pathway state map is invoked. The standard clinical pathway state map contains the set of legal states that are allowed to exist at different time points for the disease to which the target patient belongs, as well as the transition probability matrix between states. The text and image features in the semantic units of the image are analyzed, and the text and image features are quantified into the current observation state vector reflecting the current physiological indicators of the target patient; The historical treatment state vector is used as the current state input of the finite state machine model, and the expected set of legal states at the next moment is predicted based on the transition probability matrix. Determine whether the current observed state vector belongs to the expected set of legal states. If the current observed state vector belongs to the expected set of legal states, the state transition is deemed legal. If the current observed state vector does not belong to the expected set of legal states, the state transition is deemed illegal.

[0010] Optionally, after determining that the current observed state vector does not belong to the expected set of legal states, the following steps are also included: Obtain a preset modality confidence weight table. The modality confidence weight table sets the confidence priority of different modalities in different scenarios based on image recognition accuracy and speech recognition accuracy. Identify feature conflicts that lead to illegal state transitions, and compare the confidence weights of the image features and text features corresponding to the feature conflicts. If the confidence weight of the image features is higher than that of the text features, then generate an automatic correction instruction based on the image features to correct the text features. If the difference between the confidence weight of the image feature and the confidence weight of the text feature is less than a preset fuzzy threshold, an abnormal state warning mark is generated and attached to the corresponding image and text semantic unit. Apply automatic correction instructions to text-image semantic units to update the current observation state vector, or place text-image semantic units with abnormal state warning markers into the queue for manual review. The updated graphic semantic unit is then input into the finite state machine model for secondary logical closed-loop verification.

[0011] Optionally, the step of generating a structured medical record data package based on verified graphic semantic units, and performing slot filling and medical terminology standardization processing according to a preset medical record template, and writing it back to the hospital information management system, includes the following steps: The semantic category tags contained in the verified image and text semantic units are analyzed to determine the medical document paragraph attributes corresponding to the image and text semantic units; Based on the type of ward round session and the current stage of treatment of the target patient, a matching medical record template is retrieved from the template library. The medical record template defines a standard document structure tree and data slots. Iterate through all text and image semantic units, and map and populate the text and image semantic units into one or more corresponding data slots in the medical record template according to the paragraph attributes of the medical document. Using a pre-defined medical terminology ontology database, synonym retrieval and standardized mapping are performed on the natural language text populated into the data slots; In accordance with the data interaction interface specifications of the hospital information management system, the completed medical record template is serialized into a structured medical record data packet, and the structured medical record data packet is sent to the hospital information management system through an encrypted transmission channel.

[0012] In a second aspect, the present invention also provides a medical record generation system based on mobile terminal and multimodal data, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the medical record generation method based on mobile terminal and multimodal data as described in any one of the first aspects.

[0013] Thirdly, the present invention also provides a computer-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the method for generating medical records based on mobile terminal and multimodal data according to any one of the first aspects.

[0014] The beneficial effects of this invention are: This invention ensures the uniqueness and accuracy of data acquisition sources in complex ward environments by constructing a directional acquisition channel based on signal interaction features. By extracting semantic anchors and calculating their semantic similarity, non-linear associations in the time dimension are achieved, thus accurately reconstructing the inherent logical binding relationship between what is said and seen in the ward round scenario, forming a textual and graphical semantic unit with rich contextual information. Using a finite state machine model based on a standard clinical pathway state graph, the transition legitimacy of the fused observation states is verified, placing the data-driven recognition results under strong constraints of medical knowledge. This effectively filters out pseudo-associations and abnormal data that do not conform to diagnostic and treatment logic, ensuring the medical rigor of the generated content. Finally, through templated slot filling and standardization of medical terminology, the ward round interaction process is transformed into a structured and standardized medical record data package, significantly improving the efficiency and quality of clinical documentation work. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating a method for generating medical records based on mobile terminals and multimodal data in one embodiment of this application.

[0016] Figure 2 This is a flowchart illustrating the process of constructing a targeted acquisition channel and instantiating a ward round session associated with the target patient in one embodiment of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0018] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0019] Figure 1 This is a flowchart illustrating a method for generating medical records based on a mobile terminal and multimodal data in one embodiment. It should be understood that, although... Figure 1The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps. For example Figure 1 As shown, the method for generating medical records based on mobile terminals and multimodal data disclosed in this invention specifically includes the following steps: S101. Based on the signal interaction characteristics between the doctor's mobile terminal and the target patient's bed area, construct a directional acquisition channel that includes spatial fencing and acoustic filtering, and instantiate a ward round session associated with the target patient.

[0020] The process begins by analyzing the broadcast signal received by the doctor's mobile terminal from a wireless beacon deployed in the target patient's bed area. By analyzing the received signal strength and applying a log-normal distribution propagation model, the physical distance between the mobile terminal and the beacon can be estimated. Once the calculated physical distance is less than a preset near-field interaction threshold, it indicates that the doctor has entered the effective ward round range. At this point, the mobile terminal's multi-microphone array is activated, locking onto the patient's unique identifier carried in the beacon broadcast, thus instantiating a ward round session strictly bound to that patient. Next, by calculating the time difference of the sound source signal arriving at different microphones and combining it with the terminal device's attitude sensor data, the azimuth angle of the main sound source can be accurately estimated. Based on this azimuth angle, a delay-summation beamforming algorithm is used to weight and delay the signals from each microphone, forming a highly directional pickup beam that is effectively aimed at the target sound source area. This acoustic filtering technology can significantly suppress environmental noise and interference from adjacent beds outside the beam range, ultimately forming a directional acquisition channel that combines spatial physical fencing with acoustic logical isolation.

[0021] S102. During a ward round session, multiple types of discrete ward round record data streams are acquired through multiple parallel input interfaces of the mobile terminal, and each ward round record data stream is assigned an absolute timestamp and a type marker to form a ward round data sequence. The ward round record data stream includes voice data stream, image data stream, and operation instruction data stream.

[0022] After a ward round session is successfully established and instantiated through the directional acquisition channel, a multi-task parallel processing mechanism is initiated. This mechanism simultaneously captures multimodal information streams during the ward round process via multiple input interfaces on the mobile terminal. These information streams are discrete and heterogeneous, primarily consisting of doctor-patient dialogue, images of the patient's vital signs or wounds, and operational commands from the doctor's on-screen clicks or swipes. To ensure accurate alignment and correlation of these data from different sources and at varying times in subsequent analysis, each independent data stream is assigned a high-precision absolute timestamp and a clear type label the moment it is captured. This process forms a unified ward round data sequence, where each element can be abstractly represented as a data tuple containing a timestamp, data type, and original data payload.

[0023] S103. Extract features from the ward round data sequence to generate audio semantic anchors containing medical entity information and visual semantic anchors containing visual subject features.

[0024] For the audio data stream in the sequence, an automatic speech recognition model is first used to efficiently convert it into natural language text. Then, these text sequences are input into a medical entity recognition model pre-trained on a massive amount of medical literature. This model can automatically identify and extract key medical entities such as diseases, symptoms, drugs, and examination items mentioned in the text, forming a keyword set. This set is then vectorized using word embedding technology to form audio semantic anchors, each anchor being a highly condensed representation of the core medical information of a segment of audio. Simultaneously, for the image data stream, the system analyzes it frame by frame. First, image quality is improved through preprocessing steps such as image enhancement and normalization. Then, the processed images are fed into a pre-trained medical feature recognition model. This model is responsible for determining whether the image contains valuable medical features, such as wound healing status, rash morphology, or drainage fluid color. Once the target image is identified, the visual features of its key regions are extracted and encoded into a high-dimensional vector. After dimensionality reduction and the addition of image classification labels, this vector generates visual semantic anchors corresponding to the target image.

[0025] S104. Calculate the semantic similarity between audio semantic anchors and visual semantic anchors, and perform non-linear association across time dimension between audio semantic anchors and visual semantic anchors whose semantic similarity exceeds the preset similarity threshold to construct a text-image semantic unit.

[0026] After generating separate audio and visual semantic anchors, it is necessary to quantify the semantic relevance between anchors of different modalities. By mapping all audio and visual semantic anchor vectors to a unified cross-modal semantic feature space, their similarity can be directly compared within this shared space. A many-to-many semantic similarity matrix can be constructed by calculating the cosine similarity of all possible audio-visual anchor pairs. Next, an association threshold is set to filter out candidate anchor pairs with sufficiently high similarity. To find the globally optimal matching scheme, the temporal order of these anchors in the original data sequence is ignored, and a maximum-weighted matching algorithm is applied to all candidate anchor pairs. This algorithm ensures that each anchor is paired with only the most semantically relevant heterogeneous anchor, avoiding erroneous associations. The text segment corresponding to the successfully paired audio anchor and the original image corresponding to the visual anchor are formally bound and encapsulated into a logically consistent graph-text semantic unit.

[0027] S105. Obtain the historical treatment status of the target patient and load the corresponding standard clinical pathway status map, map the text and image semantic units to the current observation status, and verify the legality of the transition between the current observation status and the historical treatment status through a finite state machine model.

[0028] The process involves querying the hospital information management system via an interface to obtain the patient's historical treatment status, such as postoperative days and a summary of the previous medical record, and parsing this into a historical treatment status vector. Simultaneously, based on the patient's admission diagnosis code, a corresponding standard clinical pathway state map is loaded from the knowledge base. This map is a pre-defined model by medical experts, containing a set of all possible and legal clinical states for a specific disease at different treatment stages, along with the probabilities of transitions between states. The currently generated textual and image semantic units are then parsed, quantizing their text and image features into a current observed state vector. Using the historical treatment status vector as the initial state input to a finite state machine, and based on the transition probability matrix in the state map, all expected legal states for the next time step can be predicted. Finally, the current observed state vector is compared with this set of expected legal states. If the observed state belongs to this set, the state transition is deemed legal; otherwise, it is deemed illegal, triggering subsequent exception handling procedures.

[0029] S106. Based on the verified graphic semantic units, and according to the preset medical record template, the slots are filled and the medical terminology is standardized to generate a structured medical record data package and write it back to the hospital information management system.

[0030] The text-image semantic units, validated through clinical pathway logic, are already high-quality, highly reliable information fragments. The final step is to transform them into a structured medical record that conforms to medical standards and can be directly archived. This process begins by parsing the semantic category tags carried by each text-image semantic unit, which indicate its content attributes. Based on the type of ward round (e.g., initial ward round, postoperative ward round, or routine ward round) and the patient's specific treatment stage, the system automatically retrieves a perfectly matching medical record template from the template library. This template predefines a standardized document structure tree and a series of data slots to be filled. Subsequently, the system traverses all validated text-image semantic units, accurately mapping and filling the text descriptions and associated images contained within them into the corresponding data slots in the template based on their semantic category tags. After the initial filling is completed, to ensure professionalism and consistency, the system utilizes a large medical terminology ontology database to perform synonym retrieval and standardized mapping on all natural language text filled into the slots. Ultimately, this fully populated and terminologically standardized electronic medical record will be serialized into a structured data packet according to the data interaction interface specifications of the hospital information management system, and then securely written back to the hospital's central database through an encrypted transmission channel.

[0031] In one embodiment, reference is made to Figure 2 Based on the signal interaction characteristics between the doctor's mobile terminal and the target patient's bed area, a directional acquisition channel including spatial fencing and acoustic filtering is constructed, and a ward round session associated with the target patient is instantiated, including the following steps: S201. Analyze the received signal strength generated by the mobile terminal held by the doctor when it receives the wireless beacon broadcast in the area where the target patient's bed is located, and calculate the physical distance of the mobile terminal relative to the bed area based on the log-normal distribution propagation model; S202. When the physical distance between the mobile terminal and the bed area is less than the preset near-field interaction threshold, the mobile terminal is triggered to open the multi-microphone array and lock the unique identifier of the target patient carried by the wireless beacon to instantiate the ward round session associated with the target patient. S203. Calculate the time difference of arrival of the sound source signals received by the multi-microphone array, and estimate the azimuth angle of the main sound source relative to the mobile terminal by combining the spatial attitude parameters of the mobile terminal. S204. Calculate the weighted delay coefficient of the received signal of each microphone in the multi-microphone array based on the azimuth angle of the sound source, and use the delay summation beamforming algorithm to form a pickup beam pointing towards the bed area; S205. Based on the spatial response characteristics of the pickup beam, suppress background noise signals outside the pickup beam range, and establish a uniquely identified directional acquisition channel in the mobile terminal; S206. Continuously monitor the rate of change of the received signal strength, and terminate the ward round session and release the directional acquisition channel when the received signal strength is lower than the preset disconnection threshold.

[0032] In this embodiment, the mobile terminal continuously measures the received Signal Strength Indication (RSSI), a raw value reflecting signal attenuation. Since wireless signal propagation indoors is affected by multipath effects and obstacle obstruction, directly using the RSSI value is not reliable. Therefore, a log-normal distribution propagation model is introduced for distance estimation. This model can better fit the signal propagation characteristics of complex indoor environments. The physical distance can be calculated using the formula... This is accomplished by [the method described above], where d represents the estimated physical distance between the mobile terminal and the bedside beacon. It is a reference distance. At reference distance The reference signal strength measured at that location, It is the signal strength measured in real time. This refers to the path loss index, the value of which depends on the specific indoor environment layout and materials. Once the physical distance between the mobile terminal and the bed area is consistently less than a preset near-field interaction threshold, the mobile terminal's built-in multi-microphone array is activated, switching from standby to working mode. Simultaneously, it parses the unique identifier of the target patient carried in the wireless beacon broadcast data packet, and this identifier is locked by the mobile terminal and used to instantiate a ward round session object. The instantiation process includes allocating a dedicated data buffer in memory, establishing a corresponding data processing thread, and forcibly associating all subsequent data streams with this unique patient identifier.

[0033] After the ward round session is instantiated, in order to achieve directional sound pickup from a specific sound source, it is necessary to accurately estimate the location of the main sound source relative to the mobile terminal. This can be accomplished by analyzing the acoustic signals received by the multi-microphone array. Since the microphones are physically located at different positions on the mobile terminal, the arrival time of sound signals from the same sound source at each microphone will have a slight difference, known as the Time Difference of Arrival (TDOA). By calculating the TDOA between different microphone pairs, the direction of the sound source can be inferred. In this embodiment, the TDOA estimation method can be a generalized cross-correlation algorithm, especially a phase-transform-weighted generalized cross-correlation algorithm, which can effectively suppress reverberation interference. For a simple linear array consisting of two microphones, the relationship between the sound source azimuth angle and the TDOA can be approximately expressed as: ,in It is the estimated azimuth angle of the sound source. It is the calculated signal arrival time difference. λ is the speed of sound in air, and L is the physical distance between the two microphones. After obtaining the precise azimuth angle of the sound source, this information can be used to form a directional pickup beam, thus acoustically focusing on the target area. Delay-summation beamforming is a classic and efficient technique for achieving this goal. The core idea of ​​this algorithm is to calculate a specific delay compensation amount for the signal received by each microphone in the multi-microphone array, based on the azimuth angle of the sound source. This delay amount aligns the target sound source signals arriving at different microphones at different times in time.

[0034] For the i-th microphone in the array, the required delay is... It can be calculated through geometric relationships, specifically as follows: ,in It is the delay time that needs to be applied to the signal from the i-th microphone. It is the position vector of the microphone relative to the center of the array. It is a unit vector representing the direction of sound wave propagation, determined based on the estimated azimuth angle of the sound source. This refers to the speed of sound propagation. After precisely delaying the signal from each microphone, all the delayed signals are then summed. According to the principle of wave superposition, the signal from the target direction is significantly enhanced due to in-phase superposition, while interference signals from other directions cancel each other out due to phase discrepancies, ultimately synthesizing a highly directional output signal. The pickup beam generated by the delay-summation beamforming algorithm possesses unique spatial response characteristics, which is key to achieving background noise suppression. This beam forms a main lobe and multiple side lobes in space. The main lobe is the region with the highest signal gain, and its direction is perfectly aligned with the sound source azimuth estimated in the previous stage, efficiently capturing the sound of doctors and patients conversing. The signal gain in the side lobe regions is much lower than that of the main lobe, resulting in significant attenuation of sound source signals from these directions. This characteristic makes the pickup beam a natural spatial filter, effectively suppressing environmental noise outside the range of the main lobe, such as conversations near hospital beds, alarms from medical equipment, and footsteps in the corridor. Based on this acoustic focusing capability, the system establishes a logically independent directional acquisition channel at the mobile terminal software level, associated with a patient's unique identifier. All high signal-to-noise ratio audio data, after beamforming processing, is imported into this dedicated channel for caching and subsequent processing. This not only significantly improves the accuracy of speech recognition but also ensures the purity and specificity of the acquired data. To ensure the integrity of ward round sessions and effectively manage system resources, the session must be automatically and accurately terminated and the directional acquisition channel released when the doctor leaves the bedside area. Therefore, it is necessary to continuously monitor the rate of change of the received signal strength and terminate the ward round session and release the directional acquisition channel when the received signal strength falls below a preset disconnection threshold.

[0035] In one implementation, continuously monitoring the rate of change of the received signal strength and terminating the ward round session and releasing the directional acquisition channel when the received signal strength falls below a preset disconnection threshold includes the following steps: During the ward round session, the strength of the continuously sampled received signal is smoothed using the Kalman filter algorithm; Calculate the time derivative of the received signal strength after smoothing to determine the movement trend of the mobile terminal relative to the bed area; When the mobile trend shows that the mobile terminal is moving away from the bedside area and the received signal strength is consistently lower than the preset disconnection threshold, a timer is started to confirm the departure. If the timer reaches the preset time limit and the received signal strength is always lower than the disconnection threshold, it is determined that the mobile terminal has left the effective interaction range, triggering a session termination command, marking all data cached in the directional acquisition channel as session end state, and disabling the mobile terminal's multi-microphone array from occupying the directional acquisition channel; Generate a session integrity check code and append the session integrity check code to the end of the data packet for the room check session.

[0036] In this embodiment, the raw RSSI readings may experience irregular instantaneous fluctuations due to factors such as electromagnetic interference and personnel movement, and directly using these data would severely affect the stability of the judgment. Therefore, a Kalman filter algorithm is introduced to smooth this time series data. This algorithm treats the true signal strength as a system state that changes linearly with time, and treats each measured RSSI value as a noisy observation of that state. Through a recursive prediction-update loop, the filter can effectively estimate the signal strength closest to the true value from the noisy measurements. After processing by this algorithm, the output will be a smooth and continuous signal strength curve, which can more accurately reflect the macroscopic motion trajectory of the mobile terminal. After obtaining the smoothed received signal strength sequence, the next step is to quantitatively analyze the motion trend of the mobile terminal, that is, to determine whether it is approaching, moving away from, or remaining in the bedside area. This determination is achieved by calculating the time derivative of the smoothed signal strength curve.

[0037] At discrete sampling time points, the time derivative can be approximated using numerical differentiation methods, such as the backward difference method. The specific calculation method is as follows: In this formula It represents the rate of change of signal strength at the current time t, which is a quantitative indicator of the moving trend; It is the smoothed signal strength value at the current moment; It is the smoothed signal strength value from the previous sampling time. This is the time interval between two consecutive samples. (Calculated) The value has a clear physical meaning: if the value is positive, it means that the signal strength is increasing and the mobile terminal is moving closer to the bed; if it is negative, it means that the signal strength is decreasing and the mobile terminal is moving away from the bed; if it is close to zero, it means that the relative distance remains basically unchanged.

[0038] Relying solely on movement trends and instantaneous signal strength to determine whether to terminate a session can lead to misjudgments due to unforeseen circumstances such as a doctor briefly turning around or temporary obstruction. To improve the robustness of the decision-making, a departure confirmation mechanism based on two conditions is introduced. This mechanism is designed to start a timer of a preset duration only when two conditions are met simultaneously. The first condition is movement trend judgment, i.e., the time derivative of the signal strength calculated in the previous step continuously displays a negative value, clearly indicating that the mobile terminal is moving away from the bedside area. The second condition is absolute position judgment, i.e., the smoothed received signal strength itself has decreased and remains below a preset disconnection threshold. This threshold represents the edge of the effective ward round interaction range. Only when the trend of moving away and the state of being at the edge of the interaction area occur simultaneously does the system initially determine that the doctor may be leaving. If, during the timing, the signal strength rebounds and exceeds the disconnection threshold at any point, it indicates that the doctor may have only temporarily left or the signal may have been temporarily disturbed, the timer will be immediately reset, and the session will continue. Conversely, if the timer successfully completes the preset timeout and the signal strength remains below the disconnection threshold throughout this period, the system will make a final determination: the mobile terminal has completely left the effective interaction range. At this point, a session termination command is formally triggered. This command performs a series of resource release and data marking operations, including marking all data streams buffered in the directional acquisition channel as having ended in the session state, and issuing a command to the operating system to disable the mobile terminal's multi-microphone array, releasing it from its hardware resource occupation of the directional acquisition channel.

[0039] Finally, a standard cryptographic hash function, such as the SHA-256 algorithm, is used to serialize all data generated during the entire ward round session, including voice, images, operating instructions, and various metadata, into a unified data packet, and then calculate a fixed-length, unique digest value. The calculated checksum is appended to the end of the data packet and sent along with it. Upon receiving the data packet, the receiving system uses the same hash function to recalculate the checksum for the received data body and compares it with the original checksum appended to the end of the data packet. If the two match perfectly, it proves that the data transmission is complete and error-free, thus technically guaranteeing the authenticity and reliability of the original electronic medical record.

[0040] In one implementation, feature extraction of the ward round data sequence to generate audio semantic anchors containing medical entity information and visual semantic anchors containing visual subject features includes the following steps: The speech data stream was separated from the ward round data sequence, and the speech data stream was converted into a natural language text sequence using an automatic speech recognition model; The natural language text sequence is input into a pre-trained medical entity recognition model to extract a set of keywords containing medical entities, and the set of keywords is vectorized into audio semantic anchors; Image data streams were separated from the ward round data sequence, and image enhancement and normalization preprocessing were performed on each frame of the image data stream. The preprocessed image data stream is input into a pre-trained medical feature recognition model to identify target images with medical features in the image data stream and extract medical visual feature vectors of the regions of interest in the target images. The medical visual feature vectors are dimensionality reduced and image classification labels are added to generate visual semantic anchors corresponding to the target image.

[0041] In this implementation, the separated speech data is a continuous waveform signal, which cannot be directly analyzed by computer. Therefore, it must be converted into a structured text format. Specifically, an end-to-end model based on the Transformer architecture, such as the Conformer model, can be used. This model combines the ability of convolutional neural networks to capture local features with self-attention mechanisms to handle global dependencies, making it particularly suitable for processing continuous speech streams. To ensure high accuracy in medical scenarios, the model needs to be trained on a large-scale speech dataset containing various medical terms, different accents, and speech recorded under hospital background noise. During the recognition process, the input raw audio waveform undergoes feature extraction and is then fed into the Conformer network for acoustic modeling and decoding, ultimately outputting a word sequence with timestamps.

[0042] Text content extraction can be achieved using a pre-trained medical entity recognition model on massive amounts of medical literature and electronic medical records, such as BioBERT or ClinicalBERT models based on the BERT architecture. These models can deeply understand the context of medical text. The input text sequence is first decomposed into a series of tokens, and then the model predicts a predefined entity label for each token, such as disease, symptom, examination, treatment, body part, etc. The identified tokens with specific labels are combined to form a keyword set containing specific medical entities. To enable quantitative comparison and calculation of this keyword set, it needs to be converted into a fixed-dimensional mathematical vector, i.e., audio semantic anchors. An effective vectorization method is to perform a weighted average of the word embedding vectors of all keywords in the set.

[0043] In parallel with processing the speech data stream, the image data stream within the ward round data sequence also needs to be analyzed. Before any advanced recognition tasks, a series of preprocessing operations must be performed on the raw images to improve image quality and make them meet the input data requirements of subsequent models. First, all image data frames are separated from the sequence based on type labels in the data stream. Considering that images taken during ward rounds may have uneven lighting or low contrast, image enhancement techniques are applied. For example, a contrast-limited adaptive histogram equalization algorithm can effectively enhance the local contrast of the image, making the details of lesions or wounds more clearly visible, while avoiding excessive amplification of noise. Subsequently, normalization processing is required to eliminate pixel value differences in images taken under different devices and lighting conditions. The preprocessed image data stream is then fed into a pre-trained medical feature recognition model to automatically identify and extract the key visual information it contains. This process typically consists of two stages.

[0044] The first stage is target image screening, which involves identifying keyframes containing valid medical information from a continuous image stream. This can be achieved using an efficient image classification model (such as MobileNet or EfficientNet), trained to distinguish images containing medical features like wounds, rashes, drainage tubes, and monitor readings from blurry, irrelevant, or blank images. After selecting the target images, the second stage, feature extraction, begins. A convolutional neural network, such as a model pre-trained on a large medical image dataset (e.g., ResNet, fine-tuned on medical data after ImageNet), is used to extract deep visual features of regions of interest in the target images. Finally, the extracted high-dimensional medical visual feature vectors are transformed into visual semantic anchors that are easier to compute and store. Directly using the original high-dimensional feature vectors suffers from high computational costs, large storage overhead, and the potential inclusion of redundant information. Therefore, dimensionality reduction is necessary. Principal component analysis (PCA) can be used, which aims to find the direction with the largest variance in the data and project the original data into a new coordinate system composed of a few principal components, thereby significantly reducing the data dimensionality while preserving the original information to the greatest extent possible. After dimensionality reduction, the image classification labels output by the medical feature recognition model from the previous stage are appended to this low-dimensional vector. Ultimately, this composite structure, consisting of the low-dimensional feature vector and the classification labels, constitutes a visual semantic anchor point that precisely corresponds to the target image.

[0045] In one implementation, the semantic similarity between audio semantic anchors and visual semantic anchors is calculated, and audio semantic anchors and visual semantic anchors with semantic similarity exceeding a preset similarity threshold are non-linearly associated across the time dimension to construct a text-image semantic unit, including the following steps: A cross-modal semantic feature space is constructed by combining all audio semantic anchors and visual semantic anchors; Calculate the cosine similarity between each audio semantic anchor vector and each visual semantic anchor vector in the cross-modal semantic feature space to form a many-to-many semantic similarity matrix; Traverse the semantic similarity matrix and filter out candidate anchor pairs whose cosine similarity values ​​exceed a preset association threshold; Ignoring the original timestamp order in the ward round data sequence, the maximum weight matching algorithm is applied to all candidate anchor pairs to determine the optimal pairing relationship between audio semantic anchors and visual semantic anchors. Based on the optimal pairing relationship, the text segment corresponding to the successfully paired audio semantic anchor point is bound to the original image corresponding to the visual semantic anchor point, and encapsulated into a graphic semantic unit with independent logical meaning. For the remaining audio semantic anchors or remaining visual semantic anchors that fail to match, secondary weak association merging is performed by selecting the matched text-image semantic units with the closest time interval based on the original timestamps of the remaining audio semantic anchors or remaining visual semantic anchors.

[0046] In this implementation, a cross-modal semantic feature space is first constructed to map the inherently heterogeneous audio and visual semantic anchor vectors into the same high-dimensional vector space. Achieving this typically relies on a specially trained deep neural network model. This model generally contains two independent encoder branches: a text encoder to process the text information in the audio semantic anchors, and an image encoder to process the image information in the visual semantic anchors. The two encoders convert their respective inputs into high-dimensional feature vectors, which are then linearly or non-linearly transformed into a final semantic space with identical dimensions through a shared projection head network. By employing training strategies such as contrastive learning and training on a large amount of image-text pair data, the model learns to bring semantically related image-text pairs closer together in the space, while pushing unrelated image-text pairs further apart. In the unified cross-modal semantic feature space, all anchor vectors are comparable, allowing for the accurate calculation of the semantic relevance between any audio anchor and visual anchor. For all audio and visual semantic anchors generated during the ward round, the system performs pairwise pairing and calculates the cosine similarity between each pair, ultimately generating a many-to-many semantic similarity matrix. This matrix comprehensively depicts the potential semantic association strength between each verbal description and each visual observation record during the ward round.

[0047] After obtaining the matrix containing all pair similarity scores, it is necessary to filter out anchor pairs with high potential relevance to narrow the search scope of the subsequent optimal matching algorithm. This filtering process is achieved by setting a preset relevance threshold. The system iterates through each element in the semantic similarity matrix and compares it with the preset relevance threshold. Only anchor pairs whose cosine similarity values ​​clearly exceed the threshold are considered semantically close enough to qualify as candidates for the next round of matching. Determining the final matching relationship from the set of candidate anchor pairs cannot be done using a simple greedy strategy; instead, a globally optimal solution should be sought. To achieve this, this step introduces the maximum weight matching algorithm from graph theory. A key feature of this method is that it completely ignores the timestamp order of each anchor in the original ward round data sequence, making decisions purely based on semantic similarity, thereby enabling the establishment of non-linear associations across the time dimension. In specific implementation, all candidate audio semantic anchors and visual semantic anchors can be regarded as two sets of vertices in a bipartite graph, and the cosine similarity values ​​between candidate anchor pairs are used as the weights of the edges connecting the corresponding vertices. The goal of the maximum weight matching algorithm is to find a matching in a weighted bipartite graph, that is, a set of edges with no common vertices, such that the sum of the weights of these edges is maximized.

[0048] For each successfully paired audio and visual semantic anchor, the system backtracks to the original data, firmly binding the original natural language text segment corresponding to the audio anchor with the original image data corresponding to the visual anchor. This binding process is not merely a simple data concatenation; it encapsulates text, images, their respective original timestamps, type tags, and semantic similarity scores, among other metadata, into a text-image semantic unit. Each text-image semantic unit represents a highly semantically consistent multimodal observation event during the ward rounds; for example, a speech describing "red and swollen wounds" and a photograph clearly showing the redness of a wound are combined. After completing the optimal matching and encapsulation of text-image semantic units, there are usually some isolated anchors that failed to match—the remaining audio or visual semantic anchors. To preserve information to the maximum extent, the system employs a secondary weak association strategy based on temporal proximity to handle these remaining anchors. Specifically, for each unmatched remaining anchor, the system obtains its absolute timestamp in the original ward round data sequence. Then, iterate through all successfully constructed semantic units (SMUs) and calculate the absolute time difference between the timestamp of the remaining anchor and the audio or image timestamp contained in each SMU. Finally, the remaining anchor is merged into the matching SMU with the smallest time interval. During merging, this anchor is marked with a special tag indicating weak association or temporal proximity to distinguish it from the strong association determined by semantic similarity calculation.

[0049] In one implementation, before applying the maximum weight matching algorithm to all candidate anchor pairs to determine the optimal pairing relationship between audio semantic anchors and visual semantic anchors, the following steps are included: Obtain the original image frame corresponding to the visual semantic anchor point and the keyword feature vector corresponding to the audio semantic anchor point in the candidate anchor point pair; The keyword feature vector is used as a guiding signal for the target category and input back into the channel attention layer of the medical feature recognition model that extracts visual semantic anchors. Gradient weighted class activation mapping is calculated on the original image frame to generate a class activation heatmap that reflects the distribution of visual response of the keyword feature vector in the original image frame. An adaptive threshold segmentation algorithm is used to binarize the class activation heatmap and extract the semantically focused region in the original image frame that is excited by the semantics of the keyword feature vector. The original image frame was processed using edge detection algorithm and morphological closing operation to extract the outline mask of the physical lesions that were visible in the original image frame; Calculate the cross-union ratio (CUR) between the semantic focus region and the physical lesion contour mask, and define the CUR as the spatial alignment fidelity index of the candidate anchor point pair. The spatial alignment fidelity index is used as a gain coefficient to weight and update the cosine similarity values ​​of candidate anchor pairs in the many-to-many semantic similarity matrix, so that the subsequent maximum weight matching algorithm can use the updated cosine similarity values ​​for optimal pairing.

[0050] In this implementation, for each candidate anchor pair, the system performs two parallel retrieval operations. The first operation involves tracing back to the original ward round data sequence based on the index information stored within the visual semantic anchor, locating and loading its corresponding unprocessed original image frame. This image frame is the fundamental data source containing all pixel information and visual context. The second operation involves extracting the set of keywords identified before vectorization from the audio semantic anchor in the candidate anchor pair, and converting these keywords into corresponding feature vectors using a word embedding model. Then, an improved gradient-weighted class activation mapping technique is used to treat the keyword feature vector as a dynamic target category guiding signal. This keyword feature vector is fed back into the end of the medical recognition model previously used to extract visual features, such as the convolutional layer before the channel attention module or global average pooling layer. With the goal of maximizing the cosine similarity between the model's output visual features and the input keyword feature vector, the gradient of this similarity score is calculated relative to each feature map in the selected convolutional layer. The global average of this gradient serves as the weight of each feature map, calculated using the following formula: ,in Z is the weight of the k-th feature map, and Z is the number of pixels in the feature map. It is the similarity score between visual features and keyword vectors. This is the activation value of the k-th feature map at position (i,j). By summing all feature maps according to this weight and then processing them through the ReLU activation function, a class activation heatmap is generated.

[0051] After generating a class-activated heatmap with continuous grayscale values, it needs to be converted into a well-defined binary region for accurate geometric comparison. Directly using a globally uniform threshold for segmentation often fails to adapt to uneven activation intensity distributions in the heatmap. Therefore, adaptive thresholding algorithms, such as Otsu's method (maximum inter-class variance method), can more intelligently determine the segmentation boundaries. Otsu's method iterates through all possible grayscale levels as candidate thresholds and calculates the inter-class variance for each candidate threshold, classifying heatmap pixels into foreground (high-activation areas) and background (low-activation areas). The algorithm aims to find the grayscale threshold that maximizes this inter-class variance, as it is considered the optimal segmentation point that best distinguishes the two classes. Once the optimal threshold is determined, the entire heatmap can be binarized: all pixels with values ​​higher than the threshold are set to 1 (foreground), and those with values ​​lower than or equal to the threshold are set to 0 (background). Thus, a clearly defined binary mask is precisely extracted from the fuzzy heatmap. The area enclosed by this mask is the visual region focused by the semantics of the keyword feature vector.

[0052] Next, it is necessary to extract the physically real lesion contours from the image itself as a benchmark for comparison. First, the Canny edge detection algorithm is applied to process the original image frame. This algorithm, through a series of steps including Gaussian filtering, gradient calculation, non-maximum suppression, and double-threshold hysteresis connection, can accurately identify edge pixels in the image where brightness changes drastically, thus outlining the preliminary contours of lesions, wounds, or other significant features. However, the edges obtained in this way are often discontinuous, broken lines. To form a closed and solid region mask, morphological closing operations are then performed. The specific operation of the closing operation is to first dilate the binarized edge image to connect adjacent broken edges, and then erode it to restore the original contour size of the object. Now we have two binary masks from the same image: one is a "semantic focus region" generated by text semantics, and the other is a "physical lesion contour mask" extracted through image processing. To quantify the degree of consistency between these two in spatial location, the intersection-union ratio (IUGR) can be used as the core evaluation metric. The IUGR is calculated by dividing the area of ​​the intersection of the two regions by the area of ​​the total covered region. Its mathematical expression is: ,in It is the calculated crossover ratio (CROR) value. A mask representing the semantically focused region. This represents the mask of the physical lesion outline. The result of this calculation is a value between 0 and 1. When the value is close to 1, it indicates that the visual area corresponding to the text description highly overlaps with the actual physical lesion in the image; when the value is close to 0, it indicates that the two are almost spatially unrelated.

[0053] Finally, the newly obtained spatial alignment fidelity index is integrated into the original semantic similarity evaluation system to optimize the final matching decision. Specifically, the calculated spatial alignment fidelity index is used as a gain coefficient to weight and update the original cosine similarity value of the candidate anchor pair in the many-to-many semantic similarity matrix. This can be done directly using a multiplicative approach. This weighting method preserves or even amplifies the high semantic similarity of anchor pairs whose text semantics are highly aligned with the image content in space; however, for anchor pairs that are related in abstract semantics but cannot be aligned in the specific image space, their similarity scores will be significantly penalized. After this round of weighted updates, the values ​​in the semantic similarity matrix will more comprehensively reflect the overall quality of image-text matching.

[0054] In one implementation, the historical treatment status of the target patient is obtained and the corresponding standard clinical pathway status map is loaded. The semantic units of the graph are mapped to the current observation status. The legality of the transition between the current observation status and the historical treatment status is verified by a finite state machine model, including the following steps: By querying the hospital information management system, the postoperative days and the medical record data of the previous time point of the target patient are obtained, and the historical treatment status vector of the target patient is obtained by parsing. Based on the admission diagnosis code of the target patient, the corresponding standard clinical pathway state map is invoked. The standard clinical pathway state map contains the set of legal states that are allowed to exist at different time points for the disease to which the target patient belongs, as well as the transition probability matrix between states. The text and image features in the semantic units of the image are analyzed, and the text and image features are quantified into the current observation state vector reflecting the current physiological indicators of the target patient; The historical treatment state vector is used as the current state input of the finite state machine model, and the expected set of legal states at the next moment is predicted based on the transition probability matrix. Determine whether the current observed state vector belongs to the expected set of legal states. If the current observed state vector belongs to the expected set of legal states, the state transition is deemed legal. If the current observed state vector does not belong to the expected set of legal states, the state transition is deemed illegal.

[0055] In this implementation, the query command will be indexed by the target patient's unique identifier. The data returned by the system mainly includes two parts: first, structured key diagnostic and treatment parameters, such as postoperative days and vital sign records; second, the medical record text from the previous time point. For structured data, it can be directly extracted and assigned values. For unstructured medical record text, a natural language processing flow needs to be initiated, using medical entity recognition and relation extraction technologies to parse out key information such as symptoms, signs, and diagnoses, and map them into a standardized medical coding system. Finally, all key features representing the patient's previous state extracted from structured data and unstructured text are integrated and encoded into a fixed-dimensional numerical vector, namely the historical diagnostic and treatment state vector. After obtaining the patient's historical state, an authoritative medical knowledge model needs to be retrieved to determine whether the evolution from the historical state to the current state conforms to accepted medical practices. This knowledge model is a standard clinical pathway state map retrieved from a pre-set clinical pathway knowledge base based on the diagnostic code determined when the target patient was admitted. This atlas is essentially an expert system that formally defines and enumerates the various clinical states that may occur during the entire treatment cycle for a specific disease. For example, "24 hours post-surgery, vital signs are stable, and there is no wound exudation" is a specific state. The atlas also embeds a transition probability matrix between states, which precisely quantifies the probability of transitioning from any state to another.

[0056] For the text portion of the image-text semantic unit, keywords such as symptoms, signs, and medications can be extracted using named entity recognition and converted into standardized medical concept codes. For numerical descriptions, such as "body temperature 38.5 degrees Celsius," the numerical value is directly extracted. For the image portion, the attached classification labels and semantic information decoded from the visual feature vector are utilized. Subsequently, all features parsed from the text and image are arranged and filled according to the same dimensions and order as the historical treatment state vector. In this way, the multimodal observation results of this ward round are quantified into a current observation state vector that can be directly mathematically compared with the historical state. After vectorizing both the historical and current states, the logical reasoning process based on the finite state machine model can be initiated. In this model, the historical treatment state vector obtained in the previous step is mapped to a specific starting state in the state graph, denoted as . Then, using the transition probability matrix in the loaded state graph, it is possible to predict which states a given initial state is most likely to evolve into at the next time step. Specifically, the system searches the transition probability matrix for states that are... The state consists of all transition probability values ​​from the initial row. All target states with transition probabilities greater than a preset minimum threshold (to exclude theoretically impossible transitions) will be grouped together to form a set of expected legal states. This includes a patient who is in accordance with standard clinical pathways. The patient in this condition is most likely to, and should, exhibit all the clinical manifestations that will follow.

[0057] Finally, the current observed state vector is used to find its unique discrete state in the standard clinical pathway state graph through table lookup or nearest neighbor matching. Then, it is determined whether this discrete state exists in the set of expected legal states predicted in the previous step. If it does, the state transition from the historical state to the current observed state is considered legal. This means that the patient's condition changes recorded during this ward round fully conform to the expected evolution of the standard clinical pathway. Conversely, if it does not exist in the set, the state transition is considered illegal. This indicates that the patient's current condition deviates from the normal recovery trajectory, potentially suggesting complications or errors in the data collection and analysis process, requiring subsequent anomaly handling or manual review.

[0058] In one implementation, after determining that the current observed state vector does not belong to the expected set of legal states, the following steps are also included: Obtain a preset modality confidence weight table. The modality confidence weight table sets the confidence priority of different modalities in different scenarios based on image recognition accuracy and speech recognition accuracy. Identify feature conflicts that lead to illegal state transitions, and compare the confidence weights of the image features and text features corresponding to the feature conflicts. If the confidence weight of the image features is higher than that of the text features, then generate an automatic correction instruction based on the image features to correct the text features. If the difference between the confidence weight of the image feature and the confidence weight of the text feature is less than a preset fuzzy threshold, an abnormal state warning mark is generated and attached to the corresponding image and text semantic unit. Apply automatic correction instructions to text-image semantic units to update the current observation state vector, or place text-image semantic units with abnormal state warning markers into the queue for manual review. The updated graphic semantic unit is then input into the finite state machine model for secondary logical closed-loop verification.

[0059] In this implementation, when a state transition is deemed illegal, it means that there is at least one conflict between text and image information within the image-text semantic unit, requiring the initiation of an arbitration mechanism to resolve this contradiction. This mechanism is based on a pre-defined modality confidence weight table. This table is not static but dynamically generated based on a quantitative evaluation of the underlying AI model's performance. Specifically, the system calculates the precision and recall of the image recognition model for different clinical observation scenarios, as well as the performance metrics of the automatic speech recognition and entity extraction models on the corresponding text descriptions. Based on this performance data, a confidence weight is assigned to the image modality and text modality for each scenario. This weight W(M,C) is a function that takes the modality type M (image or text) and the clinical scenario C as input and outputs a numerical value representing the confidence level.

[0060] After obtaining the modality confidence weight table, the specific reasons for the illegal state transitions are precisely located. By comparing the current observed state vector with the expected set of legal states, the system can identify which feature dimensions or features are mismatched; these mismatched dimensions are the feature conflict terms. For example, a clinical pathway might expect a wound to be in a "mildly infected" state, while the image-text semantic unit resolves to "well-healed." The system will trace back to the source of this "well-healed" conclusion, finding its corresponding image and text features. Next, it queries the modality confidence weight table to obtain the confidence weights of the image modality and the text modality in a specific clinical scenario (i.e., wound healing assessment). A comparison is then made: if the confidence weight of the image features is significantly higher than that of the text features, the system determines that the image evidence is more reliable and generates an automatic correction instruction. This instruction explicitly states that the text portion of the image-text semantic unit should be corrected based on the true situation revealed by the image features, thereby eliminating the contradiction between modalities.

[0061] In some cases, the information provided by different modalities may contain a degree of uncertainty, or their confidence weights may be very close. When the system compares the confidence weights of image features and text features corresponding to conflicting features, if the absolute difference between them is less than a preset fuzziness threshold, the system will not rashly perform automatic correction. This small difference in weights suggests that a highly certain decision cannot be made based solely on the current algorithm and data. In this situation, the system's primary principle is to ensure medical safety and avoid potential erroneous corrections. Therefore, a specific type of abnormal state warning flag is generated and attached to the metadata of the image-text semantic unit that caused the conflict, clearly indicating that there is an unresolved data conflict.

[0062] If an automatic correction instruction is generated, its content is parsed, the text portion of the target semantic unit that needs modification is located, and more reliable semantic information extracted from image features is used to replace or update the original text description. For example, "dry wound" is corrected to "minor wound oozing." After the text content is updated, an updated current observation state vector is recalculated based on the modified semantic unit. If an abnormal state warning marker is generated, no modifications are made to the original data. Instead, the semantic unit with the warning marker is separated from the main processing flow and pushed as an independent data packet to a waiting queue designed for manual review. Whether automatic correction is performed or the problematic unit is transferred to manual review, the process is not finished for the automatic correction path. To ensure the effectiveness and correctness of the automatic correction operation, a closed-loop verification must be performed.

[0063] Specifically, the updated current observation state vector generated in the previous step based on the corrected text-text semantic unit will be fed back into the finite state machine model as input. The system will repeat the exact same logical verification process: comparing the updated current state with the historical state and determining whether the new state transition falls within the expected set of legal states defined by the standard clinical pathway. This secondary verification process constitutes a crucial quality control step. If the verification passes, it means that the automatic correction has successfully resolved the logical conflict, and the text-text semantic unit can proceed to the final medical record generation stage. If the secondary verification still fails, it indicates that the root cause of the conflict may be more complex. In this case, the system will also push this corrected but still problematic unit to the manual review queue to prevent any potential errors from entering the final medical record.

[0064] In one implementation, the process of generating a structured medical record data package and writing it back to the hospital information management system based on verified graphic semantic units, and after filling slots and standardizing medical terminology according to a preset medical record template, includes the following steps: The semantic category tags contained in the verified image and text semantic units are analyzed to determine the medical document paragraph attributes corresponding to the image and text semantic units; Based on the type of ward round session and the current stage of treatment of the target patient, a matching medical record template is retrieved from the template library. The medical record template defines a standard document structure tree and data slots. Iterate through all text and image semantic units, and map and populate the text and image semantic units into one or more corresponding data slots in the medical record template according to the paragraph attributes of the medical document. Using a pre-defined medical terminology ontology database, synonym retrieval and standardized mapping are performed on the natural language text populated into the data slots; In accordance with the data interaction interface specifications of the hospital information management system, the completed medical record template is serialized into a structured medical record data packet, and the structured medical record data packet is sent to the hospital information management system through an encrypted transmission channel.

[0065] In this implementation, each text-image semantic unit contains one or more semantic category tags during generation. These tags are assigned by analyzing the text content using a natural language understanding model (e.g., determining whether a sentence is a subjective statement or an objective description) and the visual content using an image classification model (e.g., determining whether an image is a vital signs monitor or a close-up of a wound). For example, the tags might be "subjective feelings," "vital signs," "wound assessment," or "treatment plan." The system parses these tags and matches them with standard medical document paragraph attributes according to a predefined mapping rule. Next, based on the specific context of the current ward round, a precisely matched progress note template is retrieved from the template library. The template library contains a large number of standardized document templates for different scenarios. The system considers two key dimensions when selecting a template: first, the type of ward round session, such as "admission record," "initial progress note," "postoperative progress note," or "routine progress note"; and second, the specific treatment stage of the target patient, such as "day 3 post-surgery" or "second cycle of chemotherapy." By using information from these two dimensions as query criteria, the system can accurately locate and load the most suitable medical record template from the template library. This template is not a blank document, but a predefined framework with a complete document structure tree, containing all necessary chapters, headings, and a series of data slots to be filled, each associated with a specific medical document paragraph attribute.

[0066] The system iterates through all validated text-image semantic units. For each unit, it first reads the medical document paragraph attribute assigned to it in the previous steps. Then, based on this attribute, it searches for a corresponding, unfilled data slot in the loaded medical record template. Once a matching slot is found, the system injects the natural language text fragments and associated images contained in the text-image semantic unit into that slot. In some cases, a semantic unit may contain composite information, and its content can be mapped to multiple different slots in the template simultaneously. For example, a text describing "patient's temperature 38.5℃, accompanied by chills" should have its "temperature" section filled in the vital signs slot for objective data, while its "chills" section should be filled in the symptoms slot for subjective data.

[0067] After the initial slot filling is completed, a comprehensive medical terminology standardization process must be performed. This process relies on a large and continuously updated medical terminology ontology, which stores a vast amount of standard medical terms and their synonyms, near-synonyms, colloquialisms, abbreviations, and other variant forms. The system traverses all natural language text that has been filled into each data slot, using this ontology for synonym retrieval. For example, when the text contains "stomach ache," the system will find that the corresponding standard medical term is "abdominal pain" and automatically replace it. Finally, once the entire medical record template is fully filled and all text has undergone terminology standardization, the completed medical record template is serialized according to the HIS's predefined data interaction interface specifications. Specifically, the completed medical record template is first converted into a common structured data format, such as XML or JSON. After serialization, a structured medical record data package is generated. To ensure patient privacy and data security, this data package is encrypted using an encryption algorithm (such as AES-256) before transmission. The encrypted data packet is then sent to the HIS receiving server via a secure, encrypted transmission channel (such as HTTPS or a VPN-based channel). Upon receiving the data, the server decrypts and verifies its integrity, and then successfully stores the data in the patient's electronic medical record.

[0068] The processor can be a central processing unit (CPU). Of course, depending on the actual use, it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc., and this application does not limit it.

[0069] The memory can be an internal storage unit of a computer device, such as a hard disk or RAM, or an external storage device, such as a plug-in hard disk, smart memory card (SMC), secure digital card (SD), or flash memory card (FC) provided on the computer device. Furthermore, the memory can be a combination of internal storage units and external storage devices of a computer device. The memory is used to store computer programs and other programs and data required by the computer device. The memory can also be used to temporarily store data that has been output or will be output. This application does not limit this.

[0070] The present invention also discloses a computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to be configured to perform the disease record generation method based on mobile terminal and multimodal data described in any of the above embodiments.

[0071] The computer program can be stored in a machine-readable medium. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or certain middleware. The machine-readable medium includes any entity or device capable of carrying computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the machine-readable medium includes, but is not limited to, the above-mentioned components.

[0072] The method for generating medical records based on mobile terminals and multimodal data in the above embodiments is stored in the computer-readable storage medium and loaded and executed on the processor to facilitate the storage and application of the above method.

[0073] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of protection of this application is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of one or more embodiments of this application as described above, which are not provided in detail for the sake of brevity.

[0074] One or more embodiments in this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of this application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments in this application should be included within the protection scope of this application.

Claims

1. A method for generating medical records based on mobile terminals and multimodal data, characterized in that, Includes the following steps: Based on the signal interaction characteristics between the doctor's mobile terminal and the target patient's bed area, a directional acquisition channel including spatial fencing and acoustic filtering is constructed, and a ward round session associated with the target patient is instantiated. During ward rounds, multiple types of discrete ward round record data streams are acquired through multiple parallel input interfaces of the mobile terminal, and each ward round record data stream is assigned an absolute timestamp and type label to form a ward round data sequence. The ward round record data stream includes voice data stream, image data stream, and operation instruction data stream. Feature extraction was performed on the ward round data sequence to generate audio semantic anchors containing medical entity information and visual semantic anchors containing visual subject features, respectively. Calculate the semantic similarity between audio semantic anchors and visual semantic anchors, and perform non-linear association across time dimension between audio semantic anchors and visual semantic anchors whose semantic similarity exceeds a preset similarity threshold to construct text-image semantic units; The system acquires the historical treatment status of the target patient and loads the corresponding standard clinical pathway status map. It maps the text and image semantic units to the current observation status and verifies the legality of the transition between the current observation status and the historical treatment status through a finite state machine model. Based on the validated image and text semantic units, and according to the preset medical record template, slot filling and medical terminology standardization are performed to generate a structured medical record data package and write it back to the hospital information management system.

2. The method for generating medical records based on mobile terminals and multimodal data according to claim 1, characterized in that, The process of constructing a directional acquisition channel, incorporating spatial fencing and acoustic filtering, based on the signal interaction characteristics between the doctor's mobile terminal and the target patient's bed area, and instantiating a ward round session associated with the target patient, includes the following steps: The signal strength received by the doctor's mobile terminal when it receives a wireless beacon broadcast in the area where the target patient's bed is located is analyzed, and the physical distance between the mobile terminal and the bed area is calculated based on the log-normal distribution propagation model. When the physical distance between the mobile terminal and the bed area is less than the preset near-field interaction threshold, the mobile terminal is triggered to activate the multi-microphone array and lock the unique identifier of the target patient carried by the wireless beacon to instantiate the ward round session associated with the target patient. Calculate the time difference of arrival of the sound source signals received by the multi-microphone array, and estimate the azimuth angle of the main sound source relative to the mobile terminal by combining the spatial attitude parameters of the mobile terminal. The weighted delay coefficient of the received signal of each microphone in the multi-microphone array is calculated based on the azimuth angle of the sound source, and a pickup beam pointing to the bed area is formed by using a delay summation beamforming algorithm. Based on the spatial response characteristics of the pickup beam, background noise signals outside the pickup beam range are suppressed, and a uniquely identified independent directional acquisition channel is established in the mobile terminal. The system continuously monitors the rate of change in received signal strength and terminates the ward round session and releases the directional acquisition channel when the received signal strength falls below a preset disconnection threshold.

3. The method for generating medical records based on mobile terminals and multimodal data according to claim 2, characterized in that, The process of continuously monitoring the rate of change of the received signal strength and terminating the ward round session and releasing the directional acquisition channel when the received signal strength falls below a preset disconnection threshold includes the following steps: During the ward round session, the strength of the continuously sampled received signal is smoothed using the Kalman filter algorithm; Calculate the time derivative of the received signal strength after smoothing to determine the movement trend of the mobile terminal relative to the bed area; When the mobile trend shows that the mobile terminal is moving away from the bedside area and the received signal strength is consistently lower than the preset disconnection threshold, a timer is started to confirm the departure. If the timer reaches the preset time limit and the received signal strength is always lower than the disconnection threshold, it is determined that the mobile terminal has left the effective interaction range, triggering a session termination command, marking all data cached in the directional acquisition channel as session end state, and disabling the mobile terminal's multi-microphone array from occupying the directional acquisition channel; Generate a session integrity check code and append the session integrity check code to the end of the data packet for the room check session.

4. The method for generating medical records based on mobile terminals and multimodal data according to claim 1, characterized in that, The step of extracting features from the ward round data sequence to generate audio semantic anchors containing medical entity information and visual semantic anchors containing visual subject features includes the following steps: The speech data stream was separated from the ward round data sequence, and the speech data stream was converted into a natural language text sequence using an automatic speech recognition model; The natural language text sequence is input into a pre-trained medical entity recognition model to extract a set of keywords containing medical entities, and the set of keywords is vectorized into audio semantic anchors; Image data streams were separated from the ward round data sequence, and image enhancement and normalization preprocessing were performed on each frame of the image data stream. The preprocessed image data stream is input into a pre-trained medical feature recognition model to identify target images with medical features in the image data stream and extract medical visual feature vectors of the regions of interest in the target images. The medical visual feature vectors are dimensionality reduced and image classification labels are added to generate visual semantic anchors corresponding to the target image.

5. The method for generating medical records based on mobile terminals and multimodal data according to claim 1, characterized in that, The process of calculating the semantic similarity between audio semantic anchors and visual semantic anchors, and then performing a non-linear cross-time association on audio semantic anchors and visual semantic anchors with semantic similarity exceeding a preset similarity threshold to construct a text-image semantic unit includes the following steps: A cross-modal semantic feature space is constructed by combining all audio semantic anchors and visual semantic anchors; Calculate the cosine similarity between each audio semantic anchor vector and each visual semantic anchor vector in the cross-modal semantic feature space to form a many-to-many semantic similarity matrix; Traverse the semantic similarity matrix and filter out candidate anchor pairs whose cosine similarity values ​​exceed a preset association threshold; Ignoring the original timestamp order in the ward round data sequence, the maximum weight matching algorithm is applied to all candidate anchor pairs to determine the optimal pairing relationship between audio semantic anchors and visual semantic anchors. Based on the optimal pairing relationship, the text segment corresponding to the successfully paired audio semantic anchor point is bound to the original image corresponding to the visual semantic anchor point, and encapsulated into a graphic semantic unit with independent logical meaning. For the remaining audio semantic anchors or remaining visual semantic anchors that fail to match, secondary weak association merging is performed by selecting the matched text-image semantic units with the closest time interval based on the original timestamps of the remaining audio semantic anchors or remaining visual semantic anchors.

6. The method for generating medical records based on mobile terminals and multimodal data according to claim 1, characterized in that, The process of obtaining the target patient's historical treatment status and loading the corresponding standard clinical pathway status map, mapping the textual semantic units to the current observation status, and verifying the legality of the transition between the current observation status and the historical treatment status using a finite state machine model includes the following steps: By querying the hospital information management system, the postoperative days and the medical record data of the previous time point of the target patient are obtained, and the historical treatment status vector of the target patient is obtained by parsing. Based on the admission diagnosis code of the target patient, the corresponding standard clinical pathway state map is invoked. The standard clinical pathway state map contains the set of legal states that are allowed to exist at different time points for the disease to which the target patient belongs, as well as the transition probability matrix between states. The text and image features in the semantic units of the image are analyzed, and the text and image features are quantified into the current observation state vector reflecting the current physiological indicators of the target patient; The historical treatment state vector is used as the current state input of the finite state machine model, and the expected set of legal states at the next moment is predicted based on the transition probability matrix. Determine whether the current observed state vector belongs to the expected set of legal states. If the current observed state vector belongs to the expected set of legal states, the state transition is deemed legal. If the current observed state vector does not belong to the expected set of legal states, the state transition is deemed illegal.

7. The method for generating medical records based on mobile terminals and multimodal data according to claim 6, characterized in that, After determining that the current observed state vector does not belong to the expected set of legal states, the following steps are also included: Obtain a preset modality confidence weight table. The modality confidence weight table sets the confidence priority of different modalities in different scenarios based on image recognition accuracy and speech recognition accuracy. Identify feature conflicts that lead to illegal state transitions, and compare the confidence weights of the image features and text features corresponding to the feature conflicts. If the confidence weight of the image features is higher than that of the text features, then generate an automatic correction instruction based on the image features to correct the text features. If the difference between the confidence weight of the image feature and the confidence weight of the text feature is less than a preset fuzzy threshold, an abnormal state warning mark is generated and attached to the corresponding image and text semantic unit. Apply automatic correction instructions to text-image semantic units to update the current observation state vector, or place text-image semantic units with abnormal state warning markers into the queue for manual review. The updated graphic semantic unit is then input into the finite state machine model for secondary logical closed-loop verification.

8. The method for generating medical records based on mobile terminals and multimodal data according to claim 1, characterized in that, The process of generating a structured medical record data package based on verified graphic semantic units, filling slots and standardizing medical terminology according to a preset medical record template, and writing it back to the hospital information management system includes the following steps: The semantic category tags contained in the verified image and text semantic units are analyzed to determine the medical document paragraph attributes corresponding to the image and text semantic units; Based on the type of ward round session and the current stage of treatment of the target patient, a matching medical record template is retrieved from the template library. The medical record template defines a standard document structure tree and data slots. Iterate through all text and image semantic units, and map and populate the text and image semantic units into one or more corresponding data slots in the medical record template according to the paragraph attributes of the medical document. Using a pre-defined medical terminology ontology database, synonym retrieval and standardized mapping are performed on the natural language text populated into the data slots; In accordance with the data interaction interface specifications of the hospital information management system, the completed medical record template is serialized into a structured medical record data packet, and the structured medical record data packet is sent to the hospital information management system through an encrypted transmission channel.

9. A medical record generation system based on mobile terminal and multimodal data, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method for generating medical records based on mobile terminals and multimodal data as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing instructions thereon, characterized in that, When executed by a processor, this instruction causes the processor to be configured to perform the method for generating medical records based on mobile terminal and multimodal data according to any one of claims 1 to 8.

Images