Method and system for generating a medical history based on a mobile terminal and multi-modal data

Abstract

The application provides a kind of course record generation method and system based on mobile terminal and multi-modal data, which comprises the following steps: first, through the signal interaction of mobile terminal and sickbed, directional acquisition channel and ward round conversation associated with specific patient are constructed.In the conversation, time-stamped multi-modal data stream such as voice, image and operation instruction is obtained in real time, and audio semantic anchor points and visual semantic anchor points are extracted from them respectively.Through calculation and correlation of audio and video anchor points with semantic similarity exceeding threshold, a graph-text semantic unit is constructed.The unit is taken as the current observation state, and the legality of this state transition is verified by finite state machine model.Based on the verified graph-text semantic unit, template slot filling and term standardization are carried out, structured course record data package is generated and written back to HIS system.The application has the effect of improving the efficiency and quality of clinical documentation.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of medical data processing, and particularly relates to a disease course record generation method and system based on a mobile terminal and multi-modal data. BACKGROUND

[0002] With the popularization of mobile information technology in the medical field, doctors use mobile terminals (such as smart phones or tablet computers) for ward rounds, which has become a new normal in clinical work. During the ward round, doctors will generate a large amount of multi-modal instantaneous information around the patient, including inquiry dialogue with the patient, visual observation of the wound or signs, interpretation of the monitor data, and oral or touch operation instructions. These information is the most original and most real data source for forming the disease course record. At present, these information is finally transformed into structured disease course records in the electronic medical record system, mainly relying on the doctor's memory and scattered notes for secondary transcription and manual input after the ward round is completed. This process not only consumes time and effort, but also is extremely easy to cause distortion and incompleteness of the disease course record due to memory bias, information omission or transcription errors. SUMMARY

[0003] The application provides a disease course record generation method and system based on a mobile terminal and multi-modal data to solve the above technical problems.

[0004] In a first aspect, the application provides a disease course record generation method based on a mobile terminal and multi-modal data, which comprises the following steps:

[0005] Based on the signal interaction characteristics between the mobile terminal held by the doctor and the target patient's bed area, a directional collection channel containing a spatial fence and an acoustic filter is constructed, and a ward round session associated with the target patient is instantiated;

[0006] In the ward round session, a plurality of parallel input interfaces of the mobile terminal are used to obtain multi-type and discrete ward round record data streams, and an absolute timestamp and a type mark are given to each ward round record data stream to form a ward round data sequence. The ward round record data stream includes a voice data stream, an image data stream and an operation instruction data stream;

[0007] Feature extraction is performed on the ward round data sequence to generate audio semantic anchor points containing medical entity information and visual semantic anchor points containing visual subject features;

[0008] The semantic similarity between the audio semantic anchor points and the visual semantic anchor points is calculated, and the audio semantic anchor points and the visual semantic anchor points with a semantic similarity exceeding a preset similarity threshold are nonlinearly associated across the time dimension to construct a graphic-text semantic unit;

[0009] Acquire the historical diagnosis and treatment state of the target patient and load the corresponding standard clinical path state atlas, map the graph-text semantic unit to the current observation state, and verify the transition legality of the current observation state relative to the historical diagnosis and treatment state through a finite state machine model;

[0010] Based on the verified graph-text semantic unit, and according to the preset disease course record template, slot filling and medical term standardization processing are performed to generate a structured disease course record data package and write back to the hospital information management system.

[0011] Optionally, the signal interaction characteristics between the mobile terminal held by the doctor and the target patient's bed area are used to construct a directional acquisition channel containing a spatial fence and an acoustic filter, and to instantiate a ward round session associated with the target patient, including the following steps:

[0012] The received signal strength generated by the mobile terminal held by the doctor when receiving the wireless beacon broadcast of the target patient's bed area is analyzed, and the physical distance of the mobile terminal relative to the bed area is calculated based on a lognormal distribution propagation model;

[0013] When the physical distance of the mobile terminal relative to the bed area is less than a preset near-field interaction threshold, the mobile terminal is triggered to start a multi-microphone array and lock the unique identifier of the target patient carried by the wireless beacon to instantiate a ward round session associated with the target patient;

[0014] The time difference of arrival of the sound source signal received by the multi-microphone array is calculated, and the sound source azimuth of the main sound source relative to the mobile terminal is estimated based on the spatial posture parameters of the mobile terminal;

[0015] Based on the sound source azimuth, the weighting delay coefficient of the signal received by each microphone in the multi-microphone array is calculated, and a pickup beam pointing to the bed area is formed using a delay-and-sum beamforming algorithm;

[0016] According to the spatial response characteristics of the pickup beam, background noise signals outside the range of the pickup beam are suppressed, and a directional acquisition channel independent of the unique identifier is established in the mobile terminal;

[0017] The rate of change of the received signal strength is continuously monitored, and when the received signal strength is lower than a preset disconnection threshold, the ward round session is terminated and the directional acquisition channel is released.

[0018] Optionally, the rate of change of the received signal strength is continuously monitored, and when the received signal strength is lower than a preset disconnection threshold, the ward round session is terminated and the directional acquisition channel is released, including the following steps:

[0019] During the duration of the ward round session, the continuously sampled received signal strength is smoothed through a Kalman filter algorithm;

[0020] calculating a time derivative of the smoothed received signal strength to determine a movement trend of the mobile terminal relative to the bed area;

[0021] starting a timer to confirm the mobile terminal is leaving the bed area when the movement trend shows the mobile terminal is moving away from the bed area and the received signal strength continuously falls below a preset disconnection threshold;

[0022] if the timer reaches a preset time limit and the received signal strength remains below the disconnection threshold, determining that the mobile terminal has left the effective interaction range, triggering a session termination instruction, marking all data cached in the directional collection channel as a session end state, and closing the multi-microphone array of the mobile terminal to the directional collection channel;

[0023] generating a session integrity check code and attaching the session integrity check code to the end of the data packet of the ward round session.

[0024] Optionally, the feature extraction of the ward round data sequence to generate audio semantic anchor points containing medical entity information and visual semantic anchor points containing visual subject features includes the following steps:

[0025] Separating the speech data stream from the ward round data sequence, and converting the speech data stream into a natural language text sequence using an automatic speech recognition model;

[0026] Inputting the natural language text sequence into a pre-trained medical entity recognition model to extract a keyword set containing medical entities, and vectorizing the keyword set into an audio semantic anchor point;

[0027] Separating the image data stream from the ward round data sequence, and performing image enhancement and normalization preprocessing on each frame of image in the image data stream;

[0028] Inputting the preprocessed image data stream into a pre-trained medical feature recognition model to identify target images with medical features in the image data stream, and extracting a medical visual feature vector of a region of interest in the target image;

[0029] Dimensionality reduction processing of the medical visual feature vector and attaching an image classification label to generate a visual semantic anchor point corresponding to the target image.

[0030] Optionally, the semantic similarity between the audio semantic anchor points and the visual semantic anchor points is calculated, and the audio semantic anchor points and the visual semantic anchor points with a semantic similarity exceeding a preset similarity threshold are nonlinearly associated across the time dimension to construct a graphic-text semantic unit, including the following steps:

[0031] Constructing a cross-modal semantic feature space combining all audio semantic anchor points and visual semantic anchor points;

[0032] Calculate the cosine similarity value between each audio semantic anchor point vector and each visual semantic anchor point vector in the cross-modal semantic feature space, forming a many-to-many semantic similarity matrix;

[0033] Traverse the semantic similarity matrix and select candidate anchor point pairs with a cosine similarity value exceeding a preset correlation threshold;

[0034] Under the premise of ignoring the original timestamp sequence in the ward round data sequence, apply the maximum weight matching algorithm to all candidate anchor point pairs to determine the optimal pairing relationship between the audio semantic anchor points and the visual semantic anchor points;

[0035] Based on the optimal pairing relationship, bind the text segment corresponding to the paired audio semantic anchor point to the original image corresponding to the visual semantic anchor point, and encapsulate it as a graphic-text semantic unit with independent logical meaning;

[0036] For the remaining audio semantic anchor points that failed to match or the remaining visual semantic anchor points that failed to match, select the most recently matched graphic-text semantic unit according to the original timestamp of the remaining audio semantic anchor points or the remaining visual semantic anchor points for secondary weak correlation merging.

[0037] Optionally, the method comprises the following steps of:

[0038] Obtain the postoperative days of the target patient and the disease record data of the previous time node by querying the hospital information management system, and parse the historical diagnosis and treatment state vector of the target patient;

[0039] According to the admission diagnosis code of the target patient, call the corresponding standard clinical pathway state graph, and the standard clinical pathway state graph contains the legal state set allowed to exist at different time nodes and the transition probability matrix of the target patient's disease;

[0040] Parse the text features and image features in the graphic-text semantic unit, and quantize the text features and image features into a current observation state vector reflecting the current physiological indicators of the target patient;

[0041] Input the historical diagnosis and treatment state vector as the current state of the finite state machine model, and predict the expected legal state set at the next time based on the transition probability matrix;

[0042] Determine whether the current observation state vector belongs to the expected legal state set. If the current observation state vector belongs to the expected legal state set, it is determined that the state transition is legal. If the current observation state vector does not belong to the expected legal state set, it is determined that the state transition is illegal.

[0043] Optionally, after determining that the current observation state vector does not belong to the expected legal state set, the following steps are further included:

[0044] A preset modal confidence weight table is obtained, which sets the confidence priority of different modalities in different scenarios based on image recognition accuracy and speech recognition accuracy;

[0045] The feature conflict item that causes illegal state transition is identified, and the confidence weights of the image feature and the text feature corresponding to the feature conflict item are compared. If the confidence weight of the image feature is higher than the confidence weight of the text feature, an automatic correction instruction based on the image feature to correct the text feature is generated;

[0046] If the difference between the confidence weight of the image feature and the confidence weight of the text feature is lower than a preset fuzzy threshold, an abnormal state warning mark is generated and attached to the corresponding image-text semantic unit;

[0047] The automatic correction instruction is applied to the image-text semantic unit to update the current observation state vector, or the image-text semantic unit with the abnormal state warning mark is put into a manual review queue;

[0048] The updated image-text semantic unit is re-input into the finite state machine model for secondary logical closed loop verification.

[0049] Optionally, based on the verified image-text semantic unit, slot filling and medical term standardization processing are performed according to a preset medical record template to generate a structured medical record data package and write back to a hospital information management system, including the following steps:

[0050] The semantic category label contained in the verified image-text semantic unit is parsed to determine the medical document paragraph attribute corresponding to the image-text semantic unit;

[0051] According to the type of ward round conversation and the current diagnosis and treatment stage of the target patient, a matching medical record template is retrieved from the template library, which defines a standard document structure tree and data slots;

[0052] All image-text semantic units are traversed, and the image-text semantic units are mapped and filled into one or more data slots corresponding to the medical record template according to the medical document paragraph attribute;

[0053] The natural language text filled into the data slots is subjected to synonym retrieval and standardization mapping using a preset medical term ontology library;

[0054] According to the data interaction interface specification of the hospital information management system, the filled medical record template is serialized into a structured medical record data package, and the structured medical record data package is sent to the hospital information management system through an encrypted transmission channel.

[0055] In a second aspect, the present application also provides a system for generating a progress note based on a mobile terminal and multi-modal data, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for generating a progress note based on a mobile terminal and multi-modal data according to any one of the first aspect when executing the computer program.

[0056] In a third aspect, the present application also provides a computer-readable storage medium having instructions stored thereon, wherein the instructions, when executed by a processor, cause the processor to be configured to perform the method for generating a progress note based on a mobile terminal and multi-modal data according to any one of the first aspect.

[0057] The present application has the following beneficial effects:

[0058] The present application ensures the uniqueness and accuracy of data acquisition sources in a complex ward environment by constructing a directional acquisition channel based on signal interaction features. By extracting respective semantic anchor points and calculating their semantic similarity, the present application realizes nonlinear association in the time dimension, thereby accurately reconstructing the inherent logical binding relationship between what is said and what is seen in the ward round scene, forming a graphic and semantic unit with rich context information. Using a finite state machine model based on a standard clinical path state atlas, the present application verifies the transition legality of the fused observation state, thereby placing the data-driven recognition result under the strong constraint of medical knowledge, effectively filtering pseudo-associations and abnormal data that do not conform to the diagnosis and treatment logic, and ensuring the medical rigor of the generated content. Finally, by template slot filling and medical terminology standardization, the present application converts the ward round interaction process into a structured and standardized progress note data package, significantly improving the efficiency and quality of clinical documentation work. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 FIG. 1 is a flowchart of a method for generating a progress note based on a mobile terminal and multi-modal data according to an embodiment of the present application.

[0060] Figure 2 FIG. 2 is a flowchart of constructing a directional acquisition channel and instantiating a ward round session associated with a target patient according to an embodiment of the present application. DETAILED DESCRIPTION

[0061] The technical solutions in the embodiments of the present application will be clearly described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art belong to the scope of protection of the present application.

[0062] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0063] Figure 1 This is a flowchart illustrating a method for generating medical records based on a mobile terminal and multimodal data in one embodiment. It should be understood that, although... Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps. For example Figure 1 As shown, the method for generating medical records based on mobile terminals and multimodal data disclosed in this invention specifically includes the following steps:

[0064] S101. Based on the signal interaction characteristics between the doctor's mobile terminal and the target patient's bed area, construct a directional acquisition channel that includes spatial fencing and acoustic filtering, and instantiate a ward round session associated with the target patient.

[0065] Wherein, first need to analyze the doctor holding mobile terminal received by the target patient bed area deployed wireless beacon broadcast signal issued by the signal. By analyzing the received signal strength indication, and application of lognormal distribution propagation model, the physical distance between mobile terminal and beacon can be estimated. Once the calculated physical distance is less than the preset near field interaction threshold, it indicates that the doctor has entered the effective ward round range, at this time the multi microphone array of mobile terminal is started and the patient unique identifier carried in the beacon broadcast is locked, thus instantiating a ward round session strictly bound with the patient. Then, by calculating the time difference of sound source signal arriving at different microphones, and combining the attitude sensor data of terminal device, the azimuth of main sound source can be accurately estimated. According to the azimuth, delay and sum beamforming algorithm is used to weight and delay the microphone signal, forming a highly directional pickup beam, effectively aiming at the target sound source area. This acoustic filtering technology can significantly suppress the environmental noise and adjacent bed interference outside the beam range, and finally form a directional acquisition channel with spatial physical fence and acoustic logical isolation.

[0066] S102. In the ward round session, multiple parallel input interfaces of the mobile terminal are used to obtain multiple types and discrete ward round record data streams, and each ward round record data stream is given an absolute timestamp and a type mark, forming a ward round data sequence. The ward round record data stream includes voice data stream, image data stream and operation instruction data stream.

[0067] Wherein, after the successful establishment and instantiation of ward round session in the directional acquisition channel, a multi task parallel processing mechanism will be started, and multiple input interfaces of the mobile terminal are used to synchronously capture multiple modal information streams in the ward round process. These information streams are discrete and heterogeneous, mainly including the conversation voice of doctor and patient, the image of patient's physical sign or wound, and the operation instruction of doctor's click or slide on the terminal screen. In order to ensure that these data with different sources and different time points can be accurately aligned and associated in subsequent analysis, each independent data stream will be given a high precision absolute timestamp and a clear type mark at the moment of capture. This process forms a unified ward round data sequence, each element of which can be abstractly represented as a data tuple, including timestamp, data type and original data load.

[0068] S103. Feature extraction is performed on the ward round data sequence to generate audio semantic anchor points containing medical entity information and visual semantic anchor points containing visual subject features.

[0069] Wherein, for the speech data stream in the sequence, it is first converted into natural language text efficiently by using an automatic speech recognition model. Then the text sequence is input into a medical entity recognition model pre-trained on a large amount of medical literature. The model can automatically identify and extract key medical entities such as diseases, symptoms, drugs, and examination items involved in the text, forming a keyword set. This set is then vectorized through word embedding technology to form audio semantic anchor points, each of which is a highly condensed core medical information of a segment of speech. At the same time, for the image data stream, the system will analyze frame by frame. First, the image quality is improved through preprocessing steps such as image enhancement and normalization, and then the processed image is input into a pre-trained medical feature recognition model. The model is responsible for determining whether the image contains valuable medical features, such as wound healing, rash shape, or drainage fluid color. Once the target image is identified, the visual features of its key regions are extracted and encoded into a high-dimensional vector. After dimension reduction and adding image classification labels, a visual semantic anchor point corresponding to the target image is generated.

[0070] S104. Calculate the semantic similarity between the audio semantic anchor points and the visual semantic anchor points, and perform nonlinear association between the audio semantic anchor points and the visual semantic anchor points with a semantic similarity exceeding a preset similarity threshold across the time dimension to construct a text-image semantic unit.

[0071] Wherein, after generating separate audio and visual semantic anchor points, it is necessary to quantify the semantic correlation between different modal anchor points. By mapping all audio semantic anchor point vectors and visual semantic anchor point vectors into a unified cross-modal semantic feature space, their similarity can be directly compared in this shared space. By calculating the cosine similarity of all possible audio-visual anchor point pairs, a many-to-many semantic similarity matrix can be constructed. Then, set an association threshold to filter out candidate anchor point pairs with high enough similarity. In order to find the globally optimal matching solution, the time sequence of these anchor points in the original data sequence is ignored, and the maximum weight matching algorithm is applied to all candidate anchor point pairs. This algorithm can ensure that each anchor point is paired with only one semantically most relevant heterogeneous anchor point, avoiding false associations. The text segment corresponding to the successfully paired audio anchor point and the original image corresponding to the visual anchor point are formally bound and encapsulated into a logically consistent text-image semantic unit.

[0072] S105. Obtain the historical diagnosis and treatment state of the target patient and load the corresponding standard clinical path state map, map the text-image semantic unit to the current observation state, and verify the legality of the transition of the current observation state relative to the historical diagnosis and treatment state through a finite state machine model.

[0073] Wherein, the historical diagnosis and treatment state of the patient is obtained by querying the hospital information management system through the interface, such as the postoperative days and the last course record summary, and is parsed into a historical diagnosis and treatment state vector. At the same time, according to the patient's admission diagnosis code, the corresponding standard clinical pathway state atlas is loaded from the knowledge base. This atlas is a model defined in advance by medical experts, which contains all possible and legal clinical state sets of a specific disease at different treatment stages, as well as the transition probabilities between states. Then the current generated graph-text semantic unit is parsed, and the text and image features in it are also quantified into a current observation state vector. The historical diagnosis and treatment state vector is input as the initial state of the finite state machine, and according to the transition probability matrix in the state atlas, all expected legal states at the next time can be predicted. Finally, the current observation state vector is compared with the expected legal state set. If the observation state belongs to the set, it is determined that the state transition is legal; otherwise, it is determined to be illegal, and the subsequent abnormal processing process is triggered.

[0074] S106. Based on the verified graph-text semantic unit, and according to the preset course record template, slot filling and medical term standardization processing are performed to generate a structured course record data package and write it back to the hospital information management system.

[0075] Wherein, the graph-text semantic unit that has passed the clinical pathway logic verification is a high-quality and high-credibility information segment, and the last step is to convert it into a structured course record that meets medical standards and can be directly archived. The process begins with parsing the semantic category labels carried by each graph-text semantic unit, which indicate the content attributes. According to the type of ward round (such as first course, postoperative ward round or daily ward round) and the specific diagnosis and treatment stage of the patient, the system automatically retrieves a course record template that completely matches it from the template library. The template predefines a standardized document structure tree and a series of data slots to be filled. Then, the system traverses all the verified graph-text semantic units, and according to their semantic category labels, the text description and associated images contained therein are accurately mapped and filled into the corresponding data slots in the template. After preliminary filling, to ensure professionalism and uniformity, the system will use a large medical terminology ontology library to perform synonym retrieval and standardization mapping on all natural language text filled into the slots. Finally, this complete and term-standardized electronic medical record will be serialized into a structured data package according to the data interaction interface specification of the hospital information management system, and securely written back to the central database of the hospital through an encrypted transmission channel.

[0076] In one embodiment, refer to Figure 2Based on the signal interaction characteristics between the doctor's mobile terminal and the target patient's bed area, a directional acquisition channel including spatial fencing and acoustic filtering is constructed, and a ward round session associated with the target patient is instantiated, including the following steps:

[0077] S201. Analyze the received signal strength generated by the mobile terminal held by the doctor when it receives the wireless beacon broadcast in the area where the target patient's bed is located, and calculate the physical distance of the mobile terminal relative to the bed area based on the log-normal distribution propagation model;

[0078] S202. When the physical distance between the mobile terminal and the bed area is less than the preset near-field interaction threshold, the mobile terminal is triggered to open the multi-microphone array and lock the unique identifier of the target patient carried by the wireless beacon to instantiate the ward round session associated with the target patient.

[0079] S203. Calculate the time difference of arrival of the sound source signals received by the multi-microphone array, and estimate the azimuth angle of the main sound source relative to the mobile terminal by combining the spatial attitude parameters of the mobile terminal.

[0080] S204. Calculate the weighted delay coefficient of the received signal of each microphone in the multi-microphone array based on the azimuth angle of the sound source, and use the delay summation beamforming algorithm to form a pickup beam pointing towards the bed area;

[0081] S205. Based on the spatial response characteristics of the pickup beam, suppress background noise signals outside the pickup beam range, and establish a uniquely identified directional acquisition channel in the mobile terminal;

[0082] S206. Continuously monitor the rate of change of the received signal strength, and terminate the ward round session and release the directional acquisition channel when the received signal strength is lower than the preset disconnection threshold.

[0083] In this embodiment, the mobile terminal continuously measures the received Signal Strength Indication (RSSI), a raw value reflecting signal attenuation. Since wireless signal propagation indoors is affected by multipath effects and obstacle obstruction, directly using the RSSI value is not reliable. Therefore, a log-normal distribution propagation model is introduced for distance estimation. This model can better fit the signal propagation characteristics of complex indoor environments. The physical distance can be calculated using the formula... This is accomplished by [the method described above], where d represents the estimated physical distance between the mobile terminal and the bedside beacon. It is a reference distance. At reference distance The reference signal strength measured at that location, It is the signal strength measured in real time. is the path loss exponent, whose value depends on the specific indoor environment layout and material. Once the mobile terminal is detected to be continuously within the pre-defined near field interaction threshold distance from the patient bed area, the multi-microphone array inside the mobile terminal is activated from the standby state to the working state. At the same time, the unique patient identifier carried in the wireless beacon broadcast data packet is parsed, and the identifier is locked by the mobile terminal and used to instantiate a ward round session object. The instantiation process includes opening a dedicated data buffer in memory, establishing a corresponding data processing thread, and forcibly associating all subsequent collected data streams with the unique patient identifier.

[0084] After the ward round session is instantiated, in order to realize directional sound pickup of a specific sound source, the direction of the main sound source relative to the mobile terminal must be accurately estimated, which can be done by analyzing the acoustic signals received by the multi-microphone array. Since each microphone is physically located at a different position on the mobile terminal, there will be a slight difference in the time at which the sound signal from the same sound source reaches each microphone, i.e. time difference of arrival (TDOA). By calculating the TDOA between different pairs of microphones, the direction of the sound source can be inferred. The TDOA estimation method in this embodiment can be a generalized cross-correlation algorithm, especially a generalized cross-correlation with phase transform weighting, which can effectively suppress reverberation interference. For a simple linear array consisting of two microphones, the relationship between the sound source azimuth angle and the TDOA can be approximately expressed as where is the estimated sound source azimuth angle, is the calculated signal time difference of arrival, is the speed of sound in air, and L is the physical distance between the two microphones. After obtaining the accurate sound source azimuth angle, the information can be used to form a directional sound pickup beam, thereby focusing on the target area at the acoustic level. Delay-and-sum beamforming algorithm is a classic and efficient technique to achieve this goal. The core idea of this algorithm is to calculate a specific delay compensation for the signal received by each microphone in the multi-microphone array according to the sound source azimuth angle. The role of this delay is to align the target sound source signals arriving at different microphones at different times in time.

[0085] For the i-th microphone in the array, the delay that needs to be applied is which can be calculated by geometric relationship, specifically where is the delay time required for the signal of the i-th microphone, is the position vector of the microphone relative to the center of the array, is the unit vector of the sound wave propagation direction determined according to the estimated sound source azimuth angle, is the speed of sound. After each microphone signal is delayed by such a precise amount, all the delayed signals are added together. According to the principle of wave superposition, the signal from the target direction will be significantly enhanced due to the in-phase superposition, while the interference signals from other directions will be mutually offset due to the inconsistent phase, and finally a directional output signal with strong directivity is synthesized. The pickup beam generated by the delay-and-sum beamforming algorithm has a unique spatial response characteristic, which is the key to realizing background noise suppression. The beam forms a main lobe and multiple side lobes in space. The main lobe is the area with the highest signal gain, and its pointing direction is completely consistent with the estimated sound source azimuth angle in the previous stage, which can efficiently capture the sound from the doctor-patient conversation. The signal gain in the side lobe area is much lower than that in the main lobe, and the sound source signals from these directions will be significantly attenuated. This characteristic makes the pickup beam naturally form a spatial filter, which can effectively suppress environmental noise outside the main lobe range of the pickup beam, such as conversation near the adjacent bed, alarm sound of medical equipment, and walking sound of people in the corridor. Based on this acoustic focusing ability, the system establishes a logically independent directional acquisition channel associated with the unique identifier of the patient at the software level of the mobile terminal. All high signal-to-noise ratio audio data after beamforming processing will be imported into this exclusive channel for caching and subsequent processing. This not only greatly improves the accuracy of speech recognition, but also ensures the purity and specificity of the collected data. In order to ensure the integrity of the ward round conversation and effectively manage system resources, it is necessary to automatically and accurately terminate the conversation and release the directional acquisition channel when the doctor leaves the bed area. Therefore, it is necessary to continuously monitor the rate of change of the received signal strength, and terminate the ward round conversation and release the directional acquisition channel when the received signal strength is lower than the preset disconnection threshold.

[0086] In one embodiment, continuously monitoring the rate of change of the received signal strength and terminating the ward round conversation and releasing the directional acquisition channel when the received signal strength is lower than the preset disconnection threshold comprises the following steps:

[0087] During the duration of the ward round conversation, the continuously sampled received signal strength is smoothed by a Kalman filtering algorithm;

[0088] The time derivative of the smoothed received signal strength is calculated to determine the movement trend of the mobile terminal relative to the bed area;

[0089] When the movement trend shows that the mobile terminal is moving away from the bed area and the received signal strength is continuously lower than the preset disconnection threshold, a timer is started for off-site confirmation;

[0090] If the timer reaches the preset time limit and the received signal strength is always lower than the disconnect threshold, it is determined that the mobile terminal has left the effective interaction range, a session termination instruction is triggered, all data buffered in the directional collection channel are marked as session end state, and the multi-microphone array of the mobile terminal is closed to occupy the directional collection channel;

[0091] A session integrity check code is generated and attached to the end of the data packet of the ward round session.

[0092] In the embodiment, the original RSSI readings may have irregular instantaneous fluctuations due to electromagnetic interference, personnel movement, etc. Direct use of these data will seriously affect the stability of the judgment. Therefore, the Kalman filtering algorithm is introduced to smooth the time series data. The algorithm regards the real signal strength as a linear change over time system state, and regards each measured RSSI value as a noisy observation of the state. Through a recursive prediction-update cycle, the filter can effectively estimate the signal strength closest to the true value from the noisy measurement value. After the algorithm processing, the output will be a smooth and continuous signal strength curve, which can more accurately reflect the macro motion trajectory of the mobile terminal. After obtaining the smoothed received signal strength sequence, the next step is to quantitatively analyze the motion trend of the mobile terminal, that is, to determine whether it is approaching, moving away or remaining in the bed area. This determination is achieved by calculating the time derivative of the smoothed signal strength curve.

[0093] At discrete sampling time points, the time derivative can be approximated by numerical differentiation methods, such as backward difference method. The specific calculation method is In the formula, represents the signal strength change rate at the current time t, that is, the quantitative indicator of the moving trend; is the smoothed signal strength value at the current time; is the smoothed signal strength value at the previous sampling time; is the time interval between two consecutive samplings. The calculated value has a clear physical meaning: if the value is positive, it means that the signal strength is increasing, and the mobile terminal tends to approach the bed; if it is negative, it means that the signal strength is decreasing, and the mobile terminal is moving away from the bed; if it is close to zero, it means that the relative distance remains basically unchanged.

[0094] Deciding whether to terminate a session based only on the moving trend and the instantaneous value of the signal strength can cause misjudgment due to occasional situations such as the doctor briefly turning around or being temporarily blocked. In order to improve the robustness of the decision, a double-condition-based off-site confirmation mechanism is introduced. The mechanism sets that when the two conditions are met at the same time, a preset duration timer will be started. The first condition is the moving trend judgment, that is, the time derivative of the signal strength calculated in the last step continuously shows a negative value, which clearly indicates that the mobile terminal is moving away from the bed area. The second condition is the absolute position judgment, that is, the smoothed received signal strength itself has fallen and continuously below a pre-set disconnection threshold. This threshold represents the edge of the effective ward-round interaction range. Only when the trend of moving away and the state of being at the edge of the interaction area occur at the same time, the system will preliminarily determine that the doctor may be leaving. If the signal strength rebounds and exceeds the disconnection threshold at any time during the timing process, indicating that the doctor may only be temporarily away or the signal is temporarily disturbed, the timer will be immediately reset, and the session will continue. On the contrary, if the timer successfully completes the preset whole process, and the signal strength is always below the disconnection threshold during this period, the system will make a final decision: the mobile terminal has completely left the effective interaction range. At this time, a session termination instruction is triggered. The instruction will perform a series of resource release and data marking operations, including marking all data streams cached in the directional acquisition channel as session end state, and issuing an instruction to the operating system to close the multi-microphone array of the mobile terminal and release its hardware resource occupation of the directional acquisition channel.

[0095] Finally, a standard cryptographic hash function, such as the SHA-256 algorithm, is used to calculate a fixed-length and unique digest value for all data generated during the entire ward-round session, including voice, image, operation instruction and various metadata, after being serialized into a unified data packet. The calculated check code will be attached to the tail of the data packet and sent together. The receiving end system will use the same hash function to recalculate the check code for the received data body, and compare it with the original check code attached to the tail of the data packet. If they are completely consistent, it proves that the data transmission is complete and correct, thereby technically ensuring the authenticity and reliability of the electronic medical record original record.

[0096] In one embodiment, feature extraction is performed on the ward-round data sequence to generate audio semantic anchor points containing medical entity information and visual semantic anchor points containing visual subject features, including the following steps:

[0097] Separating the voice data stream from the ward-round data sequence, and converting the voice data stream into a natural language text sequence using an automatic speech recognition model;

[0098] The natural language text sequence is input into a pre-trained medical entity recognition model to extract a keyword set containing medical entities, and the keyword set is vectorized into an audio semantic anchor point;

[0099] The image data stream is separated from the ward round data sequence, and each frame of image in the image data stream is subjected to image enhancement and normalization preprocessing;

[0100] The preprocessed image data stream is input into a pre-trained medical feature recognition model to identify target images with medical features in the image data stream, and extract medical visual feature vectors of the regions of interest in the target images;

[0101] The medical visual feature vectors are subjected to dimension reduction processing and attached with image classification labels to generate visual semantic anchor points corresponding to the target images.

[0102] In this embodiment, the separated speech data is a continuous waveform signal and cannot be directly analyzed by a computer, so it must be converted into a structured text format. Specifically, an end-to-end model based on the Transformer architecture can be used, such as the Conformer model, which combines the ability of convolutional neural networks to capture local features and self-attention mechanisms to handle global dependencies, and is particularly suitable for processing continuous speech streams. To ensure high accuracy in a medical context, the model needs to be trained on a large-scale speech dataset containing various medical terminology, different accents, and recordings in a hospital background noise. In the recognition process, the input original audio waveform is subjected to feature extraction and then fed into the Conformer network for acoustic modeling and decoding, and finally a word sequence with timestamps is output.

[0103] For text content extraction, a medical entity recognition model pre-trained on a large amount of medical literature and electronic medical records can be used, such as the BioBERT or ClinicalBERT model based on the BERT architecture. These models can deeply understand the context of medical text. The input text sequence is first decomposed into a series of word pieces, and then the model predicts a predefined entity label for each word piece, such as disease, symptom, examination, treatment, body part, etc. The identified word pieces with specific labels are combined to form a keyword set containing specific medical entities. In order to quantitatively compare and calculate this keyword set, it needs to be converted into a fixed-dimensional mathematical vector, i.e. an audio semantic anchor point. An effective vectorization method is to weight and average the word embedding vectors of all keywords in the set.

[0104] In parallel with processing the stream of speech data, the stream of image data in the sequence of ward round data also needs to be analyzed. Before any advanced recognition task is performed, a series of preprocessing operations must be performed on the raw images to improve the image quality and make it meet the requirements of the subsequent model for input data. First, all image data frames are separated from the sequence according to the type label in the data stream. Considering that the images taken during the ward round process may have uneven lighting or low contrast, image enhancement techniques will be applied. For example, using the limited contrast adaptive histogram equalization algorithm can effectively enhance the local contrast of the image, making the details of the lesions or wounds more clearly visible, while avoiding excessive amplification of noise. Subsequently, in order to eliminate the pixel value differences of images taken under different devices and different lighting conditions, normalization processing is required. The preprocessed image data stream is sent to a pre-trained medical feature recognition model to automatically identify and extract key visual information contained therein. This process is usually divided into two stages.

[0105] The first stage is target image screening, that is, to determine which frames in the continuous image stream contain key frames with valid medical information. This can be achieved through an efficient image classification model (such as MobileNet or EfficientNet) trained to distinguish images containing medical features such as wounds, rashes, drainage tubes, and monitor readings from those that are blurred, irrelevant, or blank. After screening the target images, the second stage of feature extraction is entered. A convolutional neural network, such as a model pre-trained on a large medical image dataset (such as ResNet fine-tuned on medical data after ImageNet), is used to extract deep visual features of the region of interest in the target image. Finally, the extracted high-dimensional medical visual feature vector is converted into a visual semantic anchor point that is easier to calculate and store. Directly using the original high-dimensional feature vector has the problems of high computational cost, large storage overhead, and possible inclusion of redundant information. Therefore, it needs to be reduced in dimension. Specifically, principal component analysis can be used, which aims to find the direction with the largest variance in the data and project the original data into a new coordinate system composed of a few principal components, thereby retaining the original information to the greatest extent while significantly reducing the data dimension. After completing the dimension reduction, the image classification label output by the medical feature recognition model in the previous stage is also attached to the low-dimensional vector. Finally, this composite structure composed of a low-dimensional feature vector and a classification label constitutes a visual semantic anchor point that accurately corresponds to the target image.

[0106] In one embodiment, the semantic similarity between the audio semantic anchor point and the visual semantic anchor point is calculated, and the audio semantic anchor point and the visual semantic anchor point with a semantic similarity exceeding a preset similarity threshold are nonlinearly associated across the time dimension to construct a text-image semantic unit, including the following steps:

[0107] construct a cross-modal semantic feature space combining all audio semantic anchor points and visual semantic anchor points;

[0108] calculate the cosine similarity values between each audio semantic anchor point vector and each visual semantic anchor point vector in the cross-modal semantic feature space, forming a many-to-many semantic similarity matrix;

[0109] traverse the semantic similarity matrix and filter out candidate anchor point pairs whose cosine similarity values exceed a preset correlation threshold;

[0110] apply the maximum weight matching algorithm to all candidate anchor point pairs while ignoring the original timestamp sequence in the ward round data sequence, to determine the optimal pairing relationship between audio semantic anchor points and visual semantic anchor points;

[0111] based on the optimal pairing relationship, bind the text segments corresponding to the successfully paired audio semantic anchor points with the original images corresponding to the visual semantic anchor points, and encapsulate them as graphic-text semantic units with independent logical meaning;

[0112] for the remaining audio semantic anchor points that failed to match or the remaining visual semantic anchor points that failed to match, select the matched graphic-text semantic unit with the closest time interval according to the original timestamp of the remaining audio semantic anchor points or the remaining visual semantic anchor points for secondary weak correlation merging.

[0113] In this embodiment, a cross-modal semantic feature space is first constructed, the purpose of which is to map the essentially heterogeneous audio semantic anchor point vectors and visual semantic anchor point vectors into the same high-dimensional vector space. Achieving this goal usually relies on a specially trained deep neural network model. The model generally includes two independent encoder branches: a text encoder responsible for processing text information in audio semantic anchor points and an image encoder responsible for processing image information in visual semantic anchor points. The two encoders convert their respective inputs into high-dimensional feature vectors, and then through a shared projection head network, linearly or nonlinearly transform these feature vectors into a final semantic space with the same dimension. By using training strategies such as contrastive learning, the model can learn to bring together vector representations of semantically related graphic-text pairs in space while pushing apart unrelated graphic-text pairs. In the unified cross-modal semantic feature space, all anchor point vectors are comparable, and the semantic relevance between any audio anchor point and visual anchor point can be accurately calculated. For all audio semantic anchor points and visual semantic anchor points generated in the ward round conversation, the system will pair them two by two and calculate the cosine similarity values between each pair, ultimately generating a many-to-many semantic similarity matrix. This matrix completely depicts the potential semantic correlation strength between each piece of oral description information and each visual observation record in this ward round conversation.

[0114] After obtaining the matrix containing all the pair-wise similarity scores, it is necessary to filter out the anchor pairs with high potential relevance from it to narrow down the search range for the subsequent optimal matching algorithm. This filtering process is achieved by setting a pre-defined relevance threshold. The system will iterate through each element in the semantic similarity matrix and compare it with the pre-defined relevance threshold. Only those anchor pairs whose cosine similarity values clearly exceed the threshold will be considered as semantically close enough to qualify as candidates for the next round of matching. To determine the final matching relationship from the set of candidate anchor pairs, a greedy strategy cannot be simply adopted, but a globally optimal solution should be sought. To achieve this, the maximum weight matching algorithm in graph theory is introduced in this step. A key feature of this method is that it completely ignores the time stamp order of each anchor in the original ward round data sequence and makes decisions purely based on semantic similarity, thus being able to establish non-linear associations across the time dimension. In specific implementation, all the candidate audio semantic anchors and visual semantic anchors can be regarded as two groups of vertices of a bipartite graph, and the cosine similarity values between the candidate anchor pairs are regarded as the weights of the edges connecting the corresponding vertices. The goal of the maximum weight matching algorithm is to find a matching in this weighted bipartite graph, i.e. a set of edges with no common vertices, such that the sum of the weights of the edges in this set is maximized.

[0115] For each pair of successfully paired audio semantic anchors and visual semantic anchors, the system will backtrack to the original data and firmly bind the original natural language text segment corresponding to the audio anchor with the original image data corresponding to the visual anchor. This binding process is not just a simple data splicing, but encapsulates the text, image, their original time stamps, type labels, and semantic similarity scores, etc. into a graphic-text semantic unit. Each graphic-text semantic unit represents a multi-modal observation event with high semantic consistency in the ward round process, such as a voice describing "wound red and swollen" and a clear photo showing the redness of the wound being combined together. After completing the optimal matching and encapsulating the graphic-text semantic units, there will usually be some isolated anchors that have failed to match, i.e. the remaining audio semantic anchors or visual semantic anchors. In order to maximize the preservation of information, the system will adopt a secondary weak association strategy based on time proximity to handle these remaining anchors. Specifically, for each remaining anchor that fails to match, the system will obtain its absolute time stamp in the original ward round data sequence. Then, iterate through all the successfully constructed graphic-text semantic units and calculate the absolute time difference between the time stamp of the remaining anchor and the audio or image time stamp contained in each graphic-text semantic unit. Finally, the remaining anchor will be merged into the matched graphic-text semantic unit with the smallest time interval. When merging, a special label of weak association or time proximity will be given to this anchor to distinguish it from the strong association determined by the semantic similarity calculation.

[0116] In one embodiment, before applying the maximum weight matching algorithm to all candidate anchor pairs to determine the optimal pairing relationship between the audio semantic anchor and the visual semantic anchor, the following steps are further included:

[0117] Obtaining the original image frame corresponding to the visual semantic anchor in the candidate anchor pair and the keyword feature vector corresponding to the audio semantic anchor;

[0118] Reversely inputting the keyword feature vector as a guide signal of the target category into the channel attention layer of the medical feature recognition model for extracting the visual semantic anchor, performing gradient weighted class activation mapping calculation on the original image frame, and generating a class activation heat map reflecting the visual response distribution of the keyword feature vector in the original image frame;

[0119] Using an adaptive threshold segmentation algorithm to binarize the class activation heat map, and extracting a semantic focus area in the original image frame that is semantically excited by the keyword feature vector;

[0120] Processing the original image frame using an edge detection algorithm and a morphological closing operation to extract a physical lesion contour mask that is explicitly present in the original image frame;

[0121] Calculating the intersection-over-union value between the semantic focus area and the physical lesion contour mask, and defining the intersection-over-union value as a spatial alignment fidelity index of the candidate anchor pair;

[0122] Using the spatial alignment fidelity index as a gain coefficient to weight and update the cosine similarity value of the candidate anchor pair in the many-to-many semantic similarity matrix, so that the maximum weight matching algorithm uses the updated cosine similarity value for optimal pairing.

[0123] In this embodiment, for each candidate anchor pair, the system performs two parallel retrieval operations. The first one is to backtrack to the original rounds data sequence according to the index information stored in the visual semantic anchor, locate and load the corresponding unprocessed original image frame. This image frame is the fundamental data source containing all pixel information and visual context. The second operation is to extract the keyword set identified by the audio semantic anchor in the candidate anchor pair before vectorization, and convert these keywords into corresponding feature vectors through the word embedding model. Then use an improved gradient weighted class activation mapping technique, and use the keyword feature vector as a dynamic target class guide signal. The keyword feature vector is input in reverse to the end of the medical recognition model previously used to extract visual features, such as the convolution layer before the channel attention module or the global average pooling layer. The goal is to maximize the cosine similarity between the visual features output by the model and the input keyword feature vector. The gradient of this similarity score with respect to each feature map in the selected convolution layer is calculated. The global average of this gradient is used as the weight of each feature map, and the formula is where is the weight of the kth feature map, Z is the number of pixels in the feature map, is the similarity score of the visual feature and the keyword vector, is the activation value of the kth feature map at position (i,j). After weighting all feature maps by this weight and passing through the ReLU activation function, a class activation heat map with continuous gray values is generated.

[0124] After generating the class activation heat map with continuous gray values, it needs to be converted into a clear binary region for precise geometric comparison. Directly using a global uniform threshold for segmentation often cannot adapt to the uneven distribution of activation intensity in the heat map. Therefore, adaptive threshold segmentation algorithms such as Otsu's method (maximum between-class variance method) can more intelligently determine the segmentation limit. Otsu's method will iterate through all possible gray levels as candidate thresholds, and calculate the between-class variance for each candidate threshold by dividing the heat map pixels into foreground (high activation area) and background (low activation area) two classes. The goal of the algorithm is to find the gray threshold that maximizes this between-class variance, as it is considered the best segmentation point that best distinguishes the two classes. Once the best threshold is determined, the entire heat map can be binarized: all pixels with a value higher than the threshold are set to 1 (foreground), and those with a value lower than or equal to the threshold are set to 0 (background). Thus, a clear binary mask is accurately extracted from the blurred heat map, and the range enclosed by this mask is the visual region focused on by the keyword feature vector semantics.

[0125] Next, the physically real lesion contours need to be extracted from the image itself as a benchmark for comparison. First, the Canny edge detection algorithm is applied to the original image frame. This algorithm can accurately identify the edge pixels in the image where the brightness changes dramatically through a series of steps such as Gaussian filtering, gradient calculation, non-maximum suppression, and double-threshold hysteresis connection, thus outlining the preliminary contours of lesions, wounds, or other prominent features. However, the edges obtained in this way are often discontinuous broken lines. In order to form a closed and solid area mask, morphological closing operation needs to be performed next. The specific operation of closing is to first dilate the binary edge image to connect adjacent broken edges, and then erode to restore the original size of the object. Now there are two binary masks from the same image: one is the "semantic focus area" generated by the text semantics, and the other is the "physical lesion contour mask" extracted by image processing. In order to quantify the consistency degree of the two in spatial position, the intersection-over-union ratio can be used as the core evaluation index. The calculation method of intersection-over-union ratio is the area of the intersection of the two regions, divided by the total area covered. Its mathematical expression is where is the calculated intersection-over-union ratio value, represents the mask of the semantic focus area, represents the mask of the physical lesion contour. The calculation result is a value between 0 and 1. When the value is close to 1, it means that the visual area corresponding to the text description is highly consistent with the actual physical lesion in the image; when the value is close to 0, it means that there is almost no spatial correlation between the two.

[0126] Finally, the newly obtained spatial alignment fidelity index is integrated into the original semantic similarity evaluation system to optimize the final matching decision. The specific method is to use the calculated spatial alignment fidelity index as a gain coefficient to update the original cosine similarity value of the candidate anchor pair in the many-to-many semantic similarity matrix. The update calculation can be directly performed in the form of multiplication. The effect of this weighting method is that for those anchor pairs whose text semantics and image content are highly aligned in space, their original high semantic similarity is maintained or even amplified; while for those anchor pairs that are related in abstract semantics but cannot be aligned in specific image space, their similarity score will be significantly penalized. After this round of weighted update, the values in the semantic similarity matrix will more comprehensively reflect the overall quality of the text-image matching.

[0127] In one embodiment, the historical diagnosis and treatment state of the target patient is obtained and the corresponding standard clinical path state map is loaded, the text-image semantic units are mapped to the current observation state, and the transition legality of the current observation state relative to the historical diagnosis and treatment state is verified by a finite state machine model, including the following steps:

[0128] Obtain the postoperative days of the target patient and the medical record data of the last time node by querying the hospital information management system, and parse to obtain the historical diagnosis and treatment state vector of the target patient;

[0129] According to the admission diagnosis code of the target patient, the corresponding standard clinical pathway state atlas is called, and the standard clinical pathway state atlas includes the legal state set allowed to exist at different time nodes of the disease to which the target patient belongs and the state transition probability matrix;

[0130] Parse the text features and image features in the picture-text semantic unit, and quantize the text features and image features into the current observation state vector reflecting the current physiological indicators of the target patient;

[0131] The historical diagnosis and treatment state vector is input as the current state of the finite state machine model, and the expected legal state set at the next time is predicted based on the transition probability matrix;

[0132] Determine whether the current observation state vector belongs to the expected legal state set. If the current observation state vector belongs to the expected legal state set, it is determined that the state transition is legal, and if the current observation state vector does not belong to the expected legal state set, it is determined that the state transition is illegal.

[0133] In this embodiment, the query instruction will be indexed with the unique identifier of the target patient. The data returned by the system mainly includes two parts: one is the structured key diagnosis and treatment parameters, such as postoperative days, vital sign records, etc.; the second is the medical record text of the last time node. For structured data, it can be directly extracted and assigned. For unstructured medical record text, a natural language processing process needs to be started, and medical entity recognition and relationship extraction technology is used to parse the symptoms, signs, diagnoses and other key information described therein, and map them to a standardized medical coding system. Finally, all the key features representing the patient's last state extracted from structured data and unstructured text are integrated and coded into a fixed-dimensional numerical vector, i.e. the historical diagnosis and treatment state vector. After obtaining the historical state of the patient, a authoritative medical knowledge model needs to be called to determine whether the evolution from the historical state to the current state conforms to the recognized medical routine. This knowledge model is a standard clinical pathway state atlas retrieved from a pre-set clinical pathway knowledge base according to the diagnosis code of the target patient determined at admission. The atlas is essentially an expert system that formally defines and enumerates various clinical states that a specific disease may have during the entire treatment period, for example, "24 hours after surgery, vital signs stable, no wound exudation" is a specific state. The atlas also embeds a state transition probability matrix, which accurately quantifies the probability of transitioning from any state to another state.

[0134] For the text part in the graphic-text semantic unit, key words such as symptoms, signs, and medication can be extracted through named entity recognition, and these words can be converted into standardized medical concept codes; for numerical descriptions such as "body temperature 38.5 degrees Celsius", the numerical value is directly extracted. For the image part, the classification label attached thereto and the semantic information decoded from the visual feature vector are used. Subsequently, all the features parsed from the text and the image are arranged and filled in the same dimension and order as the historical diagnosis and treatment state vector. In this way, the multi-modal observation results in the current ward round are quantified into a current observation state vector that can be directly mathematically compared with the historical state. After the historical and current states are both vectorized, the logic reasoning process based on the finite state machine model can be started. In this model, the historical diagnosis and treatment state vector obtained in the last step is mapped to a specific starting state in the state atlas, denoted as . Then, using the transition probability matrix in the loaded state atlas, it can be predicted which states are most likely to evolve from the starting state at the next time node. Specifically, the system will find all transition probability values in the transition probability matrix with the state as the starting row. All target states with a transition probability greater than a preset minimum threshold (to exclude theoretically impossible transitions) will be collected together to form a set of expected legal states. This set includes all clinical manifestations that a patient in the state is most likely to have and should have next according to the standard clinical pathway.

[0135] Finally, the current observation state vector is found in the standard clinical pathway state atlas by table lookup or nearest neighbor matching to find its unique corresponding discrete state. Then it is judged whether this discrete state exists in the set of expected legal states predicted in the last step. If it is in the set, it is determined that the state transition from the historical state to the current observation state is legal. This means that the patient's condition change recorded in the current ward round is completely consistent with the expected evolution of the standard clinical pathway. Otherwise, if it is not in the set, it is determined that the state transition is illegal. This indicates that the patient's current condition deviates from the conventional recovery trajectory, which may indicate the occurrence of a complication or an error in the data acquisition and analysis process, and needs to trigger subsequent exception handling or manual review processes.

[0136] In one embodiment, after it is determined that the current observation state vector does not belong to the set of expected legal states, the following steps are further included:

[0137] A preset modality confidence weight table is obtained, and the modality confidence weight table sets the confidence priority of different modalities in different scenarios based on image recognition accuracy and speech recognition accuracy;

[0138] If the confidence weight of the image feature is higher than that of the text feature, an automatic correction instruction based on the image feature to correct the text feature is generated;

[0139] If the difference between the confidence weight of the image feature and the confidence weight of the text feature is lower than a preset blur threshold, an abnormal state warning mark is generated and attached to the corresponding image-text semantic unit;

[0140] The automatic correction instruction is applied to the image-text semantic unit to update the current observation state vector, or the image-text semantic unit with the abnormal state warning mark is put into a manual review queue;

[0141] The updated image-text semantic unit is re-input into the finite state machine model for secondary logical closed loop verification.

[0142] In the present embodiment, when the state transition is determined to be illegal, it means that there is at least one conflict between the text information and the image information in the image-text semantic unit, and an arbitration mechanism needs to be started to solve this contradiction. The basis of this mechanism is a pre-set modal confidence weight table. This table is not fixed, but is dynamically generated based on the quantitative evaluation of the performance of the underlying artificial intelligence model. Specifically, the system will calculate the precision and recall of the image recognition model in different clinical observation scenarios, as well as the performance indicators of the automatic speech recognition and entity extraction model on the corresponding text description. Based on these performance data, a confidence weight is assigned to each image modality and text modality in each scenario. This weight W(M, C) is a function, with the input being the modality type M (image or text) and the clinical scenario C, and the output being a value representing the credibility.

[0143] After obtaining the modal confidence weight table, the specific reason for the precise positioning to cause state transition illegal is found. By comparing the current observation state vector with the expected legal state set, it can be identified which one or which several feature dimensions have mismatches. These mismatched dimensions are the feature conflict items. For example, the clinical pathway expects the wound to be in the "mild infection" state, but the state parsed by the image-text semantic unit is "good healing". The system will backtrack to the source of the "good healing" conclusion and find the corresponding image features and text features. Then, the modal confidence weight table is queried to obtain the confidence weight of the image modal and the confidence weight of the text modal in a specific clinical scenario (i.e. wound healing condition judgment). Then, comparison is made: if the confidence weight of the image feature is significantly higher than the confidence weight of the text feature, the system will determine that the image evidence is more reliable, and accordingly generate an automatic correction instruction. The instruction will clearly indicate that the real situation revealed by the image feature should be used as the basis for modifying the text part in the image-text semantic unit, so as to eliminate the contradiction between the modalities.

[0144] In some cases, the information provided by different modalities may have some uncertainty, or their confidence weights are very close. When the system compares the confidence weights of the image feature and the text feature corresponding to the feature conflict item, if the absolute difference between the two is found to be less than a preset fuzzy threshold, the system will not automatically correct rashly. This small weight difference indicates that it is impossible to make a highly certain decision based on the current algorithm and data. At this time, the primary principle of the system is to ensure medical safety and avoid potential false corrections. Therefore, an abnormal state warning marker of a specific type is generated, and the warning marker is attached to the metadata of the image-text semantic unit that caused the conflict, clearly indicating that there is a data conflict to be solved here.

[0145] If an automatic correction instruction is generated, the content of the instruction is parsed, the text part that needs to be modified in the target image-text semantic unit is located, and the more reliable semantic information extracted from the image feature is used to replace or update the original text description. For example, "dry wound" is corrected to "wound with a small amount of exudation". After the text content is updated, an updated current observation state vector is generated based on the modified image-text semantic unit. If an abnormal state warning marker is generated, no modification will be made to the original data, and the image-text semantic unit with the warning marker attached will be separated from the main processing flow and pushed to a separate data package in a processing queue designed for manual review. Whether automatic correction is performed or the problem unit is handed over to manual review, the flow does not end for the automatic correction path. In order to ensure the effectiveness and correctness of the automatic correction operation, a closed loop verification must be performed.

[0146] Specifically, the updated current observation state vector generated in the previous step based on the revised graph-text semantic unit will be re-input into the finite state machine model. The system will repeat the same logic verification process: compare the updated current state with the historical state, and determine whether this new state transition falls within the expected legal state set defined by the standard clinical pathway. This secondary verification process constitutes a key quality control link. If the verification passes, it means that the automatic revision has successfully resolved the logical conflict, and the graph-text semantic unit can enter the final progress note generation stage. If the secondary verification still fails, it indicates that the root cause of the conflict may be more complex, and at this time the system will also push this revised but still problematic unit to the manual review queue to prevent any potential false information from entering the final medical record.

[0147] In one embodiment, based on the verified graph-text semantic unit, and according to the pre-set progress note template, the structured progress note data package is generated and written back to the hospital information management system, including the following steps:

[0148] Parsing the semantic category label contained in the verified graph-text semantic unit to determine the medical document paragraph attribute corresponding to the graph-text semantic unit;

[0149] According to the type of ward round conversation and the current diagnosis and treatment stage of the target patient, the matching progress note template is retrieved from the template library, and the progress note template defines the standard document structure tree and data slot;

[0150] Iterate through all graph-text semantic units, and map and fill the graph-text semantic units into one or more data slots corresponding to the progress note template according to the medical document paragraph attribute;

[0151] Using the pre-set medical terminology ontology library to perform synonym retrieval and standardization mapping on the natural language text filled into the data slot;

[0152] According to the data interaction interface specification of the hospital information management system, the filled progress note template is serialized into a structured progress note data package, and the structured progress note data package is sent to the hospital information management system through an encrypted transmission channel.

[0153] In this embodiment, each graphic-text semantic unit, when generated, already contains one or more semantic category labels within it. These labels are assigned by the natural language understanding model analyzing its textual content (e.g. determining whether a sentence is a subjective statement or an objective description) and the image classification model analyzing its visual content (e.g. determining whether an image is a vital sign monitor or a wound close-up). For example, the labels could be “subjective feeling”, “vital sign”, “wound assessment”, or “treatment plan”. The system parses these labels and matches them with standard medical document section attributes according to a predefined mapping rule. Next, a precisely matching progress note template is retrieved from a template library according to the specific context of the current ward round. The template library pre-stores a large number of standardized document templates for different scenarios. The system considers two key dimensions to select a template: one is the type of ward round conversation, such as “admission note”, “first progress note”, “postoperative progress note”, or “daily progress note”; the other is the specific diagnosis and treatment stage that the target patient is currently in, such as “3rd day after surgery” or “2nd cycle of chemotherapy”. By using this information as query conditions, the system can accurately locate and load the most suitable progress note template in the template library. This template is not a blank document, but a framework that pre-defines a complete document structure tree, which contains all the necessary chapters, titles, and a series of data slots waiting to be filled. Each slot is associated with a specific medical document section attribute.

[0154] For each unit, the system first reads its medical document section attribute assigned in the previous step. Then it looks for the corresponding, unfilled data slot in the loaded progress note template according to this attribute. After finding the matching slot, the system injects the natural language text segment and associated image contained in the graphic-text semantic unit into this slot. In some cases, a semantic unit may contain composite information, the content of which can be mapped to multiple different slots in the template at the same time. For example, a text describing “patient's temperature 38.5°C with chills”, the “temperature” part should be filled into the vital sign slot of the objective data, while the “chills” part should be filled into the symptom slot of the subjective data.

[0155] After the initial slot filling, a comprehensive medical terminology standardization process must be performed. This process relies on a large and continuously updated medical terminology ontology, which stores a vast amount of medical standard terms and their synonyms, near-synonyms, colloquial names, abbreviations, and other variant forms. The system traverses all the natural language texts filled into the various data slots, and performs synonym retrieval using the ontology. For example, when detecting the occurrence of "tummy ache" in the text, the system discovers through retrieval that the corresponding standard medical term is "abdominal pain", and automatically replaces it. Finally, when the entire medical record template is completely filled and all the texts are subjected to terminology standardization, the filled medical record template is serialized according to the pre-defined data interaction interface specification of the HIS. Specifically, the filled medical record template is first converted into a general structured data format, such as XML or JSON. After serialization, a structured medical record data packet is generated. To protect patient privacy and data security, the data packet is encrypted by an encryption algorithm (such as AES-256) before transmission. Subsequently, the encrypted data packet is sent to the receiving server of the HIS through a secure encrypted transmission channel (such as HTTPS or a VPN-based channel). The server decrypts and performs integrity check after receiving the data, and successfully stores the data into the electronic medical record file of the patient.

[0156] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, or any other processing units, etc. The processor can be a combination of two or more of the above.

[0157] The memory can be an internal storage unit of the computer device, such as a hard disk or a memory of the computer device, or an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, or a flash memory card (FC), etc. The memory can also be a combination of the internal storage unit and the external storage device of the computer device. The memory is used to store computer programs and other programs and data required by the computer device. The memory can also be used to temporarily store data that has been output or will be output. The present application does not limit the memory.

[0158] The present application also discloses a computer readable storage medium, which stores instructions. When the instructions are executed by a processor, the processor is configured to perform the medical record generation method based on a mobile terminal and multi-modal data described in any one of the embodiments.

[0159] The computer program can be stored in the computer readable medium, and the computer program includes computer program code. The computer program code can be in the form of source code, object code, executable file or some intermediate form. The computer readable medium includes any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the computer readable medium includes but is not limited to the above-mentioned components.

[0160] The computer readable storage medium stores the disease course record generation method based on the mobile terminal and the multi-modal data in the computer readable storage medium, and is loaded and executed on the processor to facilitate the storage and application of the method.

[0161] Those skilled in the art should understand that the above discussion of any embodiment is only exemplary and is not intended to limit the scope of protection of the present application to these examples; under the idea of the present application, the above embodiments or technical features in different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes of different aspects of one or more embodiments of the present application as described above. In order to be brief, they are not provided in details.

[0162] One or more embodiments of the present application are intended to cover all such alternatives, modifications and variations falling within the broad scope of the present application. Therefore, any omission, modification, equivalent replacement, improvement, etc. made in the spirit and principles of one or more embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. A method for generating medical records based on mobile terminals and multimodal data, characterized in that, Includes the following steps: Based on the signal interaction characteristics between the doctor's mobile terminal and the target patient's bed area, a directional acquisition channel including spatial fencing and acoustic filtering is constructed, and a ward round session associated with the target patient is instantiated. During ward rounds, multiple types of discrete ward round record data streams are acquired through multiple parallel input interfaces of the mobile terminal, and each ward round record data stream is assigned an absolute timestamp and type label to form a ward round data sequence. The ward round record data stream includes voice data stream, image data stream, and operation instruction data stream. Feature extraction was performed on the ward round data sequence to generate audio semantic anchors containing medical entity information and visual semantic anchors containing visual subject features, respectively. Calculate the semantic similarity between audio semantic anchors and visual semantic anchors, and perform non-linear association across time dimension between audio semantic anchors and visual semantic anchors whose semantic similarity exceeds a preset similarity threshold to construct text-image semantic units; By querying the hospital information management system, the postoperative days and the medical record data of the previous time point of the target patient are obtained, and the historical treatment status vector of the target patient is obtained by parsing. Based on the admission diagnosis code of the target patient, the corresponding standard clinical pathway state map is invoked. The standard clinical pathway state map contains the set of legal states that are allowed to exist at different time points for the disease to which the target patient belongs, as well as the transition probability matrix between states. The text and image features in the semantic units of the image are analyzed, and the text and image features are quantified into the current observation state vector reflecting the current physiological indicators of the target patient; The historical treatment state vector is used as the current state input of the finite state machine model, and the expected set of legal states at the next moment is predicted based on the transition probability matrix. Determine whether the current observed state vector belongs to the expected set of legal states. If the current observed state vector belongs to the expected set of legal states, the state transition is deemed legal. If the current observed state vector does not belong to the expected set of legal states, the state transition is deemed illegal. Based on the validated image and text semantic units, and according to the preset medical record template, slot filling and medical terminology standardization are performed to generate a structured medical record data package and write it back to the hospital information management system.

2. The method for generating medical records based on mobile terminals and multimodal data according to claim 1, characterized in that, The process of constructing a directional acquisition channel, incorporating spatial fencing and acoustic filtering, based on the signal interaction characteristics between the doctor's mobile terminal and the target patient's bed area, and instantiating a ward round session associated with the target patient, includes the following steps: The signal strength received by the doctor's mobile terminal when it receives a wireless beacon broadcast in the area where the target patient's bed is located is analyzed, and the physical distance between the mobile terminal and the bed area is calculated based on the log-normal distribution propagation model. When the physical distance between the mobile terminal and the bed area is less than the preset near-field interaction threshold, the mobile terminal is triggered to activate the multi-microphone array and lock the unique identifier of the target patient carried by the wireless beacon to instantiate the ward round session associated with the target patient. Calculate the time difference of arrival of the sound source signals received by the multi-microphone array, and estimate the azimuth angle of the main sound source relative to the mobile terminal by combining the spatial attitude parameters of the mobile terminal. The weighted delay coefficient of the received signal of each microphone in the multi-microphone array is calculated based on the azimuth angle of the sound source, and a pickup beam pointing to the bed area is formed by using a delay summation beamforming algorithm. Based on the spatial response characteristics of the pickup beam, background noise signals outside the pickup beam range are suppressed, and a uniquely identified independent directional acquisition channel is established in the mobile terminal. The system continuously monitors the rate of change of the received signal strength and terminates the ward round session and releases the directional acquisition channel when the received signal strength falls below a preset disconnection threshold.

3. The method for generating medical records based on mobile terminals and multimodal data according to claim 2, characterized in that, The process of continuously monitoring the rate of change of the received signal strength and terminating the ward round session and releasing the directional acquisition channel when the received signal strength falls below a preset disconnection threshold includes the following steps: During the ward round session, the strength of the continuously sampled received signal is smoothed using the Kalman filter algorithm; Calculate the time derivative of the received signal strength after smoothing to determine the movement trend of the mobile terminal relative to the bed area; When the mobile trend shows that the mobile terminal is moving away from the bedside area and the received signal strength is consistently lower than the preset disconnection threshold, a timer is started to confirm the departure. If the timer reaches the preset time limit and the received signal strength is always lower than the disconnection threshold, it is determined that the mobile terminal has left the effective interaction range, triggering a session termination command, marking all data cached in the directional acquisition channel as session end state, and disabling the mobile terminal's multi-microphone array from occupying the directional acquisition channel; Generate a session integrity check code and append the session integrity check code to the end of the data packet for the room check session.

4. The method for generating medical records based on mobile terminals and multimodal data according to claim 1, characterized in that, The step of extracting features from the ward round data sequence to generate audio semantic anchors containing medical entity information and visual semantic anchors containing visual subject features includes the following steps: The speech data stream was separated from the ward round data sequence, and the speech data stream was converted into a natural language text sequence using an automatic speech recognition model; The natural language text sequence is input into a pre-trained medical entity recognition model to extract a set of keywords containing medical entities, and the set of keywords is vectorized into audio semantic anchors; Image data streams were separated from the ward round data sequence, and image enhancement and normalization preprocessing were performed on each frame of the image data stream. The preprocessed image data stream is input into a pre-trained medical feature recognition model to identify target images with medical features in the image data stream and extract medical visual feature vectors of the regions of interest in the target images. The medical visual feature vectors are dimensionality reduced and image classification labels are added to generate visual semantic anchors corresponding to the target image.

5. The method for generating medical records based on mobile terminals and multimodal data according to claim 1, characterized in that, The process of calculating the semantic similarity between audio semantic anchors and visual semantic anchors, and then performing a non-linear cross-time association on audio semantic anchors and visual semantic anchors with semantic similarity exceeding a preset similarity threshold to construct a text-image semantic unit includes the following steps: A cross-modal semantic feature space is constructed by combining all audio semantic anchors and visual semantic anchors; Calculate the cosine similarity between each audio semantic anchor vector and each visual semantic anchor vector in the cross-modal semantic feature space to form a many-to-many semantic similarity matrix; Traverse the semantic similarity matrix and filter out candidate anchor pairs whose cosine similarity values ​​exceed a preset association threshold; Ignoring the original timestamp order in the ward round data sequence, the maximum weight matching algorithm is applied to all candidate anchor pairs to determine the optimal pairing relationship between audio semantic anchors and visual semantic anchors. Based on the optimal pairing relationship, the text segment corresponding to the successfully paired audio semantic anchor point is bound to the original image corresponding to the visual semantic anchor point, and encapsulated into a graphic semantic unit with independent logical meaning. For the remaining audio semantic anchors or remaining visual semantic anchors that fail to match, secondary weak association merging is performed by selecting the matched text-image semantic units with the closest time interval based on the original timestamps of the remaining audio semantic anchors or remaining visual semantic anchors.

6. The method for generating medical records based on mobile terminals and multimodal data according to claim 1, characterized in that, After determining that the current observed state vector does not belong to the expected set of legal states, the following steps are also included: Obtain a preset modality confidence weight table. The modality confidence weight table sets the confidence priority of different modalities in different scenarios based on image recognition accuracy and speech recognition accuracy. Identify feature conflicts that lead to illegal state transitions, and compare the confidence weights of the image features and text features corresponding to the feature conflicts. If the confidence weight of the image features is higher than that of the text features, then generate an automatic correction instruction based on the image features to correct the text features. If the difference between the confidence weight of the image feature and the confidence weight of the text feature is less than a preset fuzzy threshold, an abnormal state warning mark is generated and attached to the corresponding image and text semantic unit. Apply automatic correction instructions to text-image semantic units to update the current observation state vector, or place text-image semantic units with abnormal state warning markers into the queue for manual review. The updated graphic semantic unit is then input into the finite state machine model for secondary logical closed-loop verification.

7. The method for generating medical records based on mobile terminals and multimodal data according to claim 1, characterized in that, The process of generating a structured medical record data package based on verified graphic semantic units, filling slots and standardizing medical terminology according to a preset medical record template, and writing it back to the hospital information management system includes the following steps: The semantic category tags contained in the verified image and text semantic units are analyzed to determine the medical document paragraph attributes corresponding to the image and text semantic units; Based on the type of ward round session and the current stage of treatment of the target patient, a matching medical record template is retrieved from the template library. The medical record template defines a standard document structure tree and data slots. Iterate through all text and image semantic units, and map and populate the text and image semantic units into one or more corresponding data slots in the medical record template according to the paragraph attributes of the medical document. Using a pre-defined medical terminology ontology database, synonym retrieval and standardized mapping are performed on the natural language text populated into the data slots; In accordance with the data interaction interface specifications of the hospital information management system, the completed medical record template is serialized into a structured medical record data packet, and the structured medical record data packet is sent to the hospital information management system through an encrypted transmission channel.

8. A medical record generation system based on mobile terminal and multimodal data, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method for generating medical records based on mobile terminals and multimodal data as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing instructions thereon, characterized in that, When executed by a processor, this instruction causes the processor to be configured to perform the method for generating medical records based on mobile terminals and multimodal data according to any one of claims 1 to 7.

Images