Multi-mode bedside emergency interaction management system and method based on multi-source data fusion

Abstract

The invention discloses a multi-mode bedside emergency interaction management system and method based on multi-source data fusion, and relates to the technical field of electric digital data processing. The system comprises a data fusion influence evaluation module, a data synchronization management regulation and control module, an interaction response validity evaluation module and an interaction response management regulation and control module. According to the method, the data fusion influence evaluation result is obtained through the obtained data fusion influence parameters, whether the data synchronization management regulation instruction is sent or not is judged, if yes, data synchronization management regulation is executed, and if not, the data fusion influence out-of-control prompt is sent. And obtaining an interaction response evaluation result based on the obtained interaction response quantization parameter, judging whether to send an interaction response management regulation instruction, if so, executing interaction response management regulation, and if not, sending an interaction response qualified instruction, thereby improving the fusion and response management effectiveness in multi-modal bedside emergency interaction, and improving the efficiency of the multi-modal bedside emergency interaction. The problem of low effectiveness of fusion and response management in multi-modal bedside emergency interaction in the prior art is solved.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing technology, and in particular to a multimodal bedside emergency interactive management system and method based on multi-source data fusion. Background Technology

[0002] With the development of bedside terminal interaction technology, in the initial stage of bedside emergency interaction, a high-definition camera is first used to capture the patient's subtle facial expressions and body movements to obtain facial feature data. Simultaneously, the patient's voice information is collected via a microphone. Next, in the bedside emergency interaction processing stage, facial expression recognition technology is used to analyze the image data in real time to identify the patient's current expression state, such as smiling or frowning. ASR (Automatic Speech Recognition) technology converts the patient's voice information collected by the microphone into text information. Multimodal data fusion technology integrates and correlates the identified facial expressions, emotion markers, and text information. After continuously monitoring the user's status and completing data fusion analysis, the bedside emergency interaction process begins. The user's status is continuously monitored, and key data such as emotion markers and intervention measures are transmitted to the interactive terminal in real time. The interactive terminal sends warning prompts to medical staff, who, upon receiving the warning, quickly arrive at the user's bedside.

[0003] For example, a method and system for intelligent interaction of medical information, as announced in patent publication CN113378554B, includes: obtaining the correlation between the descriptive content of sample medical information and the descriptive content of candidate key content; extracting candidate key content from the candidate key content to obtain first candidate key content; determining the interaction content difference degree between the sample medical information and the first candidate key content for each target medical information content; extracting second candidate key content from the first candidate key content where the interaction content difference degree is greater than a pre-set interaction content difference parameter; and determining the target interactive key content based on the second candidate key content.

[0004] For example, the bedside interaction method and system based on smart healthcare, as announced in patent publication CN113627165B, includes: calculating error index information based on key content and identification information of the interaction information; determining the bed location containing error information when one or more key contents of the description in the interaction information meet the fire source standard; when the bed location containing error information is determined, reading the interaction information of the target bed location identified after the current time point; extracting the content state vector of the error information in the interaction information identified at the current time point, or in the interaction information identified after the current time point, and determining whether the content state vector meets the first pre-set condition; when the content state vector meets the first pre-set condition, determining the content of the interaction information.

[0005] However, in the process of implementing the inventive technical solution in the embodiments of this application, it was found that the above-mentioned technology has at least the following technical problems: In bedside emergency scenarios involving emotional interaction, the fusion of multimodal data such as speech, facial expressions, and physiological signals presents challenges. Speech signals contain rich time-varying information, such as phonemes, and their tone changes very rapidly. While facial expressions also change rapidly, they generally contain less time-varying detail than speech. This results in different acquisition frequencies for different modalities, meaning different distribution densities of data points on the time axis. When aligning them to the same time axis, some information from high-frequency modalities is lost. This information loss during synchronous processing leads to poor data quality before fusion, resulting in inaccurate multimodal data fusion results. Intervention decisions based on inaccurate multimodal data fusion results do not match the user's actual needs and psychological state, leading to a mismatch between the system-generated interactive response and the user's actual needs and psychological state. Consequently, there is a problem of low effectiveness in fusion and response management in multimodal bedside emergency interactions. Summary of the Invention

[0006] This application provides a multimodal bedside emergency interaction management system and method based on multi-source data fusion, which solves the problem of low effectiveness of fusion and response management in multimodal bedside emergency interaction in the prior art, and improves the effectiveness of fusion and response management in multimodal bedside emergency interaction.

[0007] This application provides a multimodal bedside emergency interaction management system based on multi-source data fusion, including: a data fusion impact assessment module, a data synchronization management and control module, an interaction response effectiveness assessment module, and an interaction response management and control module. The data fusion impact assessment module is used to assess the impact of data acquisition and processing on bedside multimodal data fusion during bedside emergency interaction based on acquired data fusion impact parameters, obtaining a data fusion impact assessment result. The data synchronization management and control module is used to determine whether to send a data synchronization management and control command based on the data fusion impact assessment result; if so, it executes the data synchronization management and control; otherwise, it sends a data fusion impact out-of-control alert to preset medical personnel, and the data synchronization management and control... The control module adjusts the frame rate of image acquisition by the camera that collects data during bedside emergency interactions, and optimizes the time alignment of the processing before bedside multimodal data fusion. The interaction response effectiveness evaluation module evaluates the effectiveness of bedside emergency interactions based on the acquired interaction response quantification parameters and obtains the interaction response evaluation result. The interaction response management and control module determines whether to send an interaction response management and control command based on the interaction response evaluation result. If so, it executes the interaction response management and control; otherwise, it sends an interaction response qualification command to the preset medical staff. The interaction response management and control module optimizes the response alarm time of interaction responses during bedside emergency interactions and optimizes the fusion window for the next fusion process.

[0008] This application provides a multimodal bedside emergency interaction management method based on multi-source data fusion, including the following steps: 1) Assessing the impact of data acquisition and processing on bedside multimodal data fusion during bedside emergency interaction based on acquired data fusion impact parameters, obtaining a data fusion impact assessment result; 2) Determining whether to send a data synchronization management and control command based on the data fusion impact assessment result; if yes, executing data synchronization management and control; otherwise, sending a data fusion impact out-of-control alert to preset medical personnel. Data synchronization management and control refers to adjusting the image acquisition frame rate of the camera collecting data during bedside emergency interaction and optimizing the time alignment of the processing before bedside multimodal data fusion; 3) Assessing the effectiveness of bedside emergency interaction response based on acquired interaction response quantification parameters, obtaining an interaction response assessment result; 4) Determining whether to send an interaction response management and control command based on the interaction response assessment result; if yes, executing interaction response management and control; otherwise, sending an interaction response qualified command to preset medical personnel. Interaction response management and control refers to optimizing the response alarm time during bedside emergency interaction and optimizing the fusion window for the next fusion process.

[0009] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: 1. By acquiring data fusion impact parameters, the system obtains the data fusion impact assessment result and determines whether to send a data synchronization management and control command. If so, the system executes data synchronization management and control to improve the accuracy of data collection and processing during bedside emergency interaction; otherwise, it sends a data fusion impact out-of-control warning. Then, the system acquires interaction response quantification parameters, obtains the interaction response assessment result, and determines whether to send an interaction response management and control command. If so, the system executes interaction response management and control to improve the timeliness of interaction response during bedside emergency interaction; otherwise, it sends an interaction response qualification command. This improves the effectiveness of fusion and response management in multimodal bedside emergency interaction and solves the problem of low effectiveness of fusion and response management in existing technologies.

[0010] 2. Based on the data fusion impact assessment results, determine whether to send a data synchronization management and control command. If so, adjust the image acquisition frame rate of the camera that collects data during bedside emergency interaction to improve the accuracy of data acquisition quality during bedside emergency interaction, and optimize the time alignment of the processing process before bedside multimodal data fusion to shorten the total delay from data acquisition to processing, thereby improving the timeliness of data acquisition and processing in bedside emergency interaction. Otherwise, send a data fusion impact out of control prompt to the preset medical staff, thus achieving the synchronization of bedside emergency multimodal data fusion.

[0011] 3. Based on the evaluation results of the interactive response, determine whether to send an interactive response management and control command. If so, optimize the response alarm time during the bedside emergency interaction process to shorten the total time from triggering the event to issuing an effective alarm, thereby improving the timeliness of the bedside emergency interactive response. Also, optimize the fusion window for the next fusion process to reduce noise and errors introduced by time misalignment, ensuring the accurate reflection of multimodal data in time, and thus improving the effectiveness of the bedside emergency interaction response. Otherwise, send a qualified interactive response command to the preset medical staff. Attached Figure Description

[0012] Figure 1 A schematic diagram of the structure of the multimodal bedside emergency interactive management system based on multi-source data fusion provided in this application embodiment; Figure 2 A flowchart illustrating the time alignment optimization and adjustment process of a multimodal bedside emergency interactive management system based on multi-source data fusion, provided in this application embodiment; Figure 3 A flowchart illustrating the response alarm time optimization of a multimodal bedside emergency interactive management system based on multi-source data fusion, provided in this application embodiment; Figure 4 A flowchart of a multimodal bedside emergency interactive management method based on multi-source data fusion provided in this application embodiment. Detailed Implementation

[0013] This application provides a multimodal bedside emergency interaction management system and method based on multi-source data fusion, which solves the problem of low effectiveness of fusion and response management in existing technologies for multimodal bedside emergency interaction. By acquiring data fusion impact parameters, the system obtains data fusion impact assessment results and determines whether to send a data synchronization management and control command. If so, data synchronization management and control are executed; otherwise, a data fusion impact out-of-control prompt is sent. Then, the system obtains interaction response quantification parameters, obtains interaction response assessment results, and determines whether to send an interaction response management and control command. If so, interaction response management and control are executed; otherwise, an interaction response qualified command is sent, thereby improving the effectiveness of fusion and response management in multimodal bedside emergency interaction.

[0014] The technical solution in this application embodiment aims to address the aforementioned problem of low effectiveness in fusion and response management during multimodal bedside emergency interaction. The overall approach is as follows: By assessing the impact of data acquisition and processing on bedside multimodal data fusion during bedside emergency interactions using acquired data fusion impact parameters, a data fusion impact assessment result is obtained. Based on this assessment result, it is determined whether to send a data synchronization management and control command. If so, adjustments are made to the number of image acquisition frames and time alignment optimization; otherwise, a data fusion impact out of control alert is sent to designated medical personnel. Simultaneously, the effectiveness of the bedside emergency interaction response is assessed using interaction response quantification parameters, resulting in an interaction response assessment result. Based on this result, it is determined whether to send an interaction response management and control command. If so, optimization of the response alarm time and adjustment of the window size for the next data fusion are performed; otherwise, a qualified interaction response command is sent to designated medical personnel, thereby improving the effectiveness of fusion and response management in multimodal bedside emergency interactions.

[0015] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0016] As an example of the first aspect, such as Figure 1 The diagram shown is a structural schematic of a multimodal bedside emergency interactive management system based on multi-source data fusion provided in this application embodiment. The multimodal bedside emergency interactive management system based on multi-source data fusion provided in this application embodiment includes: a data fusion impact assessment module, a data synchronization management and control module, an interactive response effectiveness assessment module, and an interactive response management and control module.

[0017] The data fusion impact assessment module is used to evaluate the impact of data acquisition and processing on bedside multimodal data fusion during bedside emergency interaction based on the acquired data fusion impact parameters, and obtain the data fusion impact assessment results. This module specifically includes a data fusion impact parameter acquisition unit, a data fusion impact parameter quantification unit, and a data fusion impact threshold storage unit.

[0018] Specifically, the data fusion impact parameter acquisition unit is used to obtain data fusion impact parameters by monitoring the process before bedside multimodal data fusion through sensors. These parameters include ASR voice-text generation time, image processing time, bedside multimodal data time alignment deviation, and bedside multimodal data synchronization error. The data fusion impact parameter quantification unit is used to assess the impact of the bedside multimodal data acquisition and processing process based on the acquired data fusion impact parameters, and to determine the data fusion impact assessment result based on the obtained bedside multimodal data fusion impact values. The data fusion impact threshold storage unit is used to obtain data fusion impact thresholds and data fusion impact compensation factors from the constructed bedside emergency interaction database. The data fusion impact thresholds include ASR voice-text generation time thresholds, image processing time thresholds, bedside multimodal data time alignment deviation thresholds, and bedside multimodal data synchronization error thresholds. The data fusion impact compensation factors include ASR voice-text generation time compensation factors, image processing time compensation factors, bedside multimodal data time alignment deviation compensation factors, and bedside multimodal data synchronization error compensation factors.

[0019] The data synchronization management and control module determines whether to send a data synchronization management and control command based on the data fusion impact assessment results. If so, it executes the data synchronization management and control; otherwise, it sends a data fusion impact out-of-control alert to preset medical personnel. Data synchronization management and control refers to adjusting the image frame rate of the camera collecting data during bedside emergency interactions and optimizing the time alignment of the processing steps before bedside multimodal data fusion. The designed data synchronization management and control module can proactively identify and resolve issues affecting the accuracy of bedside multimodal data fusion during data acquisition and processing. By adjusting the image frame rate and optimizing time alignment, it improves the reliability of multimodal data fusion during bedside emergency interactions, avoiding misjudgments or delays caused by data quality issues. The data synchronization management and control command indicates the synchronization status of the data acquisition and processing processes during bedside emergency interactions.

[0020] The interaction response effectiveness evaluation module is used to evaluate the effectiveness of bedside emergency interactions based on the acquired interaction response quantification parameters, and obtain the interaction response evaluation results. This module specifically includes an interaction response quantification parameter acquisition unit, an interaction response quantification parameter quantification unit, and an interaction response quantification threshold storage unit.

[0021] Specifically, the interaction response quantification parameter acquisition unit is used to acquire interaction response quantification parameters during the bedside emergency interaction response process, including the impact value of bedside multimodal data fusion, interaction latency deviation, interaction command generation duration, and interaction command execution response time. The interaction response quantification parameter quantification unit is used to evaluate the effectiveness of the bedside emergency interaction response based on the acquired interaction response quantification parameters to obtain the effective value of the bedside interaction response, and to determine the interaction response evaluation result based on the obtained effective value of the bedside interaction response. The interaction response quantification threshold storage unit is used to acquire interaction response quantification thresholds and interaction response quantification compensation factors from the constructed bedside emergency interaction database. The interaction response quantification thresholds include the bedside multimodal data fusion threshold, interaction latency deviation threshold, interaction command generation duration threshold, and interaction command execution response time threshold. The interaction response quantification compensation factors include the bedside multimodal data fusion compensation factor, interaction latency deviation compensation factor, interaction command generation duration compensation factor, and interaction command execution response time compensation factor.

[0022] The interaction response management and control module determines whether to send an interaction response management and control command based on the interaction response evaluation results. If so, the interaction response management and control is executed; otherwise, a qualified interaction response command is sent to the preset medical staff. Interaction response management and control optimizes the response alarm time during bedside emergency interactions and optimizes the fusion window for the next fusion process to improve the speed and effectiveness of bedside emergency interactions. This module addresses the latency issue in bedside emergency interactions, and the interaction response management and control command represents a mechanism for interaction responses during bedside emergency interactions.

[0023] In this embodiment, four modules work together to form a closed-loop management process from multimodal data acquisition, processing, and fusion to interactive response, evaluation, and readjustment. All emotional data, voice information, facial expressions, and motion data collected through bedside terminals are stored in the hospital's electronic medical record system and provided to relevant medical staff for analysis. This enables synchronized data management and interactive response management in bedside emergency interaction scenarios, improving the effectiveness of the multimodal bedside emergency interaction management system. Bedside multimodal data includes, but is not limited to, video and voice data. Video data includes facial expression data such as patient pain and agitation, and limb movement data such as convulsions and tremors. Voice data includes language feature data such as volume, speech rate, and pitch, as well as environmental background noise data such as other patients' voices and background noise from instrument operation.

[0024] It should be further explained that in the design of the multimodal bedside emergency interaction management system based on multi-source data fusion, a dedicated bedside emergency interaction database was pre-established to store various key setting data, including various preset data required by the system, such as ASR voice-to-text generation time thresholds, image processing time thresholds, and bedside multimodal data synchronization error thresholds. The initial values ​​of these parameters are not set arbitrarily. For example, the bedside multimodal data synchronization error threshold is calculated by summing and averaging the historical multimodal data synchronization errors accumulated in the database, ensuring that the initial settings have a certain degree of objectivity and representativeness. Furthermore, to adapt to the complexity of actual application scenarios and constantly changing needs, all these values ​​in the database are not static but allow technicians to manually set, adjust, and fine-tune them based on the system's performance during actual debugging, thereby optimizing the system parameters.

[0025] Furthermore, the specific steps for obtaining the impact value of bedside multimodal data fusion are as follows: The ratio of ASR speech-text generation time to ASR speech-text generation time threshold is compensated using an ASR speech-text generation time compensation factor, and this result is denoted as the text generation time impact value. The specific formula is as follows: In the formula, This indicates the impact of text generation time. This represents the ASR voice-text generation time compensation factor obtained from the bedside emergency interaction database. The ASR speech-to-text generation time, obtained through ASR technology, refers to the time required from receiving the speech signal to outputting the final text. This represents the time threshold for ASR voice-text generation obtained from the bedside emergency interaction database.

[0026] Secondly, the ratio of image processing time to image processing time threshold is compensated by an image processing time compensation factor, denoted as the processing time influence value, and its specific expression is as follows: In the formula, This indicates the impact value on processing time. This represents the image processing time compensation factor obtained from the bedside emergency interactive database. This represents the image processing time obtained through the image signal processor, and the total time required to process image data during bedside multimodal data processing. This represents the image processing time threshold obtained from the bedside emergency interactive database.

[0027] Then, the ratio of the time alignment deviation of bedside multimodal data to the threshold of bedside multimodal data is compensated by the time alignment deviation compensation factor, and denoted as the alignment deviation influence value. The specific expression is as follows: In the formula, This indicates the impact value of alignment deviation. This represents the time alignment deviation compensation factor for bedside multimodal data obtained from the bedside emergency interactive database. This refers to the time alignment deviation of bedside multimodal data obtained through GPS (Global Positioning System) time synchronization technology. It indicates the deviation between the actual aligned position and the target position during the process of mapping data streams from different modalities to a unified time axis. This represents the time alignment deviation threshold for bedside multimodal data obtained from the bedside emergency interactive database.

[0028] Next, the ratio of the synchronization error to the threshold of the bedside multimodal data synchronization error is compensated by the bedside multimodal data synchronization error compensation factor, and this is denoted as the synchronization error impact value. The specific expression is as follows: In the formula, This indicates the impact value of synchronization error. This represents the synchronization error compensation factor for bedside multimodal data obtained from the bedside emergency interactive database. This refers to the bedside multimodal data synchronization error, obtained by adding a unified timestamp to the data using hardware clock synchronization technology such as GPS clock, and calculating the absolute error between the timestamp of the modal fusion data and the standard time. It represents the ratio of the amount of misaligned data to the total amount of data during the multimodal data fusion process. This represents the synchronization error threshold for bedside multimodal data obtained from the bedside emergency interactive database.

[0029] Finally, the impact values ​​of text generation time, processing time, alignment deviation, and synchronization error are coupled to obtain the impact value of bedside multimodal data fusion, the specific expression of which is: , This represents the impact value of bedside multimodal data fusion, which measures the degree of combined influence of data fusion impact parameters on bedside multimodal data fusion.

[0030] In this embodiment, the bedside emergency interaction database stores compensation factors corresponding to the data fusion impact parameters, namely, ASR speech-text generation time compensation factor, image processing time compensation factor, bedside multimodal data time alignment deviation compensation factor, and bedside multimodal data synchronization error compensation factor. These factors typically range from 0 to 1, and their sum is 1. A pre-defined mapping relationship exists between these compensation factors and the data fusion impact parameters. This mapping relationship can be one-to-one or many-to-one. For example, in practical applications, real-time data fusion impact parameters can be input into this mapping relationship to quickly obtain the corresponding compensation factors.

[0031] Specifically, a higher time alignment deviation in bedside multimodal data means a greater misalignment of multimodal data on the time axis, resulting in a higher synchronization error in bedside multimodal data. The longer the ASR speech-text generation time, the more new errors are introduced, leading to data loss or duplication, and a greater time alignment deviation in bedside multimodal data. The longer the image processing time, the more the differences in multimodal data processing time will amplify the difficulty of alignment, resulting in a greater time alignment deviation in bedside multimodal data.

[0032] There is a positive correlation between ASR speech-to-text generation time and the impact of bedside multimodal data fusion. Longer ASR speech-to-text generation times mean longer waiting times for the system to understand and process user voice input, resulting in a higher impact value. Similarly, there is a positive correlation between bedside multimodal data time alignment deviation and the impact value. Larger deviations indicate time misalignment between different modalities, leading to a higher impact value. Image processing time also shows a positive correlation, resulting in lower video frame rates and less smooth playback, further contributing to the impact value. Finally, there is a positive correlation between bedside multimodal data synchronization error and the impact value. Larger synchronization errors indicate incorrect or unsuccessful matching attempts when synchronizing different modalities, resulting in a higher impact value.

[0033] It is important to understand that considering the interrelationships among the four independent variables helps to improve the adaptive adjustment of the data fusion impact parameters, thereby enhancing the accuracy of multimodal data fusion. Furthermore, considering the positive and negative correlations between the four independent variables and the impact values ​​of bedside multimodal data fusion helps to conduct data synchronization management and control, thereby improving the reliability of data collection and processing during bedside emergency interaction.

[0034] Furthermore, the specific steps for determining whether to send a data synchronization management and control instruction based on the data fusion impact assessment results are as follows: If the obtained bedside multimodal data fusion impact value is less than or equal to the preset bedside multimodal data fusion threshold, the data fusion impact assessment result is recorded as qualified, and no additional processing is required; if the bedside multimodal data fusion impact value is within the preset bedside multimodal data fusion safety range, the data fusion impact assessment result is recorded as unqualified, and a data synchronization management and control instruction is sent to execute data synchronization management and control. The preset bedside multimodal data fusion safety range represents the open interval formed by the preset bedside multimodal data fusion safety threshold and the preset bedside multimodal data fusion threshold; if the bedside multimodal data fusion impact value is greater than or equal to the preset bedside multimodal data fusion threshold, the data fusion impact assessment result is recorded as data fusion out of control, and a data fusion impact out of control notification is sent to the preset medical staff.

[0035] As a further solution, the specific steps for adjusting the image acquisition frame count are as follows: A1, if the acquired average light intensity is lower than or equal to the preset light intensity safety threshold, then obtain the harmonic average result of the corresponding bedside multimodal data fusion influence and light intensity influence. If the obtained value is an integer, it is directly used as the reduction amount of the camera's image acquisition frame count; otherwise, the result is rounded up as the reduction amount of the camera's image acquisition frame count. The preset next image acquisition frame count is subtracted from the reduction amount of the image acquisition frame count to obtain the actual image acquisition frame count for the next time, thereby reducing the image acquisition burden and avoiding processing delays or data quality degradation that may be caused by excessive frame counts. If the image acquisition frame count is not adjusted, then the corresponding image acquisition frame count for each time is the preset... A2. If the average light intensity is within the preset light intensity safety range, no adjustment is made to the number of image acquisition frames. A3. If the average light intensity is greater than or equal to the preset light intensity threshold, the harmonic average of the corresponding bedside multimodal data fusion influence and light intensity correction is obtained. If the obtained value is an integer, it is directly used as the increase in the number of image frames acquired by the camera. Otherwise, the result is rounded down as the increase in the number of image frames acquired by the camera to reduce the burden of data processing. The preset number of image acquisition frames for the next time is added to the increase in the number of image frames acquired by the camera to obtain the actual number of image acquisition frames for the next time to improve the richness and quality of image data and provide a higher quality foundation for subsequent multimodal data fusion analysis.

[0036] Understandably, by adjusting the number of image frames captured by the camera used for data collection during bedside emergency interactions, the acquisition strategy can be dynamically optimized based on the average light intensity and the needs of multimodal data fusion. This effectively balances the burden of multimodal data acquisition with data quality, avoids processing delays and data quality degradation, and improves the richness and quality of image data, providing a higher quality foundation for subsequent multimodal data fusion analysis.

[0037] like Figure 2 The diagram shows a flowchart of the time alignment optimization adjustment process for a multimodal bedside emergency interactive management system based on multi-source data fusion provided in this application embodiment. The process includes the following steps: The voice-video time offset is compared with a preset voice-video time offset threshold. If the voice-video time offset is greater than or equal to the preset threshold, the time correction is obtained by harmonic averaging the corresponding bedside multimodal data fusion impact and voice-video time compensation, and applied to the next voice-video time adjustment. If the voice-video time offset is less than or equal to a preset safe threshold, the time compensation is obtained by harmonic averaging the corresponding bedside multimodal data fusion impact and voice-video time correction, and applied to the next voice-video time adjustment. If the voice-video time offset is within the preset safe range, no time alignment optimization adjustment is performed.

[0038] Specifically, the steps for time alignment optimization are as follows: B1, if the voice-video time offset is greater than or equal to the preset voice-video time offset threshold, the harmonic average of the corresponding bedside multimodal data fusion influence and voice-video time compensation is obtained as the time correction amount. The preset next voice-video time is subtracted from the time correction amount to obtain the next actual voice-video time. By subtracting the time correction amount, this deviation transmission is offset or compensated, and the next voice-video synchronization time is set earlier than the standard voice-video time value to offset the expected delay or deviation accumulation and improve the synchronization accuracy of voice and video data on the time axis. If no time alignment optimization is performed, the corresponding voice-video times are all preset to the same value. The voice-video time compensation amount represents the ratio of the difference between the voice-video time offset and the preset voice-video time offset threshold to the preset standard voice-video time value; B2, if the voice-video time offset is less than or equal to the preset voice-video time offset safety threshold, the corresponding bedside multimodal data fusion influence and voice-video time are obtained. The result of harmonic averaging of the correction amount is used as the time compensation amount. The preset next voice-video time is added to the time compensation amount to obtain the next actual voice-video time. By adding the time compensation amount, minor drifts or interferences are offset, making the synchronization adjustment smoother and more continuous. This avoids frequent and drastic switching adjustments of the voice-video time near the safe threshold of the voice-video time offset. For example, it adds or subtracts back and forth around the standard value of the voice-video time to prevent it from slowly sliding out of the safe zone. This adjusts the relative position of the voice and video data on the time axis, thereby compensating for the previously detected synchronization deviation and enabling better alignment between the two. The voice-video time correction amount represents the minimum deviation of the voice-video time. The minimum deviation amount represents the ratio of the difference between the preset safe threshold of the voice-video time offset and the voice-video time offset to the standard value of the voice-video time. B3, if the voice-video time offset is within the preset safe range of the voice-video time offset, no time alignment optimization adjustment is performed. The voice-video time offset is used to measure the difference between the voice-video positioning time and the preset voice-video positioning time on the same time axis.

[0039] It is understandable that using static time window alignment to handle voice-video time deviations is difficult to adapt to the dynamic changes in multimodal data, leading to the accumulation of voice-video time synchronization errors and the phenomenon of lip movements and voice misalignment. In this embodiment, by optimizing and adjusting the time alignment of the processing before bedside multimodal data fusion, the identification and compensation of time deviations in voice and video data are realized, thereby ensuring the accurate synchronization and consistency of multimodal data on the time axis, laying a solid foundation for subsequent high-quality multimodal data fusion and emergency interactive analysis.

[0040] Furthermore, the specific steps to obtain the effective value of the bedside interaction response are as follows: The ratio of the bedside multimodal data fusion threshold to the bedside multimodal data fusion influence value is corrected using a bedside multimodal data fusion compensation factor to obtain the effective value of data fusion. The specific expression is: In the formula, Indicates the valid value of data fusion. This represents the compensation factor for the impact value of bedside multimodal data fusion obtained from the bedside emergency interactive database. This represents the threshold for fusing bedside multimodal data obtained from the bedside emergency interaction database. This indicates the impact value of bedside multimodal data fusion. It is represented as a constant term to avoid meaningless numbers.

[0041] Secondly, the interaction delay deviation threshold and the ratio of interaction delay deviation are corrected by the interaction delay deviation compensation factor, and this is denoted as the effective value of the delay deviation. The specific expression is as follows: In the formula, Indicates the effective value of the time delay deviation. This represents the interaction delay deviation compensation factor obtained from the bedside emergency interaction database. This represents the interaction latency deviation threshold obtained from the bedside emergency interaction database. This refers to the interaction latency deviation obtained through data analysis tools such as Python, which is the difference between the actual time required to complete an interaction and a preset interaction time threshold. This is represented as a constant term. To avoid meaningless numbers, if the time required for one interaction is not equal to the preset interaction time threshold, then... If the time required for one interaction is equal to the preset interaction time threshold, then... It is a non-zero constant, for example, 0.001.

[0042] Then, the ratio of the interaction command generation time threshold to the interaction command generation time is corrected by the interaction command generation time compensation factor, and this is denoted as the effective value of the generation time. The specific expression is as follows: In the formula, Indicates the valid value of the generation duration. This represents the compensation factor for the generation time of interaction commands obtained from the bedside emergency interaction database. This represents the threshold for the generation time of interaction commands obtained from the bedside emergency interaction database. This indicates the time difference between the time of receiving multimodal data and the time of response generation, recorded by timestamps. It refers to the time required from receiving the patient's multimodal data to generating and outputting the interactive response.

[0043] Next, the ratio of the interaction command execution response time threshold to the interaction command execution response time is corrected by an interaction command execution response time compensation factor, and this is recorded as the effective response time value. The specific expression is as follows: In the formula, Indicates the valid value of the response time. This represents the response time compensation factor for interactive commands retrieved from the bedside emergency interaction database. This represents the threshold for the response time to interactive commands obtained from the bedside emergency interaction database. This indicates the response time for executing an interactive command, which is obtained by recording the timestamps of the start and completion of the interactive command. It refers to the time required to complete the execution of the interactive command.

[0044] Finally, the effective values ​​of delay deviation, response time, and response time are coupled to obtain the effective value of bedside interaction response, which is expressed as follows: ; This represents the effective value of the bedside interaction response, indicating the degree of effectiveness of the bedside emergency interaction response as measured by the interaction response quantification parameters.

[0045] In this embodiment, the bedside emergency interaction database stores compensation factors corresponding to the interaction response quantification parameters, namely, the bedside multimodal data fusion compensation factor, the interaction delay deviation compensation factor, the interaction command generation time compensation factor, and the interaction command execution response time compensation factor. These factors typically range from 0 to 1, and their sum is 1. A pre-defined mapping relationship exists between these compensation factors and the interaction response quantification parameters. This mapping relationship can be one-to-one or many-to-one. For example, in practical applications, real-time interaction response quantification parameters can be input into this mapping relationship to quickly obtain the corresponding compensation factors.

[0046] Specifically, the lower the impact value of bedside multimodal data fusion, the worse the quality of multimodal data fusion, resulting in delayed interaction response and longer interaction command generation time. The longer the interaction command generation time, the more likely the data is messy and fused incorrectly, making the system prone to misjudgment and resulting in a longer interaction command execution response time. The greater the interaction delay deviation, the worse the time alignment between different modal data, and the higher the impact value of bedside multimodal data fusion.

[0047] There is a negative correlation between the impact value of bedside multimodal data fusion and the effective value of bedside interaction response. The larger the impact value of bedside multimodal data fusion, the less accurate the patient status and environmental information obtained by the system, and the smaller the effective value of bedside interaction response. There is also a negative correlation between interaction latency deviation and the effective value of bedside interaction response. The larger the interaction latency deviation, the worse the real-time performance of the interaction process, and the lower the effective value of bedside interaction response. There is also a negative correlation between the interaction command generation time and the effective value of bedside interaction response. The longer the interaction command generation time, the longer the user has to wait to see the system response or execute an action after inputting the command. Finally, there is a negative correlation between the interaction command execution response time and the effective value of bedside interaction response. The longer the interaction command execution response time, the higher the risk of management errors, the lower the efficiency of command execution, and the smaller the effective value of bedside interaction response.

[0048] It should be noted that by considering the interrelationships among the four independent variables, we can more accurately identify and quantify the specific impact of each factor on the effectiveness of bedside interaction response, thereby shortening interaction latency and improving command execution accuracy. By analyzing the positive and negative correlations between the four independent variables and the effective value of bedside interaction response, we can quantitatively assess the effectiveness of each factor on bedside interaction response, which helps in the management and control of interaction response. This allows for targeted optimization of data fusion quality, shortening processing and execution latency, and ultimately improving the timeliness of bedside interaction response.

[0049] Furthermore, the specific steps for determining whether to send an interaction response management and control instruction based on the interaction response evaluation results are as follows: the difference between the effective value of the bedside interaction response and the average effective value of the bedside interaction response within the preset number of bedside interactions is recorded as the effective deviation value of the bedside interaction response; if the obtained effective deviation value of the bedside interaction response is not less than the preset effective deviation threshold of the bedside interaction response, the interaction response evaluation result is recorded as qualified and an interaction response qualified instruction is sent to the preset medical staff; otherwise, the interaction response evaluation result is recorded as unqualified and an interaction response management and control instruction is sent for interaction response management and control.

[0050] like Figure 3 The diagram shows a flowchart for optimizing the response alarm time of a multimodal bedside emergency interaction management system based on multi-source data fusion provided in this application embodiment. The specific process is as follows: The system judges the interaction command execution time against a preset safe threshold. If the interaction command execution time is not less than the preset safe threshold, the system obtains the harmonic average of the effective deviation of the bedside interaction response and the interaction command execution time, and adjusts the response alarm time accordingly. If the interaction command execution time is less than the preset safe threshold, the current alarm time is maintained.

[0051] Specifically, the execution steps for optimizing the alarm response time are as follows: If the execution time of the interactive command is not less than the preset threshold for the execution time of the interactive command, the harmonic average of the effective deviation of the bedside interactive response and the impact of the execution time of the interactive command is used as the increase in the alarm response time. The preset next alarm response time is added to the increase in the alarm response time to obtain the actual alarm response time for the next time. The longer the execution time of the interactive command, the higher the network latency and other issues. Extending the alarm response time essentially allows more time to process subsequent tasks (including confirming alarm conditions, issuing alarm signals, and receiving feedback). If the system processes interactive commands slowly, However, if the alarm response time is set according to the standard alarm response time, the alarm time will expire before the system has completed its processing steps, resulting in the alarm not being issued in time or the system not being fully ready when the alarm is issued, causing chaos. Extending the alarm time can avoid this situation, thus providing more sufficient response and preparation time to deal with any delays, thereby improving the fault tolerance and safety of the bedside emergency interaction process. If the alarm response time is not optimized, the alarm response time in the corresponding bedside interaction response process will be the same preset value; if the execution time of the interaction command is less than the preset threshold for the execution time of the interaction command, the alarm response time will not be optimized.

[0052] The interaction command execution time refers to the time from receiving the interaction command to triggering its execution. The response alarm indicates that when the interaction command execution time exceeds the expected requirement (i.e., the interaction command execution time is not less than the preset interaction command execution time threshold), the system will automatically send an alarm to the preset medical staff to indicate that the current interaction command execution time is abnormal and requires intervention. The corresponding response alarm time represents the time set by the preset medical staff for the interaction command execution timeout to require intervention. No response alarm indicates that the current interaction command execution time meets the expected requirements.

[0053] Understandably, by optimizing the response alarm time during bedside emergency interactions, the dynamic adaptation and adjustment of the execution time of interactive commands can be achieved. The response alarm waiting time can be flexibly increased or shortened according to the actual execution situation, thereby improving the response flexibility and processing efficiency in dealing with different interactive command execution efficiency scenarios, and ensuring the timeliness of bedside emergency interactions.

[0054] Specifically, the execution steps for fusion window optimization are as follows: If the interaction response time is less than or equal to the preset interaction response time safety threshold, fusion window optimization is not performed, and the modal fusion window for each iteration is the same preset value; if the interaction response time is within the preset interaction response time safety range, the harmonic average of the bedside interaction response effective deviation and the interaction response time correction is used as the modal fusion window increment. The preset next modal fusion window is added to the modal fusion window increment to obtain the next actual modal fusion window, thereby expanding the fusion window for multimodal data to avoid data mismatch or omission. A longer system response time means that data packets may arrive at off-peak times, multi-source data synchronization may be unbalanced, and there may be drift between voice and action. If the original modal fusion window is maintained, data from the previous moment may be misclassified as data from the next moment (mismatch), or delayed data may be directly discarded (omission), leading to fusion failure or response errors. Expanding the window within the performance allowable range allows the system to tolerate more data jitter, network latency, or computational latency to accommodate more multimodal data information over a wider time range. This improves the richness and accuracy of the next multimodal data fusion. If the interaction response time is greater than or equal to the preset interaction response time threshold, the harmonic average of the effective impact of the bedside interaction response and the interaction response time comparison is used as the modality fusion window reduction amount. The preset modality fusion window for the next time is subtracted from the modality fusion window reduction amount to obtain the actual modality fusion window for the next time, thereby reducing the fusion window of the next multimodal data. By calculating the modality fusion window reduction amount, the fusion window of the multimodal data can be adjusted according to the actual interaction response time, avoiding the problem of insufficient adaptability of multimodal data fusion caused by a fixed window size. When the interaction response time is longer, it means that the current processing burden is heavier or the data complexity is higher. Reducing the modality fusion window avoids the waste of resources caused by processing too much data, thereby reducing the complexity and latency of the next multimodal data processing and helping to improve the response speed and accuracy of interaction commands. The modality fusion window refers to the time interval used to process multimodal data when performing multimodal data fusion (e.g., fusing multiple types of data such as voice, video, and physiological signals).

[0055] Understandably, by optimizing the fusion window for the next fusion process, dynamic adaptive adjustment of the multimodal data processing range is achieved. Based on the actual performance of the previous interaction response, it can intelligently decide whether to expand, shrink, or maintain the modal fusion window, thereby improving the efficiency of multimodal data fusion. This ensures both the richness of fused information during interaction and rapid response during interaction delays, ultimately optimizing the effectiveness of bedside emergency interaction response.

[0056] As an embodiment of the second aspect, such as Figure 4The diagram shows a flowchart of a multimodal bedside emergency interaction management method based on multi-source data fusion provided in this application embodiment. The method includes the following steps: assessing the impact of data acquisition and processing on bedside multimodal data fusion during bedside emergency interaction based on acquired data fusion impact parameters, obtaining a data fusion impact assessment result; determining whether to send a data synchronization management and control instruction based on the data fusion impact assessment result; if so, executing data synchronization management and control; otherwise, sending a data fusion impact out-of-control alert to preset medical personnel, indicating that the data synchronization management and control is in control. The system adjusts the frame rate of images captured by cameras during bedside emergency interactions and optimizes the time alignment of the processing steps before bedside multimodal data fusion. It assesses the effectiveness of bedside emergency interactions based on the acquired quantitative parameters of the interaction response, obtaining an interaction response evaluation result. Based on the evaluation result, it determines whether to send an interaction response management and control command. If so, the command is executed; otherwise, a qualified interaction response command is sent to the designated medical personnel. The interaction response management and control optimizes the alarm time for interaction responses during bedside emergency interactions and optimizes the fusion window for the next fusion process.

[0057] In this embodiment, in a bedside emergency interaction scenario, an integrated camera, microphone, and sensors monitor the patient's facial expressions, body movements, and voice changes in real time. When the patient experiences significant emotional fluctuations, the emotional assistant automatically records and marks the emotional information at that moment, facilitating subsequent analysis and intervention by doctors. The camera can also monitor the patient's body movements, such as involuntary twitching, struggling, or discomfort. In this way, the system can promptly acquire the patient's emotional signals, further improving the accuracy of recognition and the timeliness of response. The microphone acquires the patient's voice and converts it into text. After the speech is converted into text, the system uses an emotion analysis algorithm to identify the emotional nuances, such as changes in tone, speaking speed, and pitch, thereby determining the patient's emotional state. After acquiring facial expression data, body movement data, and speech recognition results from the camera, the system comprehensively analyzes this information, using a multimodal emotion analysis algorithm to determine the emotional state. Upon recognizing the patient's emotional fluctuations, it provides real-time feedback based on the emotional state.

[0058] This application, through data synchronization management and control, reduces the number of image frames captured by the camera in low-light areas to reduce processing burden and avoid introducing noise. Furthermore, it optimizes the time alignment of voice and video data, compensating for time deviations in previously detected voice and video data, enabling more precise synchronization. By implementing interactive response management and control, it increases or decreases the waiting time for the next response alarm, giving medical staff more time to prepare. Simultaneously, to obtain more comprehensive information in the next round of data fusion to aid judgment, it expands or narrows the modal fusion window, integrating multimodal data over a longer period. This allows for dynamic adaptation to complex and changing environments and interaction needs, effectively improving the responsiveness of bedside emergency interactions and effectively solving the problem of low fusion and response management effectiveness in multimodal bedside emergency interactions in existing technologies.

[0059] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0060] This invention is described with reference to flowchart illustrations and / or block diagrams of systems, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0061] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0062] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0063] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0064] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A multi-modal bedside emergency interactive management system based on multi-source data fusion, characterized in that, The data fusion influence evaluation module, the data synchronization management and control module, the interactive response effectiveness evaluation module, and the interactive response management and control module are included. The data fusion influence evaluation module is configured to evaluate the influence of data acquisition and processing on bedside multi-modal data fusion in the bedside emergency interaction process according to the obtained data fusion influence parameters, and obtain a data fusion influence evaluation result. The data synchronization management and control module is configured to determine whether to send a data synchronization management and control instruction according to the data fusion influence evaluation result, and if yes, perform data synchronization management and control, otherwise send a data fusion influence out-of-control prompt to a preset medical staff, wherein the data synchronization management and control refers to adjusting the image acquisition frame number of the camera for data acquisition in the bedside emergency interaction process, and optimizing and adjusting the time alignment of the processing before bedside multi-modal data fusion. The interactive response effectiveness evaluation module is configured to evaluate the response effectiveness of the bedside emergency interaction according to the obtained interactive response quantitative parameters, and obtain an interactive response evaluation result. The interactive response management and control module is configured to determine whether to send an interactive response management and control instruction according to the interactive response evaluation result, and if yes, perform interactive response management and control, otherwise send an interactive response qualified instruction to a preset medical staff, wherein the interactive response management and control refers to optimizing the response alarm time of the interactive response in the bedside emergency interaction process, and optimizing the fusion window of the next fusion process.

2. The multi-modal bedside emergency interactive management system based on multi-source data fusion according to claim 1, wherein, The data fusion influence evaluation module includes a data fusion influence parameter acquisition unit, a data fusion influence parameter quantification unit, and a data fusion influence threshold storage unit. The data fusion influence parameter acquisition unit is configured to obtain data fusion influence parameters by monitoring the process before bedside multi-modal data fusion through a sensor, and specifically includes ASR speech text generation time, image processing time, bedside multi-modal data time alignment deviation, and bedside multi-modal data synchronization error. The data fusion influence parameter quantification unit is configured to evaluate the influence of bedside multi-modal data acquisition and processing according to the obtained data fusion influence parameters, and determine a data fusion influence evaluation result based on the obtained bedside multi-modal data fusion influence value. The data fusion influence threshold storage unit is configured to obtain a data fusion influence threshold and a data fusion influence compensation factor from a constructed bedside emergency interaction database, wherein the data fusion influence threshold includes an ASR speech text generation time threshold, an image processing time threshold, a bedside multi-modal data time alignment deviation threshold, and a bedside multi-modal data synchronization error threshold, and the data fusion influence compensation factor includes an ASR speech text generation time compensation factor, an image processing time compensation factor, a bedside multi-modal data time alignment deviation compensation factor, and a bedside multi-modal data synchronization error compensation factor.

3. The multi-modal bedside emergency interactive management system based on multi-source data fusion according to claim 2, wherein, The specific acquisition steps of the bedside multi-modal data fusion influence value are as follows: Comparing the data fusion influence parameters with the corresponding data fusion influence thresholds; Coupling the data fusion influence compensation factor with the corresponding comparison results after weighting, and obtaining the bedside multi-modal data fusion influence value. The bedside multi-modal data fusion influence value represents the influence degree of the data fusion influence parameter on the bedside multi-modal data fusion.

4. The multi-modal bedside emergency interactive management system based on multi-source data fusion of claim 3, wherein, The specific step of determining whether to send the data synchronization management control instruction according to the data fusion influence evaluation result is: If the obtained bedside multi-modal data fusion influence value is less than or equal to the preset bedside multi-modal data safety threshold, the data fusion influence evaluation result is recorded as data fusion influence qualified, and no additional processing is performed; If the bedside multi-modal data fusion influence value is within the preset bedside multi-modal data fusion safety interval, the data fusion influence evaluation result is recorded as data fusion influence unqualified, and a data synchronization management control instruction is sent to perform data synchronization management control, wherein the preset bedside multi-modal data fusion safety interval represents an open interval formed by the preset bedside multi-modal data fusion safety threshold and the preset bedside multi-modal data fusion threshold; If the bedside multi-modal data fusion influence value is greater than or equal to the preset bedside multi-modal data fusion threshold, the data fusion influence evaluation result is recorded as data fusion out of control, and a data fusion influence out of control prompt is sent to the preset medical staff.

5. The multi-modal bedside emergency interactive management system based on multi-source data fusion according to claim 4, wherein, The specific steps of the image acquisition frame number adjustment are: A1, if the obtained average illumination intensity is less than or equal to the preset illumination intensity safety threshold, the result of harmonically averaging the corresponding bedside multi-modal data fusion influence quantity and illumination intensity influence quantity is taken as an image acquisition frame number reduction amount of the camera to adjust the next image acquisition frame number, wherein the average illumination intensity represents the average illumination condition in the monitoring period corresponding to the bedside multi-modal data fusion influence value, the bedside multi-modal data fusion influence quantity represents the ratio of the bedside multi-modal data fusion influence value to the preset bedside multi-modal data fusion threshold, and the illumination intensity influence quantity is the ratio of the average illumination intensity to the preset illumination intensity safety threshold; A2, if the obtained average illumination intensity is within the preset illumination intensity safety interval, no image acquisition frame number adjustment is performed, and the preset illumination intensity safety interval represents an open interval formed by the preset illumination intensity safety threshold and the preset illumination intensity threshold; A3, if the obtained average illumination intensity is greater than or equal to the preset illumination intensity threshold, the result of harmonically averaging the corresponding bedside multi-modal data fusion influence quantity and illumination intensity correction quantity and taking the floor is taken as an image acquisition frame number increase amount of the camera to adjust the next image acquisition frame number, and the illumination intensity correction quantity is the ratio of the average illumination intensity to the preset illumination intensity threshold; The specific steps of the time alignment optimization adjustment are: B1, if the voice-video time offset quantity is greater than or equal to the preset voice-video time offset quantity threshold, the result of harmonically averaging the corresponding bedside multi-modal data fusion influence quantity and voice-video time compensation quantity is taken as a time correction quantity to adjust the next voice-video time, and the voice-video time compensation quantity represents the ratio of the deviation degree of the voice-video time offset quantity from the preset voice-video time offset quantity threshold to the preset voice-video time standard value; B2, if the voice-video time offset is less than or equal to a preset voice-video time offset safety threshold, then the result of the harmonic mean of the corresponding bedside multi-modal data fusion influence quantity and the voice-video time correction quantity is taken as the time compensation quantity to adjust the next voice-video time, the voice-video time correction quantity represents the ratio of the deviation degree of the preset voice-video time offset safety threshold and the voice-video time offset to the voice-video time standard value; B3, if the voice-video time offset is within a preset voice-video time offset safety interval, no time alignment optimization adjustment is performed, the preset voice-video time offset safety interval represents an open interval formed by the preset voice-video time offset safety threshold and the preset voice-video time offset threshold.

6. The multi-modal bedside emergency interactive management system based on multi-source data fusion of claim 1, wherein, The interaction response effectiveness evaluation module includes an interaction response quantization parameter acquisition unit, an interaction response quantization parameter quantization unit, and an interaction response quantization threshold storage unit: The interaction response quantization parameter acquisition unit is configured to acquire interaction response quantization parameters in a bedside emergency interaction response process, specifically including bedside multi-modal data fusion influence values, interaction time delay deviations, interaction instruction generation time lengths, and interaction instruction execution response times. The interaction response quantization parameter quantization unit is configured to evaluate the effectiveness of the bedside emergency interaction response based on the acquired interaction response quantization parameters to obtain bedside interaction response effective values, and determine an interaction response evaluation result based on the obtained bedside interaction response effective values. The interaction response quantization threshold storage unit is configured to acquire interaction response quantization thresholds and interaction response quantization compensation factors from a constructed bedside emergency interaction database, the interaction response quantization thresholds including bedside multi-modal data fusion thresholds, interaction time delay deviation thresholds, interaction instruction generation time length thresholds, and interaction instruction execution response time thresholds, and the interaction response quantization compensation factors including bedside multi-modal data fusion compensation factors, interaction time delay deviation compensation factors, interaction instruction generation time length compensation factors, and interaction instruction execution response time compensation factors.

7. The multi-modal bedside emergency interactive management system based on multi-source data fusion according to claim 6, wherein, The specific steps for obtaining the bedside interaction response effective values are: After comparing the interaction response quantization thresholds with the interaction response quantization parameters, weighting is performed through the interaction response quantization compensation factors, and then coupling processing is performed to obtain the bedside interaction response effective values. The bedside interaction response effective values represent the effectiveness degree of the bedside emergency interaction response jointly measured by the interaction response quantization parameters.

8. The multi-modal bedside emergency interactive management system based on multi-source data fusion according to claim 7, characterized in that, The specific steps for determining whether to send an interaction response management control instruction based on the interaction response evaluation result are: The deviation degree of the bedside interaction response effective value and the average bedside interaction response effective value within a preset number of bedside interactions is recorded as a bedside interaction response effective deviation value; If the obtained bedside interaction response effective deviation value is not less than a preset bedside interaction response effective deviation threshold, the interaction response evaluation result is recorded as an interaction response qualified instruction is sent to a preset medical staff, otherwise the interaction response evaluation result is recorded as an interaction response unqualified instruction is sent to perform interaction response management control.

9. The multi-modal bedside emergency interactive management system based on multi-source data fusion according to claim 8, wherein, The specific execution steps of the response alarm time optimization are: If the interaction instruction execution duration is not less than the preset interaction instruction execution duration threshold, the result of harmonic mean of the bedside interaction response effective deviation influence quantity and the interaction instruction execution duration influence quantity is taken as the response alarm time increase quantity to compensate the response alarm time of the next time, the bedside interaction response effective deviation influence quantity represents the ratio of the bedside interaction response effective deviation value to the preset bedside interaction response effective deviation threshold, and the interaction instruction execution duration influence quantity represents the ratio of the interaction instruction execution duration to the preset interaction instruction execution duration threshold; If the interaction instruction execution duration is less than the preset interaction instruction execution duration threshold, the response alarm time optimization is not performed; The specific execution steps of the fusion window optimization are as follows: If the interaction response time is less than or equal to the preset interaction response time safety threshold, the fusion window optimization is not performed; If the interaction response time is within the preset interaction response time safety interval, the result of harmonic mean of the bedside interaction response effective deviation influence quantity and the interaction response time correction quantity is taken as the modal fusion window increment to adjust the modal fusion window of the next time, the interaction response time correction quantity represents the ratio of the interaction response time to the preset interaction response time safety threshold, and the preset interaction response time safety interval represents an open interval formed by the preset interaction response time safety threshold and the preset interaction response time threshold; If the interaction response time is greater than or equal to the preset interaction response time threshold, the result of harmonic mean of the bedside interaction response effective influence quantity and the interaction response time comparison quantity is taken as the modal fusion window reduction quantity to correct the modal fusion window of the next time, and the interaction response time comparison quantity represents the ratio of the interaction response time to the interaction response time threshold.

10. A multi-modal bedside emergency interactive management method based on multi-source data fusion, characterized in that, The method comprises the following steps: According to the obtained data fusion influence parameter, the influence of the data acquisition and processing process on the bedside multi-modal data fusion in the bedside emergency interaction process is evaluated to obtain a data fusion influence evaluation result; According to the data fusion influence evaluation result, it is determined whether to send a data synchronization management control instruction, if yes, data synchronization management control is performed, otherwise a data fusion influence out-of-control prompt is sent to the preset medical staff, the data synchronization management control represents image acquisition frame number adjustment of a camera for data acquisition in the bedside emergency interaction process and time alignment optimization adjustment of the processing process before bedside multi-modal data fusion; According to the obtained interaction response quantitative parameter, the response effectiveness of the bedside emergency interaction is evaluated to obtain an interaction response evaluation result; According to the interaction response evaluation result, it is determined whether to send an interaction response management control instruction, if yes, interaction response management control is performed, otherwise an interaction response qualified instruction is sent to the preset medical staff, the interaction response management control represents response alarm time optimization of the interaction response in the bedside emergency interaction process and fusion window optimization of the next fusion process.

Images