Man-machine interaction dynamic control method based on multi-modal data and edge computing device

Abstract

The embodiment of the invention provides a man-machine interaction dynamic control method based on multi-modal data and an edge computing device, and the method comprises the steps: synchronously collecting the multi-modal data, including physiological data and behavior data, of a user in a process of using an electronic device with a man-machine interaction interface; determining a user experience level based on the physiological data and the behavior data; determining adjustment information of a human-computer interaction interface of the electronic equipment based on the user experience level; and adjusting elements or an interaction mode of a human-computer interaction interface of the electronic equipment according to the adjustment information, so that a user interacts with the electronic equipment based on the adjusted human-computer interaction interface or interaction mode. According to the method, the interaction mode is dynamically adjusted based on the user experience level, accurate adaptation of the interaction mode and the use requirement of the user is achieved, and the user experience and use efficiency of man-machine interaction are remarkably improved.

Description

Technical Field

[0001] This application relates to the fields of human-computer interaction and user experience, and specifically to a dynamic control method for human-computer interaction based on multimodal data and an edge computing device. Background Technology

[0002] As electronic devices play an increasingly prominent role in diagnosis and treatment, the quality of their human-computer interaction directly impacts the work efficiency, cognitive load, and patient experience of healthcare workers. However, existing technologies either focus on single operational behavior data or independent risk management, lacking the integrated collection of data on the status of both healthcare workers and patients. They also fail to establish a real-time closed-loop mechanism of "monitoring-early warning-intervention," resulting in a disconnect between the interaction methods configured on electronic devices and the actual user experience. This makes it difficult to meet the core requirements of medical scenarios for safety, timeliness, and seamless human-computer interaction throughout the entire process. Summary of the Invention

[0003] In view of this, this application provides a human-computer interaction dynamic control method, device and edge computing device based on multimodal data, so as to solve the problem that the interaction mode of electronic devices in the prior art is difficult to accurately match user needs and affect the user experience.

[0004] In a first aspect, embodiments of this application provide a human-computer interaction dynamic control method based on multimodal data, including: Simultaneously collect multimodal data of users during the use of electronic devices with human-computer interaction interfaces, including physiological data and behavioral data; Based on the physiological data and the behavioral data, the current user experience level of the electronic device is determined; The adjustment information for the human-computer interaction interface of the electronic device is determined based on the user experience level; The elements or interaction methods of the human-computer interaction interface of the electronic device are adjusted according to the adjustment information, so that the user can interact with the electronic device based on the adjusted human-computer interaction interface or interaction methods.

[0005] In one optional embodiment, before synchronously collecting multimodal data of users using electronic devices with human-computer interaction interfaces, the method further includes: determining the evaluation index of the user experience level corresponding to the user attribute based on a pre-established correlation between user attributes and user experience level evaluation, wherein the evaluation index of the user experience level is different for different user attributes. Determining the user experience level of the electronic device based on the physiological data and the behavioral data includes: determining the current user experience level of the electronic device based on the physiological data, the behavioral data, and the evaluation index of the user experience level corresponding to the user attributes.

[0006] In one optional embodiment, determining the current user experience level of the electronic device based on the physiological data, the behavioral data, and an evaluation index for the user experience level corresponding to the user attributes includes: When the user's user attribute is medical personnel, first physiological data and first behavioral data related to cognitive load, fatigue state and focus state are obtained, and the user experience level of the electronic device is determined based on the first physiological data and the first behavioral data. When the user's user attribute is a patient, second physiological data and second behavioral data related to anxiety state and operational adaptation state are acquired, and the user experience level of the electronic device is determined based on the second physiological data and the second behavioral data.

[0007] In one optional embodiment, the synchronous collection of multimodal data during the user's use of an electronic device with a human-computer interaction interface includes: During the user's use of the electronic device, the user's electroencephalogram (EEG) data, electrocardiogram (ECG) data, electrodermal activity data, eye movement data, or electromyography (EMG) data are collected. During the user's interaction with the electronic device, the user's operation time or probability of operation error is collected.

[0008] In an optional embodiment, before synchronously collecting multimodal data of the user during the use of an electronic device with a human-computer interaction interface, the method further includes: During the use of the electronic device by the tester, multimodal test data and subjective scoring data of the tester are acquired; User experience level labels are determined based on the subjective rating data; Training data is constructed using the multimodal test data and the user experience level labels; The model is trained based on the training data to obtain an experience level determination model, which is used to determine the corresponding user experience level based on the physiological data and the behavioral data.

[0009] In one optional embodiment, determining adjustment information for the human-computer interaction interface of the electronic device based on the user experience level includes: When the user's attribute is medical personnel, the guidance intensity adjustment information of the electronic device is generated based on the user experience level; When the user's attribute is a patient, simplification adjustment information for the electronic device is generated based on the user experience level; The lower the user experience level, the higher the guidance intensity for patients and the greater the simplification for medical staff after the electronic device is adjusted.

[0010] In one optional embodiment, adjusting the elements or interaction methods of the human-computer interface of the electronic device according to the adjustment information includes one or more combinations of the following: The user interface has been simplified and adjusted, including: highlighting or enlarging key controls, and hiding or collapsing non-core function menus; The operation process has been simplified and adjusted, including: skipping unnecessary confirmation steps and merging multiple operations into a single shortcut command; Enhancing the guidance strength of the electronic device includes: outputting guiding voice prompts, displaying dynamic visual guidance, and issuing tactile alerts.

[0011] Secondly, embodiments of this application provide a human-computer interaction dynamic control device based on multimodal data, comprising: The data acquisition module is used to synchronously acquire multimodal data of users during the use of electronic devices with human-computer interaction interfaces. The multimodal data includes physiological data and behavioral data. The user experience level determination module is used to determine the current user experience level of the electronic device based on the physiological data and the behavioral data. An adjustment information determination module is used to determine adjustment information for the human-computer interaction interface of the electronic device based on the user experience level. The dynamic interaction control module is used to adjust the elements or interaction methods of the human-computer interaction interface of the electronic device according to the adjustment information, so that the user can interact with the electronic device based on the adjusted human-computer interaction interface or interaction methods.

[0012] Thirdly, embodiments of this application provide an edge computing device, including a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the method described in any of the first aspects above.

[0013] Fourthly, embodiments of this application provide a computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform the method described in any of the first aspects.

[0014] Fifthly, embodiments of this application provide a computer program product comprising executable instructions that, when executed on a computer, cause the computer to perform the method described in any of the first aspects.

[0015] The solution provided in this application synchronously collects multimodal data from users during the use of electronic devices with human-computer interaction interfaces, including physiological data and behavioral data. Based on the physiological and behavioral data, a user experience level is determined. Based on the user experience level, adjustment information for the human-computer interaction interface of the electronic device is determined. The elements or interaction methods of the human-computer interaction interface of the electronic device are adjusted according to the adjustment information, so that the user interacts with the electronic device based on the adjusted human-computer interaction interface or interaction method. This application dynamically adjusts the interaction method based on the user experience level, achieving precise adaptation between the interaction method and user needs, significantly improving the user experience and efficiency of human-computer interaction. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a human-computer interaction dynamic control method based on multimodal data, provided in an embodiment of this application; Figure 2 A schematic diagram illustrating another human-computer interaction dynamic control method based on multimodal data provided in this application embodiment; Figure 3 A schematic diagram of the structure of a human-computer interaction dynamic control device based on multimodal data provided in this application embodiment; Figure 4 This is a schematic diagram of the structure of an edge computing device provided in an embodiment of this application. Detailed Implementation

[0018] To better understand the technical solution of this application, the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0019] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0020] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0021] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0022] Figure 1 This is a flowchart illustrating a human-computer interaction dynamic control method based on multimodal data, provided as an embodiment of this application. This method can be applied to electronic devices. Figure 1 As shown, the method may include: Step 101: Synchronously collect multimodal data of the user during the use of electronic devices. The multimodal data includes physiological data and behavioral data.

[0023] Optional, physiological data may include: (1) EEG data: such as the inhibition rates of prefrontal theta waves, beta waves, and alpha waves. Specifically, increased theta wave activity in the prefrontal cortex can reflect disordered allocation of cognitive resources and inattention. For example, in MRI scans, patients with claustrophobia have increased theta wave power, which is positively correlated with subjective anxiety scores. Decreased alpha wave power indicates that the brain is transitioning from a relaxed state to an alert state. For example, during a patient being scanned by an electronic device, the alpha wave inhibition rate is directly related to discomfort. Increased beta wave activity reflects sympathetic arousal and muscle tension. For example, in MRI scans, increased beta wave power in patients is significantly correlated with prolonged scan time.

[0024] (2) Electrocardiogram data: Indicators such as high frequency (HF) and low frequency (LF) can assess the balance of the autonomic nervous system. For example, fatigue accumulation caused by continuous surgery is reflected in the characteristics of low HF index, and an increase in LF / HF can also reflect that the individual is currently in a state of high cognitive load or fatigue. On the other hand, an increase in HF component and a decrease in LF / HF ratio reflect that the medical staff are in a state of focus and their attention is relatively stable.

[0025] (3) Skin conductance data: such as Skin Conductance Level (SCL) and Skin Conductance Response (SCR). An increase in SCL and an increase in the amplitude of SCR both reflect a high level of cognitive load or fatigue. On the other hand, a stable SCL and a decrease in the amplitude of SCR indicate a state of relatively concentrated attention. In addition, the peak value of SCR is positively correlated with the task difficulty in the allocation of cognitive resources for complex operations (such as surgical instrument positioning).

[0026] (4) Eye movement data: such as pupil diameter, fixation point position, saccade speed and blink frequency, can be used to analyze the visual focus distribution of the operating interface (such as the MRI console) to obtain the rationality of the device interface layout; the increase in blink frequency and the decrease in pupil diameter can also reflect the individual's current eye muscle fatigue and decreased attention maintenance ability; the activity of pupil diameter can reflect the doctor's psychological load during surgery, which expands with the increase of cognitive activity.

[0027] (5) Electromyography data: mainly refers to the activity of muscles. For example, an increase in the root mean square (RMS) and a decrease in the median frequency (MF) of muscles related to medical procedures (such as the forearm flexors and deltoids in surgical procedures) can reflect an increased risk of fatigue during surgical procedures.

[0028] Behavioral data may include: button / touchscreen interaction data of electronic device operation panels, such as operation time, operation error probability, etc.

[0029] Step 102: Based on the physiological data and the behavioral data, determine the current user experience level of the electronic device.

[0030] Electronic devices can determine the evaluation indicators for user experience levels corresponding to user attributes based on pre-established correlations between user attributes and user experience level evaluations. Then, based on physiological data, behavioral data, and the evaluation indicators corresponding to user attributes, the current user experience level of the electronic device is determined. Different user attributes correspond to different evaluation indicators for user experience levels. User attributes indicate the user's identity, such as medical personnel or patients. User experience level indicates the degree of adaptation between the current interaction method of the electronic device and the user's own state, and can be determined through different evaluation indicators. The higher the degree of adaptation, the higher the user's operational efficiency, comfort, and ease of learning when using the electronic device. Optionally, when the user is a medical personnel, the evaluation indicators for the user experience level may include: cognitive load, fatigue state, and focus state; when the user is a patient, the evaluation indicators for the user experience level may include: anxiety state and operational adaptation state.

[0031] Different evaluation metrics are associated with different multimodal data, as shown in the following examples: (1) Cognitive load: High cognitive load = increased prefrontal θ / β wave power + high α wave inhibition rate + increased LF / HF ratio + increased SCL and SCR amplitude + pupil diameter dilation + increased probability of operational errors; Low cognitive load = normal or decreased prefrontal θ / β wave power + low α wave inhibition rate + decreased LF / HF ratio + stable skin conductance SCL and small SCR amplitude + stable pupil diameter + low probability of operational errors.

[0032] (2) Fatigue state: High fatigue state = excessive theta wave power + continuous decrease in HF component + significant increase in blink frequency + decrease in pupil diameter + increase in forearm electromyography RMS and decrease in MF + prolonged operation time or slowed response; Low fatigue state = normal power in all frequency bands of EEG + stable HF component + normal blink frequency + stable pupil diameter + normal RMS and MF values ​​of related electromyography + quick operation response.

[0033] (3) Focused state: High focus = reduced alpha wave power (indicating active brain processing of information) + increased HF component and decreased LF / HF ratio (active parasympathetic nervous system, stable mind and body) + stable SCL and small SCR amplitude (stable emotions) + fixation point is concentrated on key areas of the operation interface + low operation error rate; Distracted focus = abnormally increased alpha wave power (may indicate mental relaxation) + increased LF / HF ratio (active sympathetic nervous system) + appearance of irrelevant SCR + fixation point wandering, chaotic saccade path + increased operation error rate.

[0034] (4) Anxiety state: High anxiety state = increased theta wave power + increased beta wave power (muscle tension) + high alpha wave inhibition rate (high alertness) + increased LF / HF ratio + significantly increased SCL and SCR + increased blinking frequency and pupil diameter + increased frontalis muscle electromyography RMS; Low anxiety state = normal theta wave and beta wave power + normal alpha wave power (relaxation) + lower LF / HF ratio + stable SCL + normal pupil diameter and blinking frequency + low facial electromyography RMS value.

[0035] (5) Operational Adaptability: Proficient Operation = Short self-service device operation time + Extremely low probability of operation error + Eye-tracking fixation focused on key buttons, efficient path + Low levels of related physiological indicators (such as anxiety, cognitive load); Difficult Operation = Significantly prolonged operation time + High probability of operation error (such as multiple accidental touches) + Dispersed eye-tracking fixation, repeatedly moving between different controls + High levels of related physiological indicators (such as anxiety, cognitive load). The above is an exemplary description of the correlation between various evaluation indicators of user experience level and multimodal data. In other implementations, other correlations may be used, which are not limited here.

[0036] Optionally, based on the above correlation, in scenarios where the user is a medical worker, the electronic device can acquire first physiological data and first behavioral data related to cognitive load, fatigue state, and focus state, and determine the medical user experience level based on the first physiological data and first behavioral data; in scenarios where the user is a patient, the electronic device can acquire second physiological data and second behavioral data related to anxiety state and operational adaptation state, and determine the user experience level based on the second physiological data and second behavioral data.

[0037] Step 103: Determine the adjustment information for the human-computer interaction interface of the electronic device based on the user experience level.

[0038] Optionally, in scenarios where the user is a healthcare worker, the electronic device can generate guidance intensity adjustment information based on the user experience level; when the user is a patient, the electronic device can generate simplification adjustment information based on the user experience level. Specifically, in scenarios where patients use electronic devices, guidance intensity can be used to characterize the level of detail in which the electronic device guides the user through the operation process. For example, patients who are familiar with electronic devices are more suitable for using electronic devices with low guidance intensity, which can save time on prompts and guidance between operation steps and improve the patient's operation efficiency. On the other hand, patients who are unfamiliar with electronic devices are more suitable for electronic devices with high guidance intensity, which can guide the patient to complete the operation process through voice prompts and improve the patient's user experience.

[0039] In scenarios where healthcare workers use electronic devices, simplification primarily refers to the simplification of the electronic device's user interface. For example, when healthcare workers' cognitive load and fatigue levels gradually increase while their concentration decreases, electronic devices can employ a more simplified user interface, highlighting key controls or omitting unnecessary steps. Conversely, when healthcare workers' cognitive load and fatigue levels are low, but their concentration levels are high, electronic devices can use a less simplified or more basic user interface to display all of the device's functions.

[0040] In one alternative embodiment, the electronic device may default to using a user interface without simplification or providing no process guidance. When a low user experience level is detected, the simplification level of the user interface or the intensity of guidance is gradually adjusted. The lower the user experience level, the higher the intensity of guidance for patients and the higher the simplification level for healthcare workers after adjustment.

[0041] Step 104: Adjust the elements or interaction methods of the human-computer interaction interface of the electronic device according to the adjustment information, so that the user can interact with the electronic device based on the adjusted human-computer interaction interface or interaction methods.

[0042] In one optional embodiment, adjusting the elements or interaction methods of the human-computer interaction interface of the electronic device may include one or more of the following combinations: (1) simplifying the operation interface, including: highlighting or enlarging key controls, hiding or folding non-core function menus; (2) simplifying the operation process, including: skipping unnecessary confirmation steps, merging multi-step operations into a single shortcut command; (3) increasing the guidance intensity of the electronic device, including: outputting guiding voice prompts, displaying dynamic visual guidance, and issuing tactile warnings.

[0043] This application synchronously collects physiological data such as EEG and ECG from users while using electronic devices, along with behavioral data such as operation time and error probability. Combined with predefined mapping rules, it determines the user experience level of the device in real time based on the cognitive load, fatigue, and concentration of medical staff, or the anxiety and operational adaptation of patients. It then adjusts the interaction method accordingly, achieving precise and dynamic adaptation between the interaction method and the user's state. This effectively reduces the cognitive load and fatigue of medical staff, improves their operational efficiency and concentration, alleviates patients' anxiety, and improves their operational adaptability. It comprehensively enhances the usability and user experience of electronic devices, and also improves the safety and timeliness of medical scenarios.

[0044] In one optional embodiment, the electronic device is internally configured with an experience level determination model to implement mapping rules from multimodal data to user experience levels. The training process of the experience level determination model is described below using a remote operation interface of a surgical robot as an example. (Refer to...) Figure 2 The process includes: Step 201: Acquire the multimodal test data and subjective scoring data of the testers during the use of electronic devices.

[0045] Twenty randomly recruited doctors, with no experience restrictions, were selected as test subjects. In a simulated operating room environment, they were presented with different operating interfaces of a surgical robot and performed various tasks. During this process, multimodal test data and subjective rating data were collected in real time from the test subjects. Multimodal tests included: theta wave and beta wave power and alpha wave suppression rate; LF and HF components and the LF / HF ratio; mean SCL and SCR amplitude; pupil diameter, blink frequency, and fixation point distribution; RMS and MF of the forearm flexors; operation time, error probability, and response time of critical steps. After the test subjects completed their tasks, subjective rating data for the electronic device's operating interface was manually determined. This subjective rating data characterized the test subjects' operational experience; a better experience resulted in a higher score.

[0046] Step 202: Determine user experience level labels based on subjective rating data.

[0047] Based on subjective rating data, scores below 25%, 25%-50%, and above 75% are used to distinguish between "low availability," "medium availability," and "high availability."

[0048] Step 203: Construct training data using multimodal test data and user experience level labels.

[0049] The input features are standardized to eliminate differences in units. Feature selection algorithms (such as recursive feature elimination) are used to screen core features that are strongly correlated with user experience level. Training sets and test sets are then divided for training.

[0050] Step 204: Train the experience level determination model based on the training data to construct mapping rules.

[0051] Using multimodal test data as input and user experience levels as labels, the model is trained to construct a mapping relationship between the two. For example, if a tester consistently detects theta wave power exceeding a threshold (e.g., 10 μV²) and blinking frequency increasing to 15 times / minute during interface operation, the user experience level of the interface can be determined to be "medium usability" based on the tester's subjective rating data. The mapping relationship between the two can then be constructed through training. When electronic devices are deployed in practical applications, if a medical worker is detected exhibiting "theta wave power consistently exceeding a threshold (e.g., 10 μV²) and blinking frequency increasing to 15 times / minute," the current interface can be determined to be "medium usability," and the electronic device can adaptively adjust its interaction methods to simplify the interface.

[0052] In this embodiment, the experience level determination model collects multimodal test data (including physiological and behavioral data) and subjective rating data from testers using electronic devices. After feature standardization and core feature selection, it is trained based on the user experience level label corresponding to the subjective rating. This constructs a precise mapping rule between multimodal data and device user experience level, enabling electronic devices to quickly determine user experience level and dynamically adjust interaction methods based on real-time multimodal user data in practical applications. This ensures that usability assessment matches the user's real experience and improves the accuracy and real-time performance of interaction method adaptation. It effectively supports electronic devices in achieving a closed loop of "state perception - level determination - strategy optimization," further enhancing the usability of electronic devices and the safety and efficiency of medical scenarios.

[0053] The human-computer interaction dynamic control method based on multimodal data in this application can be applied to scenarios such as "medical staff using electronic devices", "patients using electronic devices", and "medical staff and patients using electronic devices simultaneously". The following is an exemplary description of each scenario based on specific embodiments.

[0054] I. Doctors remotely operate surgical robots to perform surgery (medical scenario) (1) Task goal setting: Doctors complete three core steps, namely “fracture reduction and positioning, screw implantation planning and surgical execution confirmation”, through a remote interface. The electronic device evaluates the user experience level of the interface throughout the process and adjusts the interaction method in real time.

[0055] (2) Data synchronous collection and processing: Physiological and behavioral data of doctors are collected in real time during the operation, as shown in Table 1.

[0056] Table 1 The data preprocessing process may include: achieving data time alignment through Precision Time Protocol (PTP), with a final time error ≤0.8ms; removing power frequency interference based on bandpass filtering, and eliminating blink artifacts using Independent Component Analysis (ICA) algorithm (components with amplitude >±100μV are directly removed); separating tonic (basal skin conductance) and phasic (emotion-related fluctuations) components based on low-pass filtering and 1s window-length moving average noise reduction; and segmenting fixation points and saccades based on filters, retaining valid fixation data (fixation duration ≥100ms).

[0057] (3) User experience level determination: Electronic devices can be determined from three dimensions: cognitive load, fatigue state and focus state, as shown in Table 2.

[0058] Table 2 Based on high cognitive load, mild fatigue, and low concentration, electronic devices can determine the current operating interface as "medium usability." The main problems include: key function buttons are scattered (leading to confusion of gaze points); step jumps require 3 confirmations (leading to increased cognitive load).

[0059] (4) Real-time feedback adjustment: The electronic device can highlight or enlarge key controls, hide or collapse non-core function menus, and at the same time provide a voice reminder: "Increased cognitive load detected, please focus on key operations." If the doctor's state becomes high cognitive load + severe fatigue + low concentration as the surgery continues, the electronic device can determine that the current operating interface is "low availability" and then adjust the interaction method, including: a voice reminder "suggest a short rest", automatic adjustment of device parameters (such as reducing operating sensitivity or enabling auxiliary mode) and optimization of the layout of device operating parts (such as adjusting the position of the handle) to reduce the fatigue accumulation caused by continuous muscle exertion.

[0060] In this embodiment, the user experience level of the electronic device is determined from three dimensions: cognitive load, fatigue state, and focus state. Real-time feedback is provided for medium / low availability level output interface optimization, voice reminders, device parameter and operation component layout adjustment, etc., which effectively alleviates doctors' cognitive load and fatigue, improves operational focus, reduces the risk of surgical operation errors, and significantly improves the availability and operational safety of electronic devices in surgical scenarios.

[0061] II. Patients measuring their own blood glucose using a blood glucose meter (patient scenario) (1) Task objectives: The patient independently completes the entire process of “powering on - blood collection - test strip insertion - result reading”, and the blood glucose meter adjusts the interaction method in real time based on the patient’s experience.

[0062] (2) Data synchronous collection and processing: During the patient's use, the patient's physiological data and behavioral data are collected in real time. Physiological data can be referred to Table 1 above. Behavioral data may include: power-on time, number of incorrect test strip insertions, and waiting time after blood collection, etc., which are obtained through the built-in sensor of the blood glucose meter.

[0063] (3) User experience level determination: The blood glucose meter can be determined from two dimensions: “anxiety state” and “operational adaptation state”, as shown in Table 3.

[0064] Table 3 Based on the combination of high anxiety and operational difficulty, the blood glucose meter's user experience level can be determined as "low availability".

[0065] (4) Real-time feedback adjustment: Increase the intensity of patient-oriented guidance. For example, when the patient's SCL increases by more than 20% and the blinking frequency is more than 25 times / minute, the blood glucose meter screen can pop up a voice prompt: "Please relax. When inserting the test strip, the green side should be facing up. Align it with the slot and push it in gently." At the same time, a dynamic diagram is displayed.

[0066] In this embodiment, the blood glucose meter collects physiological and behavioral data from the patient during the blood glucose measurement process in real time. After determining the device's low availability based on anxiety and operational adaptation, it enhances guidance through voice prompts and dynamic illustrations, effectively alleviating patient anxiety, reducing operational difficulty, and significantly improving the usability of the blood glucose meter and the patient's user experience.

[0067] III. MRI Equipment Testing Scenarios (Simultaneous Medical Staff and Patients) (1) Task objectives: Doctors operate MRI equipment and patients undergo scanning. The status of both parties needs to be monitored simultaneously.

[0068] (2) Data synchronous collection and processing: Physiological and behavioral data of doctors and patients are collected synchronously, with a focus on doctors' cognitive load and patients' anxiety status.

[0069] (3) User experience level determination: Due to the complex design of the device's operating interface, doctors exhibited a state of "high cognitive load" and patients exhibited a state of "high anxiety" due to the closed scanning chamber and lack of real-time communication channels. The MRI device was ultimately determined to be "medium usability".

[0070] (4) Real-time feedback adjustment: The MRI equipment can optimize the doctor-oriented operating interface (propose key controls and merge redundant options), and at the same time provide voice prompts and real-time animation guidance (such as scanning progress prompts) in the scanning chamber to alleviate patient anxiety.

[0071] Figure 3 A schematic diagram of a human-computer interaction dynamic control device based on multimodal data provided in this application embodiment is shown below. Figure 3 As shown, the device may include: The acquisition module 310 is used to synchronously acquire multimodal data of the user during the use of electronic devices with human-computer interaction interfaces. The multimodal data includes physiological data and behavioral data.

[0072] User experience level determination module 320 is used to determine the current user experience level of the electronic device based on physiological data and the behavioral data.

[0073] The adjustment information determination module 330 is used to determine the adjustment information of the human-computer interaction interface of the electronic device based on the user experience level.

[0074] The dynamic interactive control module 340 is used to adjust the elements or interaction methods of the human-computer interaction interface of the electronic device according to the adjustment information, so that the user can interact with the electronic device based on the adjusted human-computer interaction interface or interaction methods.

[0075] The electronic device of this application can also be a device deployed at the edge (such as an edge computing device). The edge computing device includes a memory and a processor. The memory stores program instructions for implementing the functions of each module / unit in the above embodiments. The processor can implement the above-mentioned human-computer interaction dynamic control method based on multimodal data by calling the corresponding program instructions. Figure 4 This is a schematic diagram of the structure of an edge computing device provided in an embodiment of this application. The edge computing device 400 may include a processor 401, a memory 402, and a communication unit 403. These components communicate through one or more buses. Those skilled in the art will understand that the structure of the edge computing device shown in the figure does not constitute a limitation on the embodiment of this application. It can be a bus topology or a star topology, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0076] The communication unit 403 is used to establish a communication channel, enabling the edge computing device to communicate with other devices. It receives user data from other devices or sends user data to other devices.

[0077] The processor 401 serves as the control center of the edge computing device, connecting various parts of the device via interfaces and lines. It executes software programs, instructions, and / or modules stored in the memory 402, and accesses data stored in the memory to perform various functions and / or process data. The processor can be composed of integrated circuits (ICs), such as a single packaged IC or multiple packaged ICs with the same or different functions connected together. For example, the processor 401 may consist only of a central processing unit (CPU). In this embodiment, the CPU may have a single processing core or include multiple processing cores.

[0078] The memory 402 is used to store the execution instructions of the processor 401. The memory 402 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0079] When the execution instructions in memory 402 are executed by processor 401, the edge computing device 400 is able to perform some or all of the steps in the above embodiments.

[0080] In a specific implementation, this application also provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, it may include some or all of the steps of the various embodiments of the human-computer interaction dynamic control method based on multimodal data provided in this application. The storage medium may be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0081] In a specific implementation, this application also provides a computer program product, wherein the computer program product includes executable instructions, which, when executed on a computer, cause the computer to perform some or all of the steps in various embodiments of the human-computer interaction dynamic control method based on multimodal data provided in this application.

[0082] This application also provides a non-transitory computer-readable storage medium that stores computer instructions, which cause the computer to execute the human-computer interaction dynamic control method based on multimodal data provided in this application.

[0083] The aforementioned non-transitory computer-readable storage medium may be any combination of one or more computer-readable media. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in connection with an instruction execution system, apparatus, or device.

[0084] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0085] The program code contained on a computer-readable medium may be transmitted using any suitable medium, including—but not limited to—wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0086] Those skilled in the art will clearly understand that the techniques in the embodiments of this application can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application or some parts of the embodiments.

[0087] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the device embodiments and terminal embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

Claims

1. A human-computer interaction dynamic control method based on multimodal data, characterized in that, include: Simultaneously collect multimodal data of users during the use of electronic devices with human-computer interaction interfaces, including physiological data and behavioral data; Based on the physiological data and the behavioral data, the current user experience level of the electronic device is determined; The adjustment information for the human-computer interaction interface of the electronic device is determined based on the user experience level; The elements or interaction methods of the human-computer interaction interface of the electronic device are adjusted according to the adjustment information, so that the user can interact with the electronic device based on the adjusted human-computer interaction interface or interaction methods.

2. The method according to claim 1, characterized in that, Before synchronously collecting multimodal data of users using electronic devices with human-computer interaction interfaces, the method further includes: determining the evaluation index of the user experience level corresponding to the user attribute based on the pre-established correlation between user attributes and user experience level evaluation, wherein the evaluation index of the user experience level is different for different user attributes. Determining the user experience level of the electronic device based on the physiological data and the behavioral data includes: determining the current user experience level of the electronic device based on the physiological data, the behavioral data, and the evaluation index of the user experience level corresponding to the user attributes.

3. The method according to claim 2, characterized in that, Based on the physiological data, the behavioral data, and the evaluation indicators for user experience levels corresponding to the user attributes, the current user experience level of the electronic device is determined, including: When the user's user attribute is medical personnel, first physiological data and first behavioral data related to cognitive load, fatigue state and focus state are obtained, and the user experience level of the electronic device is determined based on the first physiological data and the first behavioral data. When the user's user attribute is a patient, second physiological data and second behavioral data related to anxiety state and operational adaptation state are acquired, and the user experience level of the electronic device is determined based on the second physiological data and the second behavioral data.

4. The method according to claim 1, characterized in that, The synchronous collection of multimodal data from users during the use of electronic devices with human-computer interaction interfaces includes: During the user's use of the electronic device, the user's electroencephalogram (EEG) data, electrocardiogram (ECG) data, electrodermal activity data, eye movement data, or electromyography (EMG) data are collected. During the user's interaction with the electronic device, the user's operation time or probability of operation error is collected.

5. The method according to claim 1, characterized in that, Before synchronously collecting multimodal data of users during the use of electronic devices with human-computer interaction interfaces, the method further includes: During the use of the electronic device by the tester, multimodal test data and subjective scoring data of the tester are acquired; User experience level labels are determined based on the subjective rating data; Training data is constructed using the multimodal test data and the user experience level labels; The model is trained based on the training data to obtain an experience level determination model, which is used to determine the corresponding user experience level based on the physiological data and the behavioral data.

6. The method according to claim 1, characterized in that, The adjustment information for the human-computer interaction interface of the electronic device is determined based on the user experience level, including: When the user's attribute is medical personnel, the guidance intensity adjustment information of the electronic device is generated based on the user experience level; When the user's attribute is a patient, simplification adjustment information for the electronic device is generated based on the user experience level; The lower the user experience level, the higher the guidance intensity for patients and the greater the simplification for medical staff after the electronic device is adjusted.

7. The method according to claim 1, characterized in that, Adjusting the elements or interaction methods of the human-computer interaction interface of the electronic device according to the adjustment information includes one or more of the following combinations: The user interface has been simplified and adjusted, including: highlighting or enlarging key controls, and hiding or collapsing non-core function menus; The operation process has been simplified and adjusted, including: skipping unnecessary confirmation steps and merging multiple operations into a single shortcut command; Enhancing the guidance strength of the electronic device includes: outputting guiding voice prompts, displaying dynamic visual guidance, and issuing tactile alerts.

8. A human-computer interaction dynamic control device based on multimodal data, characterized in that, include: The data acquisition module is used to synchronously acquire multimodal data of users during the use of electronic devices with human-computer interaction interfaces. The multimodal data includes physiological data and behavioral data. The user experience level determination module is used to determine the current user experience level of the electronic device based on the physiological data and the behavioral data. An adjustment information determination module is used to determine adjustment information for the human-computer interaction interface of the electronic device based on the user experience level. The dynamic interaction control module is used to adjust the elements or interaction methods of the human-computer interaction interface of the electronic device according to the adjustment information, so that the user can interact with the electronic device based on the adjusted human-computer interaction interface or interaction methods.

9. An edge computing device, characterized in that, The device includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the edge computing device performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 7.

Images