Device interaction control method and device, computer device, and storage medium

Abstract

The application relates to a device interaction control method and device, a computer device and a storage medium. The method comprises the following steps: acquiring face information and operation data when a user operates a device, determining user emotion information according to the face information, and determining operation fluency information according to the operation data. Based on the confirmation of the emotion dimension and the fluency dimension information, it is comprehensively judged whether the state of the user when operating the device is abnormal. If it is determined that the state of the user is abnormal, a preset interaction action is executed based on the device, the attention of the user when operating the device is improved through interaction with the user, the emotion of the user is improved, the problem of misoperation caused by user fatigue or low emotion and other reasons is avoided, and the reliability of the operation result of the device is improved.

Description

Technical Field

[0001] This application relates to the field of interactive technology, and in particular to a device interactive control method, apparatus, computer device, and storage medium. Background Technology

[0002] The emergence of medical imaging equipment has provided users with more visualized images of tissues and organs. For example, ultrasound imaging equipment uses ultrasound beams to scan the human body and obtains images of internal organs by receiving and processing the reflected signals.

[0003] During the implementation process, the applicant found that current ultrasound imaging equipment focuses on improving the user's operational workflow, while neglecting the problem of fatigue operation caused by the high demand for ultrasound imaging equipment. Summary of the Invention

[0004] Therefore, it is necessary to provide a device interaction control method, apparatus, computer equipment, or storage medium that can effectively prevent users from malfunctioning the device due to fatigue or other reasons, in order to address the aforementioned technical problems.

[0005] In a first aspect, this application provides a device interaction control method, including:

[0006] Acquire facial information and operation data of users when operating the device;

[0007] Determine the user's emotional information based on facial data;

[0008] Based on the operation data, determine the operation smoothness information;

[0009] If a user's state is deemed abnormal based on their emotional state and operational fluency information, a preset interactive action is executed based on the device.

[0010] In one embodiment, user emotion information includes an emotion value, and operation fluency information includes a fluency score. The step of determining an abnormal user state based on the user emotion information and operation fluency information includes:

[0011] The total score is determined by weighting the emotion value and fluency score.

[0012] When the total score is lower than the first threshold, the user's status is determined to be abnormal.

[0013] In one embodiment, determining user emotion information based on facial information includes:

[0014] The emotion value is determined based on facial information and a pre-set emotion model.

[0015] In one embodiment, determining user emotion information based on facial information further includes:

[0016] Based on facial information and a pre-defined emotion model, the confidence level of each emotion value is determined.

[0017] In one embodiment, determining operation smoothness information based on operation data includes:

[0018] The smoothness score is determined based on the operation data and the preset smoothness scoring model.

[0019] In one embodiment, the user's emotion information includes an emotion value, and based on the device, a preset interactive action is performed, including:

[0020] The model is determined based on the emotion value and the interaction action, and the preset interaction action corresponding to the emotion value is determined; wherein, the interaction action determination model is used to represent the mapping relationship between the emotion value and the preset interaction action.

[0021] The device executes preset interactive actions corresponding to the emotion value.

[0022] In one embodiment, a weighted calculation is performed on the emotion value and fluency score to determine the total score, including:

[0023] If a user is determined to be in a negative emotional state based on their emotion score, the emotion score and fluency score are weighted and calculated to determine the total score.

[0024] In one embodiment, the device performs a preset interactive action, including:

[0025] When the device is idle, perform preset interactive actions based on the device.

[0026] In one embodiment, the preset interactive action includes at least one of the following: voice prompt, displaying prompt content, displaying a preset game interface, or raising the device.

[0027] In one embodiment, the user's emotional information includes an emotional value, and the method further includes:

[0028] If the device is detected to be switching to working mode or the emotional value rises to the second threshold, the preset interactive action will be stopped.

[0029] Secondly, a device interaction control apparatus is provided, comprising:

[0030] The data acquisition module is used to acquire facial information and operation data of the user when operating the device;

[0031] The emotion information determination module is used to determine the user's emotion information based on facial information;

[0032] The smoothness information determination module is used to determine the smoothness information of the operation based on the operation data;

[0033] The interaction execution module is used to execute preset interactive actions based on the device when the user's state is determined to be abnormal based on the user's emotional information and operation fluency information.

[0034] Thirdly, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0035] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.

[0036] The aforementioned device interaction control method, apparatus, computer equipment, and storage medium have at least the following beneficial effects:

[0037] By acquiring facial information and operational data from users when operating the device, and determining user emotional information based on facial information, and operational fluency information based on operational data, the system comprehensively judges whether the user's state during device operation is abnormal based on the confirmation of emotional and fluency information. If the user's state is determined to be abnormal, the system executes preset interactive actions based on the device. Through interaction with the user, the system aims to improve the user's attention during device operation, improve the user's mood, and avoid misoperation problems caused by user fatigue or low mood, thereby improving the reliability of device operation results. Attached Figure Description

[0038] Figure 1 This is an application environment diagram of the device interaction control method in one embodiment;

[0039] Figure 2 This is one of the flowcharts illustrating a device interaction control method in one embodiment;

[0040] Figure 3 This is a schematic diagram illustrating the display of facial information in one embodiment;

[0041] Figure 4 This is a second flowchart illustrating a device interaction control method in one embodiment;

[0042] Figure 5 This is the third flowchart of a device interaction control method in one embodiment;

[0043] Figure 6 This is the fourth flowchart of a device interaction control method in one embodiment;

[0044] Figure 7 This is the fifth flowchart illustrating the device interaction control method in one embodiment;

[0045] Figure 8 This is the sixth flowchart of a device interaction control method in one embodiment;

[0046] Figure 9 This is a schematic diagram of the device interaction control device in one embodiment;

[0047] Figure 10 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0049] Due to the high demand for ultrasound imaging equipment, doctors often sit in front of it for extended periods, leading to fatigue and decreased dexterity. This increases the risk of operational errors, affecting the reliability of imaging results. To address this issue, this application provides an interactive control method for such frequently used equipment.

[0050] The device interaction control method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, computer device 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another network server. Server 104 can train various models and send the trained models to computer device 102 for use.

[0051] Computer device 102 can acquire the user's facial information and operation data in real time, and then determine the user's emotional information and the smoothness of operation. Based on these two factors, it comprehensively judges whether the user's state is abnormal. If so, computer device 102 actively controls the user's device to perform preset interactive actions, such as issuing prompts on the screen or interacting with the user in the form of a mini-game, to improve the user's attention and avoid misoperation caused by fatigue or other reasons. Computer device 102 can be, but is not limited to, controllers for various medical devices, personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. Medical devices can be various medical imaging equipment such as ultrasound imaging equipment. IoT devices can be smart vehicle devices, etc. Portable wearable devices can be head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster composed of multiple servers.

[0052] In one embodiment, such as Figure 2 As shown, a device interaction control method is provided, which is applied to... Figure 1 Taking computer device 102 as an example, the following steps are included:

[0053] S202, Acquire facial information and operation data of the user operating the device. The acquisition of facial information and operation data can be real-time to enable real-time detection of the user's mental state. Facial information refers to information that can characterize the user's emotions, such as facial point cloud data and color information of various facial features. Acquiring facial information of the user operating the device can be achieved through a camera device set on the device screen. For example, for ultrasound imaging devices, users often operate while facing the screen; based on this characteristic, facial information can be acquired through a depth camera set in front of the screen. The executing entity acquires facial information by communicating with the camera device. In one embodiment, it may further include: determining that the user is operating the device when a device operation command is detected. For example, for ultrasound imaging devices, it can be determined that the user is performing a scanning imaging operation when a scan sequence input command is detected. At this time, facial information and operation data are acquired. Alternatively, it can be determined that the user is operating the device when a designated interface is detected, for example, when entering a scan sequence selection interface.

[0054] S204. Determine user emotion information based on facial information. User emotion information refers to information that can represent various emotional types such as happiness, sadness, anger, fatigue, and distress. There are many ways to determine user emotion information based on facial information, which will not be exhaustive here. For example, when the facial information is a facial image, image features can be extracted from the facial information, geometric feature quantities can be determined based on the feature vector, and the user's emotion can be judged based on the geometric feature quantities.

[0055] Specifically, the size, position, and distance of facial features such as the iris, nose, and corners of the mouth can be determined first. Then, the geometric features of these features are calculated, forming a feature vector describing the user's facial image. Finally, the user's emotion is determined by comparing this feature vector with a preset vector. For example, ... Figure 3 As shown, if the angle θ2 between the feature vector β, which represents the upward angle of the corners of the mouth, and the reference direction γ is less than the angle θ1 between the reference feature vector α and the reference direction γ, it indicates that the user is smiling, and the user's emotion is determined to be happy. Here, the reference feature vector α is the standard reference vector representing the user's smile. When the user's emotion is determined to be happy, a higher emotion value can be assigned. For example, if the highest emotion value is 100, then the user's emotion value can be determined to be 80.

[0056] For example, based on the feature vector of the corners of the mouth, it can be determined whether the corners of the user's mouth are drooping. If they are drooping, the user can be judged to be in a sad mood. When a user is judged to be in a sad mood, a lower mood value can be assigned. For example, if the maximum mood value is 100, the user's mood value can be determined to be 10. Another example is determining the direction of skin texture on the forehead to see if there are "figure-eight" or "vertical" lines. If they are present, the user is judged to be in a sad mood, and the mood value can be determined to be 11. There can be a correspondence between user mood values ​​and mood types. One mood type can correspond to a range of mood values. For example, if the maximum mood value is 100, mood values ​​between 10 and 20 can all correspond to "sadness". Generally speaking, the higher the mood value, the better the user's mental state.

[0057] like Figure 3 As shown, the distance D between the upper and lower eyelids can also be identified based on facial information. If this distance D is consistently lower than a preset distance D0 within a first preset time period, it indicates that the eyelids are not moving closer due to blinking, but rather due to user fatigue or other reasons causing a lack of concentration. In this case, user emotional information can be determined. For example, if the user's emotional information includes an emotional value, a distance D less than the preset distance D0 indicates that the user is in a poor mental state and requires attention. The emotional value can be determined based on the difference between distance D and the preset distance D0. The emotional value can be negatively correlated with the difference between the preset distance D0 and distance D. The larger the difference between D0 and D, the smaller the determined emotional value.

[0058] It should be emphasized that the specific examples of emotion values ​​provided here are intended to help those skilled in the art understand the implementation of the technical solution of this application, but do not limit the actual scope of protection of this application. The maximum value of the emotion value and the correspondence between the emotion value and the facial information can be configured according to the actual use scenario of the device.

[0059] S206, Determine the smoothness of operation information based on the operation data.

[0060] Operational data refers to the actions performed by the device in response to user actions. This data can be obtained by acquiring the device's input and output data and executed code. For example, when the device is an ultrasound imaging device, the operational data can be the input and output data of each node in the scanning workflow, as well as the data input and output times. Based on this data, the operation time of each node in the scanning workflow can be determined. Based on the operation time interval between two adjacent scanning workflow nodes, or the operation time interval between two nodes separated by multiple workflow nodes, and the difference between these time intervals and the pre-collected average user operation time interval, smoothness information is determined. The greater the actual time interval exceeds the average operation time interval, the worse the user's operation smoothness. Operational smoothness information reflecting this situation is determined. For example, operational smoothness information can include a smoothness score; the worse the operation smoothness, the lower the smoothness score, and vice versa. Of course, operational smoothness information can include smoothness scores, as well as the operation times of scanning workflow nodes, etc. Any data that can characterize the smoothness of the user's operation of the device falls within the scope of protection of this application.

[0061] Operational data can also refer to user action data when operating the device, which can be acquired through cameras, etc. For example, operational data may include the time it takes for a user to complete a single scanning and imaging operation. If the actual scanning and imaging operation time is longer than a reference operation time (e.g., the average operation time), it can be determined that the user's actual triggering of the device action exhibits a noticeable sluggishness, i.e., poor operational smoothness. In this case, operational smoothness information can be determined based on the difference between the actual scanning and imaging operation time and the reference operation time to characterize this phenomenon. Operational smoothness information may include a smoothness score. When the actual scanning and imaging operation time is longer than the reference operation time, a smoothness score can be determined based on the difference between the actual scanning and imaging operation time and the reference operation time. The smoothness score is inversely correlated with this difference. The larger the difference, the lower the smoothness score; conversely, the smaller the difference, the higher the smoothness score.

[0062] S208, when determining an abnormal user state based on user emotion information and operation fluency information, executes a preset interactive action based on the device. An abnormal user state refers to a user's poor mental state while operating the device, potentially leading to erroneous operation. The preset interactive action can be a pre-configured action, and can be one or more actions, such as, but not limited to, voice prompts, displaying prompt content, showing a preset game interface, or raising / lowering the device. Two or more preset interactive actions can be executed simultaneously to improve the reminder effect, improve user mood, and alleviate fatigue, thereby increasing user attention while operating the device. Executing preset interactive actions based on the device can involve displaying prompt content on the device screen and / or showing a preset game interface, using text, animation, or game interaction to improve user mood and increase user attention. It can also involve broadcasting prompts based on the device's voice module to remind the user to pay attention. Alternatively, it can involve raising the device's height based on its mechanical structure, such as raising the display screen, allowing the user to stand while viewing and operating the device, alleviating mental fatigue caused by prolonged sitting in front of the device, thereby increasing user attention while operating the device.

[0063] Specifically, the device interaction control method provided in this application acquires the user's facial information and operation data when operating the device. Based on the facial information, it determines the user's emotional information, and based on the operation data, it determines the operation fluency information. Based on the confirmation of the emotional and fluency dimensions, it comprehensively judges the user's mental state when operating the device. If the mental state is determined to be poor, the device executes a preset interactive action. Through interaction with the user, it improves the user's attention when operating the device, avoids improper device operation caused by user fatigue or low mood, and thus improves the reliability of the device operation results.

[0064] In one embodiment, user emotion information includes an emotion value, and operation smoothness information includes a smoothness score. A higher emotion value indicates a better mental state, while a lower emotion value indicates a worse mental state. A higher smoothness score indicates smoother user operation of the device, and vice versa. The steps for determining abnormal user status based on user emotion information and operation smoothness information are as follows: Figure 4 As shown, it includes:

[0065] S402, a weighted calculation is performed on the emotion value and fluency score to determine the total score. Relying solely on emotion recognition results to determine whether there is a risk of abnormal device operation by the user is prone to error. Therefore, the user's emotion value and fluency score during device operation are considered together to determine whether the user's device operation is abnormal. Specifically, the emotion value and fluency score can be input into the total score determination model K1*A + K2*B = G to determine the total score G, where K1 is the first weight of the emotion value A and K2 is the second weight of the fluency score B.

[0066] The model determining the first weight, second weight, and total score can be trained using pre-collected sample data. Specifically, this can be achieved through the following steps:

[0067] The sentiment value and fluency score in the sample data are input into a pre-defined first neural network model to obtain the predicted total score; the first neural network model is a model that represents the mapping relationship between the sentiment value and fluency score and the predicted total score.

[0068] Based on the difference between the actual score and the predicted total score, adjust the first and second weights until the termination condition is met; the actual score refers to the score accepted by the user. The actual score can be pre-obtained. The termination condition can be reaching the maximum number of training iterations, or, for the sample data, if the difference between the actual score and the predicted total score of a preset proportion of the sample data falls within the allowable error range. The preset proportion can be a value between 90% and 100%, and the user can determine the accuracy requirements based on the application scenario and configure this preset proportion.

[0069] The trained first neural network model is used as the model for determining the total score. The trained model for determining the total score can be stored in server 104 for use by computer device 102, or it can be stored in the memory of computer device 102.

[0070] The actual score can also be obtained by:

[0071] The predicted total score is displayed based on the device's screen.

[0072] In response to the user's confirmation action on the predicted total score, the predicted total score is determined to be the actual score; the confirmation action can be a click on the "Confirm" control displayed on the screen or a voice confirmation, etc., which will not be listed here.

[0073] In response to the user's negative action regarding the predicted total score, display K1*A and K2*B;

[0074] In response to the user's adjustment of the target parameters, the actual score is determined. The target parameters include at least one of K1, K2, A, B, K1*A, and K2*B, and the actual score is the adjusted K1*A + K2*B.

[0075] The sample data may also include user action data regarding adjustments to the target parameters, including both upward and downward adjustments. In this case, the process of adjusting the first and second weights based on the difference between the actual score and the predicted total score can be performed by adjusting the first and second weights based on the difference between the actual score and the predicted total score, as well as the action data. The larger the difference, the greater the adjustment magnitude, and the greater the adjustment amount reflected by the action data, resulting in a larger adjustment magnitude for the first and second weights.

[0076] S404, when the total score is lower than the first threshold, the user's status is determined to be abnormal. The first threshold can also be a suitable value selected based on pre-experimentation. For example, in the pre-experimentation, for the above sample data, when the total score is lower than the selected threshold, the results of the determined user status abnormality are consistent with the actual situation, so the threshold can be determined as the first threshold that meets the requirements.

[0077] The device interaction control method provided in this application improves the reliability of the abnormal state judgment result by weighting the emotion value and fluency score and using the relationship between the weighted calculation result and a first threshold. This improves the reliability of the abnormal state judgment result. On the one hand, it is beneficial to execute preset interaction actions when the user's state is abnormal, so as to avoid the negative impact caused by user misoperation. On the other hand, it can also avoid misjudgment when judging the user's device operation state solely from the emotion aspect, thereby improving the effectiveness of the preset interaction action execution.

[0078] In one embodiment, to avoid false detections, step S404, which determines that the user's state is abnormal when the total score is lower than a first threshold, includes:

[0079] If the total score is lower than the first threshold and continues for a preset duration, the user's status is determined to be abnormal.

[0080] A longer preset duration increases the reliability of user status anomaly detection. However, an excessively long preset duration may prevent the device from executing preset interactive actions in a timely manner, increasing the risk of user misoperation. Therefore, the preset duration can be set based on the actual usage needs of the application scenario.

[0081] In one embodiment, to avoid false detections, step S404, which determines that the user's state is abnormal when the total score is lower than a first threshold, includes:

[0082] If the number of times the total score is lower than the first threshold within a preset time period exceeds a preset number, the user's status is determined to be abnormal.

[0083] The preset number of attempts and preset time periods can be adaptively set based on the needs of the application scenario. For example, the preset time period can be the time period from the start to the end of the scanning imaging. If the total score is detected to be lower than the first threshold multiple times within the preset time period, and the frequency of this occurrence is higher than the preset number of attempts, it indicates that the user may be misoperating the device due to poor mental state. In this case, by determining that the user's state is abnormal, the steps described in the above embodiment, which determine that the user's state is abnormal based on user emotion information and operation fluency information, and then executing preset interactive actions based on the device, are performed to improve the reliability and security of the user's operation of the device.

[0084] In one embodiment, user emotion information is determined based on facial information, such as... Figure 5 As shown, it includes:

[0085] S502, determine the emotion value based on the facial information and a preset emotion model. The emotion model is used to represent the mapping relationship between facial information and emotion values. For the facial information of the same user, at least one emotion value can be determined. For example, as described in the above embodiments, a first emotion value can be determined based on the difference between the distances D and D0 between the upper and lower eyelids, and a second emotion value can be determined based on the angle between the upturned corner feature vectors α and β.

[0086] When the emotion value is not unique, the above process of inputting the emotion value and fluency score into the total score determination model K1*A+K2*B=G to determine the total score G can include:

[0087] Each emotion value and fluency score is input into the total score to determine the model K11*A1+K12*A2+…+K1 i *A i +…+K1 n *A n +K2*B=G, determine the total score G, K1 i As the first weight of the i-th sentiment value, A i Let be the i-th emotion value. The training process for the total score determination model described above also applies to cases where emotion values ​​are not unique. Multiple emotion values ​​can be emotion values ​​determined based on features of different parts of the face, such as the emotion values ​​corresponding to the information reflecting eye and mouth features in the facial information, as illustrated in the above embodiment. The weights of the emotion values ​​of different parts of the face used to determine the total score may be different, and this process can be implemented through training a neural network model.

[0088] In one embodiment, the training steps of the preset emotion model may include:

[0089] The facial information in the sample data is input into the second neural network model to obtain the predicted emotion value; the second neural network model is a model that represents the mapping relationship between facial information and the predicted emotion value.

[0090] The second neural network model is adjusted based on the deviation between the predicted and actual emotion values ​​until the termination condition is met. The termination condition can be reaching the maximum number of training iterations or the deviation between the predicted and actual emotion values ​​falling within the range allowed by the user.

[0091] The trained second neural network model is used as the preset emotion model.

[0092] In one embodiment, user emotion information is determined based on facial information, such as... Figure 6 As shown, it also includes:

[0093] S602, based on facial information and a preset emotion model, determines the confidence level of each emotion value.

[0094] Confidence level refers to the level of confidence in an emotion value. A higher confidence level indicates greater reliability in using the emotion value to assess whether a user's state is abnormal. Based on this, in one embodiment, a weighted calculation can be performed using the emotion value with the highest confidence level and the fluency score to determine the total score.

[0095] The training process of a pre-defined emotion model may include:

[0096] The facial information in the sample data is input into the third neural network model to obtain the predicted emotion value and the predicted confidence level; the third neural network model is a model that represents the mapping relationship between facial information and emotion value and confidence level.

[0097] Based on the deviation between the predicted and actual emotion values, as well as the deviation between the predicted and actual confidence levels, the third neural network model is adjusted until the termination condition is met. The termination condition can be reaching the maximum number of training iterations, or the deviation between the predicted and actual emotion values, as well as the deviation between the predicted and actual confidence levels, all falling within the user-allowed deviation range.

[0098] The trained third neural network model is used as the preset emotion model.

[0099] In one embodiment, operation smoothness information is determined based on operation data, such as... Figure 7 As shown, it includes:

[0100] The S702 determines a smoothness score based on operational data and a pre-set smoothness scoring model. A higher smoothness score indicates better user experience when operating the device. The pre-set smoothness scoring model can be obtained through prior training.

[0101] Specifically, the training process for the fluency scoring model can include:

[0102] The operational data from the sample data is input into the fourth neural network model to obtain the predicted fluency score; the fourth neural network model is a model that represents the mapping relationship between operational data and fluency score;

[0103] The fourth neural network model is adjusted based on the deviation between the predicted fluency score and the actual fluency score until the termination condition is met. The termination condition can be reaching the maximum number of training iterations or the deviation between the predicted fluency score and the actual fluency score falling within the deviation range allowed by the user.

[0104] The trained fourth neural network model is used as the preset fluency scoring model.

[0105] In one embodiment, the user's emotion information includes an emotion value, and step S208 involves performing a preset interactive action based on the device, such as... Figure 8 As shown, it includes:

[0106] S802, determine the preset interactive action corresponding to the emotion value based on the emotion value and the interaction action determination model; wherein, the interaction action determination model is used to represent the mapping relationship between the emotion value and the preset interactive action.

[0107] The interaction action determination model can be a tabular mapping model, where the table has at least two columns of data: one column represents the emotion value, and the other column represents the interaction action corresponding to that emotion value. The interaction action determination model can also be a neural network model. When it is a neural network model, the training process can include:

[0108] The emotion values ​​from the sample data are input into the fifth neural network model to obtain the preset interactive actions; the fifth neural network model is a model that represents the mapping relationship between emotion values ​​and preset interactive actions.

[0109] The fifth neural network model is adjusted based on user ratings of preset interactive actions until a termination condition is met. The termination condition could be reaching the maximum number of training iterations or the rating exceeding a preset rating threshold (e.g., a maximum score of 100 points and a preset threshold of 90). User ratings of preset interactive actions can be displayed on the device screen after the action is performed, allowing users to rate whether the action effectively helps improve their attention span while operating the device. For example, the rating range could be 0-100 points, allowing users to rate the action based on their actual experience. The system collects these ratings as training sample data for the fifth neural network model.

[0110] The trained fifth neural network model is used as the preset fluency scoring model.

[0111] S804, The device executes a preset interactive action corresponding to the emotion value. As described above, the preset interactive action corresponding to the emotion value refers to interactive actions that can effectively improve the user's current emotional state, thereby effectively increasing the user's attention when operating the device. By executing preset interactive actions corresponding to the emotion value, different interactive actions can be executed for different user emotions, ensuring effective interaction under different emotional states.

[0112] For example, preset interactive actions can be mainly divided into two categories: non-behavioral interactions and behavioral interactions. Non-behavioral interactions may include, but are not limited to, providing reminders via pop-up prompts on the device screen or providing voice prompts. Behavioral interactions may include, but are not limited to, displaying preset game interfaces. By automatically launching mini-games on the interface, users can improve their attention and relax through these games.

[0113] Different emotion values ​​correspond to different emotion types. Taking ultrasound imaging equipment as an example, if the emotion value detects a doctor's fatigue, a voice or text prompt can be used to say, "You appear slightly fatigued; please take a break." Interactive games can also be used to help the doctor relax their limbs or eyes, such as rotating their view along with a ball on the screen. The equipment can also be raised, and interactive games can be provided to help the doctor stretch and relax. Different emotion values ​​have different preset interaction methods, and the correspondence between emotion values ​​and preset interaction methods can be configured. For example, when the emotion value corresponds to the fatigue emotion type, the corresponding preset interaction action is the voice prompt "You appear slightly fatigued; please take a break." When the emotion value corresponds to the sadness emotion type, the corresponding preset interaction action is the voice prompt "Smile; it's not a big deal," and so on. Preset interaction actions can also be configured in advance and can be updated. One emotion value can correspond to one or at least two preset interaction actions. For example, when the emotion value is 10, the corresponding preset interaction actions include the voice prompt "You appear slightly fatigued; please take a break" and displaying a preset game interface.

[0114] In one embodiment, the step of weighting the emotion value and fluency score to determine the total score includes:

[0115] When a user is determined to be in a negative emotional state based on their emotion score, the emotion score and fluency score are weighted and calculated to determine the total score. Negative emotions include, but are not limited to, sadness, anger, fatigue, and distress.

[0116] First, we can determine if the user is in a negative emotional state. If the user is in a good mood, abnormal smoothness in operating the device may be due to special circumstances, such as during an ultrasound examination, where an abnormal real-time imaging result is found for a certain part of the patient's body, causing the scanning operation time for that part to be much longer than the average scanning operation time. In this case, there is no need to perform preset interactive actions to improve the user's attention. When the user is judged to be in a negative emotional state, the influence of emotions may lead to erroneous operations. However, judging solely based on emotions is not reliable. By weighing both emotions and operational smoothness, an abnormal user state is only determined after the score is below a first threshold and remains below it for a period of time.

[0117] In one embodiment, step S208, which involves the device performing a preset interactive action, includes:

[0118] When the device is idle, it performs preset interactive actions. Idle state refers to the operating state of the device when it does not respond to user operations. For example, the ultrasound imaging device is in standby mode, or it has completed the current ultrasound scan but has not yet started the next ultrasound scan.

[0119] To avoid interference with normal user operation, such as pop-up prompts or preset interactive actions, when the device is in use, especially in the medical field where this could interrupt doctors' work and lead to adverse consequences, preset interactive actions are only executed when the device enters an idle state, based on user emotion information and operation fluency information indicating an abnormal user state. This aims to improve user attention.

[0120] In one embodiment, the device interaction control method further includes:

[0121] If the device is detected to have switched to working mode or the emotional value has risen to the second threshold, the preset interactive action will be stopped.

[0122] The "working state" refers to a non-idle state, such as when an ultrasound imaging device is performing a scan. The second threshold is an emotion value greater than the first threshold. When the emotion value rises to this threshold, it indicates that the user has recovered from a negative emotion to a positive one, such as happiness. At this time, no preset interactive action needs to be executed, and the user can operate the device smoothly. Therefore, by detecting that the device has switched to the working state or that the emotion value has risen to the second threshold, the execution of preset interactive actions is stopped. On the one hand, this avoids affecting the normal operation of the device; on the other hand, it intelligently stops the execution of preset interactive actions without requiring manual operation from the user, thus improving the user experience.

[0123] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0124] Based on the same inventive concept, this application also provides a device interaction control apparatus for implementing the device interaction control method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more display interaction device embodiments provided below can be found in the limitations of the display interaction method described above, and will not be repeated here.

[0125] In one embodiment, such as Figure 9 As shown, a device interaction control apparatus is provided, comprising: a data acquisition module 902, an emotion information determination module 904, a fluency information determination module 906, and an interaction execution module 908, wherein:

[0126] The data acquisition module 902 is used to acquire the user's facial information and operation data when the user operates the device. The emotion information determination module 904 determines the user's emotion information based on the facial information. The fluency information determination module 906 determines the operation fluency information based on the operation data. The interaction execution module 908 executes preset interactive actions based on the device when it is determined that the user's state is abnormal based on the user's emotion information and operation fluency information.

[0127] In one embodiment, user emotion information includes an emotion value, operation fluency information includes a fluency score, and the device interaction control device further includes:

[0128] The total score calculation module is used to calculate the total score by weighting the emotion value and fluency score.

[0129] The abnormal status determination module is used to determine that the user's status is abnormal when the total score is lower than the first threshold.

[0130] In one embodiment, the emotion information determination module 904 includes:

[0131] The emotion value determination unit is used to determine the emotion value based on facial information and a preset emotion model.

[0132] In one embodiment, the emotion information determination module 904 further includes:

[0133] The confidence determination unit is used to determine the confidence level of each emotion value based on facial information and a preset emotion model.

[0134] In one embodiment, the fluency information determination module 906 includes:

[0135] The fluency score determination unit is used to determine the fluency score based on the operation data and the preset fluency scoring model.

[0136] In one embodiment, the user's emotional information includes an emotional value, and the interaction execution module 908 includes:

[0137] The interaction action determination unit is used to determine the preset interaction action corresponding to the emotion value based on the emotion value and the interaction action determination model; wherein, the interaction action determination model is used to represent the mapping relationship between the emotion value and the preset interaction action.

[0138] The matching interaction action execution unit is used to execute preset interaction actions corresponding to the emotion value based on the device.

[0139] In one embodiment, the total score calculation module includes:

[0140] The total score determination unit is used to calculate the total score by weighting the emotion value and fluency score when the user is determined to be in a negative emotion based on the emotion value.

[0141] In one embodiment, the interactive execution module 908 includes:

[0142] The idle interaction execution unit is used to perform preset interactive actions based on the device when the device is in an idle state.

[0143] In one embodiment, the preset interactive action includes at least one of the following: voice prompt, displaying prompt content, displaying a preset game interface, or raising the device.

[0144] In one embodiment, the user's emotional information includes an emotional value, and the device interaction control device further includes:

[0145] The interaction stop module is used to stop the execution of preset interactive actions when the device is detected to switch to working mode or the emotion value rises to a second threshold.

[0146] Each module in the aforementioned device interaction control system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0147] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 10 As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a device interaction control method. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0148] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0149] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0150] S202, Obtain facial information and operation data when the user operates the device;

[0151] S204, Determine the user's emotional information based on facial information;

[0152] S206, Determine the operation smoothness information based on the operation data;

[0153] S208, when determining that the user's state is abnormal based on user emotion information and operation fluency information, executes preset interactive actions based on the device.

[0154] In one embodiment, when the processor executes the computer program, it also implements the steps in any of the above method embodiments and achieves the corresponding beneficial effects.

[0155] In one embodiment, the computer device is a controller for a medical imaging device. In the above embodiments, the device operated by the user is a medical imaging device, such as an ultrasound imaging device.

[0156] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0157] S202, Obtain facial information and operation data when the user operates the device;

[0158] S204, Determine the user's emotional information based on facial information;

[0159] S206, Determine the operation smoothness information based on the operation data;

[0160] S208, when determining that the user's state is abnormal based on user emotion information and operation fluency information, executes preset interactive actions based on the device.

[0161] In one embodiment, when the computer program is executed by a processor, it also implements the steps in the other method embodiments described above and achieves the corresponding beneficial effects.

[0162] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements some or all of the method steps in the above method embodiments and achieves corresponding beneficial effects.

[0163] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0164] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0165] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0166] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A device interactive control method, characterized in that, Applied to medical imaging equipment, wherein the medical imaging equipment is an ultrasound imaging device, the method includes: When the system detects entry into a designated interface or detects a scan sequence input command, it acquires the user's facial information and operation data when operating the device. The operation data includes the input data, output data, and data input and output time of each node in the medical imaging device's scanning workflow, or the time it takes for the user to complete a single scan imaging operation. Based on the facial information, determine the user's emotional information; Based on the operation data, determine the operation smoothness information; If the user's state is determined to be abnormal based on the user's emotional information and the operation fluency information, a preset interactive action is executed based on the device. The user emotion information includes an emotion value, and the operation smoothness information includes a smoothness score. The step of determining that the user's state is abnormal based on the user emotion information and the operation smoothness information includes: The total score is determined by weighting the emotion value and the fluency score. If the total score is lower than the first threshold and continues for a preset duration, or if the total score is lower than the first threshold more than a preset number of times within a preset time period, then the user's status is determined to be abnormal.

2. The method according to claim 1, characterized in that, The step of determining the operation smoothness information based on the operation data includes: Based on the operation data, determine the operation time of each node in the scanning workflow; The operation smoothness information is determined based on the difference between the operation time interval between two adjacent scanning workflow nodes or between two nodes separated by multiple workflow nodes, and the pre-collected average user operation time interval.

3. The method according to claim 1, characterized in that, The step of determining the user's emotional information based on the facial information includes: The emotion value is determined based on the facial information and the preset emotion model.

4. The method according to claim 3, characterized in that, The step of determining the user's emotional information based on the facial information further includes: Based on the facial information and the preset emotion model, the confidence level of each emotion value is determined.

5. The method according to any one of claims 1-4, characterized in that, The step of determining the operation smoothness information based on the operation data includes: Based on the operation data and the preset smoothness scoring model, a smoothness score is determined.

6. The method according to any one of claims 1-4, characterized in that, The user's emotional information includes an emotional value, and the execution of a preset interactive action based on the device includes: Based on the emotion value and the interaction action determination model, a preset interaction action corresponding to the emotion value is determined; wherein, the interaction action determination model is used to characterize the mapping relationship between the emotion value and the preset interaction action; The device executes a preset interactive action corresponding to the emotion value.

7. The method according to claim 2, characterized in that, The step of weighting the emotion value and the fluency score to determine the total score includes: If the user is determined to be in a negative emotional state based on the emotion value, the emotion value and the fluency score are weighted and calculated to determine the total score.

8. The method according to claim 1, characterized in that, The execution of preset interactive actions based on the device includes: When the device is in an idle state, the preset interactive action is performed based on the device.

9. The method according to claim 1, characterized in that, The preset interactive actions include at least one of the following: voice prompts, displaying prompt content, displaying a preset game interface, or raising the device.

10. The method according to claim 1, characterized in that, The user emotion information includes emotion values, and the method further includes: If the device is detected to switch to working mode or the emotion value rises to a second threshold, the preset interactive action will be stopped.

11. A device interaction control apparatus, characterized in that, Applied to medical imaging equipment, wherein the medical imaging equipment is an ultrasound imaging device, the device includes: The data acquisition module is used to acquire facial information and operation data of the user when operating the device; the operation data includes the input data, output data and data input and output time of each node of the medical imaging device scanning workflow, or the time it takes for the user to complete a scanning imaging operation. An emotion information determination module is used to determine the user's emotion information based on the facial information; The smoothness information determination module is used to determine the smoothness information of the operation based on the operation data; An interactive execution module is used to execute preset interactive actions based on the device when the user's state is determined to be abnormal according to the user's emotion information and the operation fluency information. The user emotion information includes an emotion value, the operation smoothness information includes a smoothness score, and the device interaction control device further includes: The total score calculation module is used to perform a weighted calculation of the emotion value and the fluency score to determine the total score; The abnormal status determination module is used to determine that the user status is abnormal when the total score is lower than a first threshold and continues for a preset duration, or when the total score is lower than the first threshold more than a preset number of times within a preset time period.

12. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

Images