Multimodal fusion human-computer interaction method, device, wearable device and medium

Abstract

The present application relates to the field of human-computer interaction technology. The present application discloses a multimodal fusion human-computer interaction method, apparatus, wearable device and medium, which can meet the multi-task operation requirements of interactive scenarios where natural language communication is not possible, and improve the efficiency of human-computer interaction. The multimodal fusion human-computer interaction method includes obtaining the user's head movement classification information, throat movement classification information and user status information; obtaining a first user intention label based on the head movement classification information, and adding the first user intention label to the first stack; obtaining a second user intention label based on the throat movement classification information, and adding the second user intention label to the second stack; when the first stack and / or the second stack is in an overflow state, obtaining the user's operation intention based on the first stack and the second stack; obtaining a first operation instruction based on the user's operation intention and the user status information, and issuing the first operation instruction to the underwater robot.

Description

Technical Field

[0001] The present application relates to the field of human-computer interaction technology. More specifically, the present application relates to a multi-modal fusion human-computer interaction method, apparatus, wearable device and medium. Background Art

[0002] Traditional human-computer interaction methods are usually limited to a single modal information, that is, any one of voice, image, text, eye movement or touch. This method performs well in daily environments, but its robustness and generalization capabilities are poor in interactive scenarios where natural language communication is not possible, such as underwater interaction scenarios and deaf-mute interaction scenarios. Specifically, this method usually requires the use of the user's limbs, encroaching on the user's task space, making it impossible for the user's limbs to complete human-computer interaction in the original task space, and it is difficult to meet the multi-task operation requirements of application scenarios where natural language communication is not possible. The task space refers to the spatial range of the various parts of the body (especially the limbs) when the user is performing an activity or task, and the area where they can move and operate freely. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a multimodal fusion human-computer interaction method, apparatus, wearable device and medium, which can meet the multi-task operation requirements of interactive scenarios where natural language communication is not possible and improve the efficiency of human-computer interaction. The embodiments of the present application are mainly implemented through the following technical solutions:

[0004] A first aspect of an embodiment of the present application provides a multi-modal fusion human-computer interaction method, comprising:

[0005] Obtaining the user's head movement classification information, throat movement classification information and user status information;

[0006] Obtaining a first user intention label based on the head action classification information, and adding the first user intention label to a first stack;

[0007] Obtaining a second user intention label based on the laryngeal motion classification information, and adding the second user intention label to a second stack;

[0008] When the first stack and / or the second stack is in an overflow state, obtaining a user operation intention based on the first stack and the second stack;

[0009] A first operation instruction is obtained based on the user operation intention and the user status information, and the first operation instruction is issued to the underwater robot.

[0010] According to an embodiment of the present application, the step of obtaining the user's head movement classification information includes:

[0011] Collecting the user's head posture information based on the first time series to obtain head movement data;

[0012] Performing filtering processing on the head motion data to obtain first data to be processed;

[0013] Using a segmentation method to segment the first data to be processed to obtain second data to be processed;

[0014] Characterizing the second data to be processed based on preset features to obtain third data to be processed;

[0015] The third data to be processed is compared with the head movement template to obtain the head movement classification information.

[0016] According to an embodiment of the present application, the step of obtaining the user's throat movement classification information includes:

[0017] collecting the laryngeal vibration information of the user based on the second time series to obtain laryngeal vibration data;

[0018] Performing audio signal processing on the laryngeal vibration data to obtain fourth data to be processed;

[0019] Performing discrete Fourier transform on the fourth data to be processed to obtain frequency spectrum information;

[0020] Extracting features from the frequency spectrum information to obtain fifth data to be processed;

[0021] A neural network is used to classify the fifth data to be processed to obtain the laryngeal movement classification information.

[0022] According to an embodiment of the present application, the step of obtaining the user status information of the user includes:

[0023] Obtain the user's vital signs data and external environment information;

[0024] The user status information is obtained based on the vital sign data and the external environment information.

[0025] According to an embodiment of the present application, after the step of obtaining the user status information based on the vital sign data and the external environment information, the step of obtaining the user status information of the user further includes:

[0026] Cross-validating the vital sign data and the external environment information to obtain a validation result;

[0027] The user status information is modified based on the verification result.

[0028] According to an embodiment of the present application, after the step of obtaining the user's head movement classification information, throat movement classification information and user status information, the multimodal fusion human-computer interaction method further includes:

[0029] When the user status information is in a dangerous state, automatically generating the second operation instruction using a dominant control system, and sending the second operation instruction to the underwater robot;

[0030] Before or after the second operation instruction is issued, the dominant control system generates a first escape-from-danger indication to prompt the user to escape from the dangerous state.

[0031] According to an embodiment of the present application, after the step of obtaining the user's head movement classification information, throat movement classification information and user status information, the multimodal fusion human-computer interaction method further includes:

[0032] Based on the preset shared function, determine whether to use the dominant control system to automatically generate a third operation instruction, and send the third operation instruction to the underwater robot;

[0033] Before or after the third operation instruction is issued, the dominant control system generates a second escape-from-danger indication to prompt the user to escape from the dangerous state.

[0034] A second aspect of the embodiment of the present application provides a multi-modal fusion human-computer interaction device, including:

[0035] An information acquisition module, used to acquire the user's head movement classification information, throat movement classification information and user status information;

[0036] A first user intention label obtaining module, configured to obtain a first user intention label based on the head action classification information, and add the first user intention label to a first stack;

[0037] A second user intention label obtaining module, configured to obtain a second user intention label based on the laryngeal motion classification information, and add the second user intention label to a second stack;

[0038] A user operation intention obtaining module, configured to obtain a user operation intention based on the first stack and the second stack when the first stack and / or the second stack is in an overflow state;

[0039] The first operation instruction obtaining module is used to obtain a first operation instruction based on the user operation intention and the user status information, and send the first operation instruction to the machine.

[0040] According to a third aspect of an embodiment of the present application, a wearable device is provided, comprising: a camera, an inertial measurement unit, a vital sign sensor, a temperature sensor, a laryngeal vibration sensor, a processor and a memory, wherein the camera, the inertial measurement unit, the vital sign sensor, the temperature sensor and the laryngeal vibration sensor are all connected to the processor, the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the steps of the multimodal fusion human-computer interaction method provided in the first aspect of the embodiment of the present application.

[0041] In a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium is used to store a computer program, and the computer program enables a computer to execute the steps of the multimodal fusion human-computer interaction method provided in the first aspect of the embodiment of the present application.

[0042] The beneficial effects of the embodiments of the present application include at least:

[0043] The embodiment of the present application obtains the user's head movement classification information, throat movement classification information and user status information; obtains a first user intention label based on the head movement classification information, and adds the first user intention label to the first stack; obtains a second user intention label based on the throat movement classification information, and adds the second user intention label to the second stack; when the first stack and / or the second stack are in an overflow state, obtains the user's operation intention based on the first stack and the second stack; obtains a first operation instruction based on the user's operation intention and the user status information, and sends the first operation instruction to the underwater robot. That is, the embodiment of the present application uses the user's head movement classification information, throat movement classification information and user status information as factors for generating the first operation instruction to control the underwater robot. Therefore, the implementation of the human-computer interaction of the present application does not need to occupy the user's limbs, so that the user can have enough body parts to implement other tasks while completing the human-computer interaction task. Therefore, compared with the prior art, the embodiment of the present application can meet the multi-task operation requirements of the interactive scene where natural language communication cannot be performed, and improve the efficiency of human-computer interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the conventional technology, the drawings required for use in the embodiments or the conventional technology descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1A flowchart of the multimodal fusion human-computer interaction method of the present application in some embodiments;

[0046] Figure 2 A flowchart of the multi-modal fusion human-computer interaction method of the present application in some embodiments;

[0047] Figure 3 A reference diagram of the head action classification information in this application;

[0048] Figure 4 A mapping relationship diagram between the first user intention label, the second user intention label and the user operation intention in this application;

[0049] Figure 5 A block diagram of a multi-modal fusion human-computer interaction device of the present application in some embodiments;

[0050] Figure 6 This is a structural diagram of the wearable device of the present application in some embodiments. DETAILED DESCRIPTION

[0051] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present application, so the present application is not limited by the specific embodiments disclosed below.

[0052] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0053] The terms "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0054] The terms "comprises," "includes," or any other variations thereof are intended to cover a non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or elements is not necessarily limited to those steps or elements explicitly listed but may include other steps or elements not explicitly listed or inherent to such process, method, product, or apparatus.

[0055] Unless otherwise defined, all technical and scientific terms used in the specification of this application have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used in the specification of this application includes any and all combinations of one or more related listed items.

[0056] The specific implementation of the present application is further described below in conjunction with the accompanying drawings.

[0057] like Figure 1 FIG. 1 is a flowchart of a multi-modal fusion human-computer interaction method provided in the first aspect of the embodiment of the present application. Figure 1 In the method, the multimodal fusion human-computer interaction method includes:

[0058] S1. Obtaining the user's head movement classification information, throat movement classification information and user status information.

[0059] Step S1 realizes natural human-computer interaction without relying on natural language recognition and limb control by introducing head movement and laryngeal vibration modes, so that the multimodal fusion human-computer interaction method of the present application can solve the problem of low efficiency of human-computer interaction in complex and unstructured scenarios and meet the human-computer interaction needs in multi-task operations.

[0060] Furthermore, the step of obtaining the user's head movement classification information includes:

[0061] S111, collecting the user's head posture information based on the first time sequence (i.e. Figure 2 head signal in the image), and obtain head movement data.

[0062] In the embodiment of the present application, the first time sequence is a preset time period. The embodiment of the present application uses an inertial measurement unit of the wearable device to detect and record the head posture information at each moment in the time period, so that all head posture information in the entire time period constitutes the head movement data.

[0063] The specific duration of the time period is determined by those skilled in the art according to actual needs.

[0064] The head posture information includes acceleration, angle and angular acceleration. In other implementations, the head posture information may also include other contents, which may be specifically set by those skilled in the art according to actual needs.

[0065] S112: Filter the head motion data to obtain first data to be processed.

[0066] The setting of step S112 can reduce the interference of noise and random errors in the head motion data on the head motion classification information.

[0067] S113: Use a segmentation method to segment the first data to be processed to obtain second data to be processed.

[0068] The segmentation method is an adaptive threshold segmentation method. In other embodiments, those skilled in the art can set other segmentation methods according to requirements.

[0069] It should be understood that after the processing in step S113, each head posture included in the second data to be processed is clear and the Euler angle of the head meets the preset value. For example, the preset value may be the head forward bending , Head extension , flex the head to the left , flex the head to the right , turn left and turn right The second data to be processed is a fragment with practical significance. In this application, the Euler angle is in degrees (°).

[0070] In other implementations, the Euler angle representation of the posture in the fragment having practical significance is converted into a posture quaternion.

[0071] S114. Characterize the second data to be processed based on preset features to obtain third data to be processed.

[0072] The preset features include but are not limited to speed on the x-axis, acceleration on the x-axis, angular acceleration on the x-axis, speed on the y-axis, acceleration on the y-axis, angular acceleration on the y-axis, speed on the z-axis, acceleration on the z-axis, and angular acceleration on the z-axis.

[0073] The embodiment of the present application characterizes the velocity on the x-axis, the acceleration on the x-axis, the angular acceleration on the x-axis, the velocity on the y-axis, the acceleration on the y-axis, the angular acceleration on the y-axis, the velocity on the z-axis, the acceleration on the z-axis, and the angular acceleration on the z-axis for each head posture in the second data to be processed, thereby obtaining the third data to be processed.

[0074] S115: Compare the third data to be processed with the head movement template to obtain the head movement classification information.

[0075] Specifically, the present application uses a dynamic time warping algorithm to compare the third data to be processed with the head movement template.

[0076] The head action templates include raising the head, turning the head to the left, turning the head to the right, lowering the head, turning the head to the right and turning the head to the left. When any head posture of the third data to be processed meets one of the head action templates, the head posture is added to the head action classification information.

[0077] The head movement classification information is selected from at least one of raising the head (i.e., extending the head backward), turning the head to the left (i.e., flexing the head to the left), turning the head to the right (i.e., turning the head right), lowering the head (i.e., flexing the head forward), turning the head to the right (i.e., flexing the head to the right), and turning the head to the left (i.e., turning the head left). For details, please refer to Figure 3 shown.

[0078] Furthermore, the step of obtaining the user's throat movement classification information includes:

[0079] S121, collecting the user's throat vibration information based on the second time sequence (i.e. Figure 2 ), and obtain laryngeal vibration data.

[0080] It should be understood that the duration of the second timing is the same as that of the first timing, and the second timing is the first timing.

[0081] The embodiment of the present application uses a laryngeal vibration sensor of a wearable device to collect the laryngeal vibration information. The laryngeal vibration sensor detects and records the laryngeal vibration information at each moment in the time period of the second time sequence, so that all laryngeal vibration information in the entire time period of the second time sequence constitutes the laryngeal vibration data.

[0082] S122. Perform audio signal processing on the laryngeal vibration data to obtain fourth data to be processed.

[0083] Specifically, the laryngeal vibration data is subjected to noise reduction, pre-emphasis, frame windowing, endpoint detection, and standardization, so as to obtain the fourth data to be processed.

[0084] The setting of step S122 can reduce the noise of the laryngeal vibration data and improve the data quality.

[0085] S123. Perform discrete Fourier transform on the fourth data to be processed to obtain frequency spectrum information.

[0086] The discrete Fourier transform transforms the fourth data to be processed from the time domain to the frequency domain, thereby obtaining the frequency spectrum information.

[0087] S124: extract features from the frequency spectrum information to obtain fifth data to be processed.

[0088] The S124 is used to extract frequency spectrum features, such as Mel-frequency cepstral coefficients.

[0089] The fifth data to be processed is used to train the neural network, so that the neural network can recognize different tones and sound intensities.

[0090] S125. Classify the fifth data to be processed using a neural network to obtain the laryngeal movement classification information.

[0091] Specifically, the neural network is a temporal neural network, such as a long short-term memory network (LSTM), which can recognize and classify different sound features, and be used for pitch tracking or sound intensity calculation.

[0092] Furthermore, the step of obtaining the user status information of the user includes:

[0093] S131, obtaining the user's vital signs data (i.e. Figure 2 vital signs in the body) and external environmental information (i.e. Figure 2 (environmental information in the ).

[0094] The vital sign data includes but is not limited to the user's blood pressure, blood oxygen, heart rate and body temperature. The vital sign data is used to modify the user's vital sign status.

[0095] The embodiment of the present application uses a vital sign sensor of a wearable device to collect the vital sign data.

[0096] The external environment information includes obstacle information. In other implementations, the external environment information may also include altitude, temperature and image information. The external environment information is used to modify the environment in which the user is located.

[0097] The embodiment of the present application uses a camera and a temperature sensor of a wearable device to collect the external environment information.

[0098] S132: Obtain the user status information based on the vital sign data and the external environment information.

[0099] Furthermore, the step S132 includes:

[0100] S1321. Modify the user's vital sign status based on the vital sign data.

[0101] The state of the vital signs is selected from one of a normal state, an abnormal state and a critical state. Specifically, when the vital signs data are all within the normal range, that is, the user's blood pressure, blood oxygen, heart rate and body temperature are all within the normal range, the vital signs state is modified to a normal state; when at least one of the vital signs data exceeds the normal range (that is, at least one of the blood pressure, blood oxygen, heart rate and body temperature exceeds the normal range), but has not yet reached a critical state, the vital signs state is modified to an abnormal state; when the vital signs data are seriously abnormal and may endanger the user's life, the vital signs state is modified to a critical state. When the vital signs state is a critical state, the dominant control system needs to intervene immediately to ensure the safety of the user.

[0102] It should be understood that the specific value ranges of the normal range, the range that exceeds the normal range but has not yet reached a crisis state, and the serious abnormality are determined by those skilled in the art according to actual needs.

[0103] S1322: Modify the user's environment based on the external environment information.

[0104] The environmental condition is selected from one of a dangerous state, a safe state, and a state between danger and safety. Specifically, when the external environmental information indicates that there is no obstacle, the environmental condition is a safe state; when the external environmental information indicates that the distance to the obstacle is close to a safe distance, the environmental condition is a state between danger and safety; when the external environmental information indicates that the distance to the obstacle is less than a safe distance, the environmental condition is a dangerous state. The safe distance is determined by those skilled in the art according to actual needs.

[0105] In other embodiments, the criterion for judging whether the environmental condition is a safe state may further include that the water depth of the user is within a first safety threshold, and that the user has sufficient oxygen remaining in the water. The criterion for judging whether the environmental condition is a state between dangerous and safe may further include that the water depth and water pressure of the user are about to approach a second safety threshold, and that the user has sufficient oxygen remaining in the water. The criterion for judging whether the environmental condition is a dangerous state may further include that the water depth and water pressure of the user are greater than the second safety threshold, and that the user has sufficient oxygen remaining in the water.

[0106] The specific values ​​of the first safety threshold, the second safety threshold and the third safety threshold can be determined by those skilled in the art according to actual needs.

[0107] S1323, obtaining the user status information (i.e., Figure 2 ).

[0108] The user status information is selected from any one of a normal state, a stuck state and a dangerous state.

[0109] Specifically, when the vital signs are in a normal state and the environment is in a safe state, the user status information is in a normal state; when the vital signs are in an abnormal state and the environment is in a critical state between danger and safety, the user status information is in a stagnant state; when the vital signs are in a critical state and the environment is in a dangerous state, the user status information is in a dangerous state. Further, after the step S132, the step of obtaining the user status information of the user further includes:

[0110] S133: cross-validate the vital sign data and the external environment information to obtain a validation result.

[0111] Furthermore, the step S133 includes:

[0112] S1331. Obtain the vital sign status corresponding to the vital sign data.

[0113] S1332: Obtain environmental conditions corresponding to the external environmental information.

[0114] S1333. When the vital sign status is a critical status, assign the verification result to a critical status.

[0115] S1334. When the environmental condition is in a dangerous state, assign the verification result to a dangerous state.

[0116] S134. Modify the user status information based on the verification result.

[0117] When the verification result is a critical state or a dangerous state, the user state information is modified to a dangerous state.

[0118] S2. Obtain a first user intention label based on the head movement classification information, and add the first user intention label to a first stack.

[0119] In an embodiment of the present application, each type of head movement classification information corresponds to a first user intention label. For example, when the head movement classification information is looking up, the first user intention label is a looking up label; when the head movement classification information is turning the head to the left, the first user intention label is a turning the head to the left label; when the head movement classification information is turning the head to the right, the first user intention label is a turning the head to the right label; when the head movement classification information is lowering the head, the first user intention label is a lowering label; when the head movement classification information is turning the head to the right, the first user intention label is a turning the head to the right label; when the head movement classification information is turning the head to the left, the first user intention label is a turning the head to the left label.

[0120] It should be understood that in the embodiment of the present application, when the first stack of fixed capacity overflows within 0.2s, it is considered to be a valid head intention operation, that is, when the first stack overflows within 0.2S, the head movement data collected within this time period are all valid.

[0121] S3. Obtain a second user intention label based on the laryngeal motion classification information, and add the second user intention label to a second stack.

[0122] The embodiment of the present application sets a sound threshold. When the sound signal of the laryngeal movement classification information is less than the sound threshold, the second user intention label is an invalid intention label; when the sound signal of the laryngeal movement classification information is greater than the sound threshold, the second user intention label is a valid intention label (i.e. Figure 4 S and S in 1 ).

[0123] The laryngeal movement classification information also includes the duration of the sound and the pitch of the sound.

[0124] S4: when the first stack and / or the second stack is in an overflow state, obtaining a user operation intention (i.e., Figure 2 ).

[0125] When the first stack is in an overflow state, check whether all the first user intention labels in the first stack are the same. If they are the same, it is considered that the user's intention expression during this period is stable and reliable, and the first user intention label at this time is used to judge the user's operation intention; if they are not the same, it is considered that the user's intention expression during this period is unstable, and thus, it is determined that all the first user intention labels in the first stack are invalid, and the user operation intention is not judged by the first user intention label at this time.

[0126] When the second stack is in an overflow state, check whether all second user intention labels in the second stack are the same. If they are the same, it is considered that the user's intention expression during this period is stable and reliable, and the second user intention label at this time is used to judge the user's operation intention; if they are not the same, it is considered that the user's intention expression during this period is unstable, and thus, it is determined that all second user intention labels in the second stack are invalid, and the second user intention label at this time is not used to judge the user's operation intention.

[0127] The correspondence between the first user intention label, the second user intention label and the user operation intention can be referred to Figure 4 Specifically, when the first user intention label (i.e. Figure 4 The head action in the Figure 4 ) and the second user intention label (i.e. Figure 4 The sound confirmation signal in the text is the valid intention tag S 1 , the user operation intention is that the speed of the underwater robot along the Z axis (that is, the speed of movement along the Z axis) is ,in, is the value of the Euler angle of the head movement, It is represented as the coefficient of mapping the Euler angle of the head movement to the speed of the underwater robot; when the first user intention label is a head-up label (i.e. Figure 4 and the second user intention label is a valid intention label S, the user operation intention is that the speed of the underwater robot along the Z axis (that is, the speed of movement along the Z axis) is ,in, is the value of the Euler angle of the head movement, It is represented as the coefficient of mapping the Euler angle of the head movement to the speed of the underwater robot; when the first user intention label is the left head label (that is, Figure 4 When the second user intention label is a valid intention label S, the user operation intention is that the speed of the underwater robot along the Y axis (that is, the speed of movement along the Y axis) is ,in, is the value of the Euler angle of the head movement, It is represented as the coefficient of mapping the Euler angle of the head movement to the speed of the underwater robot; when the first user intention label is the right head label (that is, Figure 4 and the second user intention label is a valid intention label S, the user operation intention is that the speed of the underwater robot along the Y axis (that is, the speed of movement along the Y axis) is ,in, is the value of the Euler angle of the head movement, It is represented as the coefficient of mapping the Euler angle of the head movement to the speed of the underwater robot; when the first user intention label is the left turn label (that is, Figure 4 When the second user intention label is a valid intention label S, the user operation intention is that the angular velocity of the underwater robot around the Z axis (that is, the angular velocity of rotation around the Z axis) is ,in, is the value of the Euler angle of the head movement, It is represented as the coefficient of mapping the Euler angle of the head movement to the angular velocity of the underwater robot; when the first user intention label is the label of turning the head to the right (that is, Figure 4 When the second user intention label is a valid intention label S, the user operation intention is that the angular velocity of the underwater robot around the Z axis (that is, the angular velocity of rotation around the Z axis) is ,in, is the value of the Euler angle of the head movement, It is expressed as the coefficient of mapping the Euler angle of the head movement to the angular velocity of the underwater robot. It should be understood that Figure 4 In the text, underwater robots are all described as robots.

[0128] The above is the correspondence between the first user intention label, the second user intention label and the user operation intention. In other implementations, the correspondence between the first user intention label, the second user intention label and the user operation intention can also be implemented through the following embodiments:

[0129] The control mode is switched through the throat movement classification information. Specifically, when the tone of the throat movement classification information is low (that is, the user makes a low tone), the embodiment of the present application is in the follow-up motion mode, and the underwater robot moves following the user's head movement; when the throat movement classification information is high-pitched (that is, the user makes a high-pitched voice), the embodiment of the present application is in the free observation mode, and the user's head movement does not affect the state of the underwater robot. In this embodiment, the user's head movement classification information (that is, the head signal) and the throat movement classification information (that is, the sound signal) can independently represent the user's operation intention.

[0130] S5: obtaining a first operation instruction based on the user operation intention and the user status information, and issuing the first operation instruction to the underwater robot (i.e. Figure 2 , “Issuing Instructions”).

[0131] When the user status information is a normal state or a stuck state, the first operation instruction is generated according to the user operation intention. Specifically, when the user operation intention is that the speed of the underwater robot along the Z axis (that is, the speed of movement along the Z axis) is When the first operation instruction is to control the speed of the underwater robot along the Z axis (that is, the speed of movement along the Z axis) to be ; When the user operation intention is that the underwater robot moves along the Z axis (that is, the speed of movement along the Z axis) is When the first operation instruction is to control the speed of the underwater robot along the Z axis (that is, the speed of movement along the Z axis) to be ; When the user operation intention is that the underwater robot moves along the Y axis (that is, the speed along the Y axis) is When the first operation instruction is to control the speed of the underwater robot along the Y axis (that is, the speed of movement along the Y axis) to be ; When the user operation intention is that the angular velocity of the underwater robot around the Z axis (that is, the angular velocity of rotation around the Z axis) is When the first operation instruction is to control the angular velocity of the underwater robot around the Z axis (that is, the angular velocity of rotation around the Z axis) to be ; When the user operation intention is that the angular velocity of the underwater robot around the Z axis (that is, the angular velocity of rotation around the Z axis) is When the first operation instruction is to control the angular velocity of the underwater robot around the Z axis (that is, the angular velocity of rotation around the Z axis) to be .

[0132] The first operation instruction is used to update the state of the underwater machine. When the underwater robot receives the first operation instruction, a feedback message is generated and sent to the wearable device to give the user corresponding feedback. The presentation mode of the feedback information includes but is not limited to visual feedback (such as a screen on land or a waterproof screen underwater) when the environment permits, vibration feedback of the device, and sound feedback of waterproof headphones.

[0133] Through the above technical solution, the embodiment of the present application uses the user's head movement classification information, throat movement classification information and user status information as factors for generating the first operation instruction to control the underwater robot. Therefore, the implementation of the human-computer interaction of the present application does not need to occupy the user's limbs, so that the user can have enough body parts to perform other tasks while completing the human-computer interaction task. Therefore, compared with the prior art, the embodiment of the present application can meet the multi-task operation requirements of the interaction scene where natural language communication is not possible, and improve the efficiency of human-computer interaction.

[0134] In some embodiments, after the step S1, the multimodal fusion human-computer interaction method further includes:

[0135] S6. When the user status information is in a dangerous state, the dominant control system is used to automatically generate the second operation instruction, and send the second operation instruction to the underwater robot.

[0136] S7. Before or after the second operation instruction is issued, the dominant control system generates a first escape from danger indication to prompt the user to escape from the dangerous state.

[0137] In the above technical solution, this application proposes a human-computer collaboration method represented by shared control (that is, using a dominant control system to automatically generate instructions) to reduce the user's operating burden and achieve natural human-computer collaboration.

[0138] In some embodiments, after the step S1, the multimodal fusion human-computer interaction method further includes:

[0139] S8. Determine whether to use the dominant control system to automatically generate a third operation instruction based on a preset shared function, and send the third operation instruction to the underwater robot.

[0140] The preset sharing function is defined as:

[0141] ;

[0142] in, The input corresponding to the user intention, that is, the user operation intention; It is the input of the dominant control system; It is the input passed to the instruction map by the user end of the human-computer interaction interface; The values ​​are as follows:

[0143] ;

[0144] in, The values ​​of are as follows:

[0145] ;

[0146] in, is the user's state space; The user's safe space, that is, the space corresponding to the normal state of the user status information; The dangerous space of the user, that is, the space corresponding to the dangerous state of the user status information; It is the user's intermediate state space, that is, the space corresponding to the user's state information being in a stuck state.

[0147] Furthermore, for any given human-initiated operation instruction , safe space , intermediate state space (also known as hysteresis space) , Hazardous Space Has the following properties: ; .

[0148] S9. Before or after the third operation instruction is issued, the dominant control system generates a second escape from danger indication to prompt the user to escape from the dangerous state.

[0149] In some embodiments, the angle data list in the head movement data (that is, the angle data list collected by the inertial measurement unit in the first time sequence) can be expressed as the following formula:

[0150] ;

[0151] in, Represents the rotation of the head in the front-to-back direction, that is, the angle at which the head tilts left and right; is the rotation of the head in the horizontal plane, that is, the angle of the head turning left and right; is the trajectory of head movement; is a dimensional real number space.

[0152] The calculation formula for calculating the angle change within a short time (the first time sequence) through the time window is: AngleChange=AngleList[-1]-AngleList[-20].

[0153] An angle threshold Threshold1 is set. When the angle change satisfies the condition abs(AngleChange[i])>Threshold1, the head motion data collected within the first time series is valid data, where abs is a function for calculating absolute values, and i is an index of the user operation intention enumeration corresponding to the head.

[0154] In some embodiments, the present application processes laryngeal vibration information (i.e., laryngeal signal) by calculating the latest laryngeal vibration intensity at a sampling frequency of 16000 Hz, and the calculation formula for the laryngeal vibration intensity is: VibrationIntensity=sum(abs(VibrationData[-1024])), where sum is a summation function, abs is a function for calculating absolute values, and VibrationData is the laryngeal vibration data. When VibrationIntensity>Threshold2, it means that the laryngeal vibration data is valid data, that is, the laryngeal vibration data collected within the second time series is valid data, where Threshold2 is the sound threshold.

[0155] In some implementations, the final determination formula for the user intended operation is (VibrationIntensity>Threshold2)·(AngleChange[i]>Threshold1+AngleChange[i]<-Threshold1), where i is the index of the corresponding user intended operation enumeration.

[0156] The multimodal fusion human-computer interaction method provided in this application uses wearable devices as a method of information input. It can be applied to the challenges faced by human-computer interaction in special environments of land, water and air. It not only has the advantages of natural interaction, high degree of intelligence, low learning cost and meeting multi-task operation needs, but also has good accuracy, versatility and robustness.

[0157] In addition, this application uses head movement data and throat vibration data for intention recognition, without occupying the limbs of the operator (ie, the user), and can meet the multi-tasking needs of the operator. The application of multimodal fusion human-computer interaction methods will help reduce the attention and psychological burden of operators in special environments when interacting with robots, improve work efficiency in complex environments, enhance the intelligence level of special working environments, and ensure the personal safety of operators as much as possible.

[0158] like Figure 5 , which is a structural block diagram of a multi-modal fusion human-computer interaction device provided by the second aspect of the embodiment of the present application. Figure 5 In the embodiment, the multi-modal fusion human-computer interaction device 200 includes:

[0159] An information acquisition module 201 is used to acquire the user's head movement classification information, throat movement classification information and user status information;

[0160] A first user intention label obtaining module 202, configured to obtain a first user intention label based on the head action classification information, and add the first user intention label to a first stack;

[0161] A second user intention label obtaining module 203, configured to obtain a second user intention label based on the laryngeal motion classification information, and add the second user intention label to a second stack;

[0162] A user operation intention obtaining module 204, configured to obtain a user operation intention based on the first stack and the second stack when the first stack and / or the second stack is in an overflow state;

[0163] The first operation instruction obtaining module 205 is used to obtain a first operation instruction based on the user operation intention and the user status information, and send the first operation instruction to the machine.

[0164] like Figure 6 , which is a structural diagram of a wearable device provided in the third aspect of the embodiment of the present application. Figure 6 In the embodiment, the wearable device 300 includes: a camera 301, an inertial measurement unit 302, a vital sign sensor 303, a temperature sensor 304, a laryngeal vibration sensor 305, a processor (not shown in the figure) and a memory (not shown in the figure), the camera 301, the inertial measurement unit 302, the vital sign sensor 303, the temperature sensor 304 and the laryngeal vibration sensor 305 are all connected to the processor, the memory is used to store a computer program, the processor is used to call and run the computer program stored in the memory, and execute the steps of the multimodal fusion human-computer interaction method provided in the first aspect of the above-mentioned embodiment of the present application.

[0165] The inertial measurement unit 302 is used to be installed on the user's head, and the z-axis of the inertial measurement unit 302 is substantially perpendicular to the plane where the user's face is located.

[0166] When using the wearable device 300, the user should place the throat vibration sensor 305 close to the throat of the user. In addition, the camera 301, the inertial measurement unit 302, the vital sign sensor 303 and the temperature sensor 304 should be worn correctly, and the camera 301, the inertial measurement unit 302, the vital sign sensor 303, the temperature sensor 304 and the throat vibration sensor 305 should be calibrated.

[0167] The wearable device 300 can integrate the human-machine interaction device in a limited space, and occupies a small space. Moreover, the wearable device 300 has good compatibility with existing devices, and can expand the application scenarios of the wearable device.

[0168] In a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium is used to store a computer program, and the computer program enables a computer to execute the steps of the multimodal fusion human-computer interaction method provided in the first aspect of the embodiment of the present application.

[0169] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0170] The technical features of the above embodiments can be combined without changing the basic principles of the present application. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0171] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the scope of patent protection of the present application shall be subject to the attached claims.

Claims

1. A multi-modal fusion human-computer interaction method, characterized in that: include: Obtaining the user's head movement classification information, throat movement classification information and user status information; Obtaining a first user intention label based on the head action classification information, and adding the first user intention label to a first stack; Obtaining a second user intention label based on the laryngeal motion classification information, and adding the second user intention label to a second stack; When the first stack and / or the second stack is in an overflow state, obtaining a user operation intention based on the first stack and the second stack; Obtaining a first operation instruction based on the user operation intention and the user state information, and issuing the first operation instruction to the underwater robot; When the first stack is in an overflow state, checking whether all first user intention labels in the first stack are the same, and if they are the same, judging the user operation intention based on the first user intention label at this time; If they are not the same, the user operation intention is not determined based on the first user intention label at this time; When the second stack is in an overflow state, checking whether all second user intention labels in the second stack are the same, and if they are the same, judging the user operation intention based on the second user intention label at this time; If they are not the same, the second user intention label at this time is not used to determine the user operation intention; The step of obtaining the user's head movement classification information includes: collecting the user's head posture information based on a first time series to obtain head movement data; filtering the head movement data to obtain first data to be processed; segmenting the first data to be processed using a segmentation method to obtain second data to be processed; characterizing the second data to be processed based on preset features to obtain third data to be processed; and comparing the third data to be processed with a head movement template using a dynamic time warping algorithm to obtain the head movement classification information; The step of obtaining the laryngeal motion classification information of the user includes: collecting the laryngeal vibration information of the user based on the second time series to obtain laryngeal vibration data; performing audio signal processing on the laryngeal vibration data to obtain fourth data to be processed; performing discrete Fourier transform on the fourth data to be processed to obtain spectrum information; performing feature extraction on the spectrum information to obtain fifth data to be processed; using a neural network to classify the fifth data to be processed to obtain the laryngeal motion classification information; the neural network is a long short-term memory network; Based on the preset shared function, determine whether to use the dominant control system to automatically generate a third operation instruction, and send the third operation instruction to the underwater robot; The preset sharing function is defined as: ;in, The input corresponding to the user intention, that is, the user operation intention; It is the input of the dominant control system; It is the input passed to the instruction map by the user end of the human-computer interaction interface; The values ​​are as follows: ； in, The values ​​of are as follows: ； in, is the user's state space; The user's safe space, that is, the space corresponding to the normal state of the user status information; The dangerous space of the user, that is, the space corresponding to the dangerous state of the user status information; It is the user's intermediate state space, that is, the space corresponding to the user's state information being in a stuck state.

2. The multimodal fusion human-computer interaction method according to claim 1, characterized in that: The step of obtaining the user status information of the user comprises: Obtain the user's vital signs data and external environment information; The user status information is obtained based on the vital sign data and the external environment information.

3. The multimodal fusion human-computer interaction method according to claim 2, characterized in that: After the step of obtaining the user status information based on the vital sign data and the external environment information, the step of obtaining the user status information of the user further includes: Cross-validating the vital sign data and the external environment information to obtain a validation result; The user status information is modified based on the verification result.

4. The multimodal fusion human-computer interaction method according to claim 1, characterized in that: After the step of obtaining the user's head movement classification information, throat movement classification information and user status information, the multimodal fusion human-computer interaction method further includes: When the user status information is in a dangerous state, automatically generating a second operation instruction using a dominant control system, and sending the second operation instruction to the underwater robot; Before or after the second operation instruction is issued, the dominant control system generates a first escape-from-danger indication to prompt the user to escape from the dangerous state.

5. The multimodal fusion human-computer interaction method according to claim 1, characterized in that: After the step of obtaining the user's head movement classification information, throat movement classification information and user status information, the multimodal fusion human-computer interaction method also includes: before the third operation instruction is issued or after the third operation instruction is issued, the dominant control system generates a second escape from danger indication to prompt the user to escape from the dangerous state.

6. A multi-modal fusion human-computer interaction device, characterized in that: include: An information acquisition module, used to acquire the user's head movement classification information, throat movement classification information and user status information; A first user intention label obtaining module, configured to obtain a first user intention label based on the head action classification information, and add the first user intention label to a first stack; A second user intention label obtaining module, configured to obtain a second user intention label based on the laryngeal motion classification information, and add the second user intention label to a second stack; A user operation intention obtaining module, configured to obtain a user operation intention based on the first stack and the second stack when the first stack and / or the second stack is in an overflow state; A first operation instruction obtaining module, configured to obtain a first operation instruction based on the user operation intention and the user state information, and send the first operation instruction to the machine; The multimodal fusion human-computer interaction device is further used to check whether all first user intention labels in the first stack are the same when the first stack is in an overflow state, and if they are the same, use the first user intention label at this time to determine the user operation intention; If they are not the same, the user operation intention is not determined based on the first user intention label at this time; When the second stack is in an overflow state, checking whether all second user intention labels in the second stack are the same, and if they are the same, judging the user operation intention based on the second user intention label at this time; If they are not the same, the second user intention label at this time is not used to determine the user operation intention; The multi-modal fusion human-computer interaction device is also used to collect the user's head posture information based on the first time series to obtain head movement data; filter the head movement data to obtain first data to be processed; Using a segmentation method to segment the first data to be processed to obtain second data to be processed; Characterizing the second data to be processed based on preset features to obtain third data to be processed; Using a dynamic time warping algorithm to compare the third data to be processed with the head movement template to obtain the head movement classification information; The multimodal fusion human-computer interaction device is further used to collect the laryngeal vibration information of the user based on the second time series to obtain laryngeal vibration data; perform audio signal processing on the laryngeal vibration data to obtain fourth data to be processed; Performing discrete Fourier transform on the fourth data to be processed to obtain frequency spectrum information; performing feature extraction on the frequency spectrum information to obtain fifth data to be processed; using a neural network to classify the fifth data to be processed to obtain the laryngeal movement classification information; the neural network is a long short-term memory network; The multi-modal fusion human-computer interaction device is also used to determine whether to use the dominant control system to automatically generate a third operation instruction based on a preset shared function, and send the third operation instruction to the underwater robot; The preset sharing function is defined as: ;in, The input corresponding to the user intention, that is, the user operation intention; It is the input of the dominant control system; It is the input passed to the instruction map by the user end of the human-computer interaction interface; The values ​​are as follows: ； in, The values ​​of are as follows: ； in, is the user's state space; The user's safe space, that is, the space corresponding to the normal state of the user status information; The dangerous space of the user, that is, the space corresponding to the dangerous state of the user status information; It is the user's intermediate state space, that is, the space corresponding to the user's state information being in a stuck state.

7. A wearable device, characterized in that: include: A camera, an inertial measurement unit, a vital sign sensor, a temperature sensor, a laryngeal vibration sensor, a processor and a memory, wherein the camera, the inertial measurement unit, the vital sign sensor, the temperature sensor and the laryngeal vibration sensor are all connected to the processor, the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the steps of the multimodal fusion human-computer interaction method described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein the computer program enables a computer to execute the steps of the multimodal fusion human-computer interaction method according to any one of claims 1 to 5.

Images