Command recognition method, apparatus and device

Abstract

This disclosure provides a command recognition method, apparatus, and device, relating to the field of head-mounted device technology. The command recognition method includes: acquiring bioelectrical signals via electrodes on a head-mounted device and acquiring motion signals via an inertial measurement unit on the head-mounted device; determining whether to trigger a recognition operation based on the bioelectrical signals and motion signals; in response to triggering the recognition operation, matching the bioelectrical signals with reference bioelectrical signals of each candidate command from a plurality of candidate commands to obtain a first confidence level for each candidate command, and matching the motion signals with reference motion signals of each candidate command from the plurality of candidate commands to obtain a second confidence level for each candidate command; and determining a target command from the plurality of candidate commands based on the first and second confidence levels of each candidate command.

Description

Technical Field

[0001] This disclosure relates to the field of head-mounted device technology, and more specifically, to a command recognition method, apparatus, and device. Background Technology

[0002] With the rapid development of information technology, some head-mounted devices can be equipped with intelligent control functions, so that users can control the head-mounted devices to perform corresponding operations through corresponding commands. Summary of the Invention

[0003] In related technologies, the voice control function of head-mounted devices faces significant privacy pressures in scenarios such as meetings, subways, and offices, and is greatly affected by noise, resulting in poor command recognition accuracy.

[0004] To address the aforementioned problems, the present disclosure proposes the following solutions.

[0005] According to a first aspect of this disclosure, a command recognition method is provided, comprising: acquiring bioelectrical signals via electrodes on a head-mounted device and acquiring motion signals via an inertial measurement unit on the head-mounted device; determining whether to trigger an execution of a recognition operation based on the bioelectrical signals and the motion signals; in response to triggering the execution of the recognition operation, matching the bioelectrical signals with reference bioelectrical signals of each candidate command among a plurality of candidate commands to obtain a first confidence level for each candidate command, and matching the motion signals with reference motion signals of each candidate command among the plurality of candidate commands to obtain a second confidence level for each candidate command; and determining a target command from the plurality of candidate commands based on the first confidence level and the second confidence level for each candidate command.

[0006] In some embodiments, determining whether to trigger an identification operation based on bioelectrical signals and motion signals includes: in response to a motion signal triggering a first identification request, recording a first start time of triggering the first identification request; in response to a bioelectrical signal triggering a second identification request, recording a second start time of triggering the second identification request; and in response to the absolute value of the difference between the first start time and the second start time being less than a time threshold, determining that an identification operation is triggered.

[0007] In some embodiments, in response to a motion signal triggering a first identification request, bioelectrical signals after a first start time are also collected to determine whether the bioelectrical signals after the first start time trigger a second identification request.

[0008] In some embodiments, the command recognition method further includes: determining that a motion signal triggers a first recognition request in response to at least one of acceleration in the motion signal being greater than an acceleration threshold and angular velocity in the motion signal being greater than an angular velocity threshold; and / or determining that a bioelectric signal triggers a second recognition request in response to the envelope amplitude of the bioelectric signal being greater than an amplitude threshold.

[0009] In some embodiments, determining whether to trigger the execution of an identification operation based on bioelectrical signals and motion signals includes: determining whether at least one of the bioelectrical signals and motion signals indicates a non-command action; and determining not to trigger the execution of an identification operation in response to at least one of the bioelectrical signals and motion signals indicating a non-command action.

[0010] In some embodiments, the command recognition method further includes: for each candidate command among a plurality of candidate commands, outputting prompt information indicating the configuration of the candidate command to the user; collecting motion signals generated by the user's action in response to the prompt information as reference motion signals for the candidate command; and collecting bioelectrical signals generated by the user in response to the prompt information as reference bioelectrical signals for the candidate command.

[0011] In some embodiments, determining the target command from the plurality of candidate commands based on a first confidence level and a second confidence level of each candidate command includes: obtaining a fused confidence level of each candidate command based on the first confidence level and the second confidence level of each candidate command; and determining the candidate command corresponding to the maximum fused confidence level as the target command in response to the maximum fused confidence level being greater than or equal to a first threshold.

[0012] In some embodiments, determining the target command from the plurality of candidate commands based on a first confidence level and a second confidence level of each candidate command further includes: generating and providing confirmation information indicating the candidate command corresponding to the maximum fusion confidence level in response to a maximum fusion confidence level being greater than a second threshold and less than a first threshold; and determining the candidate command corresponding to the maximum fusion confidence level as the target command in response to a confirmation operation on the confirmation information.

[0013] In some embodiments, the command recognition method further includes: obtaining a contact quality score between the head-mounted device and the user; obtaining a fusion confidence score of each candidate command based on a first confidence score and a second confidence score of each candidate command includes: for each candidate command, performing a weighted summation of the contact quality score, the first confidence score of the candidate command, and the second confidence score of the candidate command to obtain a fusion confidence score of each candidate command.

[0014] In some embodiments, the contact quality score is obtained based on at least one of the contact impedance between the electrode and the user, the wearing pressure exerted by the head-mounted device on the contact area with the user, and the signal-to-noise ratio of the bioelectrical signal.

[0015] In some embodiments, the command recognition method further includes: determining a weight coefficient corresponding to a first confidence level based on the user's scenario, wherein the weight coefficient corresponding to the first confidence level when the user is in a static scenario is greater than the weight coefficient corresponding to the first confidence level when the user is in a dynamic scenario; and / or determining a weight coefficient corresponding to a second confidence level based on the user's scenario, wherein the weight coefficient corresponding to the second confidence level when the user is in a static scenario is less than the weight coefficient corresponding to the second confidence level when the user is in a dynamic scenario.

[0016] In some embodiments, the head-mounted device is smart glasses, and / or the bioelectrical signal is an electromyographic signal.

[0017] According to a second aspect of this disclosure, a command recognition device is provided, comprising: a acquisition unit configured to acquire bioelectrical signals via electrodes on a head-mounted device and to acquire motion signals via an inertial measurement unit on the head-mounted device; a first determination unit configured to determine whether to trigger a recognition operation based on the bioelectrical signals and the motion signals; a matching unit configured to, in response to triggering the recognition operation, match the bioelectrical signals with reference bioelectrical signals of each candidate command among a plurality of candidate commands to obtain a first confidence level for each candidate command, and match the motion signals with reference motion signals of each candidate command among the plurality of candidate commands to obtain a second confidence level for each candidate command; and a second determination unit configured to determine a target command from the plurality of candidate commands based on the first confidence level and the second confidence level of each candidate command.

[0018] According to a third aspect of this disclosure, a command recognition device is provided, comprising: a memory; and a processor coupled to the memory, configured to execute the command recognition method described in any of the above embodiments based on instructions stored in the memory.

[0019] According to a fourth aspect of this disclosure, a head-mounted device is provided, comprising: a command recognition device as described above or a command recognition device as described above.

[0020] According to a fifth aspect of this disclosure, a computer-readable storage medium is provided, including a computer program, wherein when the computer program is executed by a processor, it implements the steps of the command recognition method described in any of the above embodiments.

[0021] According to a sixth aspect of this disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, it implements the steps of the command recognition method described in any of the above embodiments.

[0022] In this embodiment of the disclosure, by comprehensively considering bioelectrical signals and motion signals, the confidence of each candidate command is obtained more accurately using a first confidence level based on bioelectrical signals and a second confidence level based on motion signals. This allows for more accurate identification of the target command, reduces the possibility of misidentified commands, improves the accuracy of silent command recognition, and enhances the user experience.

[0023] Other features and advantages of this disclosure will become clearer from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0024] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the specification, serve to explain the principles of this disclosure.

[0025] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein: Figure 1 A flowchart illustrating a command recognition method according to some embodiments of the present disclosure is shown; Figure 2 A schematic diagram of smart glasses according to some embodiments of the present disclosure is shown; Figure 3 A flowchart illustrating the process of determining whether to trigger an identification operation according to some embodiments of the present disclosure is shown; Figure 4 A schematic diagram of a command recognition device according to some embodiments of the present disclosure is shown; Figure 5 A schematic diagram of a command recognition device according to some embodiments of the present disclosure is shown; Figure 6 A schematic block diagram of a computer system on which embodiments of the present disclosure may be implemented is shown.

[0026] For ease of understanding, the positions, dimensions, and extents of the structures shown in the accompanying drawings and other materials may not represent actual positions, dimensions, and extents. Therefore, the disclosed invention is not limited to the positions, dimensions, and extents disclosed in the accompanying drawings and other materials. Furthermore, the drawings are not necessarily drawn to scale, and some features may be enlarged to show details of specific components. Detailed Implementation

[0027] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0028] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the scope of this disclosure or its application or use. That is, the structures and methods herein are shown in an exemplary manner to illustrate different embodiments of the structures and methods in this disclosure. However, those skilled in the art will understand that they merely illustrate exemplary ways that can be used to implement this disclosure, and not exhaustive ways. Furthermore, the drawings are not necessarily drawn to scale, and some features may be enlarged to show details of specific components.

[0029] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0030] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0031] In related technologies, the voice control function of head-mounted devices faces significant privacy pressures in scenarios such as meetings, subways, and offices, and is greatly affected by noise, resulting in poor command recognition accuracy.

[0032] To address the aforementioned issues, this disclosure provides a command recognition method, apparatus, and device. By comprehensively considering bioelectrical signals and motion signals, the confidence level of each candidate command is obtained more accurately based on a first confidence level based on bioelectrical signals and a second confidence level based on motion signals. This allows for more accurate identification of target commands, reduces the possibility of misidentified commands, improves the accuracy of silent command recognition, and enhances the user experience.

[0033] like Figure 1 As shown, the command recognition method according to some embodiments of this disclosure may include: Step S110: Bioelectric signals are collected through electrodes on the head-mounted device, and motion signals are collected through the inertial measurement unit on the head-mounted device.

[0034] Here, a head-mounted device can be worn on a user's head, and electrodes on the head-mounted device can be used to contact designated locations on the user's head to collect bioelectrical signals at those locations. For example, electrodes placed at corresponding locations on the head-mounted device can collect bioelectrical signals from the user's temporalis muscle region, preauricular region, nose pad contact surface region, bridge of the nose contact surface region, and mastoid process region behind the ear.

[0035] In some embodiments, the head-mounted device may include smart glasses. As some implementations, electrodes may be disposed on at least one of the temples, bridge of the nose, nose pads below the bridge of the nose, and ear hooks at the ends of the temples of the smart glasses. Figure 2 As shown, electrodes 210 can be disposed on the left temple 220 and the right temple 230 of the smart glasses 200. Specifically, electrodes 210 can be disposed at the ends of the left temple 220 and the right temple 230. Alternatively, electrodes 210 can be disposed on the bridge 240, the left nose pad 250, and the right nose pad 260 of the smart glasses 200.

[0036] It should be understood that in some embodiments, the head-mounted device may also include other wearable head-mounted devices, such as headphones.

[0037] In some embodiments, the bioelectrical signal may include electromyography (EMG) signals. It should be understood that in some embodiments, the bioelectrical signal may also include other electrical signals, such as electrooculography (EOG) signals or electroencephalography (EEG) signals. When a user issues a silent lip-reading command or a quiet command, it causes a change in the bioelectrical signal at the corresponding location on the head, thereby enabling the recognition of the user's silent lip-reading command or quiet command based on the bioelectrical signal. In this way, it is possible to control the head-mounted device based on silent lip-reading commands or quiet commands, without needing to speak to control the head-mounted device to perform corresponding functions, effectively improving the command recognition accuracy in noisy environments.

[0038] In some embodiments, an inertial measurement unit (IMU) can be integrated into a head-mounted device to acquire motion signals of the user's head when the user wears the device. In this disclosure, the position of the IMU on the head-mounted device is not restricted. Here, the motion signals may include acceleration and angular velocity; thus, the acceleration and angular velocity of the user's head can be acquired by the IMU on the head-mounted device. By acquiring the corresponding motion signals, it is possible to detect whether the user's head has performed the specified action corresponding to the control command, thereby enabling head-mounted device control based on the specified head action, without needing to issue control sounds to control the head-mounted device to perform corresponding functions, achieving silent command control and effectively improving the command recognition accuracy in noisy environments. As some implementations, motion signals and bioelectrical signals can be combined for command recognition, reducing the possibility of misjudgment of control commands due to considering only motion signals or bioelectrical signals, and improving the recognition accuracy of silent commands.

[0039] return Figure 1 The command recognition method according to some embodiments of this disclosure may further include: Step S120: Determine whether to trigger the recognition operation based on the bioelectrical signal and the motion signal.

[0040] In this way, it is possible to first determine whether to trigger the recognition operation based on bioelectrical signals and motion signals, and then execute the subsequent recognition operation if the determination is yes, thereby improving the reliability of command recognition.

[0041] like Figure 3 As shown, in some embodiments, determining whether to trigger a recognition operation based on bioelectrical signals and motion signals may include: Step S310: In response to the motion signal triggering the first recognition request, record the first start time of triggering the first request; Step S320: In response to the bioelectric signal triggering the second identification request, record the second start time of triggering the second request; Step S330: In response to the absolute value of the difference between the first start time and the second start time being less than a time threshold, it is determined that the recognition operation is triggered.

[0042] Here, the first identification request can indicate that the currently acquired motion signal meets the triggering conditions of the candidate command identification event, and the second identification request can indicate that the currently acquired bioelectrical signal meets the triggering conditions of the candidate command identification event. By recording the first start time when the motion signal meets the triggering conditions of the candidate command identification event and the second start time when the bioelectrical signal meets the triggering conditions of the candidate command identification event, it can be determined whether the motion signal and the bioelectrical signal correspond or align in time. Specifically, if the absolute value of the difference between the first start time and the second start time is less than a time threshold, the motion signal and the bioelectrical signal can be considered to correspond or align in time. Thus, when both the motion signal and the bioelectrical signal meet the corresponding triggering conditions of the candidate command identification event and correspond in time, it can be determined to trigger the identification operation, reducing the possibility of the motion signal or bioelectrical signal erroneously triggering the identification operation, thereby improving the reliability of command identification.

[0043] As one implementation, the time threshold may be, for example, 200 milliseconds.

[0044] In some embodiments, a motion signal is determined to trigger a first identification request in response to at least one of an acceleration in the motion signal being greater than an acceleration threshold and an angular velocity in the motion signal being greater than an angular velocity threshold. As some implementations, a motion signal is determined to trigger a first identification request in response to an acceleration in the motion signal being greater than an acceleration threshold. Thus, if the acceleration is greater than the acceleration threshold, the motion signal can be considered to meet the triggering condition of a candidate command identification event, thereby determining that the motion signal triggers the first identification request. As some implementations, a motion signal is determined to trigger a first identification request in response to an angular velocity in the motion signal being greater than an angular velocity threshold. Thus, if the angular velocity is greater than the angular velocity threshold, the motion signal can be considered to meet the triggering condition of a candidate command identification event, thereby determining that the motion signal triggers the first identification request. As some implementations, a motion signal is determined to trigger a first identification request in response to both an acceleration in the motion signal being greater than an acceleration threshold and an angular velocity greater than an angular velocity threshold. Thus, if both the acceleration and angular velocity are greater than the acceleration threshold, the motion signal can be considered to meet the triggering condition of a candidate command identification event, thereby determining that the motion signal triggers the first identification request.

[0045] In some embodiments, a second identification request is determined to be triggered by a bioelectric signal if the envelope amplitude of the bioelectric signal is greater than an amplitude threshold. Here, the envelope can be obtained by processing the original bioelectric signal (e.g., moving root mean square processing, Hilbert transform processing), where the envelope amplitude can be used to indicate the strength of the bioelectric signal. If the envelope amplitude of the bioelectric signal is greater than the amplitude threshold, the bioelectric signal can be considered to meet the triggering condition of a candidate command identification event, thus determining that the bioelectric signal triggers the second identification request.

[0046] In some embodiments, determining whether to trigger the recognition operation based on bioelectrical signals and motion signals may include: determining not to trigger the recognition operation in response to the absolute value of the difference between a first start time and a second start time being greater than or equal to a time threshold. Specifically, if the absolute value of the difference between the first start time and the second start time is greater than or equal to the time threshold, even if the bioelectrical signals and motion signals respectively meet the corresponding triggering conditions of the candidate command recognition event, but the bioelectrical signals and motion signals are not aligned or correspond in time, then subsequent recognition operations may not be triggered. That is, the first confidence score and second confidence score of each candidate command among multiple candidate commands may not be generated subsequently, and the target command may not be determined from multiple candidate commands. In this way, the possibility of falsely triggering command recognition can be further reduced, and resource consumption can be reduced.

[0047] In some embodiments, in response to a motion signal triggering a first recognition request, bioelectrical signals after a first start time are also collected to determine whether the bioelectrical signals after the first start time trigger a second recognition request. Specifically, in response to a motion signal triggering a first recognition request, the first start time of triggering the first request is recorded, and bioelectrical signals after the first start time are collected. For example, a user can first control their head to perform a specified action and then perform corresponding silent lip movements to trigger the recognition operation, so that the corresponding function can be controlled and executed based on the recognized target command. Thus, when the user's head is detected to be performing a specified action and triggering the first recognition request through motion signals, the collected bioelectrical signals are more likely to trigger the second recognition request, thereby improving the efficiency and accuracy of command recognition.

[0048] In some embodiments, determining whether to trigger the execution of an identification operation based on bioelectrical signals and motion signals may include: determining whether at least one of the bioelectrical signals and motion signals indicates a non-command action; and determining not to trigger the execution of an identification operation in response to at least one of the bioelectrical signals and motion signals indicating a non-command action.

[0049] In some implementations, a motion signal is determined to indicate a non-command action when the acceleration in the motion signal is greater than a preset acceleration threshold and / or the angular velocity in the motion signal is greater than a preset angular velocity threshold. Thus, in cases where the user's head undergoes significant movement (such as rapid head turning or significant head nodding), it can be confirmed that the motion signal indicates a non-command action. Here, the preset acceleration threshold is greater than the acceleration threshold used to determine whether a first recognition request has been triggered, and the preset angular velocity threshold is greater than the angular velocity threshold used to determine whether a first recognition request has been triggered. Thus, a motion signal is determined to trigger a first recognition request when the acceleration in the motion signal is greater than the acceleration threshold and less than or equal to the preset acceleration threshold; or, a motion signal is determined to trigger a first recognition request when the angular velocity in the motion signal is greater than the acceleration threshold and less than or equal to the preset angular velocity threshold, and the acceleration in the motion signal is greater than the acceleration threshold and less than or equal to the preset acceleration threshold.

[0050] As some implementation methods, feature extraction can be performed using bioelectric signals, and the extracted features can be used to determine whether the bioelectric signal corresponds to a specified facial expression (e.g., frowning, opening the mouth). In response to the bioelectric signal corresponding to the specified facial expression, it can be determined that the bioelectric signal indicates a non-command action.

[0051] In the above embodiments, by identifying non-command actions, the possibility of erroneously triggering the identification operation can be reduced, the reliability of command identification can be improved, and subsequent identification operations can be avoided when a non-command action is identified, thereby reducing resource consumption.

[0052] like Figure 1 As shown, the command recognition method according to some embodiments of this disclosure may further include: In step S130, in response to triggering the recognition operation, the bioelectric signal is matched with the reference bioelectric signal of each candidate command among the multiple candidate commands to obtain the first confidence level of each candidate command, and the motion signal is matched with the reference motion signal of each candidate command among the multiple candidate commands to obtain the second confidence level of each candidate command.

[0053] Here, the multiple candidate commands can be pre-defined control commands used to control the head-mounted device to perform corresponding control functions. Candidate commands may include, for example, confirmation, cancellation, page turning, back, enabling a specified function, or disabling a specified function, without limitation.

[0054] In some embodiments, reference bioelectrical signals and reference motion signals for each candidate command can be acquired in advance. As some implementations, the reference bioelectrical signals and reference motion signals for each candidate command can be acquired for individual user needs. Specifically, the command recognition method according to some embodiments of this disclosure may include: for each candidate command among a plurality of candidate commands, outputting prompt information instructing the user to configure the candidate command; collecting motion signals generated by the user's actions in response to the prompt information as reference motion signals for the candidate command; and collecting bioelectrical signals generated by the user in response to the prompt information as reference bioelectrical signals for the candidate command.

[0055] The prompts can be projected onto the display device (e.g., eyeglass lenses) of the head-mounted device using appropriate projection technology to prompt the user to configure the corresponding candidate command. Alternatively, the prompts can be provided to the user as audio via the sound output device (e.g., a speaker) of the head-mounted device to prompt the user to configure the corresponding candidate command. Upon receiving the prompts, the user can perform an action corresponding to the candidate command (e.g., nodding or lip-syncing), thereby allowing the acquisition of reference motion signals and reference bioelectrical signals from the user in response to the corresponding candidate command.

[0056] In the above embodiments, by collecting reference motion signals and reference bioelectric signals of the user for the corresponding candidate commands, personalized configuration can be performed for the user to improve the command recognition accuracy for that user.

[0057] In some embodiments, in response to a user starting to wear the head-mounted device, an operation is performed to execute each of a plurality of candidate commands and output a prompt message to the user instructing them to configure the candidate command. Here, the user starting to wear the head-mounted device could be, for example, the user wearing the head-mounted device for the first time, or the user putting the head-mounted device back on after removing it. Thus, when a user starts wearing the head-mounted device, the corresponding candidate commands can be configured personalized for the user.

[0058] In some embodiments, in response to detecting that the accuracy of command recognition is less than a preset accuracy threshold, an operation is performed to output a prompt message to the user instructing them to configure the candidate command for each of the multiple candidate commands. Thus, when the accuracy of command recognition is poor, corresponding candidate commands can be personalized for the user to improve the command recognition accuracy.

[0059] In some embodiments, feature extraction can be performed on the bioelectric signal to match the extracted features with the features of the reference bioelectric signal for each candidate command, thereby obtaining a first confidence level for each candidate command. As some implementations, at least one of the time-domain and frequency-domain features of the bioelectric signal can be extracted to match the features with the reference bioelectric signal for each candidate command, thus obtaining the corresponding first confidence level. For example, at least one of the root mean square (RMS), zero-crossing rate, waveform length, peak amplitude, spectral energy, spectral centroid, spectral bandwidth, and wavelet coefficient energy distribution can be extracted to form a feature vector, which is then used to calculate the similarity with the feature vector of the reference bioelectric signal for each candidate command, thereby obtaining the first confidence level for each candidate command. It should be understood that other matching methods can also be used to obtain the first confidence level for each candidate command, such as dynamic time warping methods, which are not limited here.

[0060] In some embodiments, feature extraction can be performed on the motion signal to match the extracted features with the features of the reference motion signal of each candidate command, thereby obtaining a second confidence level for each candidate command. As some implementations, at least one of head movement amplitude, head movement direction, and head movement duration can be extracted from the motion signal to generate a feature vector of the motion signal, and then a similarity calculation can be performed with the feature vector of the reference motion signal of each candidate command to obtain a second confidence level for each candidate command. It should be understood that other matching methods can also be used to obtain the second confidence level for each candidate command, such as obtaining a head movement pattern indicated by the motion signal based on at least one of the head movement amplitude, head movement direction, and head movement duration extracted from the motion signal, and matching the head movement pattern indicated by the motion signal with the head movement pattern indicated by the reference motion signal of each candidate command to obtain a second confidence level for each candidate command; this is not limited to these methods.

[0061] like Figure 1 As shown, the command recognition method according to some embodiments of this disclosure may further include: Step S140: Determine the target command from multiple candidate commands based on the first confidence level and the second confidence level of each candidate command.

[0062] In some embodiments, determining a target command from multiple candidate commands based on a first confidence level and a second confidence level of each candidate command may include: obtaining a fused confidence level for each candidate command based on the first confidence level and the second confidence level of each candidate command; and determining the candidate command corresponding to the maximum fused confidence level as the target command in response to a maximum fused confidence level being greater than or equal to a first threshold. As some implementations, obtaining the fused confidence level of each candidate command based on the first confidence level and the second confidence level of each candidate command may include: for each candidate command, performing a weighted sum of the first confidence level and the second confidence level of that candidate command to obtain the fused confidence level of each candidate command. Thus, the confidence level of each candidate command can be obtained by fusing a first confidence level based on bioelectrical signals and a second confidence level based on motion signals.

[0063] In the above embodiments, the target command is determined from multiple candidate commands based on a first confidence level based on bioelectrical signals and a second confidence level based on motion signals. This allows for more accurate identification of user commands, enabling the execution of control functions corresponding to the target command. In particular, it allows for more accurate identification of silent user commands, enabling the corresponding control functions to be implemented without the user uttering a sound, thus reducing the user's privacy concerns.

[0064] Considering that poor contact quality between the head-mounted device and the user may reduce the reliability of the collected bioelectrical and motion signals, leading to a decrease in the accuracy of command recognition, in some embodiments, the target command can also be determined based on the contact quality score between the head-mounted device and the user.

[0065] Specifically, the command recognition method according to some embodiments of this disclosure may further include: obtaining a contact quality score between the head-mounted device and the user; obtaining a fused confidence score for each candidate command based on a first confidence score and a second confidence score may include: for each candidate command, performing a weighted summation of the contact quality score, the first confidence score of the candidate command, and the second confidence score of the candidate command to obtain a fused confidence score for each candidate command. As some implementations, the sum of the weight coefficients corresponding to the contact quality score, the first confidence score, and the second confidence score is 1.

[0066] In the above embodiments, the contact quality between the head-mounted device and the user is also considered, so as to obtain the fusion confidence of each candidate command based on the contact quality score, thereby further improving the accuracy of command recognition.

[0067] As some implementations, the contact quality score can be obtained based on at least one of the contact impedance between the electrode and the user, the wearing pressure exerted by the head-mounted device on the contact area with the user, and the signal-to-noise ratio of the bioelectrical signal. As some implementations, the contact quality score can be obtained by weighted summing of a first score corresponding to the contact impedance, a second score corresponding to the wearing pressure, and a third score corresponding to the signal-to-noise ratio. For example, the contact quality score Q can be obtained based on the following formula: Q = w1 × Q1 + w2 × Q2 + w3 × Q3; Here, Q1 represents the first score corresponding to contact impedance, Q2 represents the second score corresponding to wearing pressure, Q3 represents the third score corresponding to signal-to-noise ratio, and w1, w2, and w3 represent the corresponding weighting coefficients. As some implementations, w1 = 0.4, w2 = 0.3, and w3 = 0.3.

[0068] In some implementations, contact impedance can be negatively correlated with the first score; for example, the lower the contact impedance, the higher the first score. Here, a small AC signal can be applied to the electrodes, and the voltage response measured, then the contact impedance calculated according to Ohm's law. Wearing pressure can be used to indicate the fit between the head-mounted device and the user's head. In some implementations, wearing pressure can be measured by a pressure sensor on the head-mounted device, or it can be indirectly calculated from the contact impedance. Here, a moderate wearing pressure (e.g., close to a specified pressure value) results in a higher second score, while excessively low or high wearing pressure (e.g., far from a specified pressure value) results in a lower second score. In some implementations, the signal-to-noise ratio (SNR) of the bioelectrical signal can be positively correlated with the third score; for example, the higher the SNR of the bioelectrical signal, the higher the third score.

[0069] In the above embodiments, by comprehensively considering wearing pressure, contact impedance and signal-to-noise ratio of bioelectric signals, the contact quality between the head-mounted device and the user can be evaluated more accurately to obtain a more accurate contact quality score, thereby obtaining a more accurate fusion confidence of each candidate command and improving the accuracy of command recognition.

[0070] In some embodiments, if the maximum fusion confidence is less than a first threshold, a downgraded identification or rejection of the current command identification operation can be triggered to ensure the accuracy of command identification.

[0071] Specifically, in some embodiments, determining the target command from multiple candidate commands based on a first confidence level and a second confidence level may include: refusing to perform the identification operation in response to a maximum fusion confidence level being less than or equal to a second threshold. Here, the second threshold may be less than the first threshold. Thus, if the maximum fusion confidence level is too low, the reliability of the identification result is low, which may easily lead to misidentification. Therefore, the identification operation may be refused, meaning the target command will not be determined from the multiple candidate commands in this instance, and the corresponding bioelectrical and motion signals can be re-acquired for the next judgment and calculation.

[0072] In some embodiments, determining the target command from multiple candidate commands based on a first confidence level and a second confidence level for each candidate command may include: generating and providing confirmation information indicating the candidate command corresponding to the maximum fusion confidence level in response to a maximum fusion confidence level greater than a second threshold and less than a first threshold; and determining the candidate command corresponding to the maximum fusion confidence level as the target command in response to a confirmation operation on the confirmation information. As some implementations, in response to a rejection operation on the confirmation information, additional confirmation information indicating the candidate command corresponding to the second-highest fusion confidence level may be generated and provided, and the candidate command corresponding to the second-highest fusion confidence level may be determined as the target command in response to a confirmation operation on the additional confirmation information. Thus, when the maximum fusion confidence level is greater than the second threshold and less than the first threshold, secondary confirmation can be performed, for example, by the user confirming through corresponding head movements or through a terminal device communicatively connected to the head-mounted device, thereby improving the reliability of command recognition.

[0073] Considering that the reliability of bioelectrical signals collected by users is high in static scenarios but low in dynamic scenarios, in some embodiments, the weight coefficient corresponding to the first confidence level or the weight coefficient corresponding to the second confidence level can be dynamically adjusted according to the user's scenario.

[0074] Specifically, the command recognition method according to some embodiments of this disclosure may include: determining the user's scene based on motion signals. Here, the scene may include static scenes and dynamic scenes. Static scenes may be, for example, scenes where the user's position and posture do not change or remain substantially unchanged, such as sitting, standing, or lying down. Dynamic scenes may be, for example, scenes where at least one of the user's position and posture changes, such as walking or running.

[0075] In some embodiments, the weight coefficient corresponding to the first confidence level can be determined based on the user's scenario, wherein the weight coefficient corresponding to the first confidence level when the user is in a static scenario is greater than the weight coefficient corresponding to the first confidence level when the user is in a dynamic scenario. Thus, the weight coefficient corresponding to the first confidence level can be adjusted based on the user's scenario.

[0076] In some embodiments, the weight coefficient corresponding to the second confidence level can be determined based on the user's scenario, wherein the weight coefficient corresponding to the second confidence level when the user is in a static scenario is less than the weight coefficient corresponding to the second confidence level when the user is in a dynamic scenario. Thus, the weight coefficient corresponding to the second confidence level can be adjusted based on the user's scenario. As some implementations, the weight coefficient corresponding to the second confidence level can be determined to be zero.

[0077] In the above embodiments, at least one of the weight coefficients corresponding to the first confidence level and the second confidence level can be adaptively adjusted to adapt to changes in the user's scenario, thereby improving the command recognition accuracy in various scenarios.

[0078] Embodiments of this disclosure also provide a command recognition device that can be configured to execute the command recognition method of any of the above embodiments.

[0079] In some embodiments, the command recognition device may include a unit for performing the command recognition method of any of the above embodiments.

[0080] Figure 4 A schematic diagram of a command recognition device according to some embodiments of the present disclosure is shown. For example... Figure 4 As shown, the command recognition device 400 may include a collection unit 410, a first determination unit 420, a matching unit 430, and a second determination unit 440.

[0081] The acquisition unit 410 can be configured to acquire bioelectrical signals via electrodes on a head-mounted device and to acquire motion signals via an inertial measurement unit on the head-mounted device. The first determination unit 420 can be configured to determine whether to trigger a recognition operation based on the bioelectrical signals and motion signals. The matching unit 430 can be configured, in response to triggering the recognition operation, to match the bioelectrical signals with reference bioelectrical signals of each candidate command among a plurality of candidate commands to obtain a first confidence level for each candidate command, and to match the motion signals with reference motion signals of each candidate command among a plurality of candidate commands to obtain a second confidence level for each candidate command. The second determination unit 440 can be configured to determine a target command from the plurality of candidate commands based on the first and second confidence levels of each candidate command.

[0082] In some embodiments, the command recognition device 400 may further include other units to execute the command recognition method of any of the above embodiments.

[0083] The specific operation of the various units in the command recognition device 400 can be found in the detailed explanation of the command recognition method above, and will not be repeated here.

[0084] Figure 5A schematic diagram of a command recognition device according to some embodiments of the present disclosure is shown. For example... Figure 5 As shown, the command recognition device 500 may include a memory 510 and a processor 520 coupled to the memory 510. The processor 520 may be configured to execute the command recognition method of any of the foregoing embodiments based on instructions stored in the memory 510.

[0085] Specifically, processor 520 can perform various actions and processes according to instructions stored in memory 510. Processor 520 can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. The general-purpose processor can be a microprocessor or any conventional processor, and can be an x86 architecture or an ARM architecture, etc.

[0086] Memory 510 stores executable instructions that, when executed by processor 520, implement the command identification method described above. Memory 510 may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that the memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0087] Embodiments of this disclosure also provide a head-mounted device. In some embodiments, the head-mounted device may include the command recognition device 400 described in any of the above embodiments. Alternatively, in some embodiments, the head-mounted device may include the command recognition device 500 described in any of the above embodiments. In some embodiments, the head-mounted device may include smart glasses.

[0088] This disclosure also proposes a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps of the command recognition method described above.

[0089] Similarly, the non-transitory computer-readable storage media in the embodiments of this disclosure are intended to include, but are not limited to, the above and any other suitable types of memory.

[0090] This disclosure also proposes a computer program product that may include a computer program that, when executed by a processor, can implement the steps of the command recognition method as described above.

[0091] Instructions can be any set of instructions that will be executed directly by one or more processors, such as machine code, or any set of instructions that will be executed indirectly, such as a script. The terms “instruction,” “application,” “procedure,” “step,” “program,” and “computer program” used herein are used interchangeably. Instructions can be stored in object code format for direct processing by one or more processors, or stored in any other computer language, including scripts or sets of independent source code modules that are interpreted on demand or compiled in advance. The function, methods, and routines of instructions are explained in more detail in other parts of this document.

[0092] Figure 6A schematic block diagram of a computer system 600 on which embodiments of the present disclosure may be implemented is shown. The computer system 600 includes a bus 610 or other communication mechanism for transmitting information, and a processing means 620 coupled to the bus 610 for processing information. The computer system 600 also includes a storage means 630 coupled to the bus 610 for storing instructions to be executed by the processing means 620. The storage means 630 may include random access memory (RAM) or other dynamic storage devices. The storage means 630 may be used to store temporary variables or other intermediate information during the execution of instructions to be executed by the processing means 620. The storage means 630 may include a read-only memory (ROM) or other static storage device for storing static information and instructions for the processing means 620. The storage means 630 may include means such as a magnetic disk or optical disk for storing information and instructions. The computer system 600 may be coupled via the bus 610 to an output device 640 for providing output to a user, such as, but not limited to, a display (such as a cathode ray tube (CRT) or liquid crystal display (LCD)), a speaker, etc. Input devices 650, such as a keyboard, mouse, and microphone, are coupled to bus 610 for transmitting information and command selections to processing device 620. Computer system 600 can execute embodiments of this disclosure. Consistent with certain implementations of this disclosure, results are provided by computer system 600 in response to processing device 620 executing one or more sequences of one or more instructions contained in storage device 630. Such instructions may be read into storage device 630 from another computer-readable medium, such as storage device 630. Execution of the sequence of instructions contained in storage device 630 causes processing device 620 to perform the methods described herein. Alternatively, hard-wired circuitry may be used in place of or in combination with software instructions to implement the teachings. Therefore, implementations of this disclosure are not limited to any particular combination of hardware circuitry and software. In various embodiments, computer system 600 may be connected across a network via network interface 660 to one or more other computer systems, such as computer system 600, to form a networked system. This network may include a private network or a public network such as the Internet. In a networked system, one or more computer systems may store data and supply data to other computer systems. As used herein, the term "computer-readable medium" refers to any medium that participates in providing instructions to the processing device 620 for execution. Such media can take many forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical discs or magnetic disks. Volatile media include dynamic memory such as RAM. Transmission media include coaxial cables, copper wires, and optical fibers, including wiring that includes bus 610.Common forms of computer-readable media or computer program products include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, or any other magnetic media, CD-ROMs, digital video discs (DVDs), Blu-ray discs, any other optical media, thumb drives, memory cards, RAM, PROMs and EPROMs, fast EPROMs, any other memory chips or cartridges, or any other tangible media from which a computer can read. Various forms of computer-readable media may be involved when carrying one or more sequences of one or more instructions to processing device 620 for execution. For example, instructions may initially be carried on a disk of a remote computer. The remote computer may load the instructions into its dynamic memory and transmit the instructions over a telephone line using a modem. A modem local to computer system 600 may receive data over a telephone line and convert the data into an infrared signal using an infrared transmitter. An infrared detector coupled to bus 610 may receive the data carried in the infrared signal and place the data on bus 610. Bus 610 carries the data to storage device 630, where processing device 620 retrieves and executes the instructions. Optionally, the instructions received by the storage device 630 may be stored on the storage device 630 before or after they are executed by the processing device 620.

[0093] According to various embodiments, instructions configured to be executed by processing device 620 to perform a method are stored on a computer-readable medium. The computer-readable medium may be a device for storing digital information. For example, the computer-readable medium includes a compact disc read-only memory (CD-ROM) as known in the art for storing software. The computer-readable medium is accessed by a processor adapted to execute the instructions configured to be executed.

[0094] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0095] In general, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0096] As used herein, the term “exemplary” means “used as an example, instance, or illustration” and not as a “model” to be exactly copied. Any implementation described herein by example is not necessarily to be construed as preferred or advantageous over other implementations.

[0097] Furthermore, terms such as “first,” “second,” etc., may be used in this document for reference purposes only and are not intended to be limiting. For example, unless the context clearly indicates otherwise, the words “first,” “second,” and other such numerical terms relating to structures or elements do not imply order or sequence.

[0098] It should also be understood that when the term “including / contains” is used herein, it indicates the presence of the indicated feature, whole, step, operation, unit and / or component, but does not preclude the presence or addition of one or more other features, wholes, steps, operations, units and / or components and / or combinations thereof.

[0099] In this disclosure, the term “provide” is used broadly to cover all ways of obtaining an object, and therefore “provide an object” includes, but is not limited to, “purchasing,” “preparing / manufacturing,” “arranging / setting up,” “installing / assembling,” and / or “ordering” an object.

[0100] As used herein, the term “and / or” includes any and all combinations of one or more of the listed items in association. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of this disclosure. As used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise.

[0101] Those skilled in the art will recognize that the boundaries between the above operations are merely illustrative. Multiple operations may be combined into a single operation, a single operation may be distributed among additional operations, and operations may be performed with at least partial overlap in time. Moreover, alternative embodiments may include multiple instances of a particular operation, and the order of operations may be changed in various other embodiments. However, other modifications, variations, and substitutions are equally possible. Aspects and elements of all the embodiments disclosed above may be combined in any way and / or in combination with aspects or elements of other embodiments to provide multiple additional embodiments. Therefore, this specification and the accompanying drawings should be considered illustrative rather than restrictive.

[0102] While specific embodiments of this disclosure have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. The various embodiments disclosed herein can be combined in any way without departing from the spirit and scope of this disclosure. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. A command recognition method, characterized in that, The command recognition method includes: Bioelectrical signals are acquired through electrodes on a head-mounted device, and motion signals are acquired through an inertial measurement unit on the head-mounted device. Determine whether to trigger the recognition operation based on bioelectrical and motion signals; In response to triggering the recognition operation, the bioelectric signal is matched with the reference bioelectric signal of each candidate command in the plurality of candidate commands to obtain the first confidence level of each candidate command, and the motion signal is matched with the reference motion signal of each candidate command in the plurality of candidate commands to obtain the second confidence level of each candidate command. The target command is determined from the plurality of candidate commands based on the first and second confidence levels of each candidate command.

2. The command recognition method according to claim 1, characterized in that, Determining whether to trigger a recognition operation based on bioelectrical and motion signals includes: In response to a motion signal triggering a first recognition request, the first start time of triggering the first recognition request is recorded; In response to a bioelectrical signal triggering a second identification request, the second start time of triggering the second identification request is recorded; If the absolute value of the difference between the first start time and the second start time is less than a time threshold, the identification operation is triggered.

3. The command recognition method according to claim 2, characterized in that, In response to the motion signal triggering the first identification request, bioelectrical signals after the first start time are also collected to determine whether the bioelectrical signals after the first start time trigger the second identification request.

4. The command recognition method according to claim 2, characterized in that, The command recognition method further includes: In response to at least one of the acceleration in the motion signal being greater than an acceleration threshold and the angular velocity in the motion signal being greater than an angular velocity threshold, it is determined that the motion signal triggers a first identification request; and / or In response to the envelope amplitude of the bioelectric signal being greater than the amplitude threshold, the bioelectric signal is determined to trigger a second recognition request.

5. The command recognition method according to claim 1, characterized in that, Determining whether to trigger a recognition operation based on bioelectrical and motion signals includes: Determine whether at least one of the bioelectrical signal and the motor signal indicates an uncommanded action; In response to at least one of the bioelectrical signal and the motion signal indicating a non-command action, it is determined that no recognition operation is triggered.

6. The command recognition method according to claim 1, characterized in that, The command recognition method further includes: For each candidate command among multiple candidate commands, output a prompt message to the user instructing them to configure that candidate command; The motion signal generated by the user's action in response to the prompt information is collected and used as the reference motion signal for the candidate command; The bioelectrical signal generated by the user in response to the prompt information is collected and used as the reference bioelectrical signal for the candidate command.

7. The command recognition method according to claim 1, characterized in that, The target command is determined from the plurality of candidate commands based on a first confidence level and a second confidence level, including: The fusion confidence of each candidate command is obtained based on the first and second confidence of each candidate command. In response to the maximum fusion confidence being greater than or equal to a first threshold, the candidate command corresponding to the maximum fusion confidence is determined as the target command.

8. The command recognition method according to claim 7, characterized in that, Determining the target command from the plurality of candidate commands based on the first confidence level and the second confidence level of each candidate command further includes: In response to the maximum fusion confidence being greater than a second threshold and less than a first threshold, confirmation information indicating the candidate command corresponding to the maximum fusion confidence is generated and provided. In response to the confirmation operation on the confirmation information, the candidate command corresponding to the maximum fusion confidence is determined as the target command.

9. The command recognition method according to claim 7, characterized in that, The command recognition method further includes: obtaining a contact quality score between the head-mounted device and the user; The fusion confidence of each candidate command is obtained based on its first and second confidence scores. This includes: for each candidate command, a weighted sum of the contact quality score, the first confidence score, and the second confidence score is performed to obtain the fusion confidence of each candidate command.

10. The command recognition method according to claim 9, characterized in that, The contact quality score is obtained based on at least one of the following: the contact impedance between the electrode and the user, the wearing pressure exerted by the head-mounted device on the contact area with the user, and the signal-to-noise ratio of the bioelectrical signal.

11. The command recognition method according to claim 7 or 9, characterized in that, The command recognition method further includes: The weight coefficient corresponding to the first confidence level is determined based on the user's scenario, wherein the weight coefficient corresponding to the first confidence level when the user is in a static scenario is greater than the weight coefficient corresponding to the first confidence level when the user is in a dynamic scenario; and / or The weight coefficient corresponding to the second confidence level is determined based on the user's scenario. The weight coefficient corresponding to the second confidence level when the user is in a static scenario is less than the weight coefficient corresponding to the second confidence level when the user is in a dynamic scenario.

12. A command recognition device, characterized in that, The command recognition device includes: The acquisition unit is configured to acquire bioelectrical signals via electrodes on the head-mounted device and to acquire motion signals via an inertial measurement unit on the head-mounted device. The first determining unit is configured to determine whether to trigger the execution of a recognition operation based on bioelectrical signals and motion signals; The matching unit is configured to, in response to triggering an identification operation, match the bioelectrical signal with a reference bioelectrical signal of each candidate command among a plurality of candidate commands to obtain a first confidence level for each candidate command, and match the motion signal with a reference motion signal of each candidate command among the plurality of candidate commands to obtain a second confidence level for each candidate command. The second determining unit is configured to determine the target command from the plurality of candidate commands based on a first confidence level and a second confidence level for each candidate command.

13. A command recognition device, characterized in that, The command recognition device includes: Memory; and A processor coupled to the memory is configured to execute the command recognition method of any one of claims 1 to 11 based on instructions stored in the memory.

14. A head-mounted device, characterized in that, The head-mounted device includes: The command recognition device according to claim 12; or The command recognition device according to claim 13.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a computer program, wherein the computer program, when executed by a processor, implements the steps of the command recognition method according to any one of claims 1 to 11.

16. A computer program product, characterized in that, The computer program product includes a computer program, wherein when the computer program is executed by a processor, it implements the steps of the command recognition method according to any one of claims 1 to 11.

Images