Voice wake-up method and device, equipment and storage medium

Abstract

The invention provides a voice wake-up method and device, equipment and a storage medium, and the voice wake-up method can comprise the steps: obtaining audio perception information corresponding to a wake-up audio when it is detected that a user inputs the wake-up audio, and the audio perception information is used for representing the spatial relation between the user and first equipment; obtaining visual information of the user, the visual information being used for representing a spatial relationship between a visual focus of the user and the first device; according to the visual information and the audio perception information, a detection result is determined, and the detection result is used for indicating whether the wake-up audio is used for waking up the first equipment or not; and under the condition that the detection result indicates that the wake-up audio is used for waking up the first equipment, waking up the first equipment. By implementing the method, the accuracy of the first device in responding to the wake-up audio can be improved.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a voice wake-up method, device, equipment and storage medium. Background Technology

[0002] When a user has multiple electronic devices that can be woken up by the same wake-up audio, these devices often wake up simultaneously when the user outputs the wake-up audio, resulting in chaotic, redundant responses and a disruptive user experience. Therefore, ensuring that the wake-up audio is accurately responded to only by the user's target device, avoiding simultaneous responses from multiple devices, has become a pressing technical problem for the industry. Summary of the Invention

[0003] This application provides a voice wake-up method, apparatus, device, and storage medium, which can improve the accuracy of device response to wake-up audio.

[0004] A first aspect of this application provides a voice wake-up method, the method being applicable to a first device, the method comprising: When a user inputs a wake-up audio, the audio perception information corresponding to the wake-up audio is obtained, and the audio perception information is used to characterize the spatial relationship between the user and the first device; Acquire the user's visual information, which is used to characterize the spatial relationship between the user's visual focus and the first device; Based on the visual information and the audio perception information, a detection result is determined, and the detection result is used to indicate whether the wake-up audio is used to wake up the first device; If the detection result indicates that the wake-up audio is used to wake up the first device, then wake up the first device.

[0005] A second aspect of this application provides a voice wake-up device, the device being adapted for a first device, the device comprising: The information acquisition unit is used to acquire audio perception information corresponding to the wake-up audio when a user input wake-up audio is detected. The audio perception information is used to characterize the spatial relationship between the user and the first device. The information acquisition unit is further configured to acquire the user's visual information, which is used to characterize the spatial relationship between the user's visual focus and the first device; A detection unit is configured to determine a detection result based on the visual information and the audio perception information, wherein the detection result is used to indicate whether the wake-up audio is used to wake up the first device; A wake-up unit is configured to wake up the first device when the detection result indicates that the wake-up audio is used to wake up the first device.

[0006] A third aspect of the embodiments of this application provides an electronic device, Memory containing executable program code; and the processor coupled to the memory; The processor calls the executable program code stored in the memory, and when the executable program code is executed by the processor, the processor implements the method disclosed in the first aspect of the embodiments of this application.

[0007] A fourth aspect of this application provides a computer-readable storage medium having executable program code stored thereon, wherein when the executable program code is executed by a processor, it implements the method disclosed in the first aspect of this application.

[0008] The fifth aspect of this application discloses a computer program product that, when run on a computer, causes the computer to execute the method disclosed in the first aspect of this application.

[0009] The sixth aspect of this application discloses an application publishing platform for publishing computer program products, wherein when the computer program product is run on a computer, the computer executes the method disclosed in the first aspect of this application.

[0010] As can be seen from the above technical solutions, the embodiments of this application have at least the following advantages: Upon detecting user input wake-up audio, the system acquires audio perception information corresponding to the wake-up audio, which is used to characterize the spatial relationship between the user and the first device; and acquires visual information of the user, which is used to characterize the spatial relationship between the user's visual focus and the first device; and determines a detection result based on the visual information and the audio perception information, which is used to indicate whether the wake-up audio is used to wake up the first device; and wakes up the first device if the detection result indicates that the wake-up audio is used to wake up the first device.

[0011] By implementing this method, when the first device detects a wake-up audio input by the user, it can further determine whether the first device is the target device of the wake-up audio based on the audio perception information corresponding to the wake-up audio and the user's gaze information. If so, the first device is woken up, which helps to improve the accuracy of the first device's response to the wake-up audio. Attached Figure Description

[0012] Figure 1 This is an illustration of a voice wake-up scenario disclosed in an embodiment of this application; Figure 2 This is a flowchart illustrating a voice wake-up method disclosed in an embodiment of this application; Figure 3 This is another flowchart illustrating the voice wake-up method disclosed in the embodiments of this application; Figure 4 This is another flowchart illustrating the voice wake-up method disclosed in the embodiments of this application; Figure 5 This is a schematic diagram of activating the first voice wake-up mode as disclosed in an embodiment of this application; Figure 6 This is another flowchart illustrating the voice wake-up method disclosed in the embodiments of this application; Figure 7 This is a structural illustration of a voice wake-up device disclosed in an embodiment of this application; Figure 8 This is a structural illustration of an electronic device disclosed in an embodiment of this application. Detailed Implementation

[0013] This application provides a voice wake-up method, apparatus, device, and storage medium, which can improve the accuracy of device response to wake-up audio.

[0014] To enable those skilled in the art to better understand the present application, the technical solutions of the embodiments of the present application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. All embodiments based on the present application should fall within the scope of protection of the present application.

[0015] It should be noted that, in this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0016] "At least one" means one or more, while "more" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can mean: a, or b, or c, or a and b, or a and c, or b and c, or a, b, and c, where a, b, and c can be single or multiple.

[0017] The electronic devices (first device, second device) involved in the embodiments of this application may include general handheld screen electronic devices, such as mobile phones, smartphones, portable terminals, terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), laptops, notebooks, wireless broadband (Wibro) terminals, tablet computers (PCs), smart PCs, point of sale (POS) terminals, etc.

[0018] Electronic devices can also include wearable devices. Wearable devices are portable electronic devices that can be worn directly on the user's body or integrated into the user's clothing or accessories. Wearable devices are not just hardware devices; they can also achieve powerful intelligent functions through software support and data interaction and cloud interaction, such as computing, positioning, and alarm functions. They can also connect to mobile phones and various terminals. Wearable devices can include, but are not limited to, wrist-supported devices (such as watches, wristbands, etc.), foot-supported devices (such as shoes, socks, or other leg-wearing products), head-supported devices (such as glasses, helmets, headbands, etc.), as well as smart clothing, backpacks, canes, accessories, and other non-mainstream product forms.

[0019] Please see Figure 1 , Figure 1 This is an illustration of a voice wake-up scenario disclosed in an embodiment of this application. For example... Figure 1 As shown, a mobile phone 20, a tablet 30, and a laptop 40 are placed on a desktop 10. The sample audio pre-stored in these three devices is identical, meaning they can be woken up based on the same wake-up audio. In this scenario, if a user wants to wake up the mobile phone 20 via voice, the mobile phone 20, tablet 30, and laptop 40 will typically detect the wake-up audio and all be woken up, resulting in confusing, redundant responses and a disruptive user experience. Therefore, how to ensure that the user's wake-up audio only wakes up the mobile phone 20, without waking up the tablet 30 and laptop 40, has become a pressing technical problem that the industry needs to solve.

[0020] To address this issue, embodiments of this application provide a voice wake-up method, apparatus, device, and storage medium. In this voice wake-up method, the device (mobile phone 20, tablet 30, or laptop 40) can, upon detecting user input of a wake-up audio, further acquire audio perception information corresponding to the wake-up audio and the user's visual information. Based on the visual information and audio perception information, it determines whether it is the target device corresponding to the wake-up audio. If it is the target device corresponding to the wake-up audio, it performs a wake-up operation; otherwise, it does not perform a wake-up operation.

[0021] Understandably, mobile phone 20 can determine that the target device of the user's wake-up audio is itself by implementing this method, and then mobile phone 20 will further perform the wake-up operation. Tablet 30 and laptop 40, on the other hand, can determine that the target device of the user's wake-up audio is not themselves by implementing this method, and then tablet 30 and laptop 40 will not perform the wake-up operation, thus achieving accurate voice wake-up in multi-device scenarios.

[0022] The present application solution will be further described below with reference to specific embodiments.

[0023] Please see Figure 2 , Figure 2 This is a flowchart illustrating a voice wake-up method disclosed in an embodiment of this application. Figure 2 The voice wake-up method shown may include the following steps: 201. When the first device detects that the user has input a wake-up audio, it acquires the audio perception information corresponding to the wake-up audio.

[0024] In this embodiment of the application, the wake-up audio may refer to a specific sound signal that triggers the first device to activate the voice assistant.

[0025] It should be noted that the audio perception information corresponding to the wake-up audio can be used to characterize the spatial relationship between the user and the first device. This spatial relationship can refer to the user's position, distance, orientation, and other spatial states relative to the first device in physical space.

[0026] In this embodiment, the audio sensing information may include, but is not limited to, relative position information and / or breath information. The relative position information may include the user's distance and / or orientation relative to the first device, and the breath information may be used to indicate the presence of breath.

[0027] It should be noted that when the breath information indicates the presence of breath, the distance between the first device and the user is less than or equal to a distance threshold; when the breath information indicates the absence of breath, the distance between the first device and the user is greater than this distance threshold. It can be understood that the first device can determine the distance range between itself and the user based on the breath information.

[0028] In one possible implementation, the audio perception information includes relative position information and breath information. The first device acquires the audio perception information corresponding to the wake-up audio, which may include: the first device extracting target features of the wake-up audio, the target features may include at least one of the following: temporal features, Doppler features, and generalized cross-correlation (GCC) features between microphones; and determining the relative position information and breath information based on the target features.

[0029] The first device can determine relative position information and breath information based on target characteristics using a model or database.

[0030] In the embodiments of this application, the first device can also acquire relative position information based on the multi-microphone reception difference analysis method. The multi-microphone reception difference analysis method may include, but is not limited to, at least one of the following: Time Difference of Arrival (TDOA), Generalized CrossCorrelation with Phase Transform (GCC-PHAT), and Direction of Arrival (DOA).

[0031] Understandably, prior to step 201, the first device can also detect the audio input by the user, and if the audio is detected, match it with pre-stored sound samples. If the audio matches the sound samples, the user-input wake-up audio is determined. The matching between the sound samples and the user-input audio involves not only voiceprint matching but also content matching, which is beneficial for achieving secure voice wake-up.

[0032] 202. The first device acquires the user's visual information.

[0033] The visual information can be used to characterize the spatial relationship between the user's visual focus and the first device. The visual information may include, but is not limited to, at least one of the following: the user's gaze position, the user's facial orientation, and depth of field.

[0034] It can be understood that before step 202, the first device can capture video frames through its camera, and the first device can obtain the user's visual information based on the video frames.

[0035] The following sections describe how the first device acquires the gaze position, the user's facial orientation, and the depth of field distance.

[0036] In this embodiment, the first device obtains the user's gaze position by: identifying a face region in a video frame and determining the user's gaze position based on the image corresponding to the face region using a first lightweight network. The first lightweight network can be a MobileNet or a lightweight CNN architecture, which significantly reduces the computational load and time required to obtain the user's gaze position.

[0037] In the embodiments of this application, the first device can detect the user's gaze position in a traditional way that relies on eye-tracking sensors, or in a traditional computer vision method (e.g., based on eye feature points, etc.). This application embodiment does not limit the method.

[0038] The first device can obtain the user's facial orientation by: identifying the face region in a video frame and determining the user's facial orientation based on the image corresponding to the face region using a second lightweight network. The second lightweight network can be MobileNetV2 or Nasnet-Mobile, which is beneficial for achieving accurate and efficient facial orientation detection.

[0039] Furthermore, the first device determines the user's facial orientation based on the image corresponding to the face region through the second lightweight network, which may include: the first device determining an initial facial orientation based on the image corresponding to the face region through the second lightweight network; determining the orientation of the first device based on the inertial measurement unit of the first device; and determining the user's facial orientation based on the initial facial orientation and the orientation of the first device.

[0040] It should be noted that after the first device determines the initial facial orientation based on the image corresponding to the face region, it can also be corrected by combining the orientation of the first device, which is beneficial to further improve the detection accuracy of facial orientation.

[0041] In this embodiment, the first device can detect facial orientation in a multi-view geometric calculation manner, or it can be based on traditional computer vision methods (such as Haar cascade, HOG features, or feature point detection combined with geometric transformation, etc.). This embodiment does not limit the method.

[0042] The first device may obtain the depth of field distance by: determining the measured distance between the first facial key point and the second facial key point from the captured video frame, obtaining the physical distance between the first facial key point and the second facial key point, and determining the depth of field distance based on the ratio of the measured distance value and the physical distance value.

[0043] The first facial landmark and the second facial landmark can be the left eye and the right eye, or the tip of the nose and the chin, or other facial landmarks such as the left eye and the tip of the nose, or the right eye and the chin. This application does not limit the specific facial landmarks.

[0044] The physical distance between the first and second facial landmarks is preset.

[0045] In the embodiments of this application, the first device may obtain the depth distance in a way that is based on a depth sensor (such as ToF (Time of Flight), structured light, and binocular stereo vision) or a CNN model. This application does not limit the method.

[0046] 203. The first device determines the detection result based on visual information and audio perception information. The detection result is used to indicate whether the wake-up audio is used to wake up the first device.

[0047] In this embodiment of the application, the detection result can be numbers, letters, or text, etc., and this embodiment of the application is not limited thereto. For example, the detection result can be "0" or "1", where "0" indicates that the wake-up audio is not used to wake up the first device, and "1" indicates that the wake-up audio is used to wake up the first device. As another example, the detection result can be "wake-up" or "not wake-up", where "not wake-up" indicates that the wake-up audio is not used to wake up the first device, and "wake-up" indicates that the wake-up audio is used to wake up the first device.

[0048] In one possible implementation, the first device determines the detection result based on visual information and audio perception information, which may include: the first device fusing visual information and audio perception information to obtain fused information, and determining the detection result based on the fused information.

[0049] The first device can determine the detection result based on the fused information through the target detection model, which helps to improve the accuracy of the detection result.

[0050] The first device can also determine target reference information that matches the fusion information from the target database, and determine the result corresponding to the target reference information as the detection result. The target database includes multiple results and the reference information corresponding to each result.

[0051] It should be noted that the first device determines the detection result based on the fusion information through the database, which can reduce the amount of computation required for the first device to obtain the detection result.

[0052] In another possible implementation, the first device determines the detection result based on visual information and audio perception information, which may include: the first device determining first classification information based on audio perception information, determining second classification information based on visual information, and determining the detection result based on the first classification information and the second classification information.

[0053] The first device can determine the first classification information based on the audio perception information using the first classification model. The first classification model may include, but is not limited to, at least one of the following: classification models (such as Softmax, RandomForest, SVM, XGBoost, LightGBM, CNN, and Transformer), regression models (such as Linear Regression, SVR, XGBoost-reg, and neural networks), and detection models (such as a single fully connected layer + sigmoid (binary classification), or multi-label softmax).

[0054] In this embodiment, the classification information (first classification information, second classification information) can be a first tag or a second tag. The first tag can be used to indicate that the wake-up audio is used to wake up the first device, and the second tag can be used to indicate that the wake-up audio is not used to wake up the first device. Based on this, the first device determines the detection result according to the first classification information and the second classification information, which may include: if both the first classification information and the second classification information are the first tag, determining the first tag as the detection result; if the first classification information and the second classification information are different (the first classification information is the first tag, and the second classification information is the second tag, or the first classification information is the second tag, and the second classification information is the first tag), then determining the second tag as the detection result.

[0055] In this embodiment, the classification information (first classification information, second classification information) may also include the confidence level corresponding to the first label and the confidence level corresponding to the second label. Based on this, the first device determines the detection result according to the first classification information and the second classification information, which may include: the first device accumulating the confidence level corresponding to the first label to obtain the comprehensive confidence level corresponding to the first label, and accumulating the confidence level corresponding to the second label to obtain the comprehensive confidence level corresponding to the second label; and determining the label with the highest comprehensive confidence level as the detection result.

[0056] When the classification information (first classification information, second classification information) is the first label or the second label, In one possible implementation, the first device determines the first classification information based on the audio perception information by determining reference audio perception information that matches the aforementioned audio perception information from a first database, and then determining the label corresponding to the reference audio perception information as the first classification information. The first database may include a first label, a second label, reference audio perception information corresponding to the first label, and reference audio perception information corresponding to the second label.

[0057] It should be noted that the method by which the first device determines the second classification information based on visual information can be found in the above embodiment of the method by which the first device determines the first classification information based on audio perception information, and will not be repeated here.

[0058] 204. When the above detection result indicates that the wake-up audio is used to wake up the first device, the first device wakes up the first device.

[0059] In this context, "the first device wakes up the first device" can refer to the first device switching from a low-power standby mode to an operating mode. It should be noted that in standby mode, most functional modules of the first device (such as the CPU, screen, and hard drive) stop operating or operate at a very low frequency, retaining only the most basic functions (such as listening for wake-up signals and maintaining memory power). In operating mode, all or most of the first device's functional modules are powered and begin normal operation, ready to execute user instructions.

[0060] By implementing Figure 2 The voice wake-up method shown allows the first device to determine whether it is the target device for the wake-up audio based on the audio perception information and the user's gaze information when it detects the wake-up audio input by the user. If it is, the wake-up operation is performed; otherwise, the wake-up operation is not performed, which helps to improve the accuracy of the first device's response to the wake-up audio.

[0061] In another possible implementation, the first device can also determine the detection result through the DS evidence theory, as follows: Figure 3 As shown. Please refer to [the original text]. Figure 3 , Figure 3 This is another flowchart illustrating the voice wake-up method disclosed in the embodiments of this application. For example... Figure 3 The voice wake-up method shown may include the following steps: 301. When the first device detects that the user has input a wake-up audio, it acquires the audio perception information corresponding to the wake-up audio.

[0062] 302. The first device acquires the user's visual information.

[0063] Among them, for the detailed introduction of steps 301 - 302, reference can be made to the description of steps 201 - 202 in the above embodiments, which will not be elaborated here.

[0064] 303. The first device determines the detection result based on the visual information and audio perception information through the D-S evidence theory.

[0065] Among them, the first device determines the detection result based on the visual information and audio perception information through the D-S evidence theory, which may include: the first device respectively constructs a first basic probability assignment (BPA) function corresponding to the visual information and a second BPA function corresponding to the audio perception information based on the recognition framework; and, based on the D-S evidence theory combination rule, fuses the first BPA function and the second BPA function to obtain a target BPA function; and, makes a decision based on the target BPA function through the target decision rule to obtain the detection result.

[0066] Among them, the target decision rule can be a belief function or a plausibility function, which is not limited in the embodiments of the present application.

[0067] In the embodiments of the present application, the first device fuses the first BPA function and the second BPA function based on the D-S evidence theory combination rule to obtain a target BPA function, which may include: the first device calculates a conflict coefficient, calculates a normalization factor based on the conflict coefficient, and fuses the first BPA function and the second BPA function based on the normalization factor to obtain a target BPA function.

[0068] It should be noted that the conflict coefficient can be used to represent the conflict degree between the visual information and the audio perception information. Exemplarily, the conflict coefficient is K. If K = 0, it means that the visual information and the audio perception information have no conflict at all. If 0 < K < 1, it means that the visual information and the audio perception information have a certain degree of conflict. If K ≈ 1, it means that there is an extreme conflict between the visual information and the audio perception information. At this time, the normalization factor (1 - K) will be very small, which may cause the detection result to become unstable and counterintuitive. In this case, other combination rules (such as Yager rule, PCR rule, etc.) may need to be used to process the highly conflicting information.

[0069] It can be understood that in the case where there is a conflict between the visual information and the audio perception information, the first device processes the visual information and the audio perception information through the D-S evidence theory, can recognize and eliminate the quality of this part of the conflicting information, regard all the remaining non-conflicting information as a new whole, and determine the detection result based on this, increasing the reliability of the detection result.

[0070] 304. When the above detection result indicates that the wake-up audio is used to wake up the first device, wake up the first device.

[0071] For a description of step 304, please refer to the above description of step 204, which will not be repeated here.

[0072] By implementing Figure 3 The voice wake-up method shown allows the first device to determine whether it is the target device for the wake-up audio when it detects a wake-up audio input by the user, based on the audio perception information corresponding to the wake-up audio and the user's gaze information. If it is, a wake-up operation is performed; otherwise, no wake-up operation is performed, which helps improve the accuracy of the first device's response to the wake-up audio. Furthermore, the first device uses the DS evidence theory to make judgments based on audio perception information and the user's gaze information, which helps improve the reliability of the judgment.

[0073] In this embodiment, the first device can have two voice wake-up modes. One voice wake-up mode (first voice wake-up mode) requires the first device to continue acquiring audio perception information corresponding to the wake-up audio when it detects the wake-up audio, and then further determines whether to wake up the device based on the audio perception information. The other voice wake-up mode (second voice wake-up mode) allows the first device to wake up directly as soon as it detects the wake-up audio. As follows... Figure 4 As shown.

[0074] Please see Figure 4 , Figure 4 This is another flowchart illustrating the voice wake-up method disclosed in the embodiments of this application. For example... Figure 4 The voice wake-up method shown may include the following steps: 401. When the first device detects the wake-up audio input by the user, it checks whether the first voice wake-up mode is enabled.

[0075] It should be noted that if the judgment result of step 401 is yes, then steps 402-405 will continue to be executed; if the judgment result of step 401 is no, then step 406 will continue to be executed.

[0076] In this embodiment of the application, the first device may default to enabling the second voice wake-up mode.

[0077] In this embodiment of the application, the first device may activate the first voice wake-up mode in ways including but not limited to the following: Method 1: When the first device detects the wake-up audio input by the user, it detects the wireless signal of the second device through the near-field communication module of the first device, and activates the first voice wake-up mode when the wireless signal of the second device is detected; wherein the sample audio stored by the first device and the second device is the same.

[0078] It should be noted that if the sample sounds stored in the first device and the second device are the same, it means that the first device and the second device can be woken up by the same wake-up audio.

[0079] The near-field communication module of the first device may include, but is not limited to, a Bluetooth module, a Wireless Fidelity (Wi-Fi) module, and a Near Field Communication (NFC) module. When the near-field communication module is a Bluetooth module, the wireless signal of the second device is a Bluetooth signal; when the near-field communication module is a Wi-Fi module, the wireless signal of the second device is a Wi-Fi signal; and when the near-field communication module is an NFC module, the wireless signal of the second device is an NFC signal.

[0080] Understandably, when the first device detects the second device's wireless signal through its proximity communication module, it indicates that the two devices are close together, and the probability of the first device falsely responding to the wake-up audio is higher. Therefore, the first device automatically activates the first voice wake-up mode to improve the accuracy of its response to the wake-up audio, thus achieving automatic switching of the voice wake-up mode. When the first device does not detect the second device's wireless signal through its proximity communication module, it indicates that the two devices are far apart, and the probability of the first device falsely responding to the wake-up audio is lower. Therefore, the first device continues to maintain the second voice wake-up mode.

[0081] It should be noted that, as shown in Method 1, activating the first voice wake-up mode occurs after the wake-up audio is detected. That is, the first device determines whether the first voice wake-up mode needs to be activated based on its proximity communication module only after detecting the wake-up audio, which helps reduce unnecessary power consumption. Of course, the first device's determination of whether the first voice wake-up mode needs to be activated based on its proximity communication module can also occur before detecting the wake-up audio. In this case, when the first device detects the wake-up audio, it can directly perform the following operations to improve the response accuracy to the wake-up audio, thus improving the wake-up efficiency of the first device.

[0082] Method 2: The first device responds to the user's input target operation and activates the first voice wake-up mode.

[0083] The target operation may include, but is not limited to, at least one of the following: gesture operation, touch operation, or eye movement operation.

[0084] It is understandable that the first voice wake-up mode shown in Method 2 can be activated by the user in advance, without the first device needing to activate it when it detects the wake-up audio, which also helps to improve the wake-up efficiency of the first device.

[0085] Furthermore, before activating the first voice wake-up mode in response to the user's input of a target operation, the first device may also output first information to prompt the user to activate the first voice wake-up mode. The output method of the first information may include, but is not limited to, at least one of the following: voice, text, or animation.

[0086] The first device can detect the wireless signal of the second device through its proximity communication module, and output first information upon detecting the wireless signal of the second device. It is understood that when the first device detects the wireless signal of the second device through its proximity communication module, it indicates that the first and second devices are close together, and the probability of the first device falsely responding to the wake-up audio is relatively high. Based on this, the first device can output first information to prompt the user to activate the first voice wake-up mode.

[0087] Of course, the output conditions for the first information can be other conditions, and this application embodiment does not limit them.

[0088] In some embodiments, after activating the first voice wake-up mode, the first device may also output second information, which indicates that the first voice wake-up mode has been successfully activated. The output method of the second information may include, but is not limited to, at least one of the following: voice, text, or animation.

[0089] For example, Figure 5 This is a schematic diagram of activating the first voice wake-up mode as disclosed in the embodiments of this application. Figure 5 The interface on the left is the interface for outputting the first information, which includes first information 500. This first information 500 includes a first button 510 for triggering the display of the voice wake-up mode settings interface and a second button 520 for canceling the switching of the voice wake-up mode. When the first device detects a user's touch operation on the first button 510, it displays the voice wake-up mode settings interface, such as... Figure 5 The interface shown in the middle includes a switch button 600 in a closed state and a switch button 700 in an open state. Switch button 600 is used to turn the first voice wake-up mode on or off, and switch button 700 is used to turn the second voice wake-up mode on or off. When the first device detects a user's toggle operation on switch button 600, it switches switch button 600 from the closed state to the open state and switch button 700 from the open state to the closed state, as shown in the image. Figure 5 The interface shown on the right.

[0090] 402. The first device acquires the audio perception information corresponding to the wake-up audio.

[0091] 403. The first device acquires the user's visual information.

[0092] It should be noted that for a detailed explanation of steps 402-403, please refer to the description of steps 301-302 in the above embodiments, which will not be repeated here.

[0093] 404. The first device determines the detection result based on visual and audio perception information using the DS evidence theory.

[0094] 405. When the above detection result indicates that the wake-up audio is used to wake up the first device, the first device wakes up.

[0095] For a description of steps 404-405, please refer to the description of steps 303-304 in the above embodiments, which will not be repeated here.

[0096] 406. The first device wakes up the first device.

[0097] By implementing Figure 4 The method shown allows the first device, upon detecting a user-input wake-up audio, to further determine whether it is the target device for the wake-up audio based on the corresponding audio perception information and the user's gaze information. If so, a wake-up operation is performed; otherwise, it is not, thus improving the accuracy of the first device's response to the wake-up audio. Furthermore, the first device uses the DS evidence theory to determine the target device based on audio perception information and the user's gaze information, which enhances the reliability of the determination. Moreover, the first device features two voice wake-up modes to adapt to different usage scenarios, facilitating accurate and efficient voice wake-up.

[0098] In some embodiments, the first device may determine the ambient noise level based on the wake-up audio before acquiring the user's visual information. Furthermore, the first device may acquire the user's visual information even when the ambient noise level is greater than a noise level threshold.

[0099] The first device can also determine the detection result directly based on audio perception information when the ambient noise level is less than or equal to the noise level threshold.

[0100] The noise threshold can be preset by the first device at the factory or by the user; this application embodiment does not limit it.

[0101] In this embodiment, if the ambient noise level is greater than a noise level threshold, it indicates that the first device is in a noisy environment; if the ambient noise level is less than or equal to the noise level threshold, it indicates that the first device is in a quiet environment. It should be noted that in a noisy environment, audio perception information typically cannot accurately reflect the spatial relationship between the user and the first device. However, in a quiet environment, audio perception information can often accurately reflect the spatial relationship between the user and the first device. Based on this, the following... Figure 6 As shown, after the first device obtains the audio perception information corresponding to the wake-up audio, it can detect the noise level of the environment. If the noise level is high, it continues to obtain the user's visual information and makes a judgment based on both visual and audio perception information. This judgment result is more accurate. If the noise level is low, it makes a judgment directly based on the audio perception information, which can reduce computational complexity and improve judgment efficiency, thereby improving the efficiency of voice wake-up.

[0102] Please see Figure 6 , Figure 6 This is another flowchart illustrating the voice wake-up method disclosed in the embodiments of this application. For example... Figure 6 The voice wake-up method shown may include the following steps: 601. When the first device detects the wake-up audio input by the user, it checks whether the first voice wake-up mode is enabled.

[0103] It should be noted that if the judgment result in step 601 is yes, then steps 602-604 will continue to be executed; if the judgment result in step 601 is no, then step 609 will be executed directly.

[0104] 602. The first device acquires the audio perception information corresponding to the wake-up audio.

[0105] 603. The first device determines the ambient noise level based on the wake-up audio.

[0106] 604. The first device determines whether the wake-up noise level is greater than the noise level threshold.

[0107] If the judgment result in step 604 is yes, then continue to execute steps 605-606 and step 608; if the judgment result in step 604 is no, then continue to execute steps 607-608.

[0108] 605. The first device acquires the user's visual information.

[0109] 606. The first device determines the detection result based on visual and audio perception information using the DS evidence theory.

[0110] 607. The first device determines the detection result based on the audio perception information.

[0111] 608. When the above detection result indicates that the wake-up audio is used to wake up the first device, the first device wakes up.

[0112] 609. The first device wakes up the first device.

[0113] It should be noted that the relevant descriptions of steps 601-609 can be found in the above embodiments, and will not be repeated here.

[0114] Please see Figure 7 , Figure 7 This is a structural illustration of a voice wake-up device disclosed in an embodiment of this application. For example... Figure 7 The voice wake-up device shown may include an information acquisition unit 701, a detection unit 702, and a wake-up unit 703; wherein: The information acquisition unit 701 is used to acquire audio perception information corresponding to the wake-up audio when the user input wake-up audio is detected. The audio perception information is used to characterize the spatial relationship between the user and the first device. The information acquisition unit 701 is also used to acquire the user's visual information, which is used to characterize the spatial relationship between the user's visual focus and the first device; The detection unit 702 is used to determine the detection result based on visual information and audio perception information. The detection result is used to indicate whether the wake-up audio is used to wake up the first device. The wake-up unit 703 is used to wake up the first device when the detection result indicates that the wake-up audio is used to wake up the first device.

[0115] In some embodiments, the method by which the detection unit 702 determines the detection result based on visual information and audio perception information may specifically include: the detection unit 702 is used to determine the detection result based on visual information and audio perception information through the DS evidence theory.

[0116] In some embodiments, the method by which the detection unit 702 determines the detection result based on visual information and audio perception information using DS evidence theory may specifically include: the detection unit 702 constructing a first basic probability allocation (BPA) function corresponding to the visual information and a second BPA function corresponding to the audio perception information based on the recognition framework; fusing the first BPA function and the second BPA function based on DS evidence theory combination rules to obtain a target BPA function; and making a decision based on the target BPA function using target decision rules to obtain the detection result.

[0117] In some embodiments, the method by which the detection unit 702 determines the detection result based on visual information and audio perception information may specifically include: the detection unit 702 fusing visual information and audio perception information to obtain fused information; and determining the detection result based on the fused information.

[0118] In some embodiments, the method by which the detection unit 702 determines the detection result based on the fusion information may specifically include: the detection unit 702 is used to determine the detection result based on the fusion information through a target detection model.

[0119] In some embodiments, the method by which the detection unit 702 determines the detection result based on the fusion information may specifically include: the detection unit 702 determining target reference information that matches the fusion information from a target database, wherein the target database includes multiple results and reference information corresponding to each result; and determining the result corresponding to the target reference information as the detection result.

[0120] In some embodiments, the method by which the information acquisition unit 701 acquires audio perception information corresponding to the wake-up audio when the user input wake-up audio is detected may specifically include: the information acquisition unit 701 is used to acquire audio perception information corresponding to the wake-up audio when the user input wake-up audio is detected and the first voice wake-up mode is enabled.

[0121] In some embodiments, Figure 7 The voice wake-up device shown may also include a voice mode control unit ( Figure 7 (Not shown), the voice mode control unit is used to detect the wireless signal of the second device through the near-field communication module of the first device when the wake-up audio is detected by the user input wake-up audio before the information acquisition unit 701 acquires the audio perception information corresponding to the wake-up audio, and the sample audio stored in the first device and the second device is the same; and to activate the first voice wake-up mode when the wireless signal of the second device is detected.

[0122] In some embodiments, the voice mode control unit is further configured to activate a first voice wake-up mode in response to a target operation input by the user before the information acquisition unit 701 acquires the audio perception information corresponding to the wake-up audio.

[0123] In some embodiments, the information acquisition unit 701 is further configured to determine the ambient noise level based on the wake-up audio; further, the method by which the information acquisition unit 701 acquires the user's visual information may specifically include: the information acquisition unit 701 is configured to acquire the user's visual information when the ambient noise level is greater than a noise level threshold.

[0124] In some embodiments, the audio perception information includes relative location information and / or breath information, which is used to indicate the presence of breath.

[0125] In some embodiments, the visual information includes at least one of the following: the user's gaze position, the user's facial orientation, and the depth of field distance.

[0126] Please see Figure 8 , Figure 8 This is a structural illustration of an electronic device disclosed in an embodiment of this application. For example... Figure 8 The electronic device shown may include components such as processor 801, memory 802, display unit 803, input unit 804, sensor 805, and audio circuit 806.

[0127] The processor 801 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 802, and by calling data stored in the memory 802, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. Optionally, the processor 801 may include one or more processing units; optionally, the processor 801 may integrate an application processor, which mainly handles operating devices, user interfaces, and application programs. Of course, it may also include other processors, which are not listed here.

[0128] The memory 802 can be used to store software programs and modules. The processor 801 executes various functional applications and data processing of the electronic device by running the software programs and modules stored in the memory 802. The memory 802 may mainly include a program storage area and a data storage area. The program storage area may store the operating device and application programs required for at least one function (such as sound playback function, image playback function, etc.); the data storage area may store data created according to the use of the electronic device (such as audio data, telephone book, etc.). In addition, the memory 802 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0129] The display unit 803 can be used to display information input by the user or information provided to the user, as well as various menus of the electronic device. The display unit 803 may include a display panel, optionally configured as a Liquid Crystal Display (LCD), Organic Light-Emitting Diode (OLED), or similar display panel. Furthermore, a touch panel may cover the display panel. When the touch panel detects a touch operation on or near it, it transmits the information to the processor 801 to determine the type of touch event. Subsequently, the processor 801 provides corresponding visual output on the display panel based on the type of touch event. The touch panel and the display panel are not located on the same surface. Figure 8 As shown above, the touch panel and display panel can be used as two separate components to realize the input and output functions of the electronic device, or they can be integrated to realize the input and output functions of the electronic device.

[0130] Input unit 804 can be used to receive input digital or character information and generate key signal inputs related to user settings and function control of the electronic device. Specifically, input unit 804 may include a touch panel and other input devices. A touch panel, also known as a touchscreen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel) and drive corresponding connection devices according to a pre-set program. Furthermore, touch panels can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel, input unit 804 may also include other input devices. Specifically, other input devices may include, but are not limited to, one or more of function keys (such as volume control buttons, power buttons, etc.), trackballs, joysticks, etc.

[0131] Electronic devices may also include at least one sensor 805, such as a magnetometer, gyroscope, motion sensor, and other sensors. Specifically, a magnetometer is used to determine the orientation of the electronic device, and a gyroscope can be used to determine the motion posture of the electronic device, which can be used for image stabilization, navigation, and motion-sensing game scenarios. As a type of motion sensor, an accelerometer can detect the magnitude of acceleration in various directions and, when stationary, can detect the magnitude and direction of gravity, which can be used for applications that identify the posture of the electronic device, such as landscape / portrait switching, related games, and magnetometer posture calibration. Other sensors that may be configured in electronic devices, such as pressure gauges, barometers, hygrometers, thermometers, and infrared sensors, will not be elaborated here.

[0132] The audio circuit 806 may include a speaker and a microphone, providing an audio interface between the user and the electronic device. The audio circuit 806 can convert received audio data into electrical signals and transmit them to the speaker, where the speaker converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by the audio circuit 806, converted back into audio data, and output to the processor 801 for processing. The processed audio data is then transmitted via a video circuit to, for example, another device, or output to the memory 802 for further processing.

[0133] Although not shown, the electronic device may also include a power supply and a camera. Optionally, the camera may be positioned in the front or rear of the electronic device, and this application embodiment does not limit this.

[0134] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0135] In this embodiment of the application, the processor 801 also has the following functions: When a user input wake-up audio is detected, the audio perception information corresponding to the wake-up audio is obtained. The audio perception information is used to characterize the spatial relationship between the user and the first device. Acquire the user's visual information, which is used to characterize the spatial relationship between the user's visual focus and the first device; Based on visual and audio perception information, the detection result is determined, and the detection result is used to indicate whether the wake-up audio is used to wake up the first device; If the detection result indicates that the wake-up audio is used to wake up the first device, then the first device is woken up.

[0136] In this embodiment of the application, the processor 801 also has the following functions: The detection results are determined based on visual and audio perception information using the DS evidence theory.

[0137] In this embodiment of the application, the processor 801 also has the following functions: Based on the recognition framework, a first basic probability allocation (BPA) function corresponding to visual information and a second BPA function corresponding to audio perception information are constructed respectively. Based on the DS evidence theory combination rule, the first BPA function and the second BPA function are fused to obtain the target BPA function; The detection results are obtained by making decisions based on the target BPA function using target decision rules.

[0138] In this embodiment of the application, the processor 801 also has the following functions: By fusing visual and audio perception information, fused information is obtained. The detection results are determined based on the fused information.

[0139] In this embodiment of the application, the processor 801 also has the following functions: The detection result is determined based on the fused information using the target detection model.

[0140] In this embodiment of the application, the processor 801 also has the following functions: Target reference information that matches the fusion information is determined from the target database, wherein the target database includes multiple results and the reference information corresponding to each result; The result corresponding to the target reference information is determined as the detection result.

[0141] In this embodiment of the application, the processor 801 also has the following functions: When the user inputs a wake-up audio and the first voice wake-up mode is enabled, the audio perception information corresponding to the wake-up audio is obtained.

[0142] In this embodiment of the application, the processor 801 also has the following functions: Upon detecting a wake-up audio input by the user, the wireless signal of the second device is detected through the near-field communication module of the first device, and the sample audio stored in the first device and the second device is the same; Upon detecting the wireless signal of the second device, activate the first voice wake-up mode.

[0143] In this embodiment of the application, the processor 801 also has the following functions: In response to the user's input of a target action, the first voice wake-up mode is activated.

[0144] In this embodiment of the application, the processor 801 also has the following functions: Determine the ambient noise level based on the wake-up audio; Acquire the user's visual information when the ambient noise level exceeds the noise threshold.

[0145] In this embodiment of the application, the audio perception information includes relative position information and / or breath information, which is used to indicate the presence of breath.

[0146] In this embodiment of the application, the visual information includes at least one of the following: the user's gaze position, the user's facial orientation, and the depth of field distance.

[0147] This application discloses a computer-readable storage medium storing executable program code thereon. When the executable program code is executed by a processor, it implements the method executed by the electronic device in this application.

[0148] This application discloses a computer program product that, when run on a computer, enables the computer to implement the method executed by the electronic device in this application.

[0149] This application discloses an application publishing platform for publishing computer program products. When the computer program product is run on a computer, the computer implements the method executed by the electronic device in this application.

[0150] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium, storage medium, and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0151] It should be understood that the phrases "one embodiment," "an embodiment," or "some embodiments" mentioned throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment," "in one embodiment," or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The descriptions of the various embodiments above tend to emphasize the differences between the various embodiments; their similarities or commonalities can be referred to mutually, and for the sake of brevity, they will not be repeated here.

[0152] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three kinds of relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist simultaneously, and object B exists alone.

[0153] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0154] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or modules can be electrical, mechanical, or other forms.

[0155] The modules described above as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules. They may be located in one place or distributed across multiple network units. Some or all of the modules may be selected to achieve the purpose of this embodiment according to actual needs.

[0156] In addition, each functional module in the various embodiments of this application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the integrated modules can be implemented in hardware or in the form of hardware plus software functional units.

[0157] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0158] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0159] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0160] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0161] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0162] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A voice wake-up method, characterized in that, The method is applicable to a first device, and the method includes: When a user inputs a wake-up audio, the audio perception information corresponding to the wake-up audio is obtained, and the audio perception information is used to characterize the spatial relationship between the user and the first device; Acquire the user's visual information, which is used to characterize the spatial relationship between the user's visual focus and the first device; Based on the visual information and the audio perception information, a detection result is determined, and the detection result is used to indicate whether the wake-up audio is used to wake up the first device; If the detection result indicates that the wake-up audio is used to wake up the first device, then wake up the first device.

2. The method according to claim 1, characterized in that, Determining the detection result based on the visual information and the audio perception information includes: The detection result is determined based on the visual information and the audio perception information using the DS evidence theory.

3. The method according to claim 2, characterized in that, The determination of the detection result based on the visual information and the audio perception information using the DS evidence theory includes: Based on the recognition framework, a first basic probability allocation (BPA) function corresponding to the visual information and a second BPA function corresponding to the audio perception information are constructed respectively. Based on the DS evidence theory combination rule, the first BPA function and the second BPA function are fused to obtain the target BPA function; The detection result is obtained by making a decision based on the target BPA function using the target decision rule.

4. The method according to claim 1, characterized in that, Determining the detection result based on the visual information and the audio perception information includes: By fusing the visual information and the audio perception information, fused information is obtained; The detection result is determined based on the fusion information.

5. The method according to claim 4, characterized in that, Determining the detection result based on the fusion information includes: Target reference information matching the fusion information is determined from the target database, wherein the target database includes multiple results and reference information corresponding to each result; The result corresponding to the target reference information is determined as the detection result.

6. The method according to any one of claims 1-5, characterized in that, The step of obtaining audio perception information corresponding to the wake-up audio when user input is detected includes: When a user-inputted wake-up audio is detected and the first voice wake-up mode is enabled, the audio perception information corresponding to the wake-up audio is obtained.

7. The method according to claim 6, characterized in that, Before obtaining the audio perception information corresponding to the wake-up audio, the method further includes: Upon detecting a wake-up audio input by the user, the wireless signal of the second device is detected through the near-field communication module of the first device, and the sample audio stored in the first device and the second device is the same; Upon detecting the wireless signal, the first voice wake-up mode is activated.

8. The method according to claim 6, characterized in that, Before obtaining the audio perception information corresponding to the wake-up audio, the method further includes: In response to the user's input of a target operation, the first voice wake-up mode is activated.

9. The method according to any one of claims 1-5, characterized in that, Before acquiring the user's visual information, the method further includes: The ambient noise level is determined based on the wake-up audio. The acquisition of the user's visual information includes: When the ambient noise level is greater than a noise threshold, the user's visual information is acquired.

10. The method according to claim 1, characterized in that, The audio perception information includes relative position information and / or breath information, wherein the breath information is used to indicate the presence of breath.

11. The method according to claim 1, characterized in that, The visual information includes at least one of the following: the user's gaze position, the user's facial orientation, and depth of field distance.

12. A voice wake-up device, characterized in that, The device is adapted for a first apparatus, the device comprising: The information acquisition unit is used to acquire audio perception information corresponding to the wake-up audio when a user input wake-up audio is detected. The audio perception information is used to characterize the spatial relationship between the user and the first device. The information acquisition unit is further configured to acquire the user's visual information, which is used to characterize the spatial relationship between the user's visual focus and the first device; A detection unit is configured to determine a detection result based on the visual information and the audio perception information, wherein the detection result is used to indicate whether the wake-up audio is used to wake up the first device; A wake-up unit is configured to wake up the first device when the detection result indicates that the wake-up audio is used to wake up the first device.

13. An electronic device, characterized in that, include: Memory containing executable program code; and the processor coupled to the memory; The processor calls the executable program code stored in the memory, and when the executable program code is executed by the processor, the processor implements the method as described in any one of claims 1-11.

14. A computer-readable storage medium having executable program code stored thereon, characterized in that, When the executable program code is executed by the processor, it implements the method as described in any one of claims 1-11.

Images