Orientation determination method and related device

Abstract

An orientation determination method and a related device, relating to the field of voice. The method comprises: acquiring a first feature of a first device, wherein the first feature is obtained on the basis of M single-channel voice signals corresponding to M audio collection units in the first device, M is an integer, the M audio collection units collect audio from a same sound source, and the first feature comprises orientation-sensitive features of direct sound and / or reflected sound in a sound signal generated by the sound source; and on the basis of the first feature, determining a first orientation angle between the sound source and the first device. The first feature is calculated on the basis of the single-channel voice signals of the first device, and then the orientation angle can be determined on the basis of the first feature, without pre-training a deep learning model. The orientation determination method has low costs, strong generalization, and occupies less computing power resources.

Description

Direction determination method and related equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on August 15, 2023, with application number 202311033120.7 and application name “Orientation Determination Method and Related Equipment”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of speech, and in particular to a direction determination method and related equipment. Background Art

[0003] With the prevalence of smart devices, the demand for human-computer interaction continues to increase. Users can interact with their devices through voice, gestures, and other control methods. In voice-based human-computer interaction scenarios, in the absence of other sensors such as cameras, it is generally necessary to use the user's audio to determine their orientation. This orientation helps infer the user's interaction intent, determine the focal device, and enhance the interactive experience of voice assistants.

[0004] Audio-based orientation detection requires the initial collection of audio data from multiple directions. This data is then used to train the model to learn orientation-related information for user orientation judgment. Models used include support vector machines (SVMs), random forests, and convolutional neural networks (CNNs). These detection methods face challenges such as high data collection costs, limited model generalization, high computing power requirements, and difficulty in device-side deployment.

[0005] Summary of the Invention

[0006] The present application provides a method for determining an orientation and related equipment, which does not require pre-training, has strong generalization, and uses low computing power and has low orientation determination cost.

[0007] In a first aspect, a method for determining an orientation is provided. The method can be executed by an orientation determining device itself, or by a chip in the orientation determining device.

[0008] The above-mentioned orientation determination method includes the following steps: obtaining a first characteristic of the first device. The first characteristic is obtained based on M single-channel voice signals corresponding to M audio acquisition units in the first device. M is an integer, and the M audio acquisition units collect audio from the same sound source. The first characteristic includes orientation-sensitive characteristics of direct sound and / or reflected sound in the sound signal generated by the sound source. A first orientation angle between the sound source and the first device is determined based on the first characteristic.

[0009] It can be seen that in this solution, the first feature is calculated based on the single-channel voice signal of the first device, and the orientation angle can be determined based on the first feature. There is no need to pre-train a deep learning model. The orientation determination method has low cost, strong generalization, and occupies little computing power resources.

[0010] In a possible implementation of the first aspect, obtaining the first feature of the first device specifically includes the following steps: obtaining M single-channel voice signals from the first device; obtaining a second feature based on the M single-channel voice signals, where the second feature is a feature of direct sound and / or reflected sound in the sound signal that is sensitive to direction; and obtaining a first feature based on the second feature, where the first feature is obtained based on high-frequency energy, low-frequency energy, or the difference between high-frequency energy and low-frequency energy in the second feature.

[0011] In this solution, M single-channel voice signals are first obtained from the first device, and then the second feature is obtained based on the above M single-channel voice signals. Finally, the first feature is obtained based on the high-frequency energy, low-frequency energy, or the difference between high-frequency energy and low-frequency energy in the second feature. Since the energy is normalized, this solution is not easily affected by factors such as the distance and orientation between the sound source and the first device when determining the orientation angle, and the orientation angle has high accuracy.

[0012] In one possible implementation of the first aspect, obtaining the second feature based on M single-channel speech signals specifically includes the following steps: performing the following processing on each single-channel speech signal to obtain a first spectrum corresponding to the single-channel speech signal. The single-channel speech signal is framed and windowed to obtain Q sub-speech signals, where Q is greater than or equal to two. Fourier transforming the Q sub-speech signals obtains a time-frequency spectrum. Signal modulation is performed on the time-frequency spectrum to obtain a second spectrum. Filtering the second spectrum obtains the first spectrum. The second feature is obtained based on the M first spectra.

[0013] In a possible implementation of the first aspect, obtaining the first characteristic of the first device specifically includes the following steps: obtaining a second characteristic from the first device, where the second characteristic is a characteristic of direct sound and / or reflected sound in the sound signal that is sensitive to direction; and obtaining the first characteristic based on the second characteristic, where the first characteristic is obtained based on high-frequency energy, low-frequency energy, or a difference between high-frequency energy and low-frequency energy in the second characteristic.

[0014] In this solution, a second feature of the direct sound and / or reflected sound, which is sensitive to direction, is first acquired from the first device. The first feature is then derived based on the high-frequency energy, low-frequency energy, or the difference between high-frequency and low-frequency energy in the second feature. Processing the second feature on the first device first effectively reduces the computing power required by this solution.

[0015] In a possible implementation of the first aspect, determining the first orientation angle between the sound source and the first device based on the first feature specifically includes the following steps: determining a second orientation angle based on the first feature and a preset correspondence, where the preset correspondence is a correspondence between different first features and second orientation angles; determining the first orientation angle based on the second orientation angle and a third orientation angle; and determining the third orientation angle based on an image captured by a camera of the first device.

[0016] In this solution, the second orientation angle can be determined using the preset correspondence and the first feature. To improve the accuracy of the orientation angle between the sound source and the first device, this application also determines a third orientation angle based on an image captured by the camera of the first device, and then combines the second and third orientation angles to determine the first orientation angle.

[0017] In a possible implementation of the first aspect, the orientation determination method further includes the following steps: when the first orientation angle falls within the first range and the first command word table includes the first command word, controlling the first device to enter a wake-up state and perform a first operation corresponding to the first command word. The first command word is a command word contained in the text corresponding to the sound signal. When the first orientation angle falls within the second range and the second command word table includes the first command word, controlling the first device to enter a wake-up state and perform the first operation. The number of command words in the first command word table is greater than the number of command words in the second command word table, and the absolute value of any orientation angle in the second range is less than the absolute value of any orientation angle in the first range.

[0018] In this solution, two orientation angle ranges, the first range and the second range, and the first command word table and the second command word table with different numbers of command words are used to determine whether to wake up the first device and whether to perform a voice interaction response. The difficulty of entering the wake-up state is adjusted according to the orientation angle, which can reduce the risk of false wake-up, ensure the accuracy of voice interaction, and enhance the user's voice interaction experience.

[0019] In a possible implementation of the first aspect, the orientation determination method further includes the following step: when the first orientation angle is zero, controlling the first device to enter a wake-up state. When the first orientation angle is zero, it is assumed that the user is interacting with the first device, and the first device can be controlled to enter a wake-up state without the user speaking a wake-up word, thereby simplifying the voice interaction process and improving the user's voice interaction experience.

[0020] In a possible implementation of the first aspect, after controlling the first device to enter the awake state, the direction determination method further includes the following steps: controlling the first device to perform a first operation corresponding to a first command word. The first command word is a command word contained in the text corresponding to the sound signal.

[0021] In this solution, after controlling the first device to enter the wake-up state without the user saying the wake-up word, the first device is controlled to execute the first operation corresponding to the command word contained in the text corresponding to the sound signal, in response to the user's voice interaction command.

[0022] In a second aspect, the present application also provides a focus device determination method, which can be executed by a focus device determination apparatus, or by a chip in the focus device determination apparatus.

[0023] The focus device determination method includes the following steps: obtaining N first features corresponding to N first devices. The first features of the first devices are obtained based on M single-channel voice signals corresponding to M audio acquisition units in the first devices. The N first devices perform audio acquisition for the same sound source. The first features include direction-sensitive features of direct sound and / or reflected sound in the sound signal generated by the sound source. N is greater than or equal to two, and M is an integer. A focus device is determined from the N first devices based on the N first features.

[0024] In this solution, the corresponding first feature is calculated based on the single-channel voice signal of each first device, and the focus device can be determined from the N first devices based on the N first features. This method does not require pre-training of the deep learning model. The focus device determination method has low cost, strong generalization, and occupies less computing power resources.

[0025] In a possible implementation of the second aspect, the above-mentioned determination of the focus device from N first devices based on N first features specifically includes the following steps: determining the first device corresponding to at least one first feature with a higher feature value among the N first features as the focus device.

[0026] In this solution, the first device corresponding to the first feature with the highest feature value among the N first features can be determined as the focus device, or the first devices corresponding to the x first features with the highest feature values ​​among the N first features can be determined as the focus device, where x is greater than or equal to two.

[0027] In a possible implementation of the second aspect, obtaining N first features corresponding to N first devices specifically includes the following steps: obtaining M single-channel voice signals from the first device. Determining a second feature of the first device based on the M single-channel voice signals. The second feature is a feature of direct sound and / or reflected sound in the sound signal that is sensitive to direction. The first feature is obtained based on the second feature, where the first feature is obtained based on high-frequency energy, low-frequency energy, or the difference between high-frequency energy and low-frequency energy in the second feature.

[0028] In this scheme, M single-channel voice signals are first obtained from the first device, and then the second feature is obtained based on the above M single-channel voice signals. Finally, the first feature is obtained according to the high-frequency energy, low-frequency energy or the difference between high-frequency energy and low-frequency energy in the second feature. Since the energy is normalized, this scheme is not easily affected by factors such as the distance and orientation between the sound source and the first device when determining the focus device, and the focus device has high accuracy.

[0029] In a possible implementation of the second aspect, determining the second feature of the first device based on M single-channel voice signals specifically includes the following steps: performing the following processing on each single-channel voice signal to obtain a first spectrum corresponding to the single-channel voice signal. The single-channel voice signal is framed and windowed to obtain Q sub-voice signals, where Q is greater than or equal to two. Fourier transforming the Q sub-voice signals obtains a time-frequency spectrum. Signal modulation is performed on the time-frequency spectrum to obtain a second spectrum. Filtering the second spectrum obtains the first spectrum. The second feature is obtained based on the M first spectra.

[0030] In a possible implementation of the second aspect, obtaining the N first features corresponding to the N first devices specifically includes the following steps: obtaining a second feature from the first device. The second feature is a feature of direct sound and / or reflected sound in the sound signal that is sensitive to direction. The first feature is obtained based on the second feature, where the first feature is obtained based on high-frequency energy, low-frequency energy, or the difference between high-frequency energy and low-frequency energy in the second feature.

[0031] In this solution, a second feature of the direct sound and / or reflected sound, which is sensitive to direction, is first acquired from the first device. The first feature is then derived based on the high-frequency energy, low-frequency energy, or the difference between high-frequency and low-frequency energy in the second feature. Processing the second feature on the first device first effectively reduces the computing power required by this solution.

[0032] In a possible implementation of the second aspect, the focus device determination method further includes the following steps: controlling the focus device to enter an awake state.

[0033] In this solution, the focus device can be controlled to enter the wake-up state without the user saying the wake-up word, which simplifies the voice interaction process and improves the user's voice interaction experience.

[0034] In a possible implementation of the second aspect, after controlling the focus device to enter the awake state, the focus device determination method further includes the following steps: controlling the focus device to execute a first operation corresponding to a first command word. The first command word is a command word contained in the text corresponding to the sound signal.

[0035] In this solution, after the focus device is controlled to enter the wake-up state without the user saying the wake-up word, the focus device is controlled to perform the first operation corresponding to the command word contained in the text corresponding to the sound signal, in response to the user's voice interaction command.

[0036] In a third aspect, the present application also provides a direction determination device, including an acquisition module and a determination module.

[0037] An acquisition module is configured to acquire a first characteristic of a first device. The first characteristic is obtained based on M single-channel voice signals corresponding to M audio acquisition units in the first device. M is an integer. The M audio acquisition units acquire audio from the same sound source. The first characteristic includes direction-sensitive characteristics of direct sound and / or reflected sound in the sound signal generated by the sound source.

[0038] A determination module is configured to determine a first orientation angle between the sound source and the first device based on the first feature.

[0039] In this solution, the direction determination device calculates the first feature based on the single-channel voice signal of the first device, and can determine the direction angle based on the first feature. There is no need to pre-train a deep learning model. The direction determination method has low cost, strong generalization, and occupies little computing power resources.

[0040] In a possible implementation of the third aspect, the orientation determination device further includes a processing module.

[0041] The processing module is configured to control the first device to enter a wake-up state and execute a first operation corresponding to the first command word when the first orientation angle falls within the first range and the first command word table includes a first command word. The first command word is a command word included in the text corresponding to the sound signal.

[0042] The processing module is further configured to control the first device to enter a wake-up state and perform a first operation when the first orientation angle falls within a second range and the second command word list includes the first command word. The number of command words in the first command word list is greater than the number of command words in the second command word list, and the absolute value of any orientation angle in the second range is less than the absolute value of any orientation angle in the first range.

[0043] In a possible implementation of the third aspect, the orientation determination device further includes:

[0044] The processing module is configured to control the first device to enter a wake-up state when the first orientation angle is zero.

[0045] In a possible implementation of the third aspect, after controlling the first device to enter the wake-up state, the processing module is also used to control the first device to perform a first operation corresponding to a first command word, where the first command word is a command word contained in the text corresponding to the sound signal.

[0046] In a fourth aspect, the present application also provides a focus device determination apparatus, comprising an acquisition module and a determination module.

[0047] An acquisition module is configured to acquire N first features corresponding to N first devices. The first features of the first devices are obtained based on M single-channel voice signals corresponding to M audio acquisition units in the first devices. The N first devices perform audio acquisition for the same sound source. The first features include direction-sensitive features of direct sound and / or reflected sound in the sound signal generated by the sound source. N is greater than or equal to two, and M is an integer.

[0048] A determination module is configured to determine a focus device from N first devices based on the N first features.

[0049] In this solution, the focus device determination device calculates the corresponding first feature based on the single-channel voice signal of each first device, and can determine the focus device from N first devices based on the N first features. This method does not require pre-training of a deep learning model. The focus device determination method has low cost, strong generalization, and occupies less computing power resources.

[0050] In a possible implementation of the fourth aspect, the focus device determination apparatus further includes a processing module.

[0051] The processing module is used to control the focus device to enter the awake state.

[0052] In a possible implementation of the fourth aspect, after the focus device is controlled to enter the awake state, the processing module is further used to control the focus device to execute a first operation corresponding to a first command word, where the first command word is a command word contained in the text corresponding to the sound signal.

[0053] In a fifth aspect, the present application also provides an electronic device comprising a processor and a memory, wherein the processor and the memory are connected, wherein the memory is used to store program code, and the processor is used to call the program code to execute the method described in any one of the first aspect or the second aspect.

[0054] In a sixth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method as described in any one of the first aspect or the second aspect.

[0055] In a seventh aspect, the present application further provides a computer program product comprising instructions, which, when run on a computer, enables the computer to execute the method described in any one of the first aspect or the second aspect.

[0056] In an eighth aspect, the present application further provides a chip, comprising a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface and executes the method described in any one of the first aspect or the second aspect.

[0057] Optionally, as an implementation method, the chip may further include a memory, in which instructions are stored, and the processor is used to execute the instructions stored on the memory. When the instructions are executed, the processor is used to execute the method described in any one of the first aspect or the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The following is an introduction to the drawings used in the embodiments of this application.

[0059] FIG1A is a schematic diagram of a first device provided in an embodiment of the present application;

[0060] FIG1B is a flow chart of a sound direction-based rejection strategy according to an embodiment of the present application;

[0061] FIG1C is a flow chart of another sound direction-based rejection strategy provided in an embodiment of the present application;

[0062] FIG1D is a schematic diagram of a voice interaction reference process based on sound direction provided in an embodiment of the present application;

[0063] FIG1E is a schematic diagram of a process for generating a target object list according to an embodiment of the present application;

[0064] FIG1F is a schematic diagram of an azimuth angle and an orientation angle provided in an embodiment of the present application;

[0065] FIG1G is a schematic diagram of a process for actively recommending references based on sound direction according to an embodiment of the present application;

[0066] FIG1H is a schematic diagram of a command action object determination process provided by an embodiment of the present application;

[0067] FIG1I is a schematic diagram of a speaker enhancement process based on sound direction provided in an embodiment of the present application;

[0068] FIG1J is a schematic diagram of a cockpit sound zone provided in an embodiment of the present application;

[0069] FIG1K is a schematic diagram of a human-to-human interaction assistant orientation enhancement process provided by an embodiment of the present application;

[0070] FIG1L is a schematic diagram of a process for calculating an angle difference according to an embodiment of the present application;

[0071] FIG1M is a schematic diagram of a visible and audible process based on sound direction provided by an embodiment of the present application;

[0072] FIG1N is a schematic diagram of a system architecture of a method for determining a direction provided in an embodiment of the present application;

[0073] FIG10 is a schematic diagram of the system architecture of another orientation determination method provided in an embodiment of the present application;

[0074] FIG1P is a schematic diagram of the system architecture of a method for determining a focus device provided in an embodiment of the present application;

[0075] FIG1Q is a schematic diagram of the system architecture of another method for determining a focal device provided in an embodiment of the present application;

[0076] FIG2 is a schematic flow chart of a method for determining an orientation according to an embodiment of the present application;

[0077] FIG3A is a schematic diagram of a process for obtaining a first spectrum provided in an embodiment of the present application;

[0078] FIG3B is a schematic diagram of a process for waking up a first device according to an embodiment of the present application;

[0079] FIG3C is a schematic diagram of a user experience dynamic effect provided by an embodiment of the present application;

[0080] FIG3D is a schematic diagram of another user experience dynamic effect provided by an embodiment of the present application;

[0081] FIG4 is a flow chart of a method for determining a focus device according to an embodiment of the present application;

[0082] FIG5 is a schematic structural diagram of a device provided in an embodiment of the present application;

[0083] FIG6 is a schematic structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0084] The technical solution in this application will be described below with reference to the accompanying drawings.

[0085] Due to the applications involved in the embodiments of the present application, in order to facilitate understanding, the relevant terms and other related concepts involved in the embodiments of the present application are first introduced below.

[0086] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0087] The "at least one" mentioned in the embodiments of this application refers to one or more, and "plurality" refers to two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, (a and b), (a and c), (b and c), or (a and b and c), where a, b, c can be single or multiple. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can be represented by: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the related objects before and after are in an "or" relationship. The serial numbers of the steps in the embodiments of this application (such as step S1, step S21, etc.) are only for distinguishing different steps and do not limit the order of execution between the steps.

[0088] Furthermore, unless otherwise specified, ordinal numbers such as "first" and "second" in the embodiments of this application are used to distinguish multiple objects and are not used to limit the order, timing, priority, or importance of multiple objects. For example, the first device and the second device are only for ease of description and do not indicate differences in structure, importance, etc. between the first and second devices. In some embodiments, the first device and the second device can also be the same device.

[0089] In the above embodiments, the term "when" can be interpreted to mean "if...", "after...", "in response to determining...", or "in response to detecting...", depending on the context. The above are merely optional embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the concepts and principles of the present application shall be included in the scope of protection of the present application.

[0090] The orientation angle between the device and the user is the angle between the normal of the user's face and the first connecting line, which is equivalent to the head posture. The first connecting line is the line between the device and the user.

[0091] With the prevalence of smart devices, the demand for human-computer interaction continues to increase. Users can interact with their devices through voice, gestures, and other control methods. In voice-based human-computer interaction scenarios, in the absence of other sensors such as cameras, it is generally necessary to use the user's audio to determine their orientation. This orientation helps infer the user's interaction intent, determine the focal device, and enhance the interactive experience of voice assistants.

[0092] Audio-based orientation detection requires the initial collection of audio data from multiple directions. This data is then used to train the model to learn orientation-related information and determine the user's orientation. These methods face challenges such as high data collection costs, limited model generalization, high computing power requirements, and difficulty in device-side deployment.

[0093] Therefore, an embodiment of the present application provides a method for determining a direction. The first feature is calculated based on the single-channel voice signal of the first device, and the direction angle can be determined based on the first feature. There is no need to pre-train a deep learning model. The direction determination method has low cost, strong generalization, and occupies little computing power resources.

[0094] In addition, an embodiment of the present application also provides a focus device determination method, which calculates the corresponding first feature based on the single-channel voice signal of each first device in N first devices, and then determines the focus device from the N first devices based on the N first features. This method does not require pre-training of a deep learning model. The focus device determination method has low cost, strong generalization, and occupies little computing power resources.

[0095] The first device in the orientation determination method and focus device determination method of the embodiment of the present application refers to a device with an audio acquisition unit (such as a microphone, etc.). The number of audio acquisition units can be 1, 2, 3, etc., and the specific number of audio acquisition units is not limited. Referring to Figure 1A, Figure 1A is a schematic diagram of a first device provided in an embodiment of the present application; the first device includes but is not limited to a mobile phone, a large screen, a speaker, a central control screen, a TV, a sports watch, a Bluetooth headset, a tablet computer, a car computer, etc. The above-mentioned first device has a voice interaction function to respond to the user's voice interaction commands.

[0096] Exemplarily, when there is only one first device in the environment (or scene, or space) where the user is located, the orientation angle between the first device and the user can be determined using the above orientation determination method.

[0097] As another example, referring to Figure 1A, when there are N first devices (N is greater than or equal to two) in the user's environment (or scene, or space), the above-mentioned focus device determination method is used to determine the focus device from the N first devices to respond to the user's voice interaction command, avoiding "one call and a hundred responses" and greatly improving the voice interaction experience.

[0098] The orientation determination method and focus device determination method of the embodiments of the present application are applicable to privacy scenarios without camera coverage, such as at home, in the back seat of a car, and other scenarios without cameras. During the voice interaction between the user and the device, the user orientation is judged through audio, or the focus device is determined, thereby improving the interactive experience of the voice assistant.

[0099] The orientation determination method and focus device determination method of the embodiments of the present application are also applicable to scenes covered by cameras, and can assist the camera in orientation determination and focus device determination, thereby improving the accuracy of orientation and focus device.

[0100] The orientation determination method of the embodiment of the present application can be executed by the orientation determination device itself, or by a chip in the orientation determination device. The orientation determination device can be the first device and / or other main control device (or central control device) with data processing capabilities.

[0101] The focus device determination method of the embodiment of the present application can be executed by the focus device determination device itself, or by a chip in the focus device determination device. The focus device determination device can be a first device and / or a main control device (or central control device) with data processing capabilities.

[0102] The following describes application scenarios related to the orientation determination method or focus device determination method according to embodiments of the present application:

[0103] The first scenario is a rejection strategy based on sound direction. For example, referring to FIG1B , FIG1B is a flowchart of a rejection strategy based on sound direction provided by an embodiment of the present application; specifically, the following steps are included:

[0104] S111. The orientation determining device obtains a first feature of the device.

[0105] The first feature is obtained based on M single-channel voice signals corresponding to M audio acquisition units in the device. M is an integer. The M audio acquisition units acquire audio from the same sound source, and one audio acquisition unit acquires audio to obtain a single-channel voice signal. The first feature includes directionally sensitive features of direct sound and / or reflected sound in the sound signal generated by the sound source.

[0106] S112. The direction determining device determines the direction angle between the device and the sound source according to the first feature.

[0107] For a detailed description, please refer to the following description of step 202, which will not be repeated here.

[0108] S113: If the direction corner falls into the third range, the direction determining device controls the device to perform voice recognition on the sound signal generated by the sound source.

[0109] Specifically, the third range indicates that the user is facing the device when interacting. For example, if the third range is -45° to 45°, the orientation determination device controls the device to respond to a sound signal emitted by the user and perform voice recognition on the sound signal. If the text corresponding to the sound signal includes a device command word, the device performs the operation corresponding to the command word. Otherwise, if the text corresponding to the sound signal does not include the device command word, the device does not perform the operation.

[0110] S114: If the direction corner falls into the fourth range, the direction determining apparatus controls the device to not respond to the sound signal generated by the sound source.

[0111] Specifically, the fourth range indicates that the user is not interacting with the device. At this time, the control device does not respond to the sound signal sent by the user, that is, rejects the sound signal.

[0112] For example, the user in the main driver's seat says to the passenger seat, "It's too hot, open the window," and the vehicle computer determines based on the first feature that the heading angle is not zero. It can be determined that the user is not speaking the command directly to the central control screen in the car (the central control screen is a display screen connected to the vehicle computer, i.e., the display screen for human-computer interaction). The vehicle computer determines that the main driver user is not interacting with the vehicle computer at this time and therefore rejects the command. However, when the user in the main driver's seat says to the central control screen, "It's too hot, open the window," the vehicle computer determines based on the first feature that the heading angle is zero at this time and that the heading angle falls within the third range. It can be determined that the user is speaking the command directly to the central control screen in the car. The vehicle computer determines that the main driver user wishes to interact with the vehicle computer at this time. Therefore, the vehicle computer responds to the user's voice command and opens the window next to the main driver's seat.

[0113] Furthermore, in FIG1B , the direction determination device may also perform speech recognition on the sound signal to obtain corresponding semantics, and combine the above-mentioned speech and the direction angle to determine whether the device rejects the sound signal.

[0114] As another example, referring to FIG1C , FIG1C is a flowchart of another sound direction-based rejection strategy provided by an embodiment of the present application; specifically, the following steps are included:

[0115] S121. The focus device determining apparatus obtains N first features corresponding to N devices.

[0116] The first feature is obtained based on M single-channel voice signals corresponding to M audio acquisition units in the device. M is an integer. The M audio acquisition units acquire audio from the same sound source, and one audio acquisition unit acquires audio to obtain a single-channel voice signal. The first feature includes directionally sensitive features of direct sound and / or reflected sound in the sound signal generated by the sound source.

[0117] S122 . The focus device determining means determines a focus device from N devices based on the N first features.

[0118] For a detailed description, please refer to the following description of step 402, which will not be repeated here.

[0119] S123: The focus device determining device controls the focus device to respond to the sound signal generated by the sound source.

[0120] Specifically, when the text corresponding to the sound signal includes the command word of the focus device, the focus device performs the operation corresponding to the command word. Otherwise, when the text corresponding to the sound signal does not include the command word of the focus device, the focus device does not perform the operation.

[0121] S124: The focus device determining device controls the non-focus devices to not respond to the sound signal generated by the sound source.

[0122] Specifically, the focus device determining means controls the focus device to not respond to the sound signal sent by the user, that is, to reject the sound signal.

[0123] The second scenario is voice interaction reference based on sound direction. For example, referring to FIG1D , FIG1D is a schematic diagram of a voice interaction reference process based on sound direction provided by an embodiment of the present application; specifically, the following steps are included:

[0124] 1) The device receives a voice signal, and the orientation detection module performs orientation detection to determine the user's orientation angle β.

[0125] 2) The sound zone positioning module performs sound zone positioning to determine the user's position coordinates, such as locating the user at the main driving position.

[0126] 3) The speech recognition module performs speech recognition to extract keyword information related to the object of the command.

[0127] 4) Perform semantic understanding of the keyword information to determine if there is any ambiguity. For example, if a user says "increase the angle of the driver's side mirror," there is no ambiguity in the car's only driver's side mirror. The target can be directly determined to be the driver's side mirror and a response can be given.

[0128] 5) For another example, the user says "increase the rearview mirror angle". The keyword is rearview mirror, and there are two rearview mirrors on the car. In this case, there is ambiguity, and the target object list needs to be generated based on the user's orientation angle and position coordinates.

[0129] 6) Determine whether there is only one candidate object in the target object list. If so, designate the candidate object as the target object. The vehicle computer responds and executes the control operation corresponding to the voice signal, such as increasing the rearview mirror angle.

[0130] If the target object list is empty or contains multiple candidate objects, the vehicle computer guides the user into a second round of interaction through voice or other means to clarify the target of the command. For example, the vehicle computer may ask the user through voice or text output on the screen, "Do you want to adjust the angle of the driver's side mirror or the passenger side mirror?"

[0131] The process shown in FIG. 1D may be executed by a direction determining device.

[0132] For example, when the car computer has been awakened, the user sitting in the main driver's seat says to the rearview mirror on the passenger side: "Adjust the angle of the rearview mirror to a wider angle." The car computer executes the process shown in Figure 1D. The car computer recognizes the command and, based on the position information of the two rearview mirrors in the car and the user's direction, infers that the user wants to change the rearview mirror on the passenger side. The voice assistant responds: "OK, the angle of the passenger rearview mirror has been increased."

[0133] In another example of a voice interaction reference scenario based on sound direction, the user sitting in the main driver's seat says to the rearview mirror on the passenger side: "XXX, adjust the angle of the rearview mirror to a wider angle." "XXX" is the wake-up word of the car computer, such as "Xiaoyi Xiaoyi." At this time, the car computer executes the process shown in Figure 1D. The car computer recognizes the command, combines the position information of the two rearview mirrors on the car and the user's direction, and infers that the user wants to change the rearview mirror on the passenger side. The voice assistant responds: "OK, the angle of the passenger rearview mirror has been increased."

[0134] The target object list is generated by combining the user's location coordinates, orientation angle, and the coordinate information of the candidate object to determine whether to add the object to the target object list. The target object list is initialized to an empty list. For candidate object i, the method for determining whether to add it to the target object list is as follows. Referring to Figure 1E, Figure 1E is a schematic diagram of a process for generating a target object list provided in an embodiment of the present application; the specific steps include:

[0135] 1) The car computer has a built-in spatial coordinate information database of physical modules (such as windows, rearview mirrors, lights, seats, etc.) that can be controlled by voice. Based on the keyword information (such as rearview mirror) in Figure 1D and the above spatial coordinate information database, the two-dimensional coordinate information t of the candidate object i can be obtained. i =[x i ,y i ].

[0136] 2) Referring to FIG. 1F, FIG. 1F is a schematic diagram of an azimuth and orientation angle provided in an embodiment of the present application; combined with the two-dimensional coordinate information t of the candidate object i i And the user's position coordinates b = [b1, b2], calculate the azimuth of candidate object i where tan -1 is the inverse tangent function.

[0137] 3) According to the azimuth angle α i Calculate the angle difference from the user's orientation angle β The angle difference Compared with the preset angle threshold θ, if If the candidate object i is not in the area directly in front of the user, the target object list will not be updated; otherwise, the candidate object i will be added to the target object list.

[0138] The third scenario is active recommendation based on sound direction. For example, referring to FIG1G , FIG1G is a schematic diagram of an active recommendation based on sound direction provided by an embodiment of the present application; specifically, the following steps are included:

[0139] 1) The device receives a voice signal, and the orientation detection module performs orientation detection to determine the user's orientation angle.

[0140] 2) The speech recognition module performs speech recognition to extract keyword information related to the object of the command.

[0141] 3) Semantic understanding of the keyword information is performed, and the system combines it with map information near the user's location to determine if there is any ambiguity in the reference. For example, if a user asks, "What's the name of this neighborhood?" and there is only one residential area near the user's location (e.g., on the left side of the road), there is no ambiguity. The system can directly determine that the user is referring to the neighborhood on the left side of the road, and the vehicle computer can respond directly.

[0142] 4) If there is more than one cell near the user's location, there is ambiguity and the command target determination process needs to be entered.

[0143] When determining the target of a command, the user's orientation angle and the location information of the candidate object are combined to determine the target of the command. If the target can be determined (such as the community on the left side of the road), the vehicle computer will respond ("The community on the right side of the road is xx community").

[0144] If the target object cannot be determined, the car computer will guide the user into a second round of interaction through voice or other means to clarify the target object of the command. For example, the car computer will ask the user through voice or text on the screen: "Do you want to inquire about the community on the left side of the road or the community on the right side of the road?"

[0145] The process shown in FIG. 1G may be executed by a direction determining device.

[0146] 1H , which is a schematic diagram of a command action target determination process according to an embodiment of the present application. The command action target determination process includes the following steps:

[0147] 1) Determine the reference area based on the user's heading angle. For example, if β < -45, the reference area is the area to the left of the user's location. If β > 45, the reference area is the area to the right of the user's location. If -45 < β < 45, the reference area is the area directly in front of the user's location.

[0148] 2) Combine the map information near the user's location and the reference area to determine whether there is a candidate object in the reference area (for example, if the user is asking for the name of a community, determine whether there is a residential area in the reference area). If the candidate object exists and is the only one that exists, then that object is set as the command target; if the candidate object does not exist or there are multiple candidate objects, then the command target cannot be determined.

[0149] For example, when the car computer has been awakened, the user sitting in the main driver's seat looks out the left window and says, "What is the name of this community?" The car computer executes the process shown in Figure 1G. The car computer recognizes the command and determines based on the user's direction that the community the user is asking about is the community on the left side of the road rather than the right side, and gives an answer.

[0150] In another example of a scenario where active recommendation is provided based on the direction of the sound, the user sitting in the main driver's seat looks out the left window and says, "XXX, what is the name of this community?" "XXX" is the wake-up word for the car computer, such as "Xiaoyi Xiaoyi". At this time, the car computer executes the process shown in Figure 1G. The car computer recognizes the command and determines, based on the user's direction, that the community the user is asking about is the community on the left side of the road rather than the right side, and gives an answer.

[0151] The fourth scenario is speaker enhancement based on sound direction. For example, referring to FIG1I , FIG1I is a schematic diagram of a speaker enhancement process based on sound direction provided in an embodiment of the present application; specifically, the following steps are included:

[0152] 1) The device receives a voice signal, and the orientation detection module performs orientation detection to determine the user's orientation angle.

[0153] 2) The sound zone positioning module performs sound zone positioning to determine the user's position.

[0154] 3) Determine whether the user is facing the boundary based on the user's heading angle and position. The determination method is as follows: For a general-purpose car cabin, there are generally two or three rows of seats, with two or three seats in each row. The boundary here is defined as the intersection of two seats in the same row. If the user is sitting on the left side (referring to Figure 1J, taking three rows of seats and two seats per row as an example, the left side is the position of row 1 left, row 2 left, and row 3 left in Figure 1J), if the user's heading angle β is greater than 45 degrees, the user is considered to be facing the boundary; otherwise, the user is considered not facing the boundary. If the user is sitting on the right side (referring to Figure 1J, taking three rows of seats and two seats per row as an example, the right side is the position of row 1 right, row 2 right, and row 3 right in Figure 1J), if the user's heading angle β is less than -45 degrees, the user is considered to be facing the boundary; otherwise, the user is considered not facing the boundary. If the user is detected to be facing the boundary, the next step is performed; otherwise, only the sound zone where the user is located is enhanced.

[0155] 4) The signal-to-noise ratio detection module detects the current cabin signal-to-noise ratio (SNR). If the current cabin SNR is greater than a preset threshold, SNR0, the cabin is considered relatively quiet and both the user's audio zone and adjacent audio zones can be enhanced. Otherwise, only the user's audio zone is enhanced. Adjacent audio zones refer to those in the same row as the user's audio zone. For example, if the user is sitting on the left side of row 1, the adjacent audio zone is the right side of row 1.

[0156] Optionally, if computing power is sufficient, the signal-to-noise ratio detection module can work in parallel with the orientation detection module and the sound zone positioning module to reduce the overall link latency and improve the interactive experience.

[0157] The process shown in FIG. 1I may be executed by a direction determining device.

[0158] The fifth scenario is the enhancement of the orientation of the human-to-human interaction assistant. For example, referring to FIG1K , FIG1K is a schematic diagram of the orientation enhancement process of the human-to-human interaction assistant provided in an embodiment of the present application; specifically, the following steps are included:

[0159] 1) The device receives a voice signal, and the direction detection module performs direction detection to determine the speaker's direction angle β.

[0160] 2) The speaker localization module is used to determine the speaker's position coordinates b = [b1, b2].

[0161] 3) The interaction object recognition module recognizes the interaction object, such as interaction object B, based on the speaker's position coordinate b, the speaker's orientation angle β and the preset participant position information.

[0162] 4) The speaker recognition module is used to determine the identity of the current speaker, such as speaker A.

[0163] 5) The speech recognition module is used to recognize the current spoken content, such as "Do you have any suggestions?"

[0164] 6) Generate meeting minutes based on the speaker, interaction object and speech content, such as "A said to B, do you have any suggestions?"

[0165] The process shown in FIG. 1K may be executed by a direction determining device.

[0166] For example, the location information of the participants in step 3) is system preset information, and the acquisition method includes at least one of the following methods:

[0167] A. For some meeting scenarios with fixed seats, the names and location coordinates of the people in each position can be entered into the system in advance according to the meeting seat distribution map.

[0168] B. For meetings with non-fixed seating, participants can register after they are seated. For example, each person says their name and seat number, and the system records and generates the participant's location information.

[0169] C. If cameras are available, real-time detection of personnel information at each location during the meeting entrance can be used to obtain the location information of participants in combination with the face recognition system.

[0170] For example, one implementation of the interactive object recognition module in step 3) is: for N participants in the scene excluding the current speaker, calculate the angle difference between each participant and the speaker. Select the smallest angle difference The corresponding participant is regarded as the current interaction object B.

[0171] As another example, another implementation of the interactive object recognition module in step 3) is: for N participants in the scene excluding the current speaker, calculate the angle difference between each participant and the speaker. Determine the smallest angle difference among the above N angle differences Set the candidate object list to an empty list and iterate over all angle differences like Then add the corresponding participant i to the candidate list, is the preset angle difference threshold. When the angle difference of the participants is less than Calculate the distance between each participant and the speaker in the candidate list and select the participant with the smallest distance as the current interaction object B.

[0172] For example, referring to FIG1L, FIG1L is a flowchart of calculating an angle difference according to an embodiment of the present application; for participant i, calculating the angle difference The specific method is:

[0173] Based on the preset participant location information, the location coordinates t of participant i are obtained i =[x i ,y i ], combined with the speaker's position coordinates b = [b1, b2] to calculate the azimuth angle corresponding to participant i where tan -1 is the inverse tangent function.

[0174] According to the azimuth angle α i Calculate the angle difference from the speaker's orientation angle β

[0175] The speaker recognition module can at least be implemented based on voiceprint technology. Voiceprint registration can be performed after participants are seated. For example, after each participant says their name and seat number to register, voiceprint registration is performed at the same time. Voiceprint registration can also be completed before the meeting.

[0176] If a camera is available, the speaker localization module, direction detection module, and speaker recognition module can fuse audio and visual information for multimodal detection.

[0177] For example, the microphone array in the conference room is set in the middle of the conference room. The direction of arrival (DOA) information can be combined to determine the direction of the participants. When speaking, the speaker's conversation partner can be judged based on the direction and recorded in real time in the meeting minutes.

[0178] The sixth scenario is visible and speakable based on the direction of the sound. Visible and speakable is a voice interaction feature, which refers to what interface elements can be seen on the interface. As long as you say its name, the system will operate the element by simulating a click. For example, there is a Bluetooth switch on the interface. The user only needs to say "turn on the Bluetooth switch" or "turn off the Bluetooth switch", and the system will simulate clicking the hot area of ​​the Bluetooth switch to change the Bluetooth switch. This function is generally used on devices with screens. When the user wakes up and the device enters the wake-up state, this function will be triggered. The above hot areas are the interactive parts of the interface (that is, the clickable places).

[0179] For example, referring to FIG. 1M , FIG. 1M is a schematic diagram of a visible and audible process based on sound direction provided by an embodiment of the present application; specifically, the following steps are included:

[0180] 1) The device receives a voice signal, and the direction detection module performs direction detection to determine the speaker's direction angle β.

[0181] 2) Determine whether the heading angle falls within a third range. The third range indicates that the user is facing the device. For example, the third range is -45° to 45°. If the heading angle meets this condition, the device is considered to be directly in front of the user. If this condition is met, the voice recognition module is activated. Otherwise, the voice recognition module is rejected and no response is given.

[0182] 3) The voice recognition module extracts keywords from the voice signal and compares them with the keywords corresponding to the touchable elements on the current screen to determine whether the current screen contains the keyword. If the keyword is contained, the command is executed; otherwise, the voice is rejected and no response is given.

[0183] Optionally, if computing power is sufficient, the voice recognition module can work in parallel with the direction detection module to reduce the overall link latency and improve the interactive experience.

[0184] For example, the user is saying "Open Settings" to the device, and the device detects that the user is facing the device (the orientation angle is zero), and there is an interface element "Settings" on the current screen. The device will simulate the user clicking "Settings" and enter the settings page. On the contrary, when the user is facing the device, but there is no element indicated by the sound signal on the current display screen, the control device will guide the user into the second round of interaction through voice or other means to clarify the operation element. For example, the device asks the user by making a voice sound or outputting text on the screen: "The element is not displayed on the screen. Please indicate again or slide the display screen."

[0185] The following describes the system architecture of the embodiment of the present application:

[0186] Refer to Figure 1N, which is a schematic diagram of the system architecture of a direction determination method provided in an embodiment of the present application; in the embodiment of the present application, the system architecture of the direction determination method includes a first device, a first feature calculation unit and a direction detection unit, the first device includes m audio acquisition units, m is a positive integer, the first feature calculation unit is used to calculate the first feature based on the m single-channel voice signals of the first device, and the direction detection unit is used to determine the first direction angle between the user and the first device based on the first feature.

[0187] Exemplarily, in FIG. 1N , the first feature calculation unit and the orientation detection unit may be provided in the first device.

[0188] As another example, in FIG1N , the first feature calculation unit and the orientation detection unit may be provided in the main control device.

[0189] As another example, in FIG1N , the first feature calculation unit may be provided in the first device, while the orientation detection unit may be provided in the master device. Alternatively, the first feature calculation unit may be provided in the master device, while the orientation detection unit may be provided in the first device.

[0190] Refer to Figure 1O, which is a schematic diagram of the system architecture of another orientation determination method provided in an embodiment of the present application; the system architecture of the application orientation determination method includes a first device, a second feature calculation unit, a first feature calculation unit and a orientation detection unit, the first device includes m audio acquisition units, m is a positive integer, the second feature calculation unit is used to calculate the second feature based on the m single-channel voice signals of the first device, the first feature calculation unit is used to calculate the first feature based on the second feature, and the orientation detection unit is used to determine the first orientation angle between the user and the first device based on the first feature.

[0191] Exemplarily, in FIG1O , the second feature calculation unit, the first feature calculation unit, and the orientation detection unit may be provided in the first device.

[0192] As another example, in FIG10 , the second feature calculation unit, the first feature calculation unit, and the orientation detection unit may be provided in the main control device.

[0193] As another example, in FIG1O , the second feature calculation unit and the first feature calculation unit may be provided in the first device; and the orientation detection unit may be provided in the main control device.

[0194] Alternatively, the orientation detection unit may be provided in the first device, and the second feature calculation unit and the first feature calculation unit may be provided in the main control device.

[0195] Alternatively, the second feature calculation unit and the direction detection unit may be provided in the first device; and the first feature calculation unit may be provided in the main control device.

[0196] Alternatively, the first feature calculation unit may be provided in the first device, and the second feature calculation unit and the direction detection unit may be provided in the main control device.

[0197] Alternatively, the first feature calculation unit and the direction detection unit may be provided in the first device; and the second feature calculation unit may be provided in the main control device.

[0198] Alternatively, the second feature calculation unit may be provided in the first device; and the first feature calculation unit and the direction detection unit may be provided in the main control device.

[0199] Referring to FIG1P , FIG1P is a schematic diagram of the system architecture of a focus device determination method provided in an embodiment of the present application; the system architecture applying the focus device determination method includes N first devices (such as first device 1, first device 2, ... first device N), N first feature calculation units (such as first feature calculation unit 1, first feature calculation unit 2, ... first feature calculation unit N) and a processing unit, N is greater than or equal to two, each first device includes m audio acquisition units, m is a positive integer, the first feature calculation unit is used to calculate the first feature based on the m single-channel voice signals of each first device, and the processing unit is used to determine the focus device from the N first devices based on the first feature. Exemplarily, the processing unit is also used to control the focus device to enter the wake-up state.

[0200] Exemplarily, the N first feature calculation units mentioned above may also be combined into one first feature calculation unit, which is used to process the single-channel voice signals of the N first devices respectively to obtain N first features.

[0201] Exemplarily, when the focus device determination apparatus is N first devices, referring to FIG1P , the first feature calculation unit 1 is provided in the first device 1, and is configured to calculate the first feature based on the m single-channel speech signals of the first device 1. The first feature calculation unit 2 is provided in the first device 2, and is configured to calculate the first feature based on the m single-channel speech signals of the first device 2. Similarly, the first feature calculation unit N is provided in the first device N, and is configured to calculate the first feature based on the m single-channel speech signals of the first device N.

[0202] The processing unit is provided in each first device, that is, the first device 1, the first device 2, ... the first device N are each provided with a processing unit. Further, taking the first device 1 as an example, after the first device 1 calculates the first feature corresponding to itself, the first device 1 determines the focus device from the N first devices based on the first features of the N first devices. When the focus device is the first device 1, the first device 1 enters the awake state. Assuming that the focus device is the first device N, the first device 1 controls the first device N to enter the awake state. Similarly, the working process of the first device 2, ... the first device N is the same as that of the first device 1 and will not be repeated.

[0203] Exemplarily, when the focus device determination apparatus is a master device with data processing capability, referring to FIG1P , the first feature calculation unit 1, the first feature calculation unit 2, ... the first feature calculation unit N are set in the master device, and the processing unit is also set in the master device.

[0204] Exemplarily, when the focus device determining apparatus is N first devices and a master device with data processing capabilities, referring to FIG1P , first feature calculation unit 1 is provided in first device 1, first feature calculation unit 2 is provided in first device 2, and so on, first feature calculation unit N is provided in first device N. The processing unit is provided in the master device.

[0205] Exemplarily, when the focus device determination apparatus comprises N first devices and a master device with data processing capabilities, referring to FIG. 1P , first feature calculation unit 1, first feature calculation unit 2, ..., first feature calculation unit N are provided in the master device. The processing unit is provided in any one of the N first devices, such as first device 2.

[0206] Exemplarily, when the focus device determination apparatus is N first devices and a master device with data processing capabilities, with reference to FIG1P , Y of the N first feature calculation units are set in the master device, where Y is a positive integer, and the remaining NY first feature calculation units are respectively set in the corresponding first devices, and the processing unit is set in the master device. For example, if Y is 3, it is assumed that the three first feature calculation units corresponding to the first device 1, the first device 2, and the first device 3 (i.e., the first feature calculation unit 1, the first feature calculation unit 2, and the first feature calculation unit 3) are set in the master device, and the first feature calculation unit 4 is set in the first device 4, the first feature calculation unit 5 is set in the first device 5, and so on, the first feature calculation unit N is set in the first device N.

[0207] The specific configuration of the first feature calculation unit and the processing unit in FIG1P is not limited to the above example.

[0208] Referring to Figure 1Q, Figure 1Q is a schematic diagram of the system architecture of another focus device determination method provided in an embodiment of the present application; the system architecture applying the focus device determination method includes N first devices (such as first device 1, first device 2, ... first device N), N second feature calculation units (such as second feature calculation unit 1, second feature calculation unit 2, ... second feature calculation unit N), N first feature calculation units (such as first feature calculation unit 1, first feature calculation unit 2, ... first feature calculation unit N) and a processing unit, N is greater than or equal to two, each first device includes m audio acquisition units, m is a positive integer, the second feature calculation unit is used to calculate the second feature based on the m single-channel voice signals of each first device, the first feature calculation unit is used to calculate the first feature based on the second feature of each first device, and the processing unit is used to determine the focus device from the N first devices based on the first feature. Exemplarily, the processing unit is also used to control the focus device to enter the wake-up state.

[0209] Exemplarily, the N second feature calculation units may be combined into one second feature calculation unit for processing the single-channel voice signals of the N first devices, respectively, to obtain N second features. Similarly, the N first feature calculation units may be combined into one first feature calculation unit for processing the N second features, respectively, to obtain N first features.

[0210] Exemplarily, when the focus device determination apparatus is N first devices, referring to FIG1Q , the second feature calculation unit 1 is provided in the first device 1 for calculating the second feature based on the m single-channel voice signals of the first device 1. The second feature calculation unit 2 is provided in the first device 2 for calculating the second feature based on the m single-channel voice signals of the first device 2. Similarly, the second feature calculation unit N is provided in the first device N for calculating the second feature based on the m single-channel voice signals of the first device N. The first feature calculation unit 1 is provided in the first device 1 for calculating the first feature based on the second feature of the first device 1. The first feature calculation unit 2 is provided in the first device 2 for calculating the first feature based on the second feature of the first device 2. Similarly, the first feature calculation unit N is provided in the first device N for calculating the first feature based on the second feature of the first device N.

[0211] The processing unit is provided in each first device, that is, the first device 1, the first device 2, ... the first device N are each provided with a processing unit. Further, taking the first device 1 as an example, after the first device 1 calculates the first feature corresponding to itself, the first device 1 determines the focus device from the N first devices based on the first features of the N first devices. When the focus device is the first device 1, the first device 1 enters the awake state. Assuming that the focus device is the first device N, the first device 1 controls the first device N to enter the awake state. Similarly, the working process of the first device 2, ... the first device N is the same as that of the first device 1 and will not be repeated.

[0212] Exemplarily, when the focus device determination device is a main control device with data processing capabilities, referring to Figure 1Q, the second feature calculation unit 1, the second feature calculation unit 2,... the second feature calculation unit N, the first feature calculation unit 1, the first feature calculation unit 2,... the first feature calculation unit N are set in the main control device, and the processing unit is also set in the main control device.

[0213] Exemplarily, when the focus device determination apparatus is N first devices and a master device with data processing capabilities, referring to FIG1Q , the second feature calculation unit 1 and the first feature calculation unit 1 are provided in the first device 1, the second feature calculation unit 2 and the first feature calculation unit 2 are provided in the first device 2, and so on, the second feature calculation unit N and the first feature calculation unit N are provided in the first device N. The processing unit is provided in the master device.

[0214] For example, when the focus device determination apparatus comprises N first devices and a master device with data processing capabilities, referring to FIG1Q , second feature calculation unit 1, second feature calculation unit 2, ..., second feature calculation unit N, first feature calculation unit 1, first feature calculation unit 2, ..., first feature calculation unit N are provided in the master device. The processing unit is provided in any one of the N first devices, for example, first device 2.

[0215] Exemplarily, when the focus device determination apparatus is N first devices and a master control device with data processing capabilities, referring to FIG1Q , Y second feature calculation units out of the N second feature calculation units and Y first feature calculation units out of the N first feature calculation units are set in the master control device, where Y is a positive integer, and the remaining NY second feature calculation units and the remaining NY first feature calculation units are respectively set in the corresponding first devices, and the processing unit is set in the master control device.

[0216] The specific configuration of the second feature calculation unit, the first feature calculation unit, and the processing unit in FIG1Q is not limited to the above example.

[0217] For example, in the system architecture illustrated in FIG1P and FIG1Q above, the system architecture applying the focus device determination method further includes a communication system for enabling communication between the N first devices and / or for enabling communication between the N first devices and the master device. The communication system may be a Bluetooth communication system or a mobile hotspot communication system (such as a WiFi communication system). For example, a first device may obtain the first feature or the second feature of another first device through the communication system, or the master device may obtain the first feature or the second feature of a first device through the communication system.

[0218] The following is a detailed description of the method for determining the orientation:

[0219] Referring to FIG. 2 , FIG. 2 is a flow chart of a method for determining a direction provided in an embodiment of the present application. In the embodiment of the present application, the execution subject takes a direction determining device as an example, and the direction determining method 200 includes the following steps:

[0220] 201. A direction determining device obtains a first feature of a first device.

[0221] Specifically, the first feature is obtained based on M single-channel voice signals corresponding to M audio acquisition units in the first device. M is an integer, such as 1, 2, or 3. The M audio acquisition units acquire audio from the same sound source, and one audio acquisition unit acquires audio to obtain a single single-channel voice signal. The first feature includes directionally sensitive features of direct sound and / or reflected sound in the sound signal generated by the sound source.

[0222] 202. The direction determining device determines a first direction angle between the sound source and the first device based on the first feature.

[0223] In an embodiment of the present application, the first feature is calculated based on the single-channel voice signal of the first device, and the orientation angle can be determined based on the first feature. There is no need to pre-train a deep learning model. The orientation determination method has low cost, strong generalization, and occupies little computing power resources.

[0224] In a possible implementation, step 201 specifically includes the following steps:

[0225] 211. The direction determining apparatus obtains M single-channel voice signals from the first device.

[0226] Specifically, the first device is in a listening state (i.e., a standby state), and when a sound source appears, the audio collection unit in the first device collects audio to obtain a single-channel voice signal. In this embodiment, the direction determination device obtains M single-channel voice signals from the first device.

[0227] Exemplarily, the first device sends M single-channel voice signals to the direction determining apparatus. Also exemplarily, the direction determining apparatus sends a voice signal request to the first device, and the first device sends M single-channel voice signals to the direction determining apparatus in response to the voice signal request.

[0228] 212. The direction determining device obtains a second feature based on the M single-channel voice signals, where the second feature is a feature of the direct sound and / or reflected sound in the sound signal that is sensitive to direction.

[0229] 213. The direction determining device obtains the first feature based on the second feature, where the first feature is obtained according to the high-frequency energy, the low-frequency energy, or the difference between the high-frequency energy and the low-frequency energy in the second feature.

[0230] In an embodiment of the present application, M single-channel voice signals are first obtained from the first device, and then the second feature is obtained based on the above M single-channel voice signals. Finally, the first feature is obtained based on the high-frequency energy, low-frequency energy or the difference between high-frequency energy and low-frequency energy in the second feature. Since the energy is normalized, the embodiment of the present application is not easily affected by factors such as the distance and orientation between the sound source and the first device when determining the orientation angle, and the orientation angle has high accuracy.

[0231] In a possible implementation, step 212 specifically includes the following steps:

[0232] A1. The direction determination device performs the following processing on each single-channel speech signal to obtain a first spectrum corresponding to the single-channel speech signal. Referring to FIG3A , FIG3A is a schematic diagram of a flow chart of obtaining a first spectrum provided in an embodiment of the present application. The above processing specifically includes:

[0233] A11, frame and window the single-channel voice signal to obtain Q-segment voice signals [x1…x Q ], Q is greater than or equal to 2. The window length is L, and the length of each speech signal segment is L. In the above calculation, the window function can be selected from rectangular window, Hamming window, Hamming window, etc. according to the actual usage scenario. The frame shift can also be adjusted as needed. For example, the frame shift can be set to L / 2.

[0234] A12, perform Fourier transform on the Q segment voice signal to obtain the time spectrum F0. For example, when the P-dimensional Fourier transform is performed on the Q segment voice signal,

[0235] A13. Perform signal modulation on the time-frequency spectrum F0 to obtain a second frequency spectrum F2.

[0236] Specifically, the first filter group is used to modulate the signal. The time spectrum F0 is used as the input of the first filter group, and the second spectrum is obtained after filtering by the first filter group. in is the filter coefficient matrix of the first filter group, comprising a total of E channels (i.e., the first filter group includes E filters, E is greater than or equal to two, and the E filters are in parallel). Each channel corresponds to a different center frequency. Exemplarily, to facilitate calculation of the first feature, the first row of G corresponds to the lowest frequency channel, and the last row of G corresponds to the highest frequency channel. Optionally, the first filter group can be a gammatone filter group, a Mel filter group, a wavelet transform filter group, etc.

[0237] A14. Filter the second spectrum F2 to obtain the first spectrum F1. The filtering process filters out the characteristics of the direct sound and / or reflected sound that are sensitive to direction in the second spectrum F2. For example, the second spectrum F2 is filtered to obtain a third spectrum F3, and the first spectrum F1 is obtained based on the third spectrum F3.

[0238] Specifically, the second spectrum F2 is used as the input of a second filter bank (comprising K channels in total, where K is greater than or equal to 1), and a third spectrum F3 is obtained after filtering.

[0239] Specifically, the second spectrum F2 is used as the input of K channels, and each channel outputs a sub-spectrum Combine the above K sub-spectra to obtain the third spectrum

[0240] Exemplarily, the second filter bank is a bandpass filter, with each channel corresponding to a different parameter setting to obtain direction-sensitive direct sound and / or reflected sound. The parameters include at least one of the following: center frequency, quality factor, and order. The speech signal received by the first device includes direct sound and reflected sound at different stages. Exemplarily, the output of the low-frequency bandpass filter corresponds to the direct sound and / or early reflections in the speech signal, and the output of the high-frequency bandpass filter corresponds to the late reflections in the speech signal.

[0241] The first spectrum F1 is calculated based on the third spectrum F3, specifically including: summing and transposing the time dimension (i.e., the third dimension) of the third spectrum F3 to obtain the first spectrum

[0242] A2. The direction determining device obtains a second feature based on the M first frequency spectra.

[0243] Exemplarily, the M first spectra F1 are averaged to obtain the second feature. Specifically, the second feature T2 is obtained by adding the corresponding elements of the M first spectra F1 and dividing by M, that is, The first row corresponds to the lowest frequency channel of the first filter group, and the Eth row corresponds to the highest frequency channel of the first filter group.

[0244] In step 213, when the second feature is represented in matrix form, the high-frequency energy in the second feature is used as the first feature. Exemplarily, when processing the elements row by row, the matrix elements of the rows corresponding to the high-frequency output channels in the second feature are summed to obtain the high-frequency energy.

[0245] As another example, when processing the elements column by column, the matrix elements of the columns corresponding to the high-frequency output channels in the second feature are summed and then inverted to obtain the high-frequency energy.

[0246] In step 213, when the second feature is represented in matrix form, the low-frequency energy in the second feature is used as the first feature. For example, when processing elements row by row, the matrix elements of the rows corresponding to the low-frequency output channels in the second feature are summed and then inverted to obtain the low-frequency energy.

[0247] As another example, when processing the elements column by column, the matrix elements of the columns corresponding to the low-frequency output channels in the second feature are summed to obtain the low-frequency energy. For example, assuming that each filter in the second filter group is a low-frequency filter, that is, each matrix element in the second feature corresponds to low-frequency energy, then summing each column of the second feature T2 yields in Calculate the first feature based on matrix P Among them, the element p in the matrix P j The energy of the sound wave reflected in different stages in space, the larger the j, the later the reflection. Therefore, the output p of the low-frequency output channel j The sum corresponds to the energy of the direct sound and early reflections in the signal. This energy is greater if the user is facing the device, and smaller if the user is facing away from the device. Therefore, T1 can be used to determine the user's orientation.

[0248] In step 213, when the second feature is represented in matrix form, the difference between the high-frequency energy and the low-frequency energy in the second feature is used as the first feature. The above difference can be high-frequency energy / low-frequency energy, low-frequency energy / high-frequency energy, |high-frequency energy-low-frequency energy|, |low-frequency energy-high-frequency energy|, etc., / represents division, and || represents absolute value.

[0249] For example, the ratio of high frequency energy to low frequency energy in the second feature is used as the first feature. When processing the elements row by row, the low frequency energy part and the high frequency energy part of the second feature are summed up to obtain the low frequency energy and high-frequency energy in Based on the obtained E L and E H Calculate the ratio of high frequency energy to low frequency energy in The first feature is obtained by summing the high-frequency and low-frequency energy ratio R The above v can be set according to actual conditions without limitation. For example, v can be set to E / 2.

[0250] For example, the ratio of low frequency energy to high frequency energy in the second feature is used as the first feature. When processing the elements column by column, the sum of each column of the second feature T2 is obtained. in Assume that the first k channels in the second filter group are low-frequency channels; the k+1~Kth channels are high-frequency channels, and the first feature is calculated based on the matrix P. The energy is normalized by calculating the ratio, eliminating the differences in pickup energy caused by factors such as distance and device type, resulting in better robustness.

[0251] In a possible implementation, step 201 specifically includes the following steps:

[0252] 214. The direction determining device obtains a second feature from the first device, where the second feature is a feature of direct sound and / or reflected sound in the sound signal that is sensitive to direction.

[0253] Specifically, after the first device obtains M single-channel voice signals, it first processes the M single-channel voice signals to obtain the second feature. The direction determination device then obtains the second feature from the first device. This effectively reduces the computing power occupied by this solution. The specific process of the first device first processing the M single-channel voice signals to obtain the second feature can be referred to the relevant description of step 212 above and will not be repeated here.

[0254] 215. The direction determining device obtains the first feature based on the second feature, where the first feature is obtained according to the high-frequency energy, the low-frequency energy, or the difference between the high-frequency energy and the low-frequency energy in the second feature.

[0255] In a possible implementation, step 202 specifically includes the following steps:

[0256] The orientation determining device determines the first orientation angle according to the first feature and a preset corresponding relationship, where the preset corresponding relationship is a corresponding relationship between different first features and the first orientation angles.

[0257] Specifically, a correspondence between the first feature of the first device and the first orientation angle is pre-established, and the preset correspondences of different first devices may be the same or different. The orientation determining device determines the first orientation angle corresponding to the first feature by searching the preset correspondences.

[0258] For example, when the first device is used for the first time, its preset correspondence can be modified. For example, the user can be asked to rotate 90 degrees clockwise from facing the first device and wake it up three times, recording the first feature and the corresponding orientation angle calculated by the first device each time, to refresh the preset correspondence so that the preset correspondence is adapted to the current usage environment and the accuracy of orientation detection is increased.

[0259] In a possible implementation, step 202 specifically includes the following steps:

[0260] 221. The orientation determining device determines the second orientation angle according to the first feature and a preset correspondence relationship, where the preset correspondence relationship is a correspondence relationship between different first features and the second orientation angle.

[0261] 222. The orientation determining device determines a first orientation angle based on the second orientation angle and a third orientation angle, wherein the third orientation angle is determined based on an image captured by a camera of the first device.

[0262] In this embodiment of the present application, the second orientation angle is determined using audio. To improve the accuracy of the orientation angle between the sound source and the first device, the third orientation angle is determined based on an image captured by the camera of the first device, and the first orientation angle is determined by combining the second and third orientation angles.

[0263] The third orientation angle may be determined based on an image captured by a camera of the first device. Visual information or eye movement information may be obtained from the image, and the third orientation angle may be determined based on the visual information or eye movement information. The third orientation angle may also be obtained based on other methods (non-audio methods).

[0264] For example, the user's orientation is determined based on relevant facial features recorded by the camera. Specifically, based on the facial image captured by the camera, a 3D facial model is calculated using facial attributes, where facial attributes include facial key points, such as eyes, nose, or lips; and a third orientation angle is calculated based on the 3D facial model and the camera position.

[0265] As another example, the image captured by the camera and the eye tracking technology are used to determine the eye direction of the user to assist in determining the third orientation angle.

[0266] In a possible implementation, when it is determined that the first device has a camera, the orientation determination method of the embodiment of the present application is not performed, and the orientation between the first device and the user can be determined directly based on the camera.

[0267] For example, the essence of this embodiment is to determine the user's orientation through audio and decide whether the device enters the awake state. The above-mentioned orientation determination method also includes the following steps:

[0268] When the first orientation angle falls within the first range and the first command word table includes a first command word, the orientation determination device controls the first device to enter a wake-up state and perform a first operation corresponding to the first command word. The first command word is a command word included in the text corresponding to the sound signal.

[0269] Specifically, when the first command word is "turn on the air conditioner", the first operation is to control the first device to perform cooling.

[0270] When the first orientation angle falls within the second range and the second command word list includes the first command word, the orientation determination device controls the first device to enter a wake-up state and perform a first operation. The number of command words in the first command word list is greater than the number of command words in the second command word list, and the absolute value of any orientation angle in the second range is less than the absolute value of any orientation angle in the first range.

[0271] Specifically, the specific values ​​of the first range and the second range can be set according to actual conditions. The first command word list or the second command word list can be words such as "turn on the air conditioner" and "open the car window". The specific word list can be set according to actual conditions.

[0272] In an embodiment of the present application, two orientation angle ranges, a first range and a second range, and a first command word table and a second command word table with different numbers of command words are used to determine whether to wake up the first device and whether to perform a voice interaction response. When the first orientation angle falls into the first range, it can be considered that the user is most likely interacting with the first device, and the first command word table is used to compare the command words; when the first orientation angle falls into the second range, it can be considered that the user is least likely interacting with the first device, and the second command word table is used to compare the command words. The difficulty of entering the wake-up state is adjusted according to the orientation angle. This can reduce the risk of false wake-up, ensure the accuracy of voice interaction, and enhance the user's voice interaction experience.

[0273] Exemplarily, the orientation angle judgment and command word comparison can be performed simultaneously, or the two can be performed one after another. For example, after receiving the voice, the first device can first determine whether the first orientation angle falls within the first range. If it does, the command word range is expanded, that is, the first command word table is selected for command word comparison. When the first command word comparison passes, the first device is controlled to enter the wake-up state and execute the first operation corresponding to the first command word. When the first command word is included in the first command word table, it is considered that the first command word comparison passes. If the first orientation angle does not fall within the first range, it is determined whether the first orientation angle falls within the second range. If the first orientation angle falls within the second range, the command word range is narrowed, and the second command word table is selected for command word comparison. When the first command word comparison passes, the first device is controlled to enter the wake-up state and execute the first operation corresponding to the first command word. When the first command word is included in the second command word table, it is considered that the first command word comparison passes. This method can reduce the power consumption of the orientation determination device.

[0274] For another example, after receiving a voice message, the first device can perform a command word comparison. If the first command word successfully matches the first command word list, it determines whether the first orientation angle falls within the first range. If so, the device is controlled to enter a wake-up state and execute the first operation corresponding to the first command word. If the first command word successfully matches the second command word list, it determines whether the first orientation angle falls within the second range. If so, the device is controlled to enter a wake-up state and execute the first operation corresponding to the first command word. If the first command word does not appear in either the first command word list or the second command word list, processing stops.

[0275] For another example, after receiving a voice message, the first device can simultaneously determine the orientation angle and compare the command word. When the first orientation angle falls within a first range and the first command word list includes the first command word, the orientation determination device controls the first device to enter a wake-up state and execute a first operation corresponding to the first command word. When the first orientation angle falls within a second range and the second command word list includes the first command word, the orientation determination device controls the first device to enter a wake-up state and execute the first operation.

[0276] In a possible implementation, the above-mentioned orientation determination method further includes the following steps:

[0277] When the first orientation angle is zero, the orientation determining device controls the first device to enter the awake state.

[0278] During the interaction process, users generally face the device they want to wake up, so choosing to face the device directly to wake it up is more in line with the user's usage habits. In the embodiment of the present application, when the first orientation angle is zero, it is considered that the user is interacting with the first device, and the first device can be controlled to enter the awake state without the user saying the wake-up word. This simplifies the voice interaction process, making the voice interaction process smoother and more natural, and achieving a wake-up-free interaction experience under the condition of using only audio information, thereby improving the user's voice interaction experience.

[0279] In a possible implementation, after controlling the first device to enter the awake state, the orientation determination method further includes the following steps:

[0280] The direction determining device controls the first device to perform a first operation corresponding to a first command word. The first command word is a command word contained in the text corresponding to the sound signal.

[0281] Specifically, after the first device is controlled to enter the awake state, when the first command word appears in a command word table such as the first command word table or the second command word table, the first device is controlled to execute a first operation corresponding to the first command word.

[0282] Exemplarily, the orientation angle judgment and command word comparison can be performed simultaneously, or the two can be performed one after another. For example, after receiving the voice, the first device can first determine whether the first orientation angle is zero. If it is zero, the first device is controlled to enter the wake-up state. Then the first command word is compared with the command word, and the comparison can be performed using a command word table such as the first command word table or the second command word table. When the first command word comparison passes, the first device is controlled to perform the first operation corresponding to the first command word. Among them, when the first command word is included in a command word table such as the first command word table or the second command word table, it is considered that the first command word comparison passes. Since the power consumption of voice recognition (i.e., command word comparison) is relatively high, the power consumption of the orientation determination device can be reduced by first determining the orientation angle and then performing command word comparison.

[0283] For another example, the first device may perform a command word comparison after receiving the voice. When the first command word comparison passes, it determines whether the first orientation angle is zero. If it is zero, it controls the first device to enter the wake-up state and execute the first operation corresponding to the first command word.

[0284] For another example, after receiving the voice, the first device can simultaneously determine the orientation angle and compare the command word. When the first orientation angle is zero and the command word comparison passes, the orientation determination device controls the first device to enter the wake-up state and execute the first operation corresponding to the first command word.

[0285] In addition, when the text corresponding to the sound signal does not include the first command word, the first device waits for the user to input the next sound signal in the awake state, performs voice recognition processing on the next sound signal, and when the text corresponding to the next sound signal includes the first command word, executes the first operation corresponding to the first command word.

[0286] In order to better illustrate the process of waking up the first device, the following takes the simultaneous execution of orientation angle judgment and command word comparison (i.e., voice recognition) as an example, referring to Figure 3B, which is a flow chart of waking up the first device provided in an embodiment of the present application.

[0287] For example, the first range is (-45, 45) and the second range is less than -45 or greater than 45. In FIG3B , the first device performs audio collection to obtain M single-channel voice signals, calculates a first feature based on the M single-channel voice signals, and calculates a first orientation angle β by combining the preset correspondence relationship and the first feature.

[0288] If β=0, it is considered that the user is interacting with the first device, triggering the wake-up-free mode to directly enter the wake-up state; then call the comparison result of the command word. If the command word comparison passes, execute the first operation corresponding to the first command word, otherwise wait for the input of the next voice signal.

[0289] If -45 < β < 45, it is considered that the user is most likely interacting with the first device, and the command word list is switched to the first command word list. Check whether the current voice hits the current command word list. If it hits, the device enters the awake state and executes the command. If not, no response is given.

[0290] If β<-45 or β>45, it is considered that the user is interacting with the first device with a small probability, and the command word list is switched to the second command word list. The current voice is checked to see if it hits the current command word list. If it does, the device enters the awake state and executes the command. If not, no response is given.

[0291] The embodiment of the present application can calculate the user's orientation angle relative to the device based on the voice signal received by the device, and infer the probability of the user interacting with the device based on the orientation angle, thereby regulating the difficulty of the device entering the wake-up state.

[0292] In one possible implementation, when the first device enters the wake-up state, user feedback can be provided through images, audio, and other means, including but not limited to user experience (UX) animations, response messages, and prompt sounds. For example, referring to Figure 3C , when speaking toward the first device, feedback such as an avatar looking in the user's direction appears on the UX interface, triggering a wake-up request. Referring to Figure 3D , when speaking not directly toward the first device, the avatar looks forward or hides, and does not respond to the user.

[0293] In the embodiments of the present application, the numbering of the steps does not limit the execution order of the steps.

[0294] The following is a detailed description of the method for determining the focus device:

[0295] 4 is a flow chart of a method for determining a focus device according to an embodiment of the present application. The execution subject takes a focus device determining apparatus as an example. The focus device determining method 400 includes the following steps:

[0296] 401. A focus device determining apparatus obtains N first features corresponding to N first devices.

[0297] Specifically, the first characteristic of the first device is obtained based on M single-channel voice signals corresponding to M audio acquisition units in the first device. The N first devices perform audio acquisition for the same sound source. The first characteristic includes directionally sensitive characteristics of direct sound and / or reflected sound in the sound signal generated by the sound source. N is greater than or equal to two, and M is an integer.

[0298] 402. A focus device determining apparatus determines a focus device from N first devices based on N first features.

[0299] In an embodiment of the present application, the corresponding first feature is calculated based on the single-channel voice signal of each first device, and the focus device can be determined from the N first devices based on the N first features. This method does not require pre-training of a deep learning model. The focus device determination method has low cost, strong generalization, and occupies little computing power resources.

[0300] In a possible implementation, step 401 specifically includes the following steps:

[0301] 411. The focus device determining apparatus obtains M single-channel voice signals from the first device.

[0302] 412. The focus device determining device determines a second feature of the first device based on the M single-channel voice signals. The second feature is a feature of direct sound and / or reflected sound in the sound signal that is sensitive to direction.

[0303] 413. The focus device determining device obtains a first feature based on the second feature, where the first feature is obtained according to high-frequency energy, low-frequency energy, or the difference between high-frequency energy and low-frequency energy in the second feature.

[0304] In this embodiment of the present application, M single-channel speech signals are first obtained from the first device. A second feature is then derived based on these M single-channel speech signals. Finally, a first feature is derived based on the high-frequency energy, low-frequency energy, or the difference between the high-frequency and low-frequency energies in the second feature. Because the energy is normalized, this solution is less susceptible to factors such as the distance and orientation between the sound source and the first device when determining the focal device, and achieves high focal device accuracy. By processing the M single-channel speech signals from each first device using steps 411 to 413, the first feature of each first device can be obtained.

[0305] In a possible implementation, step 412 specifically includes the following steps:

[0306] B1. The focus device determination device performs the following processing on each single-channel speech signal to obtain a first spectrum corresponding to the single-channel speech signal.

[0307] The single-channel speech signal is framed and windowed to obtain Q sub-speech signals, where Q is greater than or equal to two. The Q sub-speech signals are Fourier transformed to obtain a time-frequency spectrum. Signal modulation is performed on the time-frequency spectrum to obtain a second frequency spectrum. The second frequency spectrum is filtered to obtain a first frequency spectrum.

[0308] B2. The focus device determination device obtains a second feature based on the M first spectra.

[0309] For the detailed description of steps 411 to 413 , please refer to the above steps 211 to 213 and will not be repeated here.

[0310] In a possible implementation, step 401 specifically includes the following steps:

[0311] 414. The focus device determining device obtains a second feature from the first device. The second feature is a feature of direct sound and / or reflected sound in the sound signal that is sensitive to direction.

[0312] 415. The focus device determining device obtains a first feature based on the second feature, where the first feature is obtained according to high-frequency energy, low-frequency energy, or the difference between high-frequency energy and low-frequency energy in the second feature.

[0313] In this embodiment, a second feature of the direct sound and / or reflected sound, which is sensitive to direction, is first acquired from the first device. The first feature is then derived based on the high-frequency energy, low-frequency energy, or the difference between high-frequency and low-frequency energy in the second feature. Processing the second feature in the first device can effectively reduce the computing power used by this solution.

[0314] For the detailed description of steps 414 to 415, please refer to the above steps 214 to 215, which will not be repeated here.

[0315] In a possible implementation, step 402 specifically includes the following steps:

[0316] The focus device determining means determines the first device corresponding to at least one first feature having a higher feature value among the N first features as the focus device.

[0317] In an embodiment of the present application, the first device corresponding to the first feature with the highest feature value among the N first features can be determined as the focus device, or the first devices corresponding to the x first features with the highest feature values ​​among the N first features can be determined as the focus devices, where x is greater than or equal to two, for example, x is 2, 3, 5, etc.

[0318] In a possible implementation, the focus device determination method 400 further includes the following steps:

[0319] The focus device determining means controls the focus device to enter an awake state.

[0320] In an embodiment of the present application, after the focus device is determined, the focus device can be controlled to enter the wake-up state without the user saying the wake-up word, which simplifies the voice interaction process and improves the user's voice interaction experience.

[0321] In a possible implementation, after controlling the focus device to enter the awake state, the focus device determination method further includes the following steps:

[0322] The focus device determining means controls the focus device to execute a first operation corresponding to a first command word, wherein the first command word is a command word contained in a text corresponding to the sound signal.

[0323] In an embodiment of the present application, after the focus device is controlled to enter the wake-up state without the user saying the wake-up word, the focus device is controlled to execute the first operation corresponding to the command word contained in the text corresponding to the sound signal, in response to the user's voice interaction command.

[0324] Specifically, the corresponding text is determined based on the sound signal. This text is then compared with the command vocabulary of the focus device to determine whether the command vocabulary includes the words in the text, i.e., the command words. If the command words are included, the focus device is controlled to perform the operation corresponding to the command words. For example, if the first command word is "turn on the air conditioner," the first operation is to turn on the air conditioner for cooling. Furthermore, the command vocabulary corresponding to the N first devices can be the same or different, and this is not limited to this.

[0325] Exemplarily, focal device determination and command word comparison can be performed simultaneously, or sequentially. For example, a first feature can be determined first, so that the focal device determination device determines the focal device based on the first feature. A command word comparison is then performed on the first command word. When the first command word comparison passes, the focal device is controlled to execute the first operation corresponding to the first command word. Because voice recognition (i.e., command word comparison) consumes a lot of power, first determining the focal device and then performing command word comparison can reduce the power consumption of the focal device determination device.

[0326] For another example, a command word comparison is first performed based on the voice and the command word table of the first device. When the first command word comparison passes, the process proceeds to determine the focus device. At this time, the focus device is determined from the first device that passes the first command word comparison.

[0327] For another example, the focus device judgment and command word comparison can be performed simultaneously, that is, the command word comparison is performed based on the voice and the command word table of the first device, and the focus device is determined from the N first devices based on the first feature. When the first command word comparison passes, the focus device determination device controls the focus device to enter the wake-up state and execute the first operation corresponding to the first command word.

[0328] In addition, when the text corresponding to the sound signal does not include the first command word, the focus device waits for the user to input the next sound signal in the awake state, performs voice recognition processing on the next sound signal, and when the text corresponding to the next sound signal includes the first command word, executes the first operation corresponding to the first command word.

[0329] In the embodiments of the present application, the numbering of the steps does not limit the execution order of the steps.

[0330] In one possible implementation, when the focus device enters the wake-up state, user feedback can be provided through images, audio, and other means, including but not limited to UX animations, response messages, and prompt sounds. For example, when there are multiple first devices, the first device (i.e., the focus device) that the user is pointing to lights up the screen or displays an avatar, triggering the wake-up-free state; the other first devices turn off their screens or hide their avatars, and do not respond.

[0331] Refer to Table 1 below. In Table 1, the first device takes the Liquid Crystal Display (LCD) central control screen as an example. The facing device refers to the device that the user is facing, that is, the device that is expected to be awakened. The non-facing device is a device that is not directly in front of the user, that is, the device that is not expected to be awakened. The last column in Table 1 shows the accuracy of the focus device determination method of the present application in different scenarios and different device states. It can be seen that the focus device determination method proposed in the embodiment of the present application does not require pre-training and has strong generalization. There is no need to predict the speaker's position, distance and other information. It is robust to distance and noise. It is not limited by the number of device channels, has low computing power occupancy, and a small communication burden, and can achieve long-term end-side operation. Long-term end-side operation means that the terminal device usually has very limited computing power and high power consumption requirements. Only algorithms with low computational complexity can perform long-term end-side operation, that is, the algorithm corresponding to the focus device determination method of the embodiment of the present application is run on the terminal device for a long time.

[0332] Table 1 Device wake-up indicator table

[0333] The above describes in detail the method of the embodiment of the present application. The following describes the device provided by the embodiment of the present application.

[0334] Figure 5 is a schematic diagram of the structure of a possible device provided in an embodiment of the present application. In the embodiment of the present application, the device can be an electronic device or a module (such as a chip) used in an electronic device.

[0335] The device 500 shown in FIG5 can be a heading angle determination device, which is used to implement the functions of the above-mentioned heading angle determination method embodiment, and thus can also achieve the beneficial effects of the above-mentioned heading angle determination method embodiment. As shown in FIG5, the heading angle determination device includes an acquisition module 501 and a determination module 502. The heading angle determination device is used to implement the functions of the heading angle determination method embodiment shown in FIG2. Alternatively, the heading angle determination device may include a module for implementing any function or operation in the heading angle determination method embodiment shown in FIG2, which module can be implemented in whole or in part through software, hardware, firmware, or any combination thereof.

[0336] When the orientation angle determination device is used to implement the functions of the method embodiment shown in Figure 2, the acquisition module 501 is used to obtain a first feature of the first device. The above-mentioned first feature is obtained based on M single-channel voice signals corresponding to M audio acquisition units in the first device. M is an integer. The M audio acquisition units perform audio acquisition for the same sound source. The first feature includes the characteristics of direct sound and / or reflected sound that are sensitive to orientation in the sound signal generated by the sound source. The determination module 502 is used to determine a first orientation angle between the sound source and the first device based on the first feature.

[0337] In an embodiment of the present application, the direction determination device calculates the first feature based on the single-channel voice signal of the first device, and can determine the direction angle based on the first feature. There is no need to pre-train a deep learning model. The direction determination method has low cost, strong generalization, and occupies little computing power resources.

[0338] In a possible implementation, referring to FIG. 5 , the orientation determination device further includes a processing module 503 .

[0339] The processing module 503 is configured to control the first device to enter a wake-up state and execute a first operation corresponding to the first command word when the first orientation angle falls within the first range and the first command word table includes a first command word. The first command word is a command word included in the text corresponding to the sound signal.

[0340] The processing module 503 is further configured to control the first device to enter a wake-up state and perform a first operation when the first orientation angle falls within the second range and the second command word list includes the first command word. The number of command words in the first command word list is greater than the number of command words in the second command word list, and the absolute value of any orientation angle in the second range is less than the absolute value of any orientation angle in the first range.

[0341] In a possible implementation manner, the processing module 503 is further configured to control the first device to enter a wake-up state when the first orientation angle is zero.

[0342] In a possible implementation, after controlling the first device to enter the wake-up state, the processing module 503 is further used to control the first device to execute a first operation corresponding to a first command word, where the first command word is a command word contained in the text corresponding to the sound signal.

[0343] In one possible implementation, when the first device enters the wake-up state, user feedback can be given through images, audio, and the like, including but not limited to user experience (UX) animations, response words, prompt sounds, and the like. Accordingly, the orientation determination device further includes a display module for providing user feedback through images when the first device enters the wake-up state. Optionally, the orientation determination device further includes an output module for providing user feedback through audio when the first device enters the wake-up state, and the output module may be a speaker.

[0344] The device 500 shown in FIG5 can also be a focus device determination device, which is used to implement the functions of the embodiment of the focus device determination method described above, and thus can also achieve the beneficial effects possessed by the embodiment of the focus device determination method described above. Referring to FIG5 , the focus device determination device includes an acquisition module 501 and a determination module 502. The focus device determination device is used to implement the functions of the embodiment of the focus device determination method described above shown in FIG4 . Alternatively, the focus device determination device may include a module for implementing any function or operation in the embodiment of the focus device determination method described above shown in FIG4 , and the module may be implemented in whole or in part by software, hardware, firmware, or any combination thereof.

[0345] When the focus device determination apparatus is used to implement the functions of the method embodiment shown in FIG4 , acquisition module 501 is configured to acquire N first features corresponding to N first devices. The first features of the first devices are obtained based on M single-channel speech signals corresponding to M audio acquisition units in the first devices. The N first devices perform audio acquisition for the same sound source. The first features include directionally sensitive features of direct sound and / or reflected sound in the sound signal generated by the sound source. N is greater than or equal to two, and M is an integer.

[0346] The determination module 502 is configured to determine a focus device from N first devices based on the N first features.

[0347] In an embodiment of the present application, the focus device determination device calculates the corresponding first feature based on the single-channel voice signal of each first device, and can determine the focus device from N first devices based on the N first features. This method does not require pre-training of a deep learning model. The focus device determination method has low cost, strong generalization, and occupies little computing power resources.

[0348] In a possible implementation, the focus device determination apparatus further includes a processing module 503 .

[0349] The processing module 503 is used to control the focus device to enter the awake state.

[0350] In a possible implementation, after the focus device is controlled to enter the awake state, the processing module 503 is further configured to control the focus device to execute a first operation corresponding to a first command word, where the first command word is a command word contained in the text corresponding to the sound signal.

[0351] In one possible embodiment, when the focus device enters the awake state, user feedback can be provided via images, audio, or other means, including but not limited to UX animations, response messages, and prompt sounds. Accordingly, the focus device determination apparatus further includes a display module for providing user feedback via images when the focus device enters the awake state. Optionally, the focus device determination apparatus further includes an output module for providing user feedback via audio when the focus device enters the awake state, and the output module can be a speaker.

[0352] The present application also provides an electronic device. Referring to Figure 6, Figure 6 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 600 includes a memory 601, a processor 602, a communication interface 604, and a bus 603. The memory 601, processor 602, and communication interface 604 are connected to each other via bus 603.

[0353] Optionally, the electronic device 600 further includes a display (not shown), which is communicatively connected to the memory 601, processor 602, and communication interface 604 via bus 603. The display is configured to output images. Optionally, the electronic device further includes an output module (not shown), which is communicatively connected to the memory 601, processor 602, and communication interface 604 via bus 603, and is configured to output audio. The output module may be a speaker.

[0354] The memory 601 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 601 may store programs. When the program stored in the memory 601 is executed by the processor 602, the processor 602 and the communication interface 604 are used to perform the various steps of the heading angle determination method or the focus device determination method according to any embodiment of the present application.

[0355] The processor 602 can adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the functions required to be performed by the units in the orientation angle determination device or the focus device determination device of any embodiment of the present application, or to execute the orientation angle determination method or the focus device determination method of any embodiment of the present application.

[0356] The processor 602 may also be an integrated circuit chip with signal processing capabilities. During implementation, the various steps of the heading angle determination method or focus device determination method of any embodiment of the present application may be completed by hardware integrated logic circuits or software instructions in the processor 602. The aforementioned processor 602 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. The various methods, steps, and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the heading angle determination method or focus device determination method in conjunction with any embodiment of the present application may be directly implemented and executed by a hardware processor, or by a combination of hardware and software modules in the processor. The software module may be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or the like. The storage medium is located in the memory 601, and the processor 602 reads the information in the memory 601 and combines its hardware to complete the functions required to be performed by the units included in the orientation angle determination device or the focus device determination device of any embodiment of the present application, or executes the orientation angle determination method or the focus device determination method of any embodiment of the present application.

[0357] The communication interface 604 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the electronic device 600 and other devices or a communication network. For example, the first feature can be obtained through the communication interface 604.

[0358] The bus 603 may include a path for transmitting information between the various components of the electronic device 600 (e.g., the memory 601, the processor 602, the communication interface 604). In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0359] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0360] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0361] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk).

[0362] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for determining a direction, characterized in that: include: Obtaining a first characteristic of a first device; The first feature is obtained according to M single-channel voice signals corresponding to M audio collection units in the first device, where M is an integer, and the M audio collection units collect audio for the same sound source, and the first feature includes features of direct sound and / or reflected sound that are sensitive to direction in the sound signal generated by the sound source; A first orientation angle between the sound source and the first device is determined based on the first feature.

2. The method according to claim 1, characterized in that The obtaining the first characteristic of the first device includes: Acquire the M single-channel voice signals from the first device; Obtaining a second feature based on the M single-channel speech signals, where the second feature is a feature of direct sound and / or reflected sound in the sound signal that is sensitive to direction; The first feature is obtained based on the second feature, and the first feature is obtained according to the high-frequency energy, the low-frequency energy, or the difference between the high-frequency energy and the low-frequency energy in the second feature.

3. The method according to claim 2, characterized in that The obtaining a second feature based on the M single-channel speech signals includes: Perform the following processing on each of the single-channel speech signals to obtain a first spectrum corresponding to the single-channel speech signal: Performing frame windowing on the single-channel voice signal to obtain Q-segment voice signals, where Q is greater than or equal to two; Performing Fourier transform on the Q segment sub-voice signal to obtain a time-frequency spectrum; Performing signal modulation on the time-frequency spectrum to obtain a second frequency spectrum; Performing filtering processing on the second spectrum to obtain the first spectrum; The second feature is obtained based on M first spectra.

4. The method according to claim 1, characterized in that: The obtaining the first characteristic of the first device includes: Acquire a second feature from the first device, where the second feature is a feature of direct sound and / or reflected sound in the sound signal that is sensitive to direction; The first feature is obtained based on the second feature, and the first feature is obtained according to the high-frequency energy, the low-frequency energy, or the difference between the high-frequency energy and the low-frequency energy in the second feature.

5. The method according to any one of claims 1 to 4, characterized in that: The determining a first orientation angle between the sound source and the first device based on the first feature includes: Determine a second orientation angle according to the first feature and a preset corresponding relationship, wherein the preset corresponding relationship is a corresponding relationship between different first features and second orientation angles; The first orientation angle is determined based on the second orientation angle and a third orientation angle, and the third orientation angle is determined based on an image captured by a camera of the first device.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: When the first orientation angle falls within the first range and the first command word table includes a first command word, controlling the first device to enter a wake-up state and executing a first operation corresponding to the first command word; the first command word is a command word included in the text corresponding to the sound signal; When the first orientation angle falls into the second range and the second command word list includes the first command word, the first device is controlled to enter a wake-up state and perform the first operation; the number of command words in the first command word list is greater than the number of command words in the second command word list, and the absolute value of any orientation angle in the second range is less than the absolute value of any orientation angle in the first range.

7. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: When the first orientation angle is zero, the first device is controlled to enter a wake-up state.

8. The method according to claim 7, characterized in that After controlling the first device to enter the awake state, the method further includes: The first device is controlled to execute a first operation corresponding to a first command word, where the first command word is a command word contained in a text corresponding to the sound signal.

9. A method for determining a focus device, characterized in that: include: Obtaining N first features corresponding to N first devices, where the first features of the first devices are obtained according to M single-channel voice signals corresponding to M audio acquisition units in the first devices, the N first devices perform audio acquisition for the same sound source, and the first features include features of direct sound and / or reflected sound that are sensitive to direction in the sound signal generated by the sound source; N is greater than or equal to two, and M is an integer; A focus device is determined from the N first devices based on the N first features.

10. The method according to claim 9, characterized in that The determining a focus device from the N first devices based on the N first features includes: A first device corresponding to at least one first feature with a higher feature value among the N first features is determined as the focus device.

11. The method according to claim 9 or 10, characterized in that: The obtaining of N first features corresponding to the N first devices includes: Acquire the M single-channel voice signals from the first device; Determine a second feature of the first device based on the M single-channel voice signals, where the second feature is a feature of direct sound and / or reflected sound in the sound signal that is sensitive to direction; The first feature is obtained based on the second feature, and the first feature is obtained according to the high-frequency energy, the low-frequency energy, or the difference between the high-frequency energy and the low-frequency energy in the second feature.

12. The method according to claim 11, characterized in that The determining the second feature of the first device based on the M single-channel voice signals includes: Perform the following processing on each of the single-channel speech signals to obtain a first spectrum corresponding to the single-channel speech signal: Performing frame windowing on the single-channel voice signal to obtain Q-segment voice signals, where Q is greater than or equal to two; Performing Fourier transform on the Q segment sub-voice signal to obtain a time-frequency spectrum; Performing signal modulation on the time-frequency spectrum to obtain a second frequency spectrum; Performing filtering processing on the second spectrum to obtain the first spectrum; The second feature is obtained based on M first spectra.

13. The method according to claim 9 or 10, characterized in that: The obtaining of N first features corresponding to the N first devices includes: Acquire a second feature from the first device, where the second feature is a feature of direct sound and / or reflected sound in the sound signal that is sensitive to direction; The first feature is obtained based on the second feature, and the first feature is obtained according to the high-frequency energy, the low-frequency energy, or the difference between the high-frequency energy and the low-frequency energy in the second feature.

14. The method according to any one of claims 9 to 13, characterized in that: The method further comprises: Control the focus device to enter a wake-up state.

15. The method according to claim 14, characterized in that After controlling the focus device to enter the awake state, the method further includes: The focus device is controlled to execute a first operation corresponding to a first command word, where the first command word is a command word contained in a text corresponding to the sound signal.

16. A device for determining a direction, characterized in that: include: An acquisition module, configured to acquire a first feature of a first device; The first feature is obtained according to M single-channel voice signals corresponding to M audio collection units in the first device, where M is an integer, and the M audio collection units collect audio for the same sound source, and the first feature includes features of direct sound and / or reflected sound that are sensitive to direction in the sound signal generated by the sound source; A determination module is used to determine a first orientation angle between the sound source and the first device based on the first feature.

17. The device according to claim 16, characterized in that The device also includes: a processing module, configured to control the first device to enter a wake-up state and execute a first operation corresponding to the first command word when the first orientation angle falls within a first range and the first command word table includes a first command word; the first command word is a command word included in the text corresponding to the sound signal; The processing module is also used to control the first device to enter a wake-up state and perform the first operation when the first orientation angle falls into the second range and the second command vocabulary includes the first command word; the number of command words in the first command vocabulary is greater than the number of command words in the second command vocabulary, and the absolute value of any orientation angle in the second range is less than the absolute value of any orientation angle in the first range.

18. The device according to claim 16, characterized in that The device also includes: The processing module is used to control the first device to enter a wake-up state when the first orientation angle is zero.

19. The device according to claim 18, characterized in that After controlling the first device to enter the wake-up state, the processing module is further used to control the first device to execute a first operation corresponding to a first command word, where the first command word is a command word contained in the text corresponding to the sound signal.

20. A focus device determination device, characterized in that: include: an acquisition module, configured to acquire N first features corresponding to N first devices, wherein the first features of the first devices are obtained according to M single-channel voice signals corresponding to M audio acquisition units in the first devices, the N first devices perform audio acquisition for the same sound source, and the first features include features of direct sound and / or reflected sound that are sensitive to direction in the sound signal generated by the sound source; N is greater than or equal to two, and M is an integer; A determination module is used to determine a focus device from the N first devices based on the N first features.

21. The device according to claim 20, characterized in that The device also includes: The processing module is used to control the focus device to enter the awake state.

22. The device according to claim 21, characterized in that After controlling the focus device to enter the awake state, the processing module is further used to control the focus device to execute a first operation corresponding to a first command word, where the first command word is a command word contained in the text corresponding to the sound signal.

23. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the processor and the memory are connected, wherein the memory is used to store program codes, and the processor is used to call the program codes to execute the method according to any one of claims 1 to 15.

24. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 15.

Images