A distributed voice interaction method, system and distributed center

Abstract

Embodiments of the present application disclose a distributed voice interaction method, system and distributed center. The method comprises: receiving a wake-up word segment audio feature and a wake-up word time range uploaded by a wake-up device; the wake-up device is a smart voice device woken up by a wake-up word among a plurality of smart voice devices, and the wake-up word segment audio feature comprises a first time difference of arrival (tdoa) feature and a first audio quality feature; selecting, according to the first tdoa feature and the wake-up word time range, a wake-up device with the earliest wake-up word arrival time as a response device; and selecting, according to the first audio quality feature, a wake-up device with the best wake-up word audio quality as a sound pickup device in a recognition stage. It can be seen that in the embodiments of the present application, the smart voice device closest to the user is always ensured to respond to the user, so that the response can be better perceived by the user; and the audio quality of the recognized statement collected is also ensured to be good, so that more accurate recognition effect can be obtained.

Description

Technical Field

[0001] This application relates to the field of voice interaction technology, and more specifically, to a distributed voice interaction method, system, and distributed center. Background Technology

[0002] With the widespread adoption of smart voice devices, multiple smart voice devices may operate simultaneously in the same environment. For example, in a home environment, appliances such as televisions, air conditioners, refrigerators, and washing machines may all have smart voice interaction functions. Because multiple smart voice devices exist in the same environment, they may respond to the user's voice commands in succession within a short period, resulting in a "one call, multiple responses" phenomenon.

[0003] Currently, the approach is to select the smart voice device with the highest signal energy during the wake word's time period as the response device, and then directly use the response device as the sound pickup device during the recognition stage, in order to avoid the phenomenon of "one call, multiple responses".

[0004] However, this method relies excessively on the signal energy during the wake-word period, which can be affected by factors such as noise and speaker orientation in practice, potentially preventing the nearest intelligent voice device from serving as the response device. Furthermore, directly using the response device as the pickup device in the recognition phase cannot guarantee the quality of the voice data acquired during this stage, resulting in poor speech recognition performance. Summary of the Invention

[0005] In view of this, embodiments of this application disclose a distributed voice interaction method, system, and distributed center, which ensures that the voice intelligent device closest to the user always responds to the user, and ensures that the audio quality of the collected recognized sentences is good.

[0006] The technical solutions provided in this application are as follows:

[0007] In a first aspect, embodiments of this application provide a distributed voice interaction method, the method comprising:

[0008] The system receives the wake-up word audio features and wake-up word time range uploaded by the wake-up device; wherein, the wake-up device is an intelligent voice device that is woken up by a wake-up word among multiple intelligent voice devices, and the wake-up word audio features include a first time difference of arrival (tdoa) feature and a first audio quality feature;

[0009] Based on the first tdoa feature and the wake word time range, the wake-up device with the earliest wake word arrival time is selected as the response device;

[0010] Based on the first audio quality feature, the wake-up device with the best wake-up word audio quality is selected as the sound pickup device in the recognition stage.

[0011] In one possible implementation, there are multiple wake-up devices, and the first tdoa feature is multiple. The step of selecting the wake-up device with the earliest wake-up word arrival time as the response device based on the first tdoa feature and the wake-up word time range includes:

[0012] Based on the wake word time ranges corresponding to multiple first tdoa features, tdoa sub-features with common time periods are extracted from the multiple first tdoa features to obtain multiple tdoa sub-features;

[0013] The multiple tdoa sub-features are input into a preset first decision model to obtain a first decision result; wherein, the first decision model is trained using multiple first audio training data with known arrival time differences, and the first audio training data includes wake words;

[0014] Based on the first judgment result, the wake-up device with the earliest wake-up word arrival time is selected as the response device.

[0015] In one possible implementation, there are multiple wake-up devices, and multiple first audio quality features. The step of selecting the wake-up device with the best wake-up word audio quality based on the first audio quality features as the sound pickup device in the recognition phase includes:

[0016] Multiple first audio quality features are input into a preset second decision model to obtain a second decision result; wherein, the second decision model is trained using multiple second audio training data with known audio quality, and the second audio training data includes wake words;

[0017] Based on the second judgment result, the wake-up device with the best wake-up word audio quality is selected as the sound pickup device in the recognition stage.

[0018] In one possible implementation, the method further includes:

[0019] The first extraction model and the first decision model are constructed through the following process:

[0020] Construct a first training set; wherein the first training set includes multiple first audio training data, each first audio training data includes audio from N arrays, each array of audio includes a wake word, the arrival time difference of the wake word in the N arrays is known, and N≥2;

[0021] The audio of each array in each first audio training data is independently input into the first initial extraction model to obtain N second TDOA features corresponding to each first audio training data.

[0022] Randomly set p second tdoa features corresponding to each first audio training data to zero to obtain p first zeroed features; wherein p satisfies 0≤p≤N-2;

[0023] With the N×1 dimensional vector as the training objective, the p first zeroed features and Np non-zeroed second tdoa features corresponding to each first audio training data are input into the first initial decision model. The first initial extraction model and the first initial decision model are jointly iteratively trained to obtain the trained first extraction model and the first decision model. The first extraction model is used to extract the first tdoa features, and the first decision model is used to select the wake-up device with the earliest wake-up word arrival time.

[0024] In one possible implementation, the method further includes:

[0025] The second extraction model and the second decision model are constructed through the following process:

[0026] Construct a second training set; wherein the second training set includes multiple second audio training data, each second audio training data includes audio from M arrays, each array of audio includes a wake word, the audio quality of the wake word in the M arrays is known, and M≥2;

[0027] The audio of each array in each second audio training data is independently input into the second initial extraction model to obtain M second audio quality features corresponding to each second audio training data.

[0028] Randomly set q second audio quality features corresponding to each second audio training data to zero to obtain q second zero-set features; wherein q satisfies 0≤q≤M-2;

[0029] With the M×1 dimensional vector as the training objective, each second audio training data point corresponds to q second zeroed features and Mq non-zeroed second audio quality features, which are then input into the second initial decision model. The second initial extraction model and the second initial decision model are jointly iteratively trained to obtain the trained second extraction model and the second decision model. The second extraction model is used to extract the first audio quality features, and the second decision model is used to select the wake-up device with the best wake-up word audio quality.

[0030] In one possible implementation, the method further includes:

[0031] The multiple intelligent voice devices are synchronized in time; wherein the synchronization error is less than d / c, where d represents the distance resolution between the intelligent voice devices and c represents the speed of sound in air.

[0032] Secondly, embodiments of this application provide a distributed center, the distributed center comprising:

[0033] A receiving unit is used to receive the wake-up word audio features and wake-up word time range uploaded by the wake-up device; wherein, the wake-up device is an intelligent voice device that is woken up by a wake-up word among multiple intelligent voice devices, and the wake-up word audio features include a first time difference of arrival (tdoa) feature and a first audio quality feature;

[0034] The first selection unit is used to select the wake-up device with the earliest wake-up time as the response device based on the first tdoa feature and the wake-up word time range.

[0035] The second selection unit is used to select the wake-up device with the best wake-up word audio quality as the sound pickup device in the recognition stage based on the first audio quality feature.

[0036] Thirdly, embodiments of this application provide a distributed voice interaction system, which includes multiple intelligent voice devices and a distributed center;

[0037] The distributed center is used to receive the wake-up word audio features and wake-up word time range uploaded by the wake-up device; wherein, the wake-up device is an intelligent voice device that is woken up by a wake-up word among multiple intelligent voice devices, and the wake-up word audio features include a first time difference of arrival (tdoa) feature and a first audio quality feature;

[0038] The distributed center is also used to select the wake-up device with the earliest wake-up time as the response device based on the first tdoa feature and the wake-up word time range;

[0039] The distributed center is also used to select the wake-up device with the best wake-up word audio quality as the sound pickup device in the recognition stage based on the first audio quality feature.

[0040] In one possible implementation, the intelligent voice device is configured to determine the time range of the wake word when awakened by the wake word;

[0041] The intelligent voice device is also used to determine the wake-up word audio segment from the collected audio data based on the wake-up word time range;

[0042] The intelligent voice device is further configured to input the wake-up word audio into a preset first extraction model to obtain the first TDOA feature; wherein, the first extraction model is trained using multiple first audio training data with known arrival time differences, and the first audio training data includes the wake-up word;

[0043] The intelligent voice device is also used to upload the first tdoa feature and the wake word time range to the distributed center.

[0044] In one possible implementation, the intelligent voice device is used to input real-time collected audio data frame by frame into a preset first extraction model to obtain multiple first TDOA sub-features; wherein, the first extraction model is trained using multiple first audio training data with known arrival time differences, and the first audio training data includes wake words;

[0045] The intelligent voice device is also used to determine the time range of the wake word when it is woken up by the wake word;

[0046] The intelligent voice device is further configured to determine the first tdoa feature from the plurality of first tdoa sub-features based on the wake word time range;

[0047] The intelligent voice device is also used to upload the first tdoa feature and the wake word time range to the distributed center.

[0048] In one possible implementation, the intelligent voice device is configured to determine the time range of the wake word when awakened by the wake word;

[0049] The intelligent voice device is also used to determine the wake-up word audio segment from the collected audio data based on the wake-up word time range;

[0050] The intelligent voice device is further configured to input the wake-up word audio into a preset second extraction model to obtain the first audio quality feature; wherein, the second extraction model is trained using multiple second audio training data with known audio quality, and the second audio training data includes the wake-up word;

[0051] The intelligent voice device is also used to upload the first audio quality feature to the distributed center.

[0052] In one possible implementation, the intelligent voice device is used to input real-time acquired audio data frame by frame into a preset second extraction model to obtain multiple first audio quality sub-features; wherein, the second extraction model is trained using multiple second audio training data with known audio quality, and the second audio training data includes wake words;

[0053] The intelligent voice device is also used to determine the time range of the wake word when it is woken up by the wake word;

[0054] The intelligent voice device is further configured to determine the first audio quality feature from the plurality of first audio quality sub-features based on the wake word time range;

[0055] The intelligent voice device is also used to upload the first audio quality feature to the distributed center.

[0056] Based on the above technical solution, this application has the following beneficial effects:

[0057] This application discloses a distributed voice interaction method, system, and distributed center. The method includes: receiving wake-up word audio features and wake-up word time range uploaded by a wake-up device; wherein the wake-up device is the intelligent voice device woken up by a wake-up word among multiple intelligent voice devices, and the wake-up word audio features include a first time difference of arrival (TDOA) feature and a first audio quality feature; selecting the wake-up device with the earliest wake-up word arrival time as the response device based on the first TDOA feature and the wake-up word time range; and selecting the wake-up device with the best wake-up word audio quality as the pickup device in the recognition stage based on the first audio quality feature. It is evident that since the intelligent voice device closest to the user generally collects the wake-up word audio first, selecting the wake-up device with the earliest wake-up word arrival time as the response device ensures that the voice intelligent device closest to the user always responds, making the response more perceptible to the user. Furthermore, since the wake word and the recognition statement are spoken by the same user, and the user is likely to utter the wake word and recognition statement in the same location, choosing the wake-up device with the best wake word audio quality as the sound pickup device in the recognition stage can ensure that the audio quality of the collected recognition statement is good, thereby obtaining a more accurate recognition effect and ensuring the correctness of voice interaction. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the published drawings without creative effort.

[0059] Figure 1 This is a flowchart of a distributed voice interaction method disclosed in an embodiment of this application;

[0060] Figure 2 This is a schematic diagram of a method for selecting a response device using a first decision model, as disclosed in an embodiment of this application.

[0061] Figure 3 This is a schematic diagram illustrating the training process of a first decision model and a first extraction model disclosed in an embodiment of this application;

[0062] Figure 4 This is a schematic diagram of a first audio training data disclosed in an embodiment of this application;

[0063] Figure 5 This is a schematic diagram illustrating the training process of a second decision model and a second extraction model disclosed in an embodiment of this application;

[0064] Figure 6 This is a schematic diagram of a second audio training data disclosed in an embodiment of this application;

[0065] Figure 7 This is a schematic diagram of the structure of a distributed center disclosed in an embodiment of this application;

[0066] Figure 8 This is a schematic diagram of a distributed voice interaction system disclosed in an embodiment of this application. Detailed Implementation

[0067] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0068] This application discloses a distributed voice interaction method, system, and distributed center. The method includes: receiving wake-up word audio features and wake-up word time range uploaded by a wake-up device; wherein the wake-up device is the intelligent voice device woken up by a wake-up word among multiple intelligent voice devices, and the wake-up word audio features include a first time difference of arrival (TDOA) feature and a first audio quality feature; selecting the wake-up device with the earliest wake-up word arrival time as the response device based on the first TDOA feature and the wake-up word time range; and selecting the wake-up device with the best wake-up word audio quality as the pickup device in the recognition stage based on the first audio quality feature. It is evident that since the intelligent voice device closest to the user generally collects the wake-up word audio first, selecting the wake-up device with the earliest wake-up word arrival time as the response device ensures that the voice intelligent device closest to the user always responds, making the response more perceptible to the user. Furthermore, since the wake word and the recognition statement are spoken by the same user, and the user is likely to utter the wake word and recognition statement in the same location, choosing the wake-up device with the best wake word audio quality as the sound pickup device in the recognition stage can ensure that the audio quality of the collected recognition statement is good, thereby obtaining a more accurate recognition effect and ensuring the correctness of voice interaction.

[0069] See Figure 1 The flowchart of a distributed voice interaction method disclosed in this application includes:

[0070] S101. Receive the wake-up word audio features and wake-up word time range uploaded by the wake-up device; wherein, the wake-up device is an intelligent voice device woken up by a wake-up word among multiple intelligent voice devices, and the wake-up word audio features include a first arrival time difference (tdoa) feature and a first audio quality feature;

[0071] It should be noted that the wake word time range refers to the time from the start to the end of the wake word. For example, a wake word time range of 09:01:00 to 09:01:01 has a duration of 1 second. It is understood that the above is merely an illustrative example and should not be construed as a limitation of this application. For Chinese, most wake words are four characters long, although there are also two, three, or five characters, etc., and the duration of the wake word is usually in the range of 0.5 to 2 seconds. It is understood that this application does not impose any limitations on this.

[0072] It should be noted that the time difference of arrival (TDOA) is a feature that includes TDOA information. For example, if a speaker is 1 meter away from a microphone on a television and 2 meters away from a microphone on an air conditioner, then the TDOA can be equal to 2m / (340m / s) - 1m / (340m / s) = 0.0029 seconds. It is understood that the above is merely an illustrative example and should not be construed as a limitation of this application. TDOA features can be manually designed, such as by performing a specific decomposition on the audio signal and extracting the intensity and relative temporal relationship of each component after decomposition, or they can be obtained by training a feature extraction model. This application does not limit the scope of the TDOA feature.

[0073] S102. Select the wake-up device with the earliest wake-up time as the response device based on the first tdoa feature and the wake-up word time range.

[0074] Understandably, for a specific wake word, if only one smart voice device among multiple smart voice devices is woken up by the wake word, then that wake-up device is used as the response device; if multiple smart voice devices are woken up by the wake word, then the tdoa features of the wake word segments of all wake-up devices are compared to find the device with the earliest wake word arrival time, which is then used as the response device.

[0075] It should be noted that the response device is used to respond to user commands. This response includes, but is not limited to, "displaying information via indicator lights or a screen" or "playing response audio via a speaker," etc., and this application does not limit this to these aspects.

[0076] S103. Select the wake-up device with the best wake-up word audio quality as the sound pickup device in the recognition stage based on the first audio quality feature.

[0077] Understandably, for a specific wake word, if only one smart voice device among multiple smart voice devices is woken up by the wake word, then that wake-up device is used as the pickup device; if multiple smart voice devices are woken up by the wake word, then the audio quality characteristics of the wake word audio segments of all wake-up devices are compared, and the device with the best audio quality is found as the pickup device.

[0078] For example, assuming there are wake-up devices 1, 2, and 3, and wake-up device 1 is identified as the responding device, and wake-up device 3 is identified as the voice pickup device, then wake-up device 3 will collect the specific voice commands for the recognition phase, such as "turn up the volume." Wake-up device 3 or the cloud will then recognize the voice commands. If the recognition result needs to be fed back to the user, wake-up device 1 will execute the command, such as using wake-up device 1 to play "The volume has been turned up." It is understood that the above is merely an illustrative example and should not be construed as a limitation of this application.

[0079] It is evident that since the smart voice device closest to the user typically captures the wake word audio first, selecting the wake-up device with the earliest wake word arrival time as the response device ensures that the closest smart voice device always responds, making the response more readily perceived by the user. Furthermore, since the wake word and the recognition statement are spoken by the same user, and the user is highly likely to utter both the wake word and the recognition statement from the same location, selecting the wake-up device with the best wake word audio quality as the pickup device in the recognition stage guarantees the highest quality audio for the acquired recognition statement, resulting in more accurate recognition. Moreover, the feature extraction of audio data is performed by the wake-up device, while the distributed center uses TDOA features to select the response device and audio quality features to select the pickup device. This significantly reduces the computational and memory requirements of the distributed center, lowering its overall demands.

[0080] In one possible implementation, the distributed voice interaction method provided in this application embodiment includes multiple wake-up devices and multiple first tdoa features. S102 specifically includes:

[0081] S1021. Based on the wake word time ranges corresponding to the multiple first tdoa features, extract the tdoa sub-features with common time periods from the multiple first tdoa features to obtain multiple tdoa sub-features;

[0082] It should be noted that since the wake-up times of multiple wake-up devices are not strictly consistent, the time ranges corresponding to the uploaded first TDOA feature cannot be guaranteed to be completely identical, but a considerable overlap is guaranteed. Furthermore, each wake-up device uploads its wake-up word time range along with its first TDOA feature. The wake-up word time range of the same wake-up device corresponds to the first TDOA feature, and the common time period can be extracted from the first TDOA feature based on the wake-up word time range.

[0083] For example, if there are two wake-up devices, wake-up device 1 wakes up at 09:00:01:01 with a wake-up word time range of 09:00:00:01 to 09:00:01:01; wake-up device 2 wakes up at 09:00:01:02 with a wake-up word time range of 09:00:00:02 to 09:00:01:02. Then, extract the portion of the common time period 09:00:00:02 to 09:00:01:01 from the first tdoa feature of wake-up device 1 to obtain the tdoa sub-feature of wake-up device 1. Similarly, extract the tdoa sub-feature of the common time period 09:00:00:02 to 09:00:01:01 from the first tdoa feature of wake-up device 2 to obtain the tdoa sub-feature of wake-up device 2. It should be understood that the above is merely an illustrative example and should not be construed as a limitation of this application.

[0084] S1022. Input the multiple tdoa sub-features into a preset first decision model to obtain a first decision result; wherein, the first decision model is trained using multiple first audio training data with known arrival time differences, and the first audio training data includes wake words;

[0085] In one possible implementation, the input to the first decision model can be N tdoa sub-features corresponding to N wake-up devices, and the output first decision result can be an N×1 dimensional vector. The position of the maximum value in this N×1 dimensional vector represents the wake-up device with the earliest wake-up word arrival time among the N wake-up devices. Here, N is the number of wake-up devices to be compared when training the first decision model, satisfying N≥2. For a specific wake-up word, the number of wake-up devices that the distributed center needs to compare is assumed to be N'. Assuming N'>N, the comparison can be performed multiple times, each time selecting the nearest wake-up device from a subset of devices with a size less than or equal to N, until the final selection result is obtained.

[0086] See Figure 2 This is a schematic diagram illustrating the selection of a response device using a first decision model, as disclosed in an embodiment of this application. Figure 2 "Model 2" in this context refers to the aforementioned "first decision model." Here, N=3, assuming there are 4 wake-up devices. Figure 2In the model, 1 represents wake-up device 1, 2 represents wake-up device 2, 3 represents wake-up device 3, and 4 represents wake-up device 4. The first comparison of wake-up devices 1, 2, and 3 determines that wake-up device 3 has the earliest wake-up word arrival time. Then, the comparison of wake-up devices 3 and 4 is performed. Since the number of comparisons is less than N, the tdoa sub-features of wake-up device 3 and 4 can be combined with an all-zero special feature and input into Model 2. The final selection result is wake-up device 4. It should be noted that the all-zero special feature can also be other forms of special features, used to indicate that the feature does not correspond to a specific wake-up device, and the model will not select the wake-up device corresponding to that feature. This application does not limit this. It is understood that the above is merely an illustrative example and should not be construed as a limitation of this application.

[0087] S1023. Based on the first judgment result, select the wake-up device with the earliest wake-up word arrival time as the response device.

[0088] For example, if the first decision result is 010, where the first 0 corresponds to wake-up device 3, the 1 corresponds to wake-up device 4, and the second 0 does not correspond to a specific wake-up device, then wake-up device 4 is selected as the response device. It should be understood that the above is merely an illustrative example and should not be construed as a limitation of this application.

[0089] As can be seen, in this embodiment, the first decision model running on the distributed center takes tdoa sub-features as input, rather than directly taking audio data as input. This allows the first decision model to be designed to be relatively small, significantly reducing the computational and memory requirements of the distributed center and lowering its demands. Furthermore, by extracting tdoa sub-features based on the wake-word time range, tdoa features within the same time period are used as the basis for selecting the response device, thus improving the accuracy of response device determination.

[0090] In one possible implementation, the distributed voice interaction method provided in this application embodiment includes multiple wake-up devices and multiple first audio quality features. S103 specifically includes:

[0091] S1031. Input multiple first audio quality features into a preset second decision model to obtain a second decision result; wherein, the second decision model is trained using multiple second audio training data with known audio quality, and the second audio training data includes wake words;

[0092] In one possible implementation, the input to the second decision model can be M audio quality features corresponding to M wake-up devices, and the output second decision result can be an M×1 dimensional vector. The position of the maximum value in this M×1 dimensional vector represents the wake-up device with the best wake-up word audio quality among the M wake-up devices. Here, M is the number of wake-up devices to be compared when training the second decision model, satisfying M≥2. For a specific wake-up word, the number of wake-up devices that the distributed center needs to compare is assumed to be M'. Assuming M'>M, the comparison can be performed multiple times, each time selecting the nearest wake-up device from a subset of devices with a size less than or equal to M, until the final selection result is obtained. This is similar to the first decision model, so it will not be elaborated further.

[0093] S1032. Based on the second judgment result, select the wake-up device with the best wake-up word audio quality as the sound pickup device in the recognition stage.

[0094] As can be seen, in this embodiment, the second decision model running on the distributed center takes audio quality features as input, rather than directly taking audio data as input. This allows the second decision model to be designed to be relatively small, thereby significantly reducing the computing power and memory requirements of the distributed center and lowering the requirements for the distributed center.

[0095] See Figure 3 This is a schematic diagram illustrating the training process of a first decision model and a first extraction model disclosed in an embodiment of this application. It should be noted that... Figure 5 "Model 1" in the text refers to the "first initial extraction model". Figure 3 "Model 2" in the text refers to the "first initial decision model". The last layer of Model 2 uses softmax as the activation function and is trained using the cross-entropy criterion. Figure 3 The alpha calculation in this context is based on the order of arrival time. In one possible implementation, the distributed voice interaction method provided in this embodiment further includes: constructing a first extraction model and a first decision model through the following construction process:

[0096] S201. Construct a first training set; wherein the first training set includes multiple first audio training data, each first audio training data includes audio from N arrays, each array of audio includes a wake word, the arrival time difference of the wake word in the N arrays is known, and N≥2;

[0097] In the embodiments of this application, audio training data may be obtained by means of simulation or recording, and this application does not limit this.

[0098] S202. Input the audio of each array in each first audio training data independently into the first initial extraction model to obtain N second tdoa features corresponding to each first audio training data.

[0099] S203. Randomly set the p second tdoa features corresponding to each first audio training data to zero to obtain p first zeroed features; wherein p satisfies 0≤p≤N-2;

[0100] S204. Taking the obtained N×1 dimensional vector as the training target, input the p first zeroed features and Np non-zeroed second tdoa features corresponding to each first audio training data into the first initial decision model, and perform joint iterative training on the first initial extraction model and the first initial decision model to obtain the trained first extraction model and the first decision model; wherein, the first extraction model is used to extract the first tdoa features, and the first decision model is used to select the wake-up device with the earliest wake-up word arrival time.

[0101] After obtaining the trained first extraction model and first decision model in this embodiment, the first extraction model can be deployed in multiple intelligent voice devices, and the first decision model can be deployed in a distributed center. In one possible implementation, the s-th value of the N×1 dimensional vector is 1, and the rest are 0, where s corresponds to the index of the array with the earliest arrival time in the N inputs in the array without zeros.

[0102] The first extraction model in this application embodiment can adopt various structures, such as CNN, RNN, U-net, etc., and there is no limitation on it.

[0103] See Figure 4 This is a schematic diagram of a first audio training data disclosed in an embodiment of this application. For example: assuming each piece of first audio training data contains audio from three arrays, and it is known that the wake word of array 1 arrives earliest, and the wake word of array 3 arrives earliest; the audio from arrays 1, 2, and 3 is independently processed through a first initial extraction model to obtain second tdoa feature 1, second tdoa feature 2, and second tdoa feature 3; one tdoa feature is randomly set to zero, such as setting second tdoa feature 1 to zero, resulting in a first zeroed feature; the first zeroed feature, second tdoa feature 2, and second tdoa feature 3 are used as inputs to a first initial decision model, and the output is an N×1 dimensional vector of 010. Since second tdoa feature 1 was randomly set to zero, second tdoa feature 2 is the wake word that arrives earliest. It should be understood that the above is only an illustrative example and should not be construed as a limitation of this application.

[0104] As can be seen, in this embodiment, the first extraction model and the first decision model are obtained through joint training, enabling the first decision model to accurately select the response device based on the TDOA features extracted by the first extraction model in practical applications. Furthermore, during training, the array in the audio training data is randomly set to zero. Thus, in practical applications, if the number of wake-up devices is less than N, it can be padded by setting a special matrix of all zeros. If the number of wake-up devices is greater than N, it can be compared multiple times, and padded accordingly by setting a special matrix of all zeros. This ensures that the first extraction model and the first decision model can accurately determine the response device in practical applications.

[0105] See Figure 5 This is a schematic diagram illustrating the training process of a second decision model and a second extraction model disclosed in an embodiment of this application. It should be noted that... Figure 5 "Model 3" in the text refers to the "second initial extraction model". Figure 5 Model 4 is the "second initial decision model". The last layer of Model 5 uses softmax as the activation function and is trained using the cross-entropy criterion. Figure 5 The alpha calculation in this context is based on audio quality. In one possible implementation, the distributed voice interaction method provided in this embodiment further includes constructing a second extraction model and a second decision model through the following construction process:

[0106] S301. Construct a second training set; wherein the second training set includes multiple second audio training data, each second audio training data includes audio from M arrays, each array of audio includes a wake word, the audio quality of the wake word in the M arrays is known, and M≥2;

[0107] S302. Input the audio of each array in each second audio training data independently into the second initial extraction model to obtain M second audio quality features corresponding to each second audio training data.

[0108] S303. Randomly set q second audio quality features corresponding to each second audio training data to zero to obtain q second zeroed features; wherein q satisfies 0≤q≤M-2;

[0109] S304. Using the obtained M×1 dimensional vector as the training target, input the q second zeroed features and Mq non-zeroed second audio quality features corresponding to each second audio training data into the second initial decision model, and perform joint iterative training on the second initial extraction model and the second initial decision model to obtain the trained second extraction model and the second decision model; wherein, the second extraction model is used to extract the first audio quality features, and the second decision model is used to select the wake-up device with the best wake-up word audio quality.

[0110] After obtaining the trained second extraction model and second decision model in this embodiment, the second extraction model can be deployed in multiple smart voice devices, and the second decision model can be deployed in a distributed center. In one possible implementation, the g-th value of the M×1 dimensional vector is 1, and the rest are 0, where g corresponds to the index of the array with the best audio quality in the M inputs among the unzeroed array.

[0111] It should be noted that the second extraction model may or may not have the same structure as the first extraction model. Similarly, the second decision model may or may not have the same structure as the first decision model; their output layer structures may be the same, and they may be used in the same way during testing, but their input features may differ. This application does not impose any limitations on this.

[0112] It should be noted that the methods for evaluating audio quality include, but are not limited to, signal-to-noise ratio, objective speech intelligibility, and acoustic model score of speech recognition. These can be selected and set according to actual needs, and this application does not impose any restrictions on them.

[0113] See Figure 6 This is a schematic diagram of a second audio training data disclosed in an embodiment of this application. For example: assuming each piece of second audio training data contains audio from three arrays, the signal-to-noise ratio (SNR) of array 1 is known to be 'a', the SNR of array 2 is 'b', and the SNR of array 3 is 'c', where a < b < c, meaning the audio quality of array 3 is known to be the best; the audio from arrays 1, 2, and 3 is independently processed through a second initial extraction model to obtain second audio quality feature 1, second audio quality feature 2, and second audio quality feature 3; one audio quality feature is randomly set to zero, such as setting second audio quality feature 2 to zero, resulting in a second zeroed feature; second audio quality feature 1, the second zeroed feature, and second audio quality feature 3 are used as inputs to a second initial decision model, outputting an M×1 dimensional vector of 001, meaning second audio quality feature 3 has the best wake-up word audio quality. It should be understood that the above is merely an illustrative description and should not be construed as a limitation of this application.

[0114] As can be seen, in this embodiment, a second extraction model and a second decision model are obtained through joint training. This enables the second decision model to accurately determine the pickup device based on the audio quality features extracted by the second extraction model in practical applications. Furthermore, during training, the array in the audio training data is randomly set to zero. Thus, in practical applications, if the number of awakened devices is less than M, it can be padded by setting a special matrix of all zeros. If the number of awakened devices is greater than M, it can be compared multiple times and padded accordingly by setting a special matrix of all zeros. Therefore, in practical applications, the second extraction model and the second decision model can accurately determine the pickup device.

[0115] In one possible implementation, the distributed voice interaction method provided in this application embodiment further includes:

[0116] The multiple intelligent voice devices are synchronized in time; wherein the synchronization error is less than d / c, where d represents the distance resolution between the intelligent voice devices and c represents the speed of sound in air.

[0117] It should be noted that the time asynchrony among multiple smart voice devices manifests in two main ways: sampling time deviation and sampling rate deviation. For voice interaction tasks, the duration of a single interaction is relatively short. Therefore, this embodiment requires synchronization of the sampling time deviation at the beginning of each interaction, while the sampling rate deviation can be calibrated or left unprocessed.

[0118] Sampling time deviation refers to the difference in the start time of recording between different devices. For example, if a television starts recording at 9:01:01 and an air conditioner starts recording at 9:05:05, assuming the television's 9:01:01:0000 corresponds to the air conditioner's 9:01:01:0025, then the sampling time deviation between the air conditioner and the television is 4 minutes and 3.9975 seconds. It should be understood that the above is merely an illustrative example and should not be construed as a limitation of this application.

[0119] During microphone recording, there is an analog-to-digital signal conversion module. The number of values ​​this module outputs per second corresponds to the sampling rate. The sampling rate is usually determined by the crystal oscillator of the hardware system, but these hardware components are not ideal. For example, the typical sampling rate for voice interaction is 16000Hz, meaning it can sample 16000 values ​​per second. Although different devices are labeled as 16000Hz, some devices may actually be 16000.1Hz, meaning one extra sampling point is added every 10 seconds. This results in 60 extra sampling points over 10 minutes. However, when the system processes data, it processes it in segments of fixed data length, such as segment 1 containing points 1-60, segment 2 containing points 61-120, and so on. If the initial times are aligned, that is, segment 1 of device A and segment 1 of device B are aligned, but because A's sampling rate is different, after a period of time, the times of segment × of A and segment × of B will be different, and the error will increase. It is understood that the above is only an illustrative example and should not be construed as a limitation of this application.

[0120] In the actual implementation of a distributed voice interaction system, there are two approaches: (1) Do not estimate the sampling rate difference, but estimate the sampling time deviation every certain period of time to ensure that the cumulative time deviation caused by the sampling rate difference within the interval is small and does not have a significant impact on the system. (2) Estimate the sampling time deviation and sampling rate deviation at the same time, and use the sampling rate deviation to calculate the change of the sampling time deviation over time, so that the time interval for calculating the synchronization parameters can be longer. Ideally, the synchronization parameters can be estimated only once after power-on.

[0121] For multiple intelligent voice devices, the synchronization accuracy requirement is at least milliseconds, meaning the time error is no greater than one-thousandth of a second. Since the speed of sound is 340 meters per second, the time difference that the distributed voice interaction system needs to utilize is the time it takes for sound to travel several tens of centimeters, i.e., one or two milliseconds. Therefore, the time error between different intelligent voice devices must be less than 1 millisecond. In other words, the synchronization accuracy of the sampling time deviation needs to be less than the distance resolution of the nearest response. Distance resolution is defined as follows: for any device A and device B, when the distance difference between these two devices and the sound source is greater than or equal to d, the system needs to be able to distinguish which device is closest to the sound source. Here, d is the distance resolution of the nearest response. Corresponding to the distance resolution d, the synchronization error of the sampling time deviation needs to be less than d / c, where c is the speed of sound in air.

[0122] It should be noted that this application imposes restrictions on the time synchronization method, requiring only that the synchronization error be less than d / c. Any existing method can be used for synchronization. For example, an analog signal can be generated using a specific device and wired to all smart voice devices. The smart voice devices can synchronously sample this signal along with the audio signal received by their microphones, and then compare the synchronization signals collected by different smart voice devices. Any method, such as maximizing cross-correlation, can be used to obtain the sampling time deviation between devices. It is understood that the above is merely an illustrative example and should not be construed as a limitation of this application. For distributed arrays that inherently satisfy synchronization, such as those using a unified clock for the entire distributed system, the device synchronization step can be skipped. It should be noted that the steps described above in the embodiments of this application are based on the premise that all smart voice devices have already been synchronized.

[0123] It should be noted that after time synchronization, the smart voice device collects audio data. The smart voice device itself possesses independent voice interaction capabilities and typically enhances the collected audio before sending the enhanced audio to the wake-up engine. In this embodiment, each device runs a processing algorithm to enhance and wake up the collected audio data. This embodiment does not limit the specific algorithms used for voice enhancement and wake-up in a single smart voice device.

[0124] As can be seen, by synchronizing the time of multiple smart voice devices in this embodiment, the accuracy of the subsequently determined response device and pickup device is further guaranteed.

[0125] See Figure 7 This application discloses a schematic diagram of the structure of a distributed center, which includes:

[0126] The receiving unit 701 is used to receive the wake-up word audio features and wake-up word time range uploaded by the wake-up device; wherein, the wake-up device is an intelligent voice device that is woken up by a wake-up word among multiple intelligent voice devices, and the wake-up word audio features include a first time difference of arrival (tdoa) feature and a first audio quality feature;

[0127] The first selection unit 702 is used to select the wake-up device with the earliest wake-up time as the response device based on the first tdoa feature and the wake-up word time range.

[0128] The second selection unit 703 is used to select the wake-up device with the best wake-up word audio quality as the sound pickup device in the recognition stage based on the first audio quality feature.

[0129] It is evident that since the smart voice device closest to the user typically captures the wake word audio first, selecting the wake-up device with the earliest wake word arrival time as the response device ensures that the closest smart voice device always responds, making the response more readily perceived by the user. Furthermore, since the wake word and the recognition statement are spoken by the same user, and the user is highly likely to utter both the wake word and the recognition statement from the same location, selecting the wake-up device with the best wake word audio quality as the pickup device in the recognition stage guarantees the highest quality audio for the acquired recognition statement, resulting in more accurate recognition. Moreover, the feature extraction of audio data is performed by the wake-up device, while the distributed center uses TDOA features to select the response device and audio quality features to select the pickup device. This significantly reduces the computational and memory requirements of the distributed center, lowering its overall demands.

[0130] In one possible implementation, the first selection unit 702 in the distributed center provided in this embodiment includes:

[0131] The extraction unit is used to extract tdoa sub-features with a common time period from the multiple first tdoa features according to the wake word time ranges corresponding to the multiple first tdoa features, so as to obtain multiple tdoa sub-features.

[0132] The first result acquisition unit is used to input the multiple tdoa sub-features into a preset first decision model to obtain a first decision result; wherein, the first decision model is trained using multiple first audio training data with known arrival time differences, and the first audio training data includes wake words;

[0133] The first selection subunit is used to select the wake-up device with the earliest wake-up word arrival time as the response device based on the first decision result.

[0134] In one possible implementation, the second selection unit 703 in the distributed center provided in this application embodiment includes:

[0135] The second result acquisition unit is used to input multiple first audio quality features into a preset second decision model to obtain a second decision result; wherein, the second decision model is trained using multiple second audio training data with known audio quality, and the second audio training data includes wake words;

[0136] The second selection subunit is used to select the wake-up device with the best wake-up word audio quality as the pickup device in the recognition stage based on the second decision result.

[0137] In one possible implementation, the distributed center provided in this embodiment further includes: a construction unit, the construction unit comprising:

[0138] A construction unit is used to construct a first training set; wherein the first training set includes multiple first audio training data, each first audio training data includes audio from N arrays, each array of audio includes a wake word, the arrival time difference of the wake word in the N arrays is known, and N≥2;

[0139] The feature extraction unit is used to independently input the audio of each array in each first audio training data into the first initial extraction model to obtain N second TDOA features corresponding to each first audio training data.

[0140] The zeroing unit is used to randomly zero out the p second TDOA features corresponding to each first audio training data, thereby obtaining p first zeroed features; wherein p satisfies 0≤p≤N-2;

[0141] The training unit is used to obtain an N×1 dimensional vector as the training objective. It inputs the p first zeroed features and Np non-zeroed second tdoa features corresponding to each first audio training data into the first initial decision model, and performs joint iterative training on the first initial extraction model and the first initial decision model to obtain the trained first extraction model and the first decision model. The first extraction model is used to extract the first tdoa features, and the first decision model is used to select the wake-up device with the earliest wake-up word arrival time.

[0142] In one possible implementation, the distributed center provided in this embodiment further includes: a construction unit, the construction unit comprising:

[0143] A construction unit is used to construct a second training set; wherein the second training set includes multiple second audio training data, each second audio training data includes audio from M arrays, each array of audio includes a wake word, the audio quality of the wake word in the M arrays is known, and M≥2;

[0144] The feature extraction unit is used to independently input the audio of each array in each second audio training data into the second initial extraction model to obtain M second audio quality features corresponding to each second audio training data.

[0145] The zeroing unit is used to randomly zero out q second audio quality features corresponding to each second audio training data, resulting in q second zeroed features; wherein q satisfies 0≤q≤M-2;

[0146] The training unit is used to obtain an M×1 dimensional vector as the training objective. It inputs q second zeroed features and Mq non-zeroed second audio quality features corresponding to each second audio training data into the second initial decision model, and performs joint iterative training on the second initial extraction model and the second initial decision model to obtain the trained second extraction model and the second decision model. The second extraction model is used to extract the first audio quality features, and the second decision model is used to select the wake-up device with the best wake-up word audio quality.

[0147] In one possible implementation, the distributed center provided in this application embodiment further includes:

[0148] A synchronization unit is used to synchronize the time of the plurality of intelligent voice devices; wherein the synchronization error is less than d / c, where d represents the distance resolution between the intelligent voice devices and c represents the speed of sound in the air.

[0149] See Figure 8This application discloses a schematic diagram of a distributed voice interaction system. This application also provides a distributed voice interaction system, which includes multiple intelligent voice devices 1 and a distributed center 2.

[0150] The distributed center 2 is used to receive the wake-up word audio features and wake-up word time range uploaded by the wake-up device; wherein, the wake-up device is an intelligent voice device that is woken up by a wake-up word among multiple intelligent voice devices, and the wake-up word audio features include a first arrival time difference (tdoa) feature and a first audio quality feature;

[0151] The distributed center 2 is also used to select the wake-up device with the earliest wake-up time as the response device based on the first tdoa feature and the wake-up word time range;

[0152] The distributed center 2 is also used to select the wake-up device with the best wake-up word audio quality as the sound pickup device in the recognition stage based on the first audio quality feature.

[0153] It is evident that since the smart voice device closest to the user typically captures the wake word audio first, selecting the wake-up device with the earliest wake word arrival time as the response device ensures that the closest smart voice device always responds, making the response more readily perceived by the user. Furthermore, since the wake word and the recognition statement are spoken by the same user, and the user is highly likely to utter both the wake word and the recognition statement from the same location, selecting the wake-up device with the best wake word audio quality as the pickup device in the recognition stage guarantees the highest quality audio for the acquired recognition statement, resulting in more accurate recognition. Moreover, the feature extraction of audio data is performed by the wake-up device, while the distributed center uses TDOA features to select the response device and audio quality features to select the pickup device. This significantly reduces the computational and memory requirements of the distributed center, lowering its overall demands.

[0154] In one possible implementation, the distributed voice interaction system provided in this application embodiment is specifically used for:

[0155] Based on the wake word time ranges corresponding to multiple first tdoa features, tdoa sub-features with common time periods are extracted from the multiple first tdoa features to obtain multiple tdoa sub-features;

[0156] The multiple tdoa sub-features are input into a preset first decision model to obtain a first decision result; wherein, the first decision model is trained using multiple first audio training data with known arrival time differences, and the first audio training data includes wake words;

[0157] Based on the first judgment result, the wake-up device with the earliest wake-up word arrival time is selected as the response device.

[0158] In one possible implementation, the distributed voice interaction system provided in this application embodiment is specifically used for:

[0159] Multiple first audio quality features are input into a preset second decision model to obtain a second decision result; wherein, the second decision model is trained using multiple second audio training data with known audio quality, and the second audio training data includes wake words;

[0160] Based on the second judgment result, the wake-up device with the best wake-up word audio quality is selected as the sound pickup device in the recognition stage.

[0161] In one possible implementation, the distributed voice interaction system provided in this application embodiment is further configured to:

[0162] The first extraction model and the first decision model are constructed through the following process:

[0163] Construct a first training set; wherein the first training set includes multiple first audio training data, each first audio training data includes audio from N arrays, each array of audio includes a wake word, the arrival time difference of the wake word in the N arrays is known, and N≥2;

[0164] The audio of each array in each first audio training data is independently input into the first initial extraction model to obtain N second TDOA features corresponding to each first audio training data.

[0165] Randomly set p second tdoa features corresponding to each first audio training data to zero to obtain p first zeroed features; wherein p satisfies 0≤p≤N-2;

[0166] With the N×1 dimensional vector as the training objective, the p first zeroed features and Np non-zeroed second tdoa features corresponding to each first audio training data are input into the first initial decision model. The first initial extraction model and the first initial decision model are jointly iteratively trained to obtain the trained first extraction model and the first decision model. The first extraction model is used to extract the first tdoa features, and the first decision model is used to select the wake-up device with the earliest wake-up word arrival time.

[0167] In one possible implementation, the distributed voice interaction system provided in this application embodiment is further configured to:

[0168] The second extraction model and the second decision model are constructed through the following process:

[0169] Construct a second training set; wherein the second training set includes multiple second audio training data, each second audio training data includes audio from M arrays, each array of audio includes a wake word, the audio quality of the wake word in the M arrays is known, and M≥2;

[0170] The audio of each array in each second audio training data is independently input into the second initial extraction model to obtain M second audio quality features corresponding to each second audio training data.

[0171] Randomly set q second audio quality features corresponding to each second audio training data to zero to obtain q second zero-set features; wherein q satisfies 0≤q≤M-2;

[0172] With the M×1 dimensional vector as the training objective, each second audio training data point corresponds to q second zeroed features and Mq non-zeroed second audio quality features, which are then input into the second initial decision model. The second initial extraction model and the second initial decision model are jointly iteratively trained to obtain the trained second extraction model and the second decision model. The second extraction model is used to extract the first audio quality features, and the second decision model is used to select the wake-up device with the best wake-up word audio quality.

[0173] In one possible implementation, in the distributed voice interaction system provided in this application embodiment, the distributed center 2 is further used to: synchronize the multiple intelligent voice devices in time; wherein the synchronization error is less than d / c, where d represents the distance resolution between the intelligent voice devices and c represents the speed of sound in the air.

[0174] In one possible implementation, in the distributed voice interaction system provided in this application embodiment, the intelligent voice device 1 is used to determine the time range of the wake word when it is woken up by the wake word;

[0175] The intelligent voice device 1 is also used to determine the wake-up word segment audio from the collected audio data according to the wake-up word time range;

[0176] The intelligent voice device 1 is further configured to input the wake-up word audio into a preset first extraction model to obtain the first TDOA feature; wherein, the first extraction model is trained using multiple first audio training data with known arrival time differences, and the first audio training data includes the wake-up word;

[0177] The intelligent voice device 1 is also used to upload the first tdoa feature and the wake word time range to the distributed center.

[0178] It should be noted that the wake-up word time range in this embodiment can be determined based on the start and end points of the wake-up word provided by the wake-up engine, or a pre-set time range can be used, which needs to include the length of most wake-up words. In this embodiment, the first TDOA feature is calculated frame by frame, and all wake-up devices must use a time-aligned frame-by-frame method to calculate the first TDOA feature.

[0179] It should be noted that in this embodiment, the first TDOA feature is not extracted before the wake-up device is activated, but the collected audio data is cached. The cached audio data must be long enough to ensure that at the moment of wake-up, the cached audio data contains the complete wake-up phrase audio. Then, the TDOA feature is extracted from the cached audio data. In this embodiment, the average computational cost of extracting the TDOA feature is relatively small.

[0180] As can be seen, in this application, the first feature extraction model runs on each wake-up device. This distributes the computation of the entire response device selection process across the wake-up devices, allocating less computational load to the distributed center. Consequently, the computational power and memory requirements of the distributed center are significantly reduced, lowering the demands on it. Moreover, the TDOA feature has a smaller dimension compared to the wake-up word audio, which reduces the amount of data uploaded and transmission latency.

[0181] In one possible implementation, in the distributed voice interaction system provided in this application embodiment, the intelligent voice device 1 is used to input the real-time collected audio data frame by frame into a preset first extraction model to obtain multiple first TDOA sub-features; wherein, the first extraction model is trained using multiple first audio training data with known arrival time differences, and the first audio training data includes wake words;

[0182] The intelligent voice device 1 is also used to determine the time range of the wake word when it is woken up by the wake word;

[0183] The intelligent voice device 1 is further configured to determine the first tdoa feature from the plurality of first tdoa sub-features based on the wake word time range;

[0184] The intelligent voice device 1 is also used to upload the first tdoa feature and the wake word time range to the distributed center.

[0185] It should be noted that in this embodiment, the first tdoa sub-feature is selected within a certain frame range based on the wake-up word time range. In this embodiment, tdoa features are extracted frame by frame, distributing the computational load across each frame and preventing a burst of feature extraction computation at the wake-up moment.

[0186] As can be seen, in this application, the first feature extraction model runs on each wake-up device. This distributes the computation of the entire response device selection process across the wake-up devices, allocating less computational load to the distributed center. Consequently, the computational power and memory requirements of the distributed center are significantly reduced, lowering the demands on it. Moreover, the TDOA feature has a smaller dimension compared to the wake-up word audio, which reduces the amount of data uploaded and transmission latency.

[0187] In one possible implementation, in the distributed voice interaction system provided in this application embodiment, the intelligent voice device 1 is used to determine the time range of the wake word when it is woken up by the wake word;

[0188] The intelligent voice device 1 is also used to determine the wake-up word segment audio from the collected audio data according to the wake-up word time range;

[0189] The intelligent voice device 1 is further configured to input the wake-up word audio into a preset second extraction model to obtain the first audio quality feature; wherein, the second extraction model is trained using multiple second audio training data with known audio quality, and the second audio training data includes the wake-up word;

[0190] The intelligent voice device 1 is also used to upload the first audio quality feature to the distributed center.

[0191] It should be noted that in this embodiment, the first audio quality feature is not extracted before the wake-up device is woken up, but the collected audio data is cached. The length of the cached audio data must be long enough to ensure that at the moment of wake-up, the cached audio data contains the complete wake-up phrase audio, and then the audio quality feature is extracted from the cached audio data. In this embodiment, the average computational load for extracting audio quality features is relatively small.

[0192] As can be seen, in this application, the second feature extraction model runs on each wake-up device. This distributes the computation of the entire response device selection process across the wake-up devices, allocating less computational load to the distributed center. Consequently, the computational power and memory requirements of the distributed center are significantly reduced, lowering the demands on it. Furthermore, audio quality features have smaller dimensions compared to wake-up word audio, thus reducing the amount of data uploaded and transmission latency.

[0193] In one possible implementation, in the distributed voice interaction system provided in this application embodiment, the intelligent voice device 1 is used to input the real-time collected audio data frame by frame into a preset second extraction model to obtain multiple first audio quality sub-features; wherein, the second extraction model is trained using multiple second audio training data with known audio quality, and the second audio training data includes wake words;

[0194] The intelligent voice device 1 is also used to determine the time range of the wake word when it is woken up by the wake word;

[0195] The intelligent voice device 1 is further configured to determine the first audio quality feature from the plurality of first audio quality sub-features based on the wake word time range;

[0196] The intelligent voice device 1 is also used to upload the first audio quality feature to the distributed center.

[0197] As can be seen, in this application, the second feature extraction model runs on each wake-up device. This distributes the computation of the entire response device selection process across the wake-up devices, allocating less computational load to the distributed center. Consequently, the computational power and memory requirements of the distributed center are significantly reduced, lowering the demands on it. Furthermore, audio quality features have smaller dimensions compared to wake-up word audio, thus reducing the amount of data uploaded and transmission latency.

[0198] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0199] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the devices and systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

[0200] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0201] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A distributed voice interaction method, characterized in that, The method includes: The system receives the wake-up word audio features and wake-up word time range uploaded by the wake-up device; wherein, the wake-up device is an intelligent voice device that is woken up by a wake-up word among multiple intelligent voice devices, and the wake-up word audio features include a first time difference of arrival (tdoa) feature and a first audio quality feature; Based on the first tdoa feature and the wake word time range, the wake-up device with the earliest wake word arrival time is selected as the response device; Based on the first audio quality feature, the wake-up device with the best wake-up word audio quality is selected as the sound pickup device in the recognition stage.

2. The method according to claim 1, characterized in that, The wake-up devices are multiple, and the first TDOA feature is multiple. The step of selecting the wake-up device with the earliest wake-up word arrival time as the response device based on the first TDOA feature and the wake-up word time range includes: Based on the wake word time ranges corresponding to multiple first tdoa features, tdoa sub-features with common time periods are extracted from the multiple first tdoa features to obtain multiple tdoa sub-features; The multiple tdoa sub-features are input into a preset first decision model to obtain a first decision result; wherein, the first decision model is trained using multiple first audio training data with known arrival time differences, and the first audio training data includes wake words; Based on the first judgment result, the wake-up device with the earliest wake-up word arrival time is selected as the response device.

3. The method according to claim 1, characterized in that, The wake-up devices are multiple, and the first audio quality features are multiple. The step of selecting the wake-up device with the best wake-up word audio quality based on the first audio quality features as the sound pickup device for the recognition stage includes: Multiple first audio quality features are input into a preset second decision model to obtain a second decision result; wherein, the second decision model is trained using multiple second audio training data with known audio quality, and the second audio training data includes wake words; Based on the second judgment result, the wake-up device with the best wake-up word audio quality is selected as the sound pickup device in the recognition stage.

4. The method according to claim 1, characterized in that, The method further includes: The first extraction model and the first decision model are constructed through the following process: Construct a first training set; wherein the first training set includes multiple first audio training data, each first audio training data includes audio from N arrays, each array of audio includes a wake word, the arrival time difference of the wake word in the N arrays is known, and N≥2; The audio of each array in each first audio training data is independently input into the first initial extraction model to obtain N second TDOA features corresponding to each first audio training data. Randomly set p second tdoa features corresponding to each first audio training data to zero to obtain p first zeroed features; wherein p satisfies 0≤p≤N-2; With the N×1 dimensional vector as the training objective, the p first zeroed features and Np non-zeroed second tdoa features corresponding to each first audio training data are input into the first initial decision model. The first initial extraction model and the first initial decision model are jointly iteratively trained to obtain the trained first extraction model and the first decision model. The first extraction model is used to extract the first tdoa features, and the first decision model is used to select the wake-up device with the earliest wake-up word arrival time.

5. The method according to claim 1, characterized in that, The method further includes: The second extraction model and the second decision model are constructed through the following process: Construct a second training set; wherein the second training set includes multiple second audio training data, each second audio training data includes audio from M arrays, each array of audio includes a wake word, the audio quality of the wake word in the M arrays is known, and M≥2; The audio of each array in each second audio training data is independently input into the second initial extraction model to obtain M second audio quality features corresponding to each second audio training data. Randomly set q second audio quality features corresponding to each second audio training data to zero to obtain q second zero-set features; wherein q satisfies 0≤q≤M-2; With the M×1 dimensional vector as the training objective, each second audio training data point corresponds to q second zeroed features and Mq non-zeroed second audio quality features, which are then input into the second initial decision model. The second initial extraction model and the second initial decision model are jointly iteratively trained to obtain the trained second extraction model and the second decision model. The second extraction model is used to extract the first audio quality features, and the second decision model is used to select the wake-up device with the best wake-up word audio quality.

6. The method according to claim 1, characterized in that, The method further includes: The multiple intelligent voice devices are synchronized in time; wherein the synchronization error is less than d / c, where d represents the distance resolution between the intelligent voice devices and c represents the speed of sound in air.

7. A distributed center, characterized in that, The distributed center includes: A receiving unit is used to receive the wake-up word audio features and wake-up word time range uploaded by the wake-up device; wherein, the wake-up device is an intelligent voice device that is woken up by a wake-up word among multiple intelligent voice devices, and the wake-up word audio features include a first time difference of arrival (tdoa) feature and a first audio quality feature; The first selection unit is used to select the wake-up device with the earliest wake-up time as the response device based on the first tdoa feature and the wake-up word time range. The second selection unit is used to select the wake-up device with the best wake-up word audio quality as the sound pickup device in the recognition stage based on the first audio quality feature.

8. A distributed voice interaction system, characterized in that, The voice interaction system includes multiple intelligent voice devices and a distributed center; The distributed center is used to receive the wake-up word audio features and wake-up word time range uploaded by the wake-up device; wherein, the wake-up device is an intelligent voice device that is woken up by a wake-up word among multiple intelligent voice devices, and the wake-up word audio features include a first time difference of arrival (tdoa) feature and a first audio quality feature; The distributed center is also used to select the wake-up device with the earliest wake-up time as the response device based on the first tdoa feature and the wake-up word time range; The distributed center is also used to select the wake-up device with the best wake-up word audio quality as the sound pickup device in the recognition stage based on the first audio quality feature.

9. The system according to claim 8, characterized in that, The intelligent voice device is used to determine the time range of the wake word when it is woken up by the wake word; The intelligent voice device is also used to determine the wake-up word audio segment from the collected audio data based on the wake-up word time range; The intelligent voice device is further configured to input the wake-up word audio into a preset first extraction model to obtain the first TDOA feature; wherein, the first extraction model is trained using multiple first audio training data with known arrival time differences, and the first audio training data includes the wake-up word; The intelligent voice device is also used to upload the first tdoa feature and the wake word time range to the distributed center.

10. The system according to claim 8, characterized in that, The intelligent voice device is used to input real-time collected audio data frame by frame into a preset first extraction model to obtain multiple first TDOA sub-features; wherein, the first extraction model is trained using multiple first audio training data with known arrival time differences, and the first audio training data includes wake words; The intelligent voice device is also used to determine the time range of the wake word when it is woken up by the wake word; The intelligent voice device is further configured to determine the first tdoa feature from the plurality of first tdoa sub-features based on the wake word time range; The intelligent voice device is also used to upload the first tdoa feature and the wake word time range to the distributed center.

11. The system according to claim 8, characterized in that, The intelligent voice device is used to determine the time range of the wake word when it is woken up by the wake word; The intelligent voice device is also used to determine the wake-up word audio segment from the collected audio data based on the wake-up word time range; The intelligent voice device is further configured to input the wake-up word audio into a preset second extraction model to obtain the first audio quality feature; wherein, the second extraction model is trained using multiple second audio training data with known audio quality, and the second audio training data includes the wake-up word; The intelligent voice device is also used to upload the first audio quality feature to the distributed center.

12. The system according to claim 8, characterized in that, The intelligent voice device is used to input real-time collected audio data frame by frame into a preset second extraction model to obtain multiple first audio quality sub-features; wherein, the second extraction model is trained using multiple second audio training data with known audio quality, and the second audio training data includes wake words; The intelligent voice device is also used to determine the time range of the wake word when it is woken up by the wake word; The intelligent voice device is further configured to determine the first audio quality feature from the plurality of first audio quality sub-features based on the wake word time range; The intelligent voice device is also used to upload the first audio quality feature to the distributed center.

Images