Distributed voice processing system and method
Through the distributed voice processing system, audio signal preprocessing and recognition is used to use multiple node devices to solve the problems of cross-room recognition and network instability in the prior art, and improve the reliability of voice control and user privacy security.
Patent Information
- Application Number
- CN202411067272.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-31
- Publication Date
- 2024-11-26
AI Technical Summary
Existing voice recognition technologies have shortcomings in cross-room recognition, network instability and user privacy security, especially the method of relying on a single device or cloud server can easily lead to identification failure and privacy leakage.
A distributed voice processing system is adopted to form a network through multiple node devices to pre-process and identify audio signals. Each node device in the network participates in the identification process, reduces dependence on the central device, and transmits pre-processing results to reduce the network. Bandwidth requirements.
It realizes the expansion of recognition distance without relying on the Internet, improves the reliability and recognition rate of cross-room voice control, enhances the stability and robustness of recognition, reduces dependence on the network, and improves user privacy and security.
Smart Images

Figure CN119028343A_ABST
Abstract
Description
[0001] Divisional application
[0002] This application is a divisional application of the invention application submitted to the State Intellectual Property Office, with an application date of December 31, 2020, titled "Distributed Speech Processing System and Method", and application number CN 202011627998.X. Technical Field
[0003] The present application relates to the field of distributed speech processing, and in particular to a distributed speech processing system and method. Background Art
[0004] Voice recognition technology and keyword recognition technology are becoming more and more mature and are being used more and more widely in the market. For example, voice recognition functions have been added to children's toys, educational products, smart homes and other products to achieve voice interactive control functions.
[0005] There are currently two common methods for speech recognition: one is based on local recognition on a single device, and the other is based on local recognition combined with server cloud recognition.
[0006] For the first method, a single device commonly used in the smart home market implements local voice control. The voice recognition process is to collect the original voice into one device and perform calculations on this device to obtain the recognition result. In the application process of this method, if the user moves in the spatial environment or needs to recognize across rooms, the voice recognition is often not completed smoothly due to the limitation of the sound pickup distance, resulting in the situation that the recognition cannot be completed or the recognition effect is poor.
[0007] For the second method, voice recognition is usually performed on the market through smart speakers or smart gateways. These devices serve as the control center and are also the only entrance to voice recognition. The device must first be connected to the Internet, and then the corresponding cloud server must also be connected to the Internet. These devices obtain voice recognition results from the cloud to complete voice recognition or voice control. There are problems with this method. For example, if the device that is the only entrance to voice recognition fails or the network fluctuates, it will cause the problem of voice recognition failure. In particular, when the network stability is poor, it is easy to cause slow recognition response speed. In addition, this type of voice recognition method uploads the voice to the cloud, and the device needs to monitor the surrounding environment sound in real time, which is easy to cause user privacy and security issues.
[0008] At the same time, both methods have problems such as inability to achieve voice recognition control across rooms.
[0009] In addition to the above two methods, there is another local center recognition method, which collects the original voice from multiple points and transmits it to a central device for voice recognition. This method can be used to solve problems such as short pickup distance, difficulty in cross-room recognition, and personnel movement. However, this method is heavily dependent on the central device. When the central device fails, the voice recognition function of the entire system will fail. In addition, since the direct transmission of original voice data has high network requirements and large data transmission delay, its actual recognition effect is poor.
[0010] The Chinese patent (CN111415658A) discloses a decentralized voice-controlled multi-device system and its control method. In this scheme, the device first identifies the wake-up word in the voice, then sends the identified wake-up word to all devices in the system, and simultaneously receives the wake-up word sent by other devices in the system. The device screens all the wake-up words and selects the wake-up words that match the device. In this scheme, if the voice received by the device contains a wake-up word (i.e., a voice command) that it does not support, voice control may fail.
[0011] Chinese patent (CN110136708A) discloses a distributed voice control system and control method based on Bluetooth mesh, which includes a Bluetooth Mesh network, a voice controller, and a Bluetooth node device; the voice controller includes voice collection, voice noise reduction, voice recognition, a Bluetooth module, and an optional WiFi module. The voice controllers communicate with each other via Bluetooth and keep data synchronization in real time. Any voice controller can control the Bluetooth node device in the network; the Bluetooth node device communicates with the voice controller via the Bluetooth Mesh network and responds to operations based on the received Mesh data or its own key events. In this solution, each voice controller collects voice, performs voice noise reduction and echo cancellation, and then performs local or online voice recognition, semantic understanding and analysis of the information to be controlled, encapsulates it into Mesh data, and sends it to the Mesh network through the Bluetooth module. If the voice controller does not support the current control command, it may cause the device to be unable to recognize the voice command that it does not support, and ultimately cause the voice control to fail.
[0012] In summary, an improved distributed speech processing solution is needed in the prior art to solve the above problems existing in the prior art. It should be understood that the above-mentioned technical problems are only examples and not limitations of the present invention, and the present invention is not limited to the technical solution that solves all the above-mentioned technical problems at the same time. The technical solution of the present invention can be implemented to solve one or more of the above-mentioned or other technical problems. Summary of the invention
[0013] In order to overcome the defects of the prior art, the present invention discloses a distributed speech processing system and a processing method thereof.
[0014] In one aspect of the present invention, a distributed speech processing system is provided, comprising: a plurality of node devices, the plurality of node devices forming a network, wherein each node device comprises a processor, a memory, a communication module and a sound processing module, and at least one of the plurality of node devices comprises a sound collection module; wherein the sound collection module is configured to collect an audio signal; the sound processing module is configured to preprocess the audio signal to obtain a first sound preprocessing result, wherein the preprocessing comprises extracting a sound feature value of the audio signal; wherein the communication module is further configured to receive one or more second sound preprocessing results from at least one other node device in the network; and the sound processing module is further configured to perform speech recognition based on the first sound preprocessing result and / or the one or more second sound preprocessing results to obtain a first speech recognition result.
[0015] Preferably, the communication module is further configured to send the first sound preprocessing result to one or more node devices in the network.
[0016] Optionally, the communication module is further configured to send the first speech recognition result to one or more node devices in the network.
[0017] Optionally, the communication module is also configured to receive one or more second voice recognition results from at least one other node device from the network; and the sound processing module is also configured to perform voice recognition based on the first voice recognition result and the one or more second voice recognition results to obtain a final voice recognition result.
[0018] Preferably, each of the first sound preprocessing result and the one or more second sound preprocessing results includes a sound feature value, sound quality and sound time information.
[0019] In another aspect of the present invention, a distributed speech processing method is provided, which is executed by a node device in a network, comprising: if the node device includes a sound collection module, executing the following steps: collecting an audio signal; and preprocessing the audio signal to obtain a first sound preprocessing result, wherein the preprocessing includes extracting a sound feature value of the audio signal; receiving one or more second sound preprocessing results from at least one other node device in the network; and performing speech recognition based on the first sound preprocessing result and / or the one or more second sound preprocessing results to obtain a first speech recognition result.
[0020] In another aspect of the present invention, a distributed speech processing system is provided, comprising: a plurality of node devices, the plurality of node devices forming a network, wherein each node device comprises a processor, a memory, a communication module, a sound collection module and a sound processing module; wherein the sound collection module is configured to collect an audio signal; the sound processing module is configured to pre-process the audio signal to obtain a first sound pre-processing result; wherein the communication module is further configured to receive one or more second sound pre-processing results from at least one other node device from the network; each of the first sound pre-processing result and the one or more second sound pre-processing results comprises one or more data blocks; each of the one or more data blocks comprises time information, the time information identifies the time when the sound processing module completes the pre-processing of the data block; each of the one or more data blocks comprises an ascending sequence number, the ascending sequence number is allocated according to the time information in the data block; the sound processing module is further configured to splice the data blocks of the first sound pre-processing result and / or the one or more second sound pre-processing results in ascending order of the ascending sequence number to obtain a complete third sound pre-processing result; and the sound processing module is further configured to process the third sound pre-processing result to obtain a final speech recognition result.
[0021] Preferably, the communication module is configured to send the first sound preprocessing result to one or more node devices in the network.
[0022] Optionally, each data block in the first or second sound preprocessing result is configured to have the same duration.
[0023] Optionally, the incremental serial number is allocated to the data block by a sound processing module of each node device among the multiple node devices when pre-processing the audio signal.
[0024] Optionally, the incremental serial number is allocated by the sound processing module of each node device among the multiple node devices to the data block of the second sound preprocessing result after receiving the second sound preprocessing result from at least one other node device from the network.
[0025] Optionally, the sound processing module is configured to detect the time difference of the data blocks, and if the time difference is within a specified threshold, assign the same increasing sequence number.
[0026] Optionally, the sound processing module is configured to select data blocks with the best sound quality from data blocks with the same increasing sequence number for splicing.
[0027] In another aspect of the present invention, a distributed speech processing method is provided, which is executed by a node device in a network, comprising: if the node device includes a sound collection module, executing the following steps: collecting an audio signal; and preprocessing the audio signal to obtain a first sound preprocessing result, the method also comprising: receiving one or more second sound preprocessing results from at least one other node device from the network; each of the first sound preprocessing result and the one or more second sound preprocessing results includes one or more data blocks, wherein each of the one or more data blocks includes time information, the time information identifies the time when the sound processing module completes the preprocessing of the data block, and wherein each of the one or more data blocks also includes an increasing sequence number, the increasing sequence number is allocated according to the time information in the data block; splicing the data blocks of the first sound preprocessing result and / or one or more second sound preprocessing results in increasing order of the increasing sequence numbers to obtain a complete third sound preprocessing result; and processing the third sound preprocessing result to obtain a final speech recognition result.
[0028] Preferably, the distributed voice processing method further comprises sending the first sound preprocessing result to one or more node devices in the network.
[0029] Optionally, each data block in the first sound preprocessing result is configured to have the same duration.
[0030] Optionally, the incremental sequence number is allocated to the data block when the sound processing module of each node device in the multiple node devices pre-processes the audio signal, wherein the incremental sequence number is allocated according to the time information.
[0031] Optionally, the incremental serial number is assigned to the data block of the second sound preprocessing result after the sound processing module of each node device among the multiple node devices receives the second sound preprocessing result from the network from at least one other node device, wherein the incremental serial number is assigned according to the time information.
[0032] Optionally, the sound processing module is configured to detect the time difference of the data blocks, and if the time difference is within a threshold, assign the same increasing sequence number.
[0033] Optionally, the sound processing module is configured to select a data block with the best sound quality from data blocks with the same increasing sequence number.
[0034] The solution provided in this application can extend the recognition distance without Internet access, improve the recognition rate when people are moving, and easily realize cross-room voice control. At the same time, it can also make voice recognition closer to user habits and more suitable for life usage scenarios.
[0035] In addition, the present invention can realize voice recognition control over ultra-long distances or across multiple rooms by performing distributed voice recognition through node devices in the network. The technical solution of the present invention enables each node device in the network to participate in the process of voice recognition. On the one hand, a decentralized design is realized, thereby reducing recognition failures caused by failures of key central nodes, and the design can realize concurrent voice recognition of devices in the network, which can improve the efficiency of voice recognition; on the other hand, the information transmitted during the recognition process is sound preprocessing information, that is, non-original audio data, so the bandwidth requirements of the network are not high, which improves the stability of voice recognition. By transmitting non-original audio data, two advantages can be generated: first, compared with the voice recognition method that directly transmits the original data, the amount of data required to be transmitted by the solution in the present invention is reduced; second, compared with the voice recognition method that directly transmits the recognition result, the solution in the present invention transmits the sound preprocessing result, which can avoid recognition failures caused by unsupported instructions, and improve the stability and robustness of voice recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Hereinafter, the present application will be further explained based on embodiments with reference to the accompanying drawings.
[0037] Figure 1 A block diagram schematically illustrates an embodiment of a distributed speech processing system according to the present invention;
[0038] Figure 2 A block diagram schematically showing another embodiment of a distributed speech processing system according to the present invention;
[0039] Figure 3 A block diagram schematically shows a node device of an embodiment of a distributed speech processing system according to the present invention;
[0040] Figure 4 A flow chart schematically illustrates an embodiment of a distributed speech processing method according to the present invention;
[0041] Figure 5 A flow chart schematically illustrates another embodiment of the distributed speech processing method according to the present invention;
[0042] Figure 6 A flowchart schematically illustrating a specific embodiment of the distributed speech processing method according to the present invention; and
[0043] Figure 7 The following is a flow chart schematically showing another specific embodiment of the distributed speech processing method according to the present invention. DETAILED DESCRIPTION
[0044] The following will be combined with the accompanying drawings and specific implementation methods to describe in detail the distributed speech recognition processing system and processing method of the present application. It should be understood that the embodiments shown in the accompanying drawings and described below are merely illustrative and not intended to limit the present application. In addition, it should be understood that in the present disclosure, ordinal numbers such as "first", "second", "third", etc., unless explicitly specified or determined by the technical context, are only used to indicate different or identical elements in the technical solution, and do not imply any limitation on the order or importance of these elements.
[0045] Figure 1 A block diagram of an embodiment of a distributed speech processing system 100 according to the present invention is shown, and the system includes multiple node devices 102, 104, 106 and 112, 114 and 116 in a network 110. It should be understood that the network 100 can be, for example, a wired, wireless and / or wired-wireless hybrid network used in a home and / or office, including but not limited to a wireless network commonly used in a smart home scenario. The various devices that form the network 110 can communicate in a wired or wireless manner. Among them, the wired method can adopt a communication method such as a network cable or a power line carrier, and the wireless method can adopt a communication method such as Wi-Fi, BLE, Zigbee, etc. to achieve networking communication between various devices.
[0046] In a specific embodiment, each node device has the ability to connect to other node devices. Each node device can form a self-organizing network to form a self-organizing network or a group network. Each device can also form a Mesh network, which can realize that any device node in the Mesh network can also serve as a router, that is, each node in the network can send and receive signals, and each node can communicate directly with one or more peer nodes.
[0047] Figure 2 A block diagram of another embodiment of a distributed speech processing system 200 according to the present invention is schematically shown, in which some node devices form a group, so that the system of the present invention can also send messages to the group in a broadcast or multicast manner. It should be understood that the node devices can be in one or more groups, and the groups can be dynamic and user-defined, without requiring the node devices between the groups to have fixed hardware or communication connection relationships.
[0048] exist Figure 1 and Figure 2 In the system shown, the distances between users and node devices may be different. For example, user 108 is between device B and device C and is within the sound pickup distance of device B and device C, but user 108 is far away from device A and other devices, and the voice signal sent by user 108 cannot be directly received by device A and other devices.
[0049] Figure 3 The block diagram of the node device 300 according to one embodiment of the distributed speech processing system of the present invention is schematically shown. Figure 3 As shown, each node device 300 may include a processor 302, a memory 304, a communication module 306, and a sound processing module 312. At least one of the multiple node devices includes a sound collection module 308. Optionally, the node device 300 may also include an output module 312, wherein the processor 302 may provide a μs-level accurate clock; the communication module 306 may be wired (e.g., network cable / power line carrier, etc.) or wireless (e.g., Wi-Fi / BLE / Zigbee, etc.) for networking communication with other devices; the memory 304 may record networking information and recognition model parameters; the output module 312 may be, for example, a speaker, a switch device, etc.; the sound collection module 308 may be, for example, a single microphone, multiple microphones, or a microphone array.
[0050] The sound collection module 308 can be configured to collect audio signals. The sound processing module 310 can be configured to pre-process the audio signal to obtain a locally generated sound pre-processing result. The communication module 306 can be configured to send the locally generated sound pre-processing result to one or more node devices in the network 110. The communication module 306 can also be configured to receive one or more sound pre-processing results from at least one other node device from the network 110. It should be understood that in the context of the present application, the locally generated sound pre-processing result can be referred to as a "first sound pre-processing result", and the sound pre-processing result received from other node devices through the network can be referred to as a "second sound pre-processing result". The sound processing module 310 can also be configured to perform speech recognition based on the locally generated sound pre-processing result and / or one or more sound pre-processing results received through the network 110. In this way, the node device 300 can obtain a locally generated speech recognition result.
[0051] The voice recognition performed by the sound processing module of the node device may include but is not limited to wake-up word detection, keyword recognition, continuous voice recognition, etc. As a non-limiting example, the voice recognition result obtained by the voice processing module of the node device performing voice recognition may include device identification, recognition result, effective time of the recognition result, recognition start time, and sound quality. The first voice recognition result may also include instruction information and device identification to instruct the target device to perform corresponding operations.
[0052] In one or more embodiments, the distributed speech recognition scheme of the present invention can utilize the sound preprocessing results generated locally and from the network on the one hand, and can also utilize the speech recognition results generated locally and from the network on the other hand.
[0053] In one embodiment of the present invention, the node device of the present invention can arbitrate the speech recognition results from different sources. As a non-limiting example, the communication module 306 can also be configured to send the locally generated speech recognition results to one or more node devices in the network. The communication module 306 is also configured to receive one or more speech recognition results from at least one other node device from the network. It should be understood that in the context of the present application, the locally generated speech recognition result can be referred to as the "first speech recognition result", and the speech recognition result received from other node devices through the network can be referred to as the "second speech recognition result". The sound processing module 310 is also configured to perform speech recognition based on the first speech recognition result and one or more second speech recognition results to obtain the final speech recognition result. In addition, in a specific embodiment, the sound processing module 310 is also configured to perform weighted processing on the first speech recognition result and one or more second speech recognition results to obtain the final speech recognition result.
[0054] For example, the sound processing module of the node device can be configured to assign weights based on the sound quality of the first voice recognition result and one or more second voice recognition results, and the higher the sound quality, the greater the weight assigned. For another example, the sound processing module of the node device can be configured to assign weights based on the source device of the first voice recognition result and one or more second voice recognition results, and if the source device is the node device, the greater the weight assigned.
[0055] In the context of the present invention, the sound preprocessing result is an intermediate result generated in the recognition process from the original voice to the voice recognition result. In a specific embodiment, each of the first sound preprocessing result and one or more second sound preprocessing results includes a sound feature value, a sound quality and a sound time information. In a specific embodiment, the communication module 306 of the node device receives one or more second sound preprocessing results from at least one other node device from the network, wherein the second sound preprocessing result includes a sound feature value, a sound quality and a sound time information, and may also include an increasing sequence number of an audio signal. Among them, the sound feature value in the preprocessing result is an MFCC feature value or a PLP feature value of an audio signal. The sound quality may include a signal-to-noise ratio and an amplitude of an audio signal. The sound time information may include the start time and the end time of an audio signal. The sound time information may include the start time and the duration of an audio signal. It should be understood by those skilled in the art that the implementation of the present invention is not limited thereto. On the contrary, those skilled in the art may adopt any suitable sound preprocessing result to implement the scheme of the present invention based on existing and future developed speech recognition and processing technologies.
[0056] Those skilled in the art should understand that the preprocessing technology applicable to the present invention may include but is not limited to signal framing, pre-emphasis, Fast Fourier Transform (FFT) and other preprocessing. The preprocessing method may obtain audio parameters according to the audio signal, generate frequency domain signals, or perform Mel-Frequency Cepstral Coefficients (MFCC) algorithm or Perceptual Linear Predictive (PLP) algorithm extraction to characterize the content of the voice information.
[0057] In a specific embodiment, the sound processing module 310 of the node device is further configured to determine whether the sound quality of each of the first sound preprocessing result and one or more second sound preprocessing results exceeds a predetermined threshold, and if not, discard the speech preprocessing result.
[0058] In a specific embodiment, the sound processing module 310 of the node device is further configured to select one or more sound preprocessing results with the highest sound quality from the first sound preprocessing result and one or more second sound preprocessing results for speech recognition to obtain a first speech recognition result.
[0059] As an example but not a limitation, the first speech recognition result obtained by the sound processing module of the node device performing speech recognition may include instruction information, wherein the instruction information is a specific numerical value, such as 011, and the instruction information can be understood and executed by the node device supporting the corresponding instruction. In addition, the first speech recognition result obtained by the sound processing module of the node device performing speech recognition may include instruction information, wherein different node devices support different ranges of instruction information.
[0060] In addition, the sound processing module of the node device can also be configured to select one or more sound preprocessing results with the highest sound quality from the first sound preprocessing result and one or more second sound preprocessing results for speech recognition to obtain a first speech recognition result.
[0061] In one embodiment, the sound processing module of the node device can determine whether the sound quality of the first sound preprocessing result exceeds a predetermined threshold. If so, the first sound preprocessing result is selected for speech recognition to obtain a first speech recognition result.
[0062] As an example embodiment, the communication module of the node device may send the first speech recognition result to one or more node devices in the network by unicast, multicast and / or broadcast.
[0063] In one embodiment, for a first voice recognition result in which the device identification in the first voice recognition result is inconsistent with the device identification of the node device, the first voice recognition result is sent to one or more node devices in the network through the communication module of the node device. On the contrary, for a first voice recognition result in which the device identification in the first voice recognition result is consistent with the device identification of the node device, the first voice recognition result is not sent to one or more node devices in the network.
[0064] In one embodiment, the sound processing module of the node device can judge the time validity of the final speech recognition result obtained. If the validity time of the recognition result is exceeded, the corresponding operation corresponding to the recognition result is not performed. In addition, the sound processing module of the node device can judge the device identification of the final speech recognition result obtained. If the device identification is the node device, the corresponding operation corresponding to the recognition result is performed.
[0065] As a non-limiting example, the sound processing module of the node device can determine the device identification of the final speech recognition result obtained. If the device identification is the node device, feedback information is output and sent to one or more other node devices in the network through the communication module.
[0066] In addition, the sound processing module of the node device can determine the device identification for the final speech recognition result obtained, and output feedback information if the device identification is the node device, wherein the output feedback information at least includes the recognition time, the recognition result and the maximum value of the incrementing sequence number.
[0067] Figure 4 A flowchart of an embodiment of a distributed speech processing method 400 according to the present invention is schematically shown. The distributed speech processing method is executed by a node device in a network. In step 402, determine whether the node device includes a sound collection module. If yes, go to step 404. If not, go to step 410. In step 404, collect an audio signal. In step 406, preprocess the audio signal to obtain a first sound preprocessing result. In step 408, send the first sound preprocessing result to one or more node devices in the network. In step 410, receive one or more second sound preprocessing results from at least one other node device from the network. In step 412, perform speech recognition based on the first sound preprocessing result and / or one or more second sound preprocessing results.
[0068] In one embodiment, it may be determined whether the sound quality of each of the first sound preprocessing result and the one or more second sound preprocessing results exceeds a predetermined threshold. If not, the sound preprocessing result is discarded.
[0069] In another embodiment, one or more sound preprocessing results with the highest sound quality may be selected from the first sound preprocessing result and one or more second sound preprocessing results to perform speech recognition to obtain the first speech recognition result.
[0070] As a non-limiting example, the method of the present invention can combine the local recognition result with the recognition result from the network to obtain the final speech recognition result. For example, the node device can send the first speech recognition result to one or more node devices in the network. The node device can receive one or more second speech recognition results from at least one other node device from the network. The node device can perform speech recognition based on the first speech recognition result and the one or more second speech recognition results to obtain the final speech recognition result.
[0071] In another specific embodiment, a weighted average may be performed on the first speech recognition result and one or more second speech recognition results to obtain a final speech recognition result.
[0072] In one or more embodiments, the solution of the present invention also uses fragments of sound preprocessing results generated locally and from the network to be spliced to obtain.
[0073] In one embodiment, a distributed speech processing system is provided, comprising: a plurality of node devices, the plurality of node devices forming a network, wherein each node device comprises a processor, a memory, a communication module and a sound processing module, and at least one of the plurality of node devices comprises a sound collection module; wherein the sound collection module is configured to collect an audio signal; the sound processing module is configured to pre-process the audio signal to obtain a first sound pre-processing result; wherein the communication module is further configured to receive one or more second sound pre-processing results from at least one other node device in the network; the first sound pre-processing result and the one or more second sound pre-processing results are received by the user; Each of the processing results includes one or more data blocks; each of the one or more data blocks includes time information, which identifies the time when the sound processing module completes the preprocessing of the data block; each of the one or more data blocks also includes an ascending sequence number, which is assigned according to the time information in the data block; the sound processing module is also configured to splice the data blocks of the first sound preprocessing result and / or one or more second sound preprocessing results in ascending order of the ascending sequence numbers to obtain a complete third sound preprocessing result; and the sound processing module is also configured to process the third sound preprocessing result to obtain a final speech recognition result.
[0074] In one embodiment, the communication module is configured to send the first sound preprocessing result to one or more node devices in the network.
[0075] In one embodiment, each data block in the first and / or second sound preprocessing results is configured to have the same duration.
[0076] In one embodiment, the incremental sequence number is assigned to the data block by the sound processing module of each node device in the plurality of node devices when pre-processing the audio signal.
[0077] In one embodiment, the incremental serial number is assigned by the sound processing module of each node device among the multiple node devices to the data block of the second sound preprocessing result after receiving the second sound preprocessing result from at least one other node device from the network.
[0078] In one embodiment, the sound processing module is configured to detect the time difference of the data blocks and assign the same increasing sequence number if the time difference is within a specified threshold.
[0079] In one embodiment, the sound processing module is configured to select data blocks with the best sound quality from data blocks with the same increasing sequence number for splicing.
[0080] Figure 5 A flowchart of another embodiment of a distributed speech processing method 500 according to the present invention is schematically shown. The method is executed by a node device in a network. In step 502, it is determined whether the node device includes a sound collection module. If yes, go to step 504. If not, go to step 508. In step 504, an audio signal is collected. In step 506, the audio signal is preprocessed to obtain a first sound preprocessing result. In step 508, one or more second sound preprocessing results from at least one other node device are received from the network, wherein each of the first sound preprocessing result and the one or more second sound preprocessing results includes one or more data blocks, wherein each of the one or more data blocks includes time information, the time information identifies the time when the sound processing module completes the data block preprocessing, and wherein each of the one or more data blocks also includes an increasing sequence number, which is assigned according to the time information in the data block. In step 510, the data blocks of the first sound preprocessing result and / or the one or more second sound preprocessing results are spliced in increasing order of the increasing sequence number to obtain a complete third sound preprocessing result. In step 512, the third sound preprocessing result is processed to obtain a final speech recognition result.
[0081] It should be understood that in the context of the present invention, the sound preprocessing result obtained by concatenating the data blocks of the first sound preprocessing result and / or one or more second sound preprocessing results is referred to as a "third sound preprocessing result".
[0082] In one embodiment, the distributed voice processing method further includes sending the first sound preprocessing result to one or more node devices in the network.
[0083] In one embodiment, each data block in the first sound preprocessing result is configured to have the same duration.
[0084] In one embodiment, the incremental sequence number is assigned to the data block when the sound processing module of each node device in the multiple node devices pre-processes the audio signal, wherein the incremental sequence number is assigned according to time information.
[0085] In one embodiment, the incremental serial number is assigned to the data block of the second sound preprocessing result by the sound processing module of each node device among multiple node devices after receiving the second sound preprocessing result from at least one other node device from the network, wherein the incremental serial number is assigned according to time information.
[0086] In one embodiment, the sound processing module is configured to detect the time difference of the data blocks and assign the same increasing sequence number if the time difference is within a threshold.
[0087] In one embodiment, the sound processing module is configured to select a data block with the best sound quality from the data blocks with the same increasing sequence number.
[0088] Figure 6 The flowchart of a specific embodiment of the distributed speech processing method 600 according to the present invention is schematically shown. In this embodiment, node devices are self-networked to establish a group, and each networked node device performs speech recognition and exchanges recognition information within the group, thereby improving the speech recognition system originally composed of one node device to a speech recognition system distributed on multiple node devices, thereby solving the problems of relying on a single control center, relying on a network server, not being able to cross regions, and insecurity of privacy information in speech recognition, keyword recognition and speech control scenarios.
[0089] In step 604, when the node device is powered on, it is found whether there is a group network. If there is no group network, a group network is created in step 606. If a group network already exists, the group network is joined in step 608. After accessing the group network, the node device first updates the function points of other devices in the network in step 610 to find out whether the function points supported by other devices in the network have been modified. At the same time or afterwards, the function points of its own device are broadcast in the group network in step 612. It should be understood that in the context of the present invention, "function points" are used to inform other node devices that access the group of what aspects of their input and output functions they have. It should be understood that in the context of the present invention, "group network" refers to a network composed of node devices that support broadcast and / or multicast, including but not limited to Wi-Fi, BLE and ZigBee networks with various topologies (such as Mesh topology), and can be wired, wireless or hybrid networks.
[0090] In step 614, the node device obtains the recognition result through distributed recognition, and determines in step 616 whether the device identification of the recognition result is the present device. If the device identification is not the present device, the recognition information is sent in step 622. As a non-limiting example, the recognition information may include the recognition device identification, the recognition time, the recognition result, and the credibility of the recognition result. If the device identification is the present device, the output is executed in step 618, and then the execution result information is sent to other node devices in the network in step 620. As a non-limiting example, the execution result information may include the device identification, the recognition time, the recognition result, the execution result, etc.
[0091] Figure 7 The flowchart of another specific embodiment of the distributed speech processing method 700 according to the present invention is schematically shown. Figure 7 As shown, the distributed speech processing method in this embodiment includes three inputs and one output. The three inputs are: the sound collected by the local microphone in step 702, the sound preprocessing information collected from the network in step 708, and the speech recognition information collected from the network in step 714; one output is the speech recognition result output in step 720.
[0092] In this embodiment, the distributed speech processing method 700 is divided into three stages: a pre-processing stage, an analysis and decision stage, and a recognition and arbitration stage.
[0093] In the preprocessing stage, the sound collected by the local microphone will first undergo sound preprocessing in step 704 to obtain preprocessing information, and then the preprocessing information will be sent to the group network in step 706. The preprocessing information, for example, includes feature information of the collected sound that can be used for recognition model recognition. The preprocessing information, for example, also includes information such as the signal-to-noise ratio and amplitude of the collected sound that can be used to evaluate the sound quality. The preprocessing information, for example, also includes an increasing sequence number of the information. As an example and not a limitation, the preprocessing information may also include start time information and end time information.
[0094] In the analysis and decision stage of step 710, the sound quality of the sound preprocessing information collected from the network and the sound preprocessing information obtained locally is sorted, and the best quality preprocessing information is analyzed and selected to be sent to the subsequent speech recognition step 712. In step 712, speech recognition is performed to output local recognition information. The local or network recognition information may include, but is not limited to, the recognition result, the device identification of the speech recognition, the effective time of the recognition result, the recognition start time, and the translator or multiple of the sound quality.
[0095] In the recognition arbitration stage, the recognition information collected from the network in step 714 is first analyzed and judged in step 716, and the expired information is removed according to the timeliness of the recognition information. Then, recognition arbitration is performed in step 718 together with the output of the local speech recognition in step 712. The recognition arbitration in step 718 sorts the sound quality carried by the network speech recognition results and the local speech recognition results to select the better speech recognition results to generate the final speech recognition results. For example, a specified number of speech recognition results with higher sound quality can be selected and weighted to obtain the final recognition result.
[0096] The following uses scenario examples to further illustrate the principle of the present invention. In the first scenario, refer to Figure 2 As shown, device A, device B, and device C are powered on in sequence. After device A is powered on first, it is found that there is no group network, so a group network is created. After devices B and C are powered on, they find that the group network already exists, so they join the group network. After devices B and C access the group network, they first update whether the function points of other devices in the network (i.e., device A) have been modified, and at the same time, the function points of the devices (i.e., devices B and C) are broadcasted in the group to inform other node devices that access the group of their own input and output functions.
[0097] The user is between device B and device C and sends a voice signal. Device B and device C collect the audio signal sent by the user, but device A cannot collect the audio signal sent by the user because the distance between the device A and the user is far, which exceeds the sound collection distance of the sound collection module of device A.
[0098] Device B and device C preprocess the received audio signal, and the obtained preprocessing result at least contains the sound feature information of the collected audio signal that can be applied to the speech recognition model. The preprocessing result also contains information such as the signal-to-noise ratio and amplitude of the collected sound that can be used to evaluate the sound quality. The preprocessing result also contains an increasing serial number of the audio signal. Taking device B as an example, the preprocessed data it sends includes N blocks of data, all of which have increasing serial numbers. The preprocessing result also contains start time information and end time information, and the start time information is used to distinguish different sound information.
[0099] Device B and device C preprocess the collected voice, obtain relevant preprocessing information and send it to the network. The communication module of device A also receives the sound preprocessing results from device B and device C from the network. The communication module of device B receives the sound preprocessing results from device C from the network. The communication module of device C receives the sound preprocessing results from device B from the network.
[0100] The signal quality of the first sound preprocessing result obtained by device B based on its own sound collection module and sound preprocessing module has exceeded the predetermined threshold. However, the quality of the collected audio signal in the second sound preprocessing result obtained from device C and received by device B through the network is better. Based on the sound quality of the first and second sound preprocessing results, device B selects the sound preprocessing result with the highest sound quality (i.e., the second sound preprocessing result of device C here) for subsequent speech recognition.
[0101] exist Figure 2 In the scenario shown, for example, in another situation, the signal quality of the first sound preprocessing result obtained by device B based on its own sound collection module and sound preprocessing module has exceeded a predetermined threshold. Even if the quality of the collected audio signal in the second sound preprocessing result obtained from device C is better, device B will still use the preprocessed signal obtained by its own sound collection module and sound preprocessing module for subsequent speech recognition.
[0102] exist Figure 2 In the scenario shown, for example, in another situation, it is assumed that device A is a TV, device B is an air conditioner, and device C is a desk lamp. Devices A, B, and C can support some common command information. For example, these three devices jointly support general command information with a command value range of 000-099, such as the wake-up command "Hello A". In addition, these three devices also support different types of commands. For example, device A supports the command information "Increase TV volume (111)", while devices B and C do not support this command.
[0103] exist Figure 2In the scenario shown, for example, in another situation, device A does not collect audio signals through the sound collection module of device A because the distance from the user exceeds the sound collection distance. However, device A receives the second sound preprocessing results from device B and device C through the network. Device A sorts the sound quality in the second sound preprocessing results and selects the second sound preprocessing result with the highest sound quality for subsequent speech recognition.
[0104] exist Figure 2 In the scenario shown, for example, in one situation, device B performs voice recognition to obtain a first voice recognition result of "Hello A", and determines that the device identifier in the first voice recognition result is device A, so the first voice recognition result is forwarded to other devices in the network (ie, device A and device C).
[0105] exist Figure 2 In the scenario shown, for example, in another situation, device A performs voice recognition to obtain a first voice recognition result of "Hello A", and determines that the device identifier in the first voice recognition result is device A, so the first voice recognition result is no longer forwarded to other devices in the network.
[0106] exist Figure 2 In the scenario shown, for example, device A obtains a first speech recognition result ("Hello A") based on its own sound processing module. Device A receives a second speech recognition result ("Hello C") from device B via the network. Device A receives another second speech recognition result ("Hello A") from device C via the network. Weighted processing is performed on these three speech recognition results, where the weight allocation considers two factors, namely, the sound quality in the recognition result and the source device of the recognition result. The higher the sound quality, the greater the weight allocated to the corresponding speech recognition result. If the source device of the recognition result is this device, the greater the weight allocated. For example, device A in this example, according to the sound quality, assigns different weight values (B: 0.6, C: 0.4) to the second speech recognition results received from device B and device C, and assigns a higher weight value (A: 0.8) to the first speech recognition result from device A itself. Therefore, the final weighted results for these three speech recognition results are "Hello A": 1.2, "Hello C": 0.6. And thus the final speech recognition result is "Hello A".
[0107] exist Figure 2In the scenario shown, for example, device A determines the time validity of the voice recognition result based on the final voice recognition result ("Hello A"). If it is found that it is still within the valid time range, it is further determined whether the device identifier is this device, and it is found that the device identifier in the voice recognition result is "A", that is, this device, so the operation corresponding to the instruction information is performed. At the same time, device A sends feedback information to other devices in the network (that is, device B and device C). The feedback information includes at least the recognition time, recognition result and the maximum value of the incrementing sequence number. Device B and device C receive the feedback information and learn that the recognition result has been executed, so device B and device C stop the voice recognition work and sending work of their own devices respectively.
[0108] like Figure 2 As shown, when the user sends an audio signal "Hello A" between device B and device C, device B and device C will collect the user's audio signal through their respective sound collection modules. In addition, device B and device C perform preprocessing through their respective sound processing modules to obtain a sound preprocessing result. Because its distance from the user exceeds the sound pickup distance of the sound collection module, device A and other devices in the network cannot collect sound through the sound collection module of the device itself. Among them, the sound preprocessing result includes sound feature values, sound quality information, and sound time information. Among them, the sound feature value is the MFCC feature value or PLP feature value of the audio signal, that is, the sound feature value obtained by the MFCC algorithm, which is used to characterize the content of the voice information. The sound quality information includes the signal-to-noise ratio and amplitude of the audio signal. The sound time information includes the start time and end time of the audio signal, or the start time and duration of the audio signal. The sound preprocessing result also includes an increasing sequence number. Taking device B as an example, the first sound preprocessing data sent by it may include N blocks of data, each of which carries an increasing sequence number. The sound time information may include start time information and / or end time information of the audio signal to distinguish different sound information.
[0109] like Figure 2As shown, device B and device C send the obtained sound preprocessing results to all devices in the network. Device A receives the sound preprocessing results sent by device B and device C respectively through the network. Device B receives the sound preprocessing results sent by device C through the network. Device C receives the sound preprocessing results sent by device B through the network. Device A prioritizes the sound preprocessing information sent by device B and the sound preprocessing information sent by device C received from the network to select the sound preprocessing result for subsequent speech recognition. Device B prioritizes the sound preprocessing information obtained locally and the sound preprocessing information sent from device C received from the network to select the sound preprocessing result for subsequent speech recognition. Device C prioritizes the sound preprocessing information obtained locally and the sound preprocessing information sent from device B received from the network to select the sound preprocessing result for subsequent speech recognition.
[0110] In another embodiment, the solution of the present invention can be applied to a continuous speech recognition scenario. In this scenario, also refer to Figure 2 , suppose the user walks from device B to device C while issuing a voice command "turn on the kitchen light". In this embodiment, device A is the kitchen light, the voice collected by device B is "turn on the kitchen", and the voice collected by device C is "room light".
[0111] Devices B and C preprocess the voice information, and B obtains the feature information corresponding to the preprocessed information "open the kitchen" and the preprocessed information described above; device C obtains the feature information corresponding to the preprocessed information "room light" and the preprocessed information described above. B and C both send their preprocessed information to group 1.
[0112] As described above, the preprocessed data may include N blocks, each of which has an increasing serial number. The duration of each block of data may be, for example, 30 ms, and each block of data is assigned an increasing serial number. The increasing serial number is related to the time point when the device completes the preprocessing. It is generally believed that if the user issues a voice command indoors, the time point when each device (A, B, C) in the network completes the preprocessing is close. For different devices, the serial numbers of the data blocks that complete the preprocessing at the same / similar time point (actually there may be a time difference of about 10 ms) are the same.
[0113] In actual applications, after processing each block of data, the device will send it to the group. After receiving each data block, the device will select the best data block from the data blocks with the same sequence number (the method is similar to preprocessing sorting), and splice the data blocks with different sequence numbers to form a complete preprocessing result.
[0114] For example, the voice command "Hello A" can be divided into 10 blocks of data (serial numbers 000-009) after voice preprocessing. Device A will select a data block from the multiple data blocks received with serial number 000; select a data block from the multiple data blocks received with serial number 000; ...; and splice the multiple data blocks in the order of serial numbers to form the final preprocessing result (first preprocessing result).
[0115] This can solve the problem of personnel movement. If the user moves from device A to device B while issuing a voice command, each device can only collect part of the voice information. For example, device A can collect data blocks with serial numbers 000-006, and device B can collect data blocks with serial numbers 003-009. Through the above method, both devices A and B can receive data blocks from 000-009, thereby completing the splicing of the voice data (preprocessing results).
[0116] It should be understood that the above distributed speech recognition system and method are provided only as examples, rather than limitations of the present invention. Those skilled in the art should understand that the principles of the present invention can be applied to distributed speech recognition systems and methods different from those described above without departing from the scope of the present invention. Although various embodiments of various aspects of the present application have been described for the purpose of the present disclosure, it should not be understood that the teachings of the present disclosure are limited to these embodiments. The features disclosed in a specific embodiment are not limited to the embodiment, but can be combined with the features disclosed in different embodiments. For example, one or more features and / or operations of the method according to the present application described in one embodiment may also be applied individually, in combination or as a whole in another embodiment. The description of the system / device embodiment is also applicable to the method embodiment, and vice versa. Those skilled in the art should understand that there are also possible more optional implementations and variations, and various changes and modifications can be made to the above system without departing from the scope defined by the claims of the present application.
Claims
1. A distributed speech processing system, characterized in that: include: A plurality of node devices, wherein the plurality of node devices form a network, wherein each node device comprises a processor, a memory, a communication module and a sound processing module, and at least one node device among the plurality of node devices comprises a sound collection module; Wherein the sound collection module is configured to collect audio signals; The sound processing module is configured to preprocess the audio signal to obtain a first sound preprocessing result, wherein the preprocessing includes extracting a sound feature value of the audio signal; wherein, The communication module is further configured to receive one or more second sound preprocessing results from at least one other node device from the network; The sound processing module is further configured to perform speech recognition based on the first sound preprocessing result and / or the one or more second sound preprocessing results to obtain a first speech recognition result.
2. The distributed speech processing system according to claim 1, characterized in that: The communication module is also configured to send the first sound preprocessing result to one or more node devices in the network.
3. The distributed speech processing system according to claim 1, characterized in that: The communication module is also configured to send the first speech recognition result to one or more node devices in the network.
4. The distributed speech processing system according to claim 1, characterized in that: The communication module is further configured to receive one or more second speech recognition results from at least one other node device from the network; and The sound processing module is further configured to perform speech recognition based on the first speech recognition result and the one or more second speech recognition results to obtain a final speech recognition result.
5. The distributed speech processing system according to claim 1, characterized in that: Each of the first sound preprocessing result and the one or more second sound preprocessing results includes a sound feature value, a sound quality, and sound time information.
6. A distributed speech processing method, executed by a node device in a network, characterized in that: include: If the node device includes a sound collection module, perform the following steps: Collect audio signals; and Preprocessing the audio signal to obtain a first sound preprocessing result, wherein the preprocessing includes extracting a sound feature value of the audio signal; receiving one or more second sound pre-processing results from at least one other node device from the network; Speech recognition is performed based on the first sound preprocessing result and / or the one or more second sound preprocessing results to obtain a first speech recognition result.
7. The distributed voice processing method according to claim 6 further includes sending the first sound preprocessing result to one or more node devices in the network.
8. The distributed speech processing method according to claim 6 further comprises sending the first speech recognition result to one or more node devices in the network.
9. The distributed speech processing method according to claim 6, further comprising receiving one or more second speech recognition results from at least one other node device from the network; and Speech recognition is performed based on the first speech recognition result and the one or more second speech recognition results to obtain a final speech recognition result.
10. The distributed speech processing method according to claim 6, characterized in that: Each of the first sound preprocessing result and the one or more second sound preprocessing results includes a sound feature value, a sound quality, and sound time information.
