Audio data processing system and audio data processing method
By using a first device and a server to collaboratively process audio data and leveraging near-field and long-field communication protocols, efficient audio data processing is achieved, solving the balance between device cost, power consumption, and audio quality, and improving the efficiency and quality of audio processing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI QIANWEN ZHILIAN ARTIFICIAL INTELLIGENCE TECHNOLOGY CO LTD
- Filing Date
- 2026-04-07
- Publication Date
- 2026-07-14
Smart Images
Figure CN122392548A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of communication technology, and more specifically, to an audio data processing system and an audio data processing method. Background Technology
[0002] In daily life, users can use devices, such as wearable devices, to collect audio data. To improve audio quality, some devices process the raw audio data. However, audio data processing places high demands on the device's hardware performance. If the processing is complex, it increases the device's manufacturing cost and power consumption, and may also reduce its portability. Conversely, if the processing is simple, the device cannot perform refined audio processing, resulting in generally lower audio quality. Therefore, accurately balancing manufacturing costs, power consumption, and portability while ensuring audio quality has become a pressing issue. Summary of the Invention
[0003] In view of this, embodiments of the present invention provide an audio data processing system and an audio data processing method, which realize the audio data processing process through the collaboration of a first device and a server, while forwarding encoded frame packets through a second device, thereby ensuring audio quality while controlling the manufacturing cost, power consumption and portability of the first device.
[0004] In a first aspect, embodiments of the present invention provide an audio data processing system, the system comprising: A first device is configured to acquire first audio data, preprocess the first audio data, acquire second audio data, encode the second audio data, acquire multiple corresponding audio encoded frames, determine the encoded frame packet corresponding to each audio encoded frame, and send each encoded frame packet to a second device via a near-field communication protocol. The first audio data is multi-channel audio stream data. The second device is configured to receive each of the encoded frame packets via a short-range communication protocol and to send the encoded frame packets to a server via a remote communication protocol. The server is configured to receive the encoded frame packets via a remote communication protocol, decode the audio encoded frames in each encoded frame packet to obtain third audio data, perform post-processing on the third audio data, and obtain the corresponding post-processing result.
[0005] Optionally, the first device is further configured to: The second audio data is encoded according to the encoding parameters to obtain a plurality of corresponding audio encoded frames. The encoding parameters include encoding format, encoding interval, number of channels and bit rate.
[0006] Optionally, the encoding parameters further include signal grouping relationships, which are used to characterize the grouping relationships of audio signals in each channel of the second audio data; The first device is further configured as follows: Multiple audio signal groups are determined based on the correlation of the audio signals, and each audio signal group is encoded to obtain the audio encoded frame corresponding to each audio signal combination. A encoded frame packet is determined based on each audio encoded frame within the same encoding period.
[0007] Optionally, the first device is further configured to: The first audio data is subjected to echo cancellation processing based on the reference audio signal to obtain the second audio data.
[0008] Optionally, the first device is further configured to: The header of the encoded frame packet is determined, the header including the frame length of the audio encoded frame and configuration parameters, the configuration parameters being used to characterize the encoded frame packet as a custom data packet, and the encoded frame packet is generated based on the header and the audio encoded frame.
[0009] Optionally, the server is further configured to: The audio encoded frames are decoded according to the decoding parameters to obtain the third audio data. The decoding parameters include the encoding format, encoding interval, number of channels, and bit rate.
[0010] Optionally, the server is further configured to: The third audio data is subjected to speech enhancement processing to obtain target audio data. The speech enhancement processing includes at least one of beamforming, voice separation and noise reduction.
[0011] Optionally, the server is further configured to: The target audio data is subjected to speech recognition to obtain the corresponding speech recognition text, and the speech recognition text is translated into the target text.
[0012] Optionally, the server is further configured to: The post-processing results are sent to the second device via a remote communication protocol; The second device is also configured to: In response to receiving the post-processing result, the post-processing result is output.
[0013] Secondly, embodiments of the present invention provide an audio data processing method, the method comprising: Collect first audio data, which is multi-channel audio stream data; The first audio data is preprocessed to obtain the second audio data; The second audio data is encoded to obtain multiple corresponding audio encoded frames; Determine the encoded frame packet corresponding to each of the aforementioned audio encoded frames; Each of the coded frame packets is sent to the second device via a short-range communication protocol; Each of the coded frame packets is received via a short-range communication protocol; The encoded frame packet is sent to the server via a remote communication protocol; The encoded frame packet is received via a remote communication protocol; Decode the audio encoded frames in each encoded frame packet to obtain third audio data; The third audio data is then post-processed to obtain the corresponding post-processing results.
[0014] Thirdly, embodiments of the present invention provide an audio data processing apparatus, the apparatus comprising: An audio acquisition unit is used to acquire first audio data, wherein the first audio data is multi-channel audio stream data; The preprocessing unit is used to preprocess the first audio data to obtain the second audio data; The encoding unit is used to encode the second audio data and obtain multiple corresponding audio encoded frames. A frame packet determination unit is used to determine the encoded frame packet corresponding to each of the audio encoded frames; The first frame packet sending unit is used to send each of the coded frame packets to the second device via a short-range communication protocol. The first frame packet receiving unit is used to receive each of the coded frame packets through a short-range communication protocol; The second frame packet sending unit is used to send the encoded frame packet to the server via a remote communication protocol; The second frame packet receiving unit is used to receive the encoded frame packet via a remote communication protocol; The decoding unit is used to decode the audio encoded frames in each encoded frame packet to obtain third audio data; An audio processing unit is used to perform post-processing on the third audio data and obtain the corresponding post-processing results.
[0015] Fourthly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in the second aspect.
[0016] Fifthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the second aspect.
[0017] In a sixth aspect, embodiments of the present invention provide a computer program product, the computer program product including a computer program / instruction, which, when executed by a processor, implements the method described in the second aspect.
[0018] The audio data processing system of this invention includes a first device, a second device, and a server. After acquiring audio data, the first device preprocesses and encodes the acquired audio data to obtain multiple audio encoded frames and generates encoded frame packets for each audio encoded frame. These encoded frame packets are then sequentially sent to the second device via a short-range communication protocol. The second device sequentially forwards the encoded frame packets sent by the first device to the server via a long-range communication protocol. The server decodes and performs post-processing on each encoded frame packet. This invention achieves audio data processing through collaboration between the first device and the server, while using the second device as a forwarding device for encoded frame packets. Therefore, it can ensure audio quality while controlling the manufacturing cost, power consumption, and portability of the first device. Attached Figure Description
[0019] The above and other objects, features and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which: Figure 1 This is a system block diagram of the audio data processing system according to an embodiment of the present invention; Figure 2 This is a flowchart of an audio data processing method according to an embodiment of the present invention; Figure 3 This is a flowchart of an audio data processing method according to an embodiment of the present invention; Figure 4 This is a data flow diagram of the audio data processing method according to an embodiment of the present invention; Figure 5 This is a flowchart of the audio data processing method of the present invention on the first device side according to an embodiment of the present invention; Figure 6 This is a flowchart of the audio data processing method of the present invention on the second device side according to an embodiment of the present invention; Figure 7 This is a flowchart of the audio data processing method on the server side according to an embodiment of the present invention; Figure 8 This is a schematic diagram of an audio data processing device according to an embodiment of the present invention; Figure 9 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation
[0020] The present application is described below based on embodiments, but it is not limited to these embodiments. In the detailed description of the present application below, certain specific details are described in detail. Those skilled in the art can fully understand the present application without these details. To avoid obscuring the substance of the present application, well-known methods, processes, flows, elements, and electrical channels are not described in detail.
[0021] Furthermore, those skilled in the art should understand that the accompanying drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.
[0022] Unless the context explicitly requires it, words such as "including" or "contains" throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, meaning "including but not limited to".
[0023] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.
[0024] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.
[0025] During audio acquisition, the device may pick up ambient noise and echoes generated by reflections of its own audio output while simultaneously capturing the target speech. If the recorded audio data includes ambient noise and echoes, noise reduction processing is typically required to minimize the ambient noise, and preprocessing is necessary to eliminate echoes as much as possible. However, if the audio data processing is complex, it increases the device's manufacturing cost and power consumption, reducing battery life, and also increases computational overhead, thus reducing audio processing efficiency. Conversely, if the audio data processing is too simple, it cannot provide refined audio processing, making it difficult to guarantee recording quality.
[0026] To address the aforementioned problems, this invention provides an audio data processing system and an audio data processing method. Figure 1 This is a system block diagram of the audio data processing system according to an embodiment of the present invention. Figure 1 As shown, the audio data processing system of this embodiment includes a first device 11, a second device 12, and a server 13.
[0027] The first device 11 and the second device 12 can establish a communication connection through short-range communication methods such as Bluetooth Low Energy (BLE), while the second device 12 and the server 13 can establish a communication connection through long-range communication methods such as a network, thereby realizing the exchange of information and data. It should be understood that, although... Figure 1 Only a certain number of first devices 11, second devices 12 and servers 13 are shown, but this does not mean that the number of each is limited. This system may include multiple first devices 11, second devices 12 and servers 13.
[0028] The first device 11 is a communication terminal capable of running computer programs. These communication terminals can be wearable devices, speakers, voice recorders, or other terminal devices. The first device 11 has multiple microphones, which can be arranged in an array for collecting audio data. In some embodiments, the microphones can be analog microphones, such as dynamic microphones, electret microphones, or micro-electro-mechanical system (MEMS) analog microphones; in some embodiments, the microphones can be digital microphones, such as MEMS digital microphones. The first device 11 has at least one short-range communication module, such as a communication channel for Bluetooth or Zigbee short-range communication.
[0029] The second device 12 is a communication terminal capable of running computer programs. These communication terminals can be mobile phones, tablets, handheld computers, vehicle-mounted terminals, or other similar devices. The second device 12 has a communication module, which may include at least one remote communication module, such as a communication channel for WLAN, GPRS, or 2G / 3G / 4G / 5G remote communication, or at least one short-range communication module, such as a communication channel for Bluetooth or Zigbee short-range communication.
[0030] Server 13 should be understood as a device that provides data processing, database, and communication facilities. For example, server 13 may refer to a single physical server with associated communication, data storage, and database facilities, or it may refer to a networked or aggregated collection of processors, associated networks, and storage devices that operate on software and one or more database systems and application software that supports the services provided by the server. Server 13 may be a monolithic server or a distributed server spanning multiple computers or computer data centers, or it may be various types of cloud servers. In some embodiments, each server may include hardware, software, or embedded logic components for performing suitable functions supported or implemented by the server, or a combination of two or more such components.
[0031] In this embodiment of the invention, after acquiring first audio data, the first device 11 preprocesses the first audio data to obtain second audio data, and encodes the second audio data to obtain multiple corresponding audio encoded frames. It then determines the encoded frame packet corresponding to each audio encoded frame and sends each encoded frame packet to the second device via a short-range communication protocol. After receiving each encoded frame packet via the short-range communication protocol, the second device 12 sends the encoded frame packet to the server via a remote communication protocol. After receiving each encoded frame packet via the remote communication protocol, the server 13 decodes the audio encoded frames in each encoded frame packet to obtain third audio data, and then performs post-processing on the third audio data to obtain the corresponding post-processing result. The first audio data is multi-channel audio stream data.
[0032] In an optional implementation of this invention, the first device 11 is further configured to encode the second audio data according to encoding parameters to obtain a plurality of corresponding audio encoded frames. The encoding parameters include encoding format, encoding interval, number of channels, and bitrate.
[0033] In an optional implementation of this invention, the encoding parameters further include signal grouping relationships, which characterize the grouping relationships of audio signals in each channel of the second audio data. The first device 11 is further configured to determine multiple audio signal groups based on the correlation of each audio signal, encode each audio signal group separately, obtain audio encoded frames corresponding to each audio signal combination, and determine an encoded frame packet based on each audio encoded frame within the same encoding period.
[0034] In an optional implementation of this invention, the first device 11 is further configured to perform echo cancellation processing on the first audio data based on a reference audio signal to obtain the second audio data.
[0035] In an optional implementation of this invention, the first device 11 is further configured to determine the header of the encoded frame packet, the header of the encoded frame packet including the frame length of the audio encoded frame and configuration parameters, the configuration parameters being used to characterize the encoded frame packet as a custom data packet, and to generate the encoded frame packet based on the header and the audio encoded frame.
[0036] In an optional implementation of this invention, server 13 is further configured to decode each audio encoded frame according to decoding parameters to obtain third audio data. The decoding parameters include encoding format, encoding interval, number of channels, and bit rate.
[0037] In an optional implementation of this invention, server 13 is further configured to perform speech enhancement processing on third audio data to obtain target audio data. The speech enhancement processing includes at least one of beamforming, voice separation, and noise reduction.
[0038] In an optional implementation of this invention, server 13 is further configured to perform speech recognition on target audio data, obtain corresponding speech recognition text, and translate the speech recognition text into target text.
[0039] In an optional implementation of this invention, the server 13 is further configured to send post-processing results to the second device 12 via a remote communication protocol; the second device 12 is further configured to receive post-processing results and output post-processing results.
[0040] The embodiments of the present invention implement the audio data processing process through the collaboration of a first device and a server, while using a second device as a forwarding device for encoded frame packets. Therefore, audio quality can be guaranteed while controlling the manufacturing cost, power consumption and portability of the first device.
[0041] The following description uses method embodiments. In this embodiment of the invention, a head-mounted device (i.e., a wearable device) is used as an example for illustration. Figure 2 This is a flowchart of an audio data processing method according to an embodiment of the present invention. Figure 2 As shown, the method in this embodiment includes the following steps: Step S100: Collect the first audio data.
[0042] In this embodiment, the control module of the head-mounted device can control each microphone to simultaneously acquire audio signals. Each microphone is an audio channel, so the control unit can determine the first audio data based on the multi-channel audio stream data acquired by each microphone.
[0043] Optionally, if the microphone is a digital microphone, the digital microphone has a built-in analog-to-digital converter (ADC) that can directly output multi-channel audio stream data with a digital signal type. Therefore, the control module can directly use the multi-channel audio stream data as the first audio data. If the microphone is an analog microphone, the multi-channel audio stream data output by the analog microphone has an analog signal type. Therefore, the control module can use an analog-to-digital converter to convert the signal type of the multi-channel audio stream data from an analog signal to a digital signal, and then determine the multi-channel audio stream data of the digital signal type as the first audio data.
[0044] Optionally, to reduce the power consumption and improve the battery life of the head-mounted device, the control module can control the microphone array to collect audio stream data after receiving an audio acquisition command. In one optional implementation, if the head-mounted device includes a voice interaction component, the wearer can issue an audio acquisition command through the voice interaction component. The head-mounted device may include at least one of a voice interaction component and a command trigger button. The voice interaction component may include a pickup unit, a voice recognition unit, and an intent recognition unit. The pickup unit can be at least one microphone in the microphone array, used to collect voice information issued by the wearer; the voice recognition unit is used to perform voice recognition on the voice information collected by the pickup unit to determine the text content of the voice information; the intent recognition unit is used to perform intent recognition on the text content of the voice information to determine the intent information of the voice information. Furthermore, the voice interaction component can send the intent information to the control module. The control module can pre-store the mapping relationship between each intent information and the command. Therefore, in this case, after receiving the intent information of the voice information, the control module can determine whether the intent information is used to represent audio acquisition. If so, the control module can determine that the wearer has issued an audio acquisition command.
[0045] In one optional implementation, if the head-mounted device includes a command trigger button, the wearer can issue an audio acquisition command via the command trigger button. The command trigger button can be a mechanical switch or a pressure sensing unit; this embodiment is not limited in this regard. The command trigger button can convert the wearer's trigger operation (such as a pressing operation) into a trigger signal and send it to the control module. The control module can pre-store the mapping relationship between each trigger signal and command. Therefore, in this case, after receiving a trigger signal, the control module can determine whether the trigger signal corresponds to an audio acquisition command. If so, the control module can determine that the wearer has issued an audio acquisition command.
[0046] Depending on the hardware structure of the head-mounted device, the control module can also determine whether an audio acquisition command has been received in other ways. For example, if the head-mounted device has a touchpad, such as the temples being configured as a touchpad, the wearer can perform specific gesture operations using the touchpad. If the control module recognizes a specific gesture operation used to trigger magnetic calibration, it can determine that the wearer has issued an audio acquisition command.
[0047] Step S200: Preprocess the first audio data to obtain the second audio data.
[0048] In this embodiment, to reduce the computational and power consumption overhead of the head-mounted device, the device only needs to preprocess the first audio data. In practical applications, head-mounted devices typically include audio playback components (such as speakers). In some scenarios, such as video conferencing and live streaming, the sound played by the audio playback component may be picked up by a microphone array, converted into an audio signal, and sent back to the audio playback component, creating an echo. If the head-mounted device is recording, the echo will be recorded simultaneously, significantly reducing audio quality. Therefore, in some embodiments, the processing of the first audio data may include echo cancellation processing, that is, the head-mounted device can perform echo cancellation processing on the first audio data to obtain the second audio data.
[0049] Optionally, the head-mounted device can perform echo cancellation processing on the initial audio stream data based on various existing methods. For example, the head-mounted device can acquire the sound played by the audio playback component as a reference audio signal, and generate a cancellation signal based on the reference audio signal and the first audio data using an adaptive filter, and then superimpose the cancellation signal onto the first audio data to achieve echo cancellation of the first audio data.
[0050] In some embodiments, the control module may also preprocess the first audio data in other ways. For example, the control module may perform DC offset removal on the audio stream data of each channel in the first audio data to avoid signal distortion and improve the signal-to-noise ratio; or it may perform at least one of high-pass filtering, low-pass filtering and band-pass filtering on the first audio data to remove useless frequency bands in the first audio data that do not belong to human voice.
[0051] Step S300: Encode the second audio data to obtain multiple corresponding audio encoded frames.
[0052] In this step, the control module can determine the encoding parameters of the second audio data. These encoding parameters may include the encoding format, encoding interval, number of channels, and bitrate.
[0053] The encoding format can be Pulse Code Modulation (PCM), Opus (Opus Interactive Audio Codec), Advanced Audio Coding, etc. Optionally, to reduce data transmission latency and improve the real-time nature of audio data post-processing, this embodiment can use Opus as the encoding format.
[0054] The encoding interval is used to characterize the duration of each original audio frame. For example, an encoding interval of 20 milliseconds indicates that the second audio data will be segmented into multiple original audio frames, each with a duration of 20 milliseconds, and then each original audio frame will be encoded into a single audio encoded frame. To avoid excessive consumption of the head-mounted device's storage resources (such as memory) and computing resources (such as CPU) during the audio encoding process, and to avoid excessive consumption of the head-mounted device's data transmission resources by the audio encoded frame, the encoding interval can be determined based on the data transmission efficiency of the short-range communication protocol, the utilization rate of the head-mounted device's storage resources, and the utilization rate of its computing resources (or the encoding time per frame). For example, an encoding interval that simultaneously satisfies the following conditions can be used as the encoding interval for the second audio data: the data transmission efficiency of the short-range communication protocol is not lower than a first threshold, the utilization rate of the head-mounted device's storage resources is not higher than a second threshold, and the utilization rate of its computing resources is not higher than a third threshold.
[0055] The number of channels can be used to characterize the number of microphones, and under normal circumstances, the number of channels does not change.
[0056] The bitrate determines the compression ratio of the second audio data. A lower bitrate results in a higher compression ratio but also higher signal loss, while a higher bitrate results in a lower compression ratio and less signal loss. To balance the transmission rate of the audio encoded frames and the quality of the encoded signal, the bitrate of the second audio data can be determined based on the signal loss rate and the data transmission bandwidth of the head-mounted device. For example, a mapping relationship between the signal loss rate and the bitrate can be established, and the bitrate of the second audio data can be determined based on this mapping relationship, provided that the signal loss rate is not higher than a fourth threshold and the corresponding transmission bandwidth is not higher than the data transmission bandwidth.
[0057] After determining the encoding parameters, the control module can encode the second audio data according to the encoding parameters to obtain multiple audio encoded frames. The Opus encoding framework mainly supports encoding and decoding of single-channel audio stream data and dual-channel audio stream data. Therefore, if the number of channels in the second audio data is higher than two, the control module can also combine the audio signals in the second audio data and then encode them through the Opus encoding framework.
[0058] Figure 3This is a flowchart of an audio data processing method according to an embodiment of the present invention. Figure 3 As shown, in an optional implementation of this embodiment, step S300 may include the following steps: Step S310: Determine multiple audio signal groups based on the correlation of each audio signal.
[0059] In this step, the control module can group the audio signals based on their correlation. The correlation between the audio signals can be represented by cross-correlation coefficients, such as the Pearson correlation coefficient. For any two audio signals, a higher cross-correlation coefficient indicates a higher correlation between the two audio signals, and a lower cross-correlation coefficient indicates a lower correlation between the two audio signals. Therefore, the control module can group two audio signals with cross-correlation coefficients higher than a preset correlation threshold into one audio signal group. For example, if the number of channels in the second audio data is 5, the audio signals of each audio channel can be divided into three audio signal groups, where two audio signal groups contain two audio signals each, and the remaining audio signal group contains one audio signal.
[0060] In some embodiments, when the correlation between audio signals is represented by the Pearson correlation coefficient, the cross-correlation coefficient between audio signal x and audio signal y is... This can be expressed by the following formula:
[0061] in, Represents each sampling point in the audio signal x The mean, Represents each sampling point in the audio signal y The mean of the sampling points is N, where N represents the total number of sampling points.
[0062] Step S320: Encode each audio signal group separately to obtain the audio encoded frame corresponding to each audio signal combination.
[0063] After determining multiple audio signal groups, the control module can encode each audio signal group according to the encoding parameters to obtain the audio encoded frame corresponding to each audio signal combination.
[0064] Step S400: Determine the encoding frame packet corresponding to each audio encoding frame.
[0065] In this step, to avoid excessive consumption of data transmission resources of the head-mounted device by the audio encoded frame, the control unit does not encapsulate the audio encoded frame after determining the packet header. Instead, it generates an encoded frame packet based on the packet header and the audio encoded frame, so that the encoded frame packet does not carry additional information. Therefore, it can effectively reduce the complexity of the encoded frame packet encapsulation, reduce the amount of data transmitted by the head-mounted device in a single transmission, and reduce the data transmission latency.
[0066] In this embodiment, the packet header may include only two fields, used to characterize the frame length of the audio encoded frame and the configuration parameters, respectively. The frame length is used for length verification of the audio encoded frame, and the configuration parameters indicate that the encoded frame packet is a custom data packet. For example, the encoded frame packet might be named XXXXYYYYDDDDDDDD, where DDDDDDDD represents the audio encoded frame, XXXXYYYY is the packet header, XXXX represents the configuration parameters, and YYYY represents the frame length of the audio encoded frame.
[0067] In one alternative implementation, the control module can determine an encoded frame packet based on an audio encoded frame. This processing method ensures that the amount of data transmitted by the head-mounted device remains relatively stable each time; and since the duration of the original audio frame corresponding to each audio encoded frame is short, it effectively reduces the negative impact on the continuity of audio data caused by packet loss during data transmission.
[0068] In one alternative implementation, if multiple audio encoded frames corresponding to each audio signal group in the second audio data are obtained by using block coding, when determining the original audio frame packet, the control module can determine an encoded frame packet based on the audio encoded frames corresponding to each audio signal group in the same coding period, thereby reducing the complexity of audio processing by the subsequent server.
[0069] For example, the audio signal groups corresponding to the second audio data are audio signal group G1, audio signal group G2 and audio signal group G3, with an encoding interval of 20 milliseconds. After the control module obtains multiple audio encoded frames corresponding to each audio signal group, it can determine an encoded frame packet based on the audio encoded frames corresponding to each audio signal group from 0 milliseconds to 19 milliseconds, determine an encoded frame packet based on the audio encoded frames corresponding to each audio signal group from 20 milliseconds to 39 milliseconds, determine an encoded frame packet based on the audio encoded frames corresponding to each audio signal group from 40 milliseconds to 59 milliseconds, and so on.
[0070] In this optional implementation, the encoding parameters of the second audio data may also include the signal grouping relationship of each audio signal, and the signal grouping relationship is used to characterize the grouping relationship of each audio signal.
[0071] In step S500, each encoded frame packet is sent to the second device via a short-range communication protocol.
[0072] In this step, the control module can send each encoded frame packet to the second device via a short-range communication protocol when the head-mounted device establishes a short-range communication link with the second device. Optionally, the control module can send each encoded frame packet via the short-range communication protocol as it is generated, thereby reducing data transmission latency while ensuring the transmission order of audio encoded frames.
[0073] Step S600: Receive each encoded frame packet via a short-range communication protocol.
[0074] In this step, the second device can receive the encoded frame packets sent by the head-mounted device via a near-field communication protocol.
[0075] In step S700, an encoded frame packet is sent to the server via a remote communication protocol.
[0076] In this step, the second device can send each encoded frame packet to the server via a remote communication protocol when establishing a remote communication link with the server. Optionally, the second device can send the encoded frame packet via the remote communication protocol each time it receives one. In practical applications, the power consumption generated during short-range communication between devices is usually significantly lower than that generated during long-range communication. Therefore, this embodiment uses a method where the head-mounted device sends the encoded frame packets via a short-range communication protocol, and the second device forwards the encoded frame packets via a remote communication protocol to reduce the power consumption of the head-mounted device.
[0077] Step S800: Receive encoded frame packets via a remote communication protocol.
[0078] In this step, the server can receive encoded frame packets sent by the second device via a remote communication protocol.
[0079] Step S900: Decode the audio encoded frames in each encoded frame packet to obtain the third audio data.
[0080] After receiving the encoded frame packet, the server can extract the audio encoded frame from the encoded frame packet, and then decode the audio encoded frame according to the decoding parameters to obtain the third audio data.
[0081] Similar to encoding parameters, decoding parameters can also include encoding format, encoding interval, number of channels, and bitrate, and all parameters in the decoding parameters are identical to those in the encoding parameters. In some embodiments, the decoding parameters can be predetermined, so the server can query the corresponding encoding parameters based on the device identifier of the head-mounted device; in some embodiments, after determining the encoding parameters, the head-mounted device can send the decoding parameters to a second device via a short-range communication protocol, and the second device can then send the decoding parameters to the server via a long-range communication protocol.
[0082] Optionally, the server can determine the parsing rules for the encoded frame packet based on the configuration parameters in the packet header to parse the encoded frame packet, and perform length verification on the audio encoded frame based on the frame length in the packet header to prevent data transmission anomalies and decoding failures. Then, the audio encoded frame is decoded when the length verification is passed.
[0083] In one alternative implementation, if the audio encoded frame is obtained by block encoding, the decoding parameters also include the signal grouping relationship. The server can decode each audio encoded frame in the same encoded frame packet and then determine each decoded audio frame corresponding to the same encoded frame packet as a third audio data.
[0084] Step S1000: Perform post-processing on the third audio data to obtain the corresponding post-processing results.
[0085] After acquiring the third audio data, the server can perform post-processing on the third audio data to improve audio quality. In some embodiments, the post-processing of the third audio data may include speech enhancement processing, that is, the head-mounted device may perform speech enhancement processing on the third audio data to obtain the target audio data. Speech enhancement processing may include at least one of beamforming, voice separation, and noise reduction.
[0086] When performing beamforming processing on the third audio data, the server can determine the time difference of arrival (TDOA) of the sound reaching different microphones. By superimposing the corresponding delays on the audio signals of each channel of the third audio signal, the sound waveform in the target direction is aligned to a unified time. Then, the aligned audio signals of each channel are superimposed, so that the waveforms in the target direction can be superimposed in the same direction, achieving the purpose of sound enhancement, while the waveforms in the non-target direction cancel each other out, achieving the purpose of noise suppression.
[0087] The human voice separation processing of the third audio data can be achieved in various ways. In some embodiments, the server can achieve human voice separation based on the beamforming principle; in other embodiments, the server can perform frequency domain transformation on each decoded audio frame in the third audio data, such as Fast Fourier Transform, to obtain the frequency domain signal of each decoded audio frame, and determine the sound source direction based on the arrival time difference of the sound to different microphones. Then, combining the spatial characteristics of the sound source direction, the human voice characteristics and noise characteristics of each frequency domain signal, the human voice time-frequency block and noise time-frequency block in each frequency domain signal are marked, thereby generating a multi-channel joint filtering mask for each frequency domain signal. Based on this multi-channel joint filtering mask, each frequency domain signal is weighted and filtered, and the filtering... The subsequent frequency domain signals undergo time-domain transformation, such as inverse Fourier transform, to obtain time-domain audio frames. These time-domain audio frames are then superimposed to achieve voice separation. In some embodiments, the server can utilize a pre-trained neural network to achieve voice separation. For example, the server can perform frequency-domain transformation on the third audio data to determine the corresponding spectrogram and output the spectrogram to a neural network trained based on the time-frequency masking method. This neural network can output a mask corresponding to the spectrogram, which includes the probability that each time-frequency grid in the spectrogram represents a human voice. Furthermore, the server can multiply the spectrogram and the mask, and then perform a time-domain transformation on the product to achieve voice separation.
[0088] Noise reduction of third-party audio data can also be achieved in various ways. Furthermore, the noise reduction process for third-party audio data can be referenced from the human voice separation process for third-party audio data, and will not be elaborated here.
[0089] Optionally, the server can determine the execution order of each speech enhancement process and process the third audio data or the processing result of the previous process in sequence to obtain the target audio data.
[0090] Optionally, after obtaining the target audio data, the server can further process the target audio data according to the wearer's actual needs. In some embodiments, the server can perform speech recognition on the target audio data to obtain speech-recognized text. Optionally, the server can perform speech recognition on the target audio data based on natural language processing models, such as Gaussian Mixture Model (GMM)-Hidden Markov Model (HMM), Connectionist Temporal Classification (CTC) model, Attention-Based model, Generative Pre-trained Transformer (GPT) series models, PaLM series models, etc. Furthermore, the server can perform speech recognition on the target audio data after receiving a speech recognition request sent by the wearer through a head-mounted device or a second device.
[0091] In some embodiments, the server can also translate the speech-recognized text to obtain the target text. Optionally, the server can translate the speech-recognized text based on a natural language processing model. Further, the server can translate the speech-recognized text after receiving a text translation request sent by the wearer through a head-mounted device or a second device. The target text is text written in a target language, which can be specified in advance by the wearer or when sending the text translation request.
[0092] In this step, the server can determine the post-processing result of the third audio data based on at least one of the target audio data, the corresponding speech recognition text, and the target text.
[0093] In some optional implementations, this embodiment may also include the following steps: Step S1100: Send the post-processing results to the second device via a remote communication protocol.
[0094] After determining the post-processing result of the third audio data, the server can send the post-processing result to the second device via a remote communication protocol.
[0095] Step S1200: In response to receiving the post-processing result, output the post-processing result.
[0096] In this step, the second device can output the post-processing result in various ways after receiving it, so as to synchronize the post-processing result with the wearer. For example, if the second device includes a display component, it can display the post-processing result through the display component; if the second device includes an audio playback component, it can broadcast the post-processing result through the audio playback component.
[0097] Figure 4 This is a data flow diagram of the audio data processing method according to an embodiment of the present invention. Figure 4As shown, the first device 11 includes an audio acquisition module 111, a preprocessing module 112, and an encoding module 113. The audio acquisition module 111 can output first audio data, i.e., audio data 41, to the preprocessing module 112. The preprocessing module 112 can preprocess the audio data 41 and output second audio data, i.e., audio data 42, to the encoding module 113. The encoding module 113 can encode the second audio data to obtain multiple audio encoded frames and generate an encoded frame packet 43 including a packet header and audio encoded frames, and then send the encoded frame packet 43 to the second device 12. The second device 12 includes a forwarding module 121. The forwarding module 121 can receive the encoded frame packet 43 through a short-range communication protocol and send the encoded frame packet 43 to the server 13 through a remote communication protocol. The server 13 includes a decoding module 131 and a post-processing module 132. The decoding module 131 can decode the audio encoded frames in the received encoded frame packet to obtain third audio data, i.e., audio data 44, and then output the audio data 44 to the post-processing module 132. The post-processing module 132 can perform post-processing on the audio data 44 and send the post-processing result 45 to the second device 12 via a remote communication protocol.
[0098] The audio data processing system of this invention includes a first device, a second device, and a server. After acquiring audio data, the first device preprocesses and encodes the acquired audio data to obtain multiple audio encoded frames and generates encoded frame packets for each audio encoded frame. These encoded frame packets are then sequentially sent to the second device via a short-range communication protocol. The second device sequentially forwards the encoded frame packets sent by the first device to the server via a long-range communication protocol. The server decodes and performs post-processing on each encoded frame packet. This invention achieves audio data processing through collaboration between the first device and the server, while using the second device as a forwarding device for encoded frame packets. Therefore, it can ensure audio quality while controlling the manufacturing cost, power consumption, and portability of the first device.
[0099] Figure 5 This is a flowchart of an audio data processing method according to an embodiment of the present invention on the first device side. Figure 5 As shown, the method in this embodiment includes the following steps in the first device: Step S100: Collect the first audio data.
[0100] Step S200: Preprocess the first audio data to obtain the second audio data.
[0101] Step S300: Encode the second audio data to obtain multiple corresponding audio encoded frames.
[0102] Step S400: Determine the encoding frame packet corresponding to each audio encoding frame.
[0103] In step S500, each encoded frame packet is sent to the second device via a short-range communication protocol.
[0104] In this embodiment of the audio data processing system, after acquiring audio data, the first device preprocesses and encodes the acquired audio data to obtain multiple audio encoded frames. It then generates encoded frame packets for each audio encoded frame and sequentially sends these packets to a second device via a short-range communication protocol. The second device then forwards these encoded frame packets sequentially to a server, whereby the server decodes and performs post-processing on each packet. This embodiment of the invention achieves audio data processing through collaboration between the first device and the server, while using the second device as a forwarding device for the encoded frame packets. Therefore, it can ensure audio quality while controlling the manufacturing cost, power consumption, and portability of the first device.
[0105] Figure 6 This is a flowchart of an audio data processing method according to an embodiment of the present invention on the second device side. Figure 6 As shown, the method of this embodiment includes the following steps in the second device: Step S600: Receive each encoded frame packet via a short-range communication protocol.
[0106] In step S700, an encoded frame packet is sent to the server via a remote communication protocol.
[0107] In this embodiment of the audio data processing system, the second device sequentially forwards the encoded frame packets sent by the first device via a short-range communication protocol to the server through a remote communication protocol, enabling the server to decode and perform post-processing on the encoded frame packets sent by the first device. In this embodiment, the encoded frame packets are generated by the first device based on multiple audio encoded frames, and the audio encoded frames are obtained by the first device through preprocessing and encoding of the acquired audio data. Therefore, this embodiment of the invention achieves the audio data processing process through the collaboration of the first device and the server, while using the second device as a forwarding device for the encoded frame packets. Thus, it is possible to ensure audio quality while controlling the manufacturing cost, power consumption, and portability of the first device.
[0108] Figure 7 This is a flowchart of the audio data processing method on the server side according to an embodiment of the present invention. Figure 7 As shown, the method in this embodiment includes the following steps on the server: Step S800: Receive encoded frame packets via a remote communication protocol.
[0109] Step S900: Decode the audio encoded frames in each encoded frame packet to obtain the third audio data.
[0110] Step S1000: Perform post-processing on the third audio data to obtain the corresponding post-processing results.
[0111] In this embodiment of the invention, the server decodes and processes each encoded frame packet sent by the second device. The encoded frame packet is generated by the first device based on multiple audio encoded frames, and the audio encoded frames are obtained by the first device through preprocessing and encoding of the acquired audio data. At the same time, the audio frame packet is received by the second device through a short-range communication protocol and sent through a long-range communication protocol. Therefore, this embodiment of the invention realizes the audio data processing process through the collaboration of the first device and the server, and uses the second device as a forwarding device for the encoded frame packet. Thus, it can ensure audio quality while controlling the manufacturing cost, power consumption and portability of the first device.
[0112] Figure 8 This is a schematic diagram of an audio data processing apparatus according to an embodiment of the present invention. Figure 8 As shown, the audio data processing device in this embodiment includes an audio acquisition unit 801, a preprocessing unit 802, an encoding unit 803, a frame packet determination unit 804, a first frame packet sending unit 805, a first frame packet receiving unit 806, a second frame packet sending unit 807, a second frame packet receiving unit 808, a decoding unit 809, and an audio processing unit 810.
[0113] The system includes: an audio acquisition unit 801 for acquiring first audio data, which is a multi-channel audio stream; a preprocessing unit 802 for preprocessing the first audio data to obtain second audio data; an encoding unit 803 for encoding the second audio data to obtain multiple corresponding audio encoded frames; a frame packet determination unit 804 for determining the encoded frame packet corresponding to each audio encoded frame; a first frame packet sending unit 805 for sending each encoded frame packet to a second device via a short-range communication protocol; a first frame packet receiving unit 806 for receiving each encoded frame packet via a short-range communication protocol; a second frame packet sending unit 807 for sending the encoded frame packet to a server via a remote communication protocol; a second frame packet receiving unit 808 for receiving the encoded frame packet via a remote communication protocol; a decoding unit 809 for decoding the audio encoded frames in each encoded frame packet to obtain third audio data; and an audio processing unit 810 for post-processing the third audio data to obtain corresponding post-processing results.
[0114] The audio data processing system of this invention includes a first device, a second device, and a server. After acquiring audio data, the first device preprocesses and encodes the acquired audio data to obtain multiple audio encoded frames and generates encoded frame packets for each audio encoded frame. These encoded frame packets are then sequentially sent to the second device via a short-range communication protocol. The second device sequentially forwards the encoded frame packets sent by the first device to the server via a long-range communication protocol. The server decodes and performs post-processing on each encoded frame packet. This invention achieves audio data processing through collaboration between the first device and the server, while using the second device as a forwarding device for encoded frame packets. Therefore, it can ensure audio quality while controlling the manufacturing cost, power consumption, and portability of the first device.
[0115] Figure 9 This is a schematic diagram of an electronic device according to an embodiment of the present invention. In this embodiment, the electronic device 9 includes a server, a terminal, etc. Figure 9 As shown, the electronic device 9 includes at least one processor 901; a memory 902 communicatively connected to at least one processor 901; and a communication component 903 communicatively connected to a scanning device, wherein the communication component 903 receives and transmits data under the control of the processor 901; wherein the memory 902 stores instructions executable by at least one processor 901, the instructions being executed by at least one processor 901 to implement the above-described audio data processing method.
[0116] Specifically, the electronic device includes: one or more processors 901 and a memory 902. Figure 9 Taking a processor 901 as an example, the processor 901 and the memory 902 can be connected via a bus or other means. Figure 9 Taking a bus connection as an example, memory 902, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Processor 901 executes various functional applications and data processing of the device by running the non-volatile software programs, instructions, and modules stored in memory 902, thereby implementing the aforementioned audio data processing method.
[0117] Memory 902 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store an option list, etc. Furthermore, memory 902 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 902 may optionally include memory remotely located relative to processor 901, and these remote memories may be connected to external devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0118] One or more modules are stored in memory 902, and when executed by one or more processors 901, they perform the audio data processing method in any of the above method embodiments.
[0119] The above-mentioned products can perform the methods provided in the embodiments of this application, and have the corresponding functional modules and beneficial effects of performing the methods. For technical details not described in detail in this embodiment, please refer to the methods provided in the embodiments of this application.
[0120] The audio data processing system of this invention includes a first device, a second device, and a server. After acquiring audio data, the first device preprocesses and encodes the acquired audio data to obtain multiple audio encoded frames and generates encoded frame packets for each audio encoded frame. These encoded frame packets are then sequentially sent to the second device via a short-range communication protocol. The second device sequentially forwards the encoded frame packets sent by the first device to the server via a long-range communication protocol. The server decodes and performs post-processing on each encoded frame packet. This invention achieves audio data processing through collaboration between the first device and the server, while using the second device as a forwarding device for encoded frame packets. Therefore, it can ensure audio quality while controlling the manufacturing cost, power consumption, and portability of the first device.
[0121] Another embodiment of the present invention relates to a non-volatile storage medium for storing a computer-readable program for use by a computer to execute some or all of the above-described method embodiments.
[0122] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
[0123] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. An audio data processing system, characterized in that, The system includes: A first device is configured to acquire first audio data, preprocess the first audio data, acquire second audio data, encode the second audio data, acquire multiple corresponding audio encoded frames, determine the encoded frame packet corresponding to each audio encoded frame, and send each encoded frame packet to a second device via a near-field communication protocol. The first audio data is multi-channel audio stream data. The second device is configured to receive each of the encoded frame packets via a short-range communication protocol and to send the encoded frame packets to a server via a remote communication protocol. The server is configured to receive the encoded frame packets via a remote communication protocol, decode the audio encoded frames in each encoded frame packet to obtain third audio data, perform post-processing on the third audio data, and obtain the corresponding post-processing result.
2. The system according to claim 1, characterized in that, The first device is further configured as follows: The second audio data is encoded according to the encoding parameters to obtain a plurality of corresponding audio encoded frames. The encoding parameters include encoding format, encoding interval, number of channels and bit rate.
3. The system according to claim 2, characterized in that, The encoding parameters also include signal grouping relationships, which are used to characterize the grouping relationships of audio signals in each channel of the second audio data; The first device is further configured as follows: Multiple audio signal groups are determined based on the correlation of the audio signals, and each audio signal group is encoded to obtain the audio encoded frame corresponding to each audio signal combination. A encoded frame packet is determined based on each audio encoded frame within the same encoding period.
4. The system according to claim 1, characterized in that, The first device is further configured as follows: The first audio data is subjected to echo cancellation processing based on the reference audio signal to obtain the second audio data.
5. The system according to claim 1, characterized in that, The first device is further configured as follows: The header of the encoded frame packet is determined, the header including the frame length of the audio encoded frame and configuration parameters, the configuration parameters being used to characterize the encoded frame packet as a custom data packet, and the encoded frame packet is generated based on the header and the audio encoded frame.
6. The system according to claim 1, characterized in that, The server is further configured as follows: The audio encoded frames are decoded according to the decoding parameters to obtain the third audio data. The decoding parameters include the encoding format, encoding interval, number of channels, and bit rate.
7. The system according to claim 1, characterized in that, The server is further configured as follows: The third audio data is subjected to speech enhancement processing to obtain target audio data. The speech enhancement processing includes at least one of beamforming, voice separation and noise reduction.
8. The system according to claim 7, characterized in that, The server is further configured as follows: The target audio data is subjected to speech recognition to obtain the corresponding speech recognition text, and the speech recognition text is translated into the target text.
9. The system according to claim 1, characterized in that, The server is also configured to: The post-processing results are sent to the second device via a remote communication protocol; The second device is also configured to: In response to receiving the post-processing result, the post-processing result is output.
10. An audio data processing method, characterized in that, The method includes: Collect first audio data, which is multi-channel audio stream data; The first audio data is preprocessed to obtain the second audio data; The second audio data is encoded to obtain multiple corresponding audio encoded frames; Determine the encoded frame packet corresponding to each of the aforementioned audio encoded frames; Each of the coded frame packets is sent to the second device via a short-range communication protocol; Each of the coded frame packets is received via a short-range communication protocol; The encoded frame packet is sent to the server via a remote communication protocol; The encoded frame packet is received via a remote communication protocol; Decode the audio encoded frames in each encoded frame packet to obtain third audio data; The third audio data is then post-processed to obtain the corresponding post-processing results.
11. An audio data processing apparatus, characterized in that, The device includes: An audio acquisition unit is used to acquire first audio data, wherein the first audio data is multi-channel audio stream data; The preprocessing unit is used to preprocess the first audio data to obtain the second audio data; The encoding unit is used to encode the second audio data and obtain multiple corresponding audio encoded frames. A frame packet determination unit is used to determine the encoded frame packet corresponding to each of the audio encoded frames; The first frame packet sending unit is used to send each of the coded frame packets to the second device via a short-range communication protocol. The first frame packet receiving unit is used to receive each of the coded frame packets through a short-range communication protocol; The second frame packet sending unit is used to send the encoded frame packet to the server via a remote communication protocol; The second frame packet receiving unit is used to receive the encoded frame packet via a remote communication protocol; The decoding unit is used to decode the audio encoded frames in each encoded frame packet to obtain third audio data; An audio processing unit is used to perform post-processing on the third audio data and obtain the corresponding post-processing results.
12. An electronic device comprising a memory and a processor, characterized in that, The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in claim 10.
13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in claim 10.
14. A computer program product, characterized in that, The computer program product includes a computer program / instruction that, when executed by a processor, implements the method as described in claim 10.