A voice interaction method, device, medium and electronic equipment

Abstract

The specification provides a voice interaction method, device, medium and electronic equipment. In the method, first, a voice interaction triggering instruction of a user is acquired, a session establishment process with a server and a local audio acquisition process are executed concurrently, and in the local audio acquisition process, the acquired audio data frames are stored in a specified buffer. Then, when it is monitored that the session establishment is completed, the audio data frames stored in the specified buffer are sent to the server at a first sending rate greater than a real-time acquisition rate of the audio data frames, and when the audio data frames in the specified buffer are sent completely, the newly acquired audio data frames are sent to the server in real time at a second sending rate equal to the real-time acquisition rate. Thus, while ensuring that the user can speak immediately after clicking without waiting for the session establishment to be completed, it is ensured that the server can receive complete starting voice data after the session establishment is completed, so as to improve the stability and response speed of voice interaction.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of voice interaction technology, and more particularly to a voice interaction method, apparatus, medium and electronic device. Background Technology

[0002] With the rapid development of artificial intelligence technology, multimodal voice interaction has been widely applied in smart assistants, in-vehicle systems, and various smart terminals, becoming an important entry point for users to obtain services. Typically, when a user initiates a voice question, the client usually first performs an end-to-cloud session establishment process. Only after the session is successfully established does it activate the local microphone to collect audio data and upload it to the cloud for Automatic Speech Recognition (ASR). Due to network fluctuations and protocol interaction time, this session establishment process often involves a delay of hundreds of milliseconds or even longer, and the smart terminal will be unable to receive audio data before the session is established.

[0003] When a smart terminal is unable to receive audio because a session has not yet been established, a prompt such as "Connecting" or "Voice preparing" is usually displayed on the user interface to inform the user to wait. However, in practical applications, such prompts are often ignored due to their short display time, inconspicuous location, or distracted user attention. Many users are accustomed to starting to speak immediately after clicking the trigger button, resulting in the voice data sent by the user in the initial stage not being collected. This causes the cloud recognition results to show missing first words or sentences, ultimately leading to deviations in the subsequent understanding and response of the large model. It can also easily make users mistakenly believe that there is a malfunction or performance defect in the product, seriously affecting the smoothness of voice interaction and user experience. Summary of the Invention

[0004] In view of the above, one or more embodiments of this specification provide the following technical solutions: According to a first aspect of one or more embodiments of this specification, a voice interaction method is provided, comprising: Based on the user's voice interaction trigger command, the session establishment process with the server and the local audio acquisition process are executed concurrently; wherein, in the local audio acquisition process, the acquired audio data frames are stored in a designated buffer. When the session is detected to be established, the audio data frames stored in the specified buffer are sent to the server at a first sending rate; wherein, the first sending rate is greater than the real-time acquisition rate of the audio data frames; After the audio data frames in the designated buffer have been sent, the newly acquired audio data frames are sent to the server in real time at a second sending rate; wherein the second sending rate is equal to the real-time acquisition rate.

[0005] According to a second aspect of one or more embodiments of this specification, a voice interaction device is provided, comprising: The execution module is used to trigger commands based on user voice interaction and concurrently execute the session establishment process with the server and the local audio acquisition process; wherein, in the local audio acquisition process, the acquired audio data frames are stored in a specified buffer. The sending module is used to send the audio data frames stored in the specified buffer to the server at a first sending rate when the session is detected to be established; wherein the first sending rate is greater than the real-time acquisition rate of the audio data frames; after the audio data frames in the specified buffer have been sent, the newly acquired audio data frames are sent to the server in real time at a second sending rate; wherein the second sending rate is equal to the real-time acquisition rate.

[0006] According to a third aspect of one or more embodiments of this specification, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor implements the steps of the voice interaction method described above by executing the executable instructions.

[0007] According to a fourth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the voice interaction method described above.

[0008] According to a fifth aspect of one or more embodiments of this specification, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the above-described voice interaction method.

[0009] In this method, the session establishment process with the server and the local audio acquisition process are executed concurrently based on the user's voice interaction trigger command. In the local audio acquisition process, the acquired audio data frames are stored in a specified buffer. Then, when the session establishment is detected to be complete, the audio data frames stored in the specified buffer are sent to the server at a first transmission rate greater than the real-time acquisition rate of the audio data frames. After the audio data frames in the specified buffer have been sent, the newly acquired audio data frames are sent to the server in real time at a second transmission rate equal to the real-time acquisition rate.

[0010] As can be seen from the above method, during the local audio acquisition process, the client can store the acquired audio data frames in a designated buffer. Upon detecting that the session has been established, it quickly sends the audio data frames stored in the designated buffer to the server at a first transmission rate greater than the real-time acquisition rate. Then, after all the audio data frames in the designated buffer have been sent, the newly acquired audio data frames are sent to the server in real-time at a second transmission rate equal to the real-time acquisition rate. This ensures that users can speak immediately after clicking without waiting for the session to complete, while also ensuring that the server receives complete initial voice data after the session is established, ultimately improving the stability and response speed of voice interaction. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the architecture of a voice interaction system provided in an exemplary embodiment.

[0012] Figure 2 This is a flowchart illustrating a voice interaction method provided in an exemplary embodiment.

[0013] Figure 3 This is a timing diagram of a voice interaction process provided in an exemplary embodiment.

[0014] Figure 4 This is a schematic diagram of a transmission process provided in an exemplary embodiment.

[0015] Figure 5 This is a timing diagram of a voice interaction process provided in an exemplary embodiment.

[0016] Figure 6 This is a schematic structural diagram of a device provided in an exemplary embodiment.

[0017] Figure 7 This is a block diagram of a voice interaction device provided in an exemplary embodiment. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0019] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals shall be provided for users to choose to authorize or refuse.

[0020] With the rapid development of artificial intelligence technology, multimodal voice interaction has been widely applied in smart assistants, in-vehicle systems, and various smart terminals, becoming an important entry point for users to obtain services. Typically, when a user initiates a voice question, the client usually adopts a "connection first, data collection later" processing logic. This means that the end-to-cloud session establishment process is first executed (including DNS resolution, TLS handshake, authentication, and media negotiation steps). Only after the session is successfully established does the local microphone activate to collect audio data and upload it to the cloud for automatic speech recognition. However, due to network fluctuations and protocol interaction time, this session establishment process often involves delays of hundreds of milliseconds or even longer, and the smart terminal will be unable to receive audio data before the connection is established.

[0021] When a smart terminal is unable to receive audio because a session has not yet been established, a prompt such as "Connecting" or "Voice preparing" is usually displayed on the user interface to inform the user to wait. However, in practical applications, such prompts are often ignored due to their short display time, inconspicuous location, or distracted user attention. Many users are accustomed to starting to speak immediately after clicking the trigger button, resulting in the voice data sent by the user in the initial stage not being collected. This causes the cloud recognition results to show missing first words or sentences, ultimately leading to deviations in the subsequent understanding and response of the large model. It can also easily make users mistakenly believe that there is a malfunction or performance defect in the product, seriously affecting the smoothness of voice interaction and user experience.

[0022] In addition, forcing users to speak only after a conversation has been established, or prompting users with text messages that indicate that a conversation can only be spoken after it has been established, both require users to wait, which in turn leads to a decline in user experience.

[0023] To address the aforementioned technical problems, this specification provides a voice interaction method. The core of this method lies in concurrently executing a session establishment process with the server and a local audio acquisition process after the user triggers a voice interaction command. Specifically, during the local audio acquisition process, the client can store the acquired audio data frames in a designated buffer. Upon detecting that the session establishment is complete, the client rapidly sends the audio data frames stored in the designated buffer to the server at a first transmission rate greater than the real-time audio data frame acquisition rate. Then, after all the audio data frames in the designated buffer have been sent, the newly acquired audio data frames are sent to the server in real-time at a second transmission rate equal to the real-time acquisition rate. This ensures that users can speak immediately after clicking without waiting for the session to complete, while also ensuring that the server receives complete initial voice data after the session is established, ultimately improving the stability and response speed of voice interaction.

[0024] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0025] Figure 1 This is a schematic diagram of the architecture of a voice interaction system provided in an exemplary embodiment. For example... Figure 1 As shown, the system may include a server 11, a network 12, and at least one terminal device (such as a personal computer PC 13, a mobile phone 14, etc.).

[0026] The server 11 can be a physical server containing an independent host, or a virtual server hosted in a host cluster. In practical applications, this server can be deployed in the cloud to form a cloud platform (i.e., the "cloud" side) providing voice interaction services, such as real-time audio and video communication (RTC) access, automatic speech recognition (ASR), and large-scale model inference. During operation, the server 11 runs the server program for the voice interaction service to implement the relevant functions of the application. Specifically, the server 11 is responsible for receiving audio data streams from terminal devices, performing speech-to-text processing, and generating response content based on the context to return to the terminal device.

[0027] The terminal device can be a client-side component for voice interaction. Besides personal computers (PC13) and mobile phones (14), the terminal device can also be electronic devices such as tablets, PDAs (Personal Digital Assistants), wearable devices (such as smart glasses and smartwatches), in-vehicle terminals, and smart speakers. This specification does not limit this to any particular embodiment. During operation, the electronic device can run a client-side program for a specific application to implement the application's related functions. For example, when running a voice interaction service program, it can act as a client for that voice interaction service. This client-side program can be a native application installed on the electronic device, or it can be a mini-program, quick app, or other similar form. Of course, when using HTML5 or similar web technologies, the related functions can be implemented through a browser-displayed page. The browser here can be a standalone browser application or a browser module embedded in some applications. In this embodiment, the client is mainly responsible for executing the voice interaction method described in this application, including: responding to user trigger commands, concurrently starting local audio acquisition and session establishment, managing the local audio buffer, and performing logic such as audio data caching, encoding, emptying, and real-time transmission.

[0028] As for the network 12 that enables interaction between the terminal device and the server 11, communication can be achieved using either wired or wireless networks, depending on the communication methods supported by the corresponding electronic device. This specification does not impose any restrictions on this. For example, a PC can support both wired and wireless communication, so it can use either wired or wireless networks as needed. A mobile phone typically only supports wireless communication, so it can use a wireless network (such as 4G / 5G, Wi-Fi, etc.) for communication. Through network 12, a real-time communication connection is established between the terminal device and the server 11, transmitting audio data and interaction results.

[0029] Figure 2 This is a flowchart illustrating a voice interaction method provided in an exemplary embodiment. In this embodiment, the executing entity for implementing the voice interaction method can be a terminal device with voice interaction capabilities, such as a smartphone, tablet computer, in-vehicle terminal, smart speaker, etc. For ease of description, the following explanation uses a client as the executing entity. The method includes the following steps: S200: Based on the user's voice interaction trigger command, concurrently execute the session establishment process with the server and the local audio acquisition process; wherein, in the local audio acquisition process, the acquired audio data frames are stored in a specified buffer.

[0030] In this manual, users can initiate voice interaction in various ways. For example, users can click the virtual "Press and hold to speak" button or the "microphone" icon on the graphical user interface displayed on the terminal device to initiate voice interaction; another example is that users can trigger voice interaction by pressing a physical button (such as the voice button on the car steering wheel or the multi-function button on the headset); of course, users can also trigger voice interaction by saying a specific wake word, etc., and this manual does not limit this.

[0031] When the client detects a voice interaction trigger command sent by the user, it can initiate two concurrent sub-processes: the concurrent execution of the session establishment process with the server and the local audio acquisition process, as detailed below. Figure 3 As shown.

[0032] Figure 3 This is a timing diagram of a voice interaction process provided in an exemplary embodiment.

[0033] Combination Figure 3 As can be seen, in the session establishment process with the server, the client can send a connection request to the server, perform Domain Name System Resolution (DNS) to obtain the server address, perform a Transport Layer Security Handshake (TLS) to establish a secure transmission channel, and complete identity authentication and media parameter negotiation (such as Session Description Protocol Exchange (SDP)) to establish a stable data transmission channel from the client to the server (such as a real-time communication channel based on the WebRTC protocol).

[0034] In the local audio acquisition process, the client can call the local audio device module (ADM) to start the microphone to acquire the audio data input by the user.

[0035] The aforementioned audio data can consist of several audio data frames, each stored in a designated buffer. The audio data can be in uncompressed or pre-encoded raw audio formats such as Pulse Code Modulation (PCM) or Waveform Audio File Format (WAV). In one embodiment, the audio data frames can be in a 16kHz sampling rate, 16-bit quantization, mono PCM format, with one frame generated every 10ms.

[0036] It is important to emphasize that, for the local audio device module, regardless of whether the session is successfully established, it will activate the microphone to collect the user's voice signal. This ensures that even if the user speaks immediately after clicking the trigger, the initial audio data can be completely recorded locally and stored in the designated buffer, thus avoiding the problem of losing the first segment of voice due to network connection delay.

[0037] It should be noted that the specified buffer mentioned above can be implemented using various data structures, such as linear buffers, linked buffers, pooled buffers, etc.

[0038] However, in practical applications, the acquisition and transmission of audio data is a continuous and high-frequency process. If a linear buffer is used, a large number of memory movement operations are required to free up space after the data is read out, or frequent memory allocation and release are required. This will bring high CPU overhead and memory fragmentation risk. If a chained buffer is used, although memory movement is avoided, the additional overhead of node management and the problem of poor cache locality may affect processing efficiency.

[0039] Therefore, in order to ensure the high efficiency of continuous data writing and reading, while fixing memory usage and avoiding frequent memory allocation and copy operations, the specified buffer mentioned above can be a circular buffer. The size of the circular buffer is determined based on the preset maximum session establishment time, the preset network fluctuation protection time, and the real-time acquisition rate. For details, please refer to the following formula.

[0040]

[0041] In the above formula, That is, the determined size of the circular buffer. This is the preset maximum session establishment time (which is also the longest session establishment time allowed by the system). This is the preset network fluctuation protection time, which is used to deal with situations where session establishment time exceeds the boundary due to network jitter or other reasons. This refers to the real-time acquisition rate of the audio data, which can be set according to actual needs. For example, for 16kHz sampling rate, 16-bit depth, mono audio data, AudioBytesPerSec is approximately 32KB / s; for 48kHz sampling rate, 16-bit depth, mono audio data, AudioBytesPerSec is approximately 96KB / s.

[0042] As can be seen from the above formula, the client can set a specified buffer size in this way, thereby ensuring that there is still enough space to cache the user's voice data at the beginning stage when the connection delay is large or the network fluctuates, preventing word loss while avoiding excessive memory allocation and waste of storage resources.

[0043] In practical applications, since the raw audio data is large in size, direct storage will quickly consume a lot of memory, limit the caching time or increase hardware costs. Therefore, the client can also encode at least a portion of the raw audio data frames in the acquired raw audio data frames to obtain compressed audio data frames, and then store the compressed audio data frames and / or the raw audio data frames in a specified buffer.

[0044] Specifically, the client can also determine whether the remaining storage space of the circular buffer is greater than a preset space threshold. If so, the acquired raw audio data frame can be stored in the specified buffer; otherwise, the acquired raw audio data frame can be encoded to obtain a compressed audio data frame, which can then be stored in the specified buffer.

[0045] In addition, the client can also encode all raw audio data frames to obtain compressed audio data frames, and store the compressed audio data frames in a designated buffer. During this process, the client can acquire raw audio data frames according to a preset acquisition period, and at the same time, it can merge and encode multiple frames of raw audio data into a single compressed audio data frame according to a preset encoding period and store it in a designated buffer.

[0046] For example, if the acquisition period for audio data is 10ms, meaning one frame of raw audio data is acquired every 10ms, and efficient encoding formats such as Opus Interactive Audio Codec (Opus) or Advanced Audio Coding (AAC) are used for pre-encoding and storage of the audio, the encoding period can be 20ms. This means the client can combine the raw audio data frames acquired in every two acquisition periods into a single encoding unit for Opus encoding, generating a compressed audio data frame. For instance, for 16kHz sampling rate, 16-bit depth, mono audio, the size of the raw PCM data for 20ms is approximately 320 bytes. After Opus encoding, the size of the generated compressed audio data frame may only be about 64 bytes (depending on the bitrate setting), achieving a compression ratio of 5:1 or even higher.

[0047] As can be seen from the above, this method of multiple acquisitions and one encoding process not only reduces the frequency of encoder calls and CPU overhead, but also reduces the size of a single frame of data stored in the specified buffer, thus enabling the caching of audio data with a longer time span within a limited buffer space.

[0048] Based on this, since mobile terminal devices (such as mobile phones, wearable devices, etc.) have limited hardware resources and storage space, in order to reduce the memory usage of the reserved specified buffer, the client can also use an upper limit control strategy based on the audio duration to determine the size of the specified buffer.

[0049] Specifically, the client can set a maximum cache duration threshold (MAX_CACHE_DURATION_MS). When the cumulative duration of all audio data frames stored in the specified buffer reaches this threshold, it can be determined that the current network environment is abnormal or the connection has timed out.

[0050] For example, the initial value of the maximum buffer duration threshold can be set to 5000ms (i.e., 5 seconds). Under this configuration, if efficient encoding formats such as Opus are used to pre-encode and store the audio, a 5-second audio buffer will only occupy about 16KB of memory space. In this case, the size of the specified buffer can be set to 16KB, thereby greatly reducing memory usage.

[0051] Furthermore, if the preset abnormal conditions are met, it indicates that there is an anomaly in the current network environment. In this case, the client can display a preset error message to the user. Such error messages could be, for example, "Network is poor, please try again later," or "Connection failed, please check your network."

[0052] The aforementioned abnormal conditions can be set according to actual needs. For example, if it is determined that a specified buffer has reached its upper limit, it can be considered as meeting a preset abnormal condition. Another example: if it is determined that the cumulative duration of audio data frames stored in a specified buffer exceeds a preset maximum duration threshold, it can be considered as meeting a preset abnormal condition.

[0053] If the session is detected to be established successfully and no preset abnormal conditions are met during the connection process, the audio data frames stored in the specified buffer and the real-time acquired user input audio data frames can be transmitted to the server, as follows: Figure 4 As shown.

[0054] Figure 4 This is a schematic diagram of a transmission process provided in an exemplary embodiment.

[0055] Combination Figure 4It can be seen that the process of the client transmitting the audio data frames stored in the specified buffer and the audio data frames of the user input collected in real time to the server can be divided into two transmission stages: the emptying stage and the pass-through stage. The following will combine steps S202, S204, and... Figure 4 The specific sending logic for these two stages will be explained in detail.

[0056] S202: When the session is detected to be established, the audio data frames stored in the specified buffer are sent to the server at a first sending rate; wherein the first sending rate is greater than the real-time acquisition rate of the audio data frames.

[0057] In this specification, the client can continuously monitor the status of the session establishment process. When the client detects that the session with the server has been established, it can enter the drain phase. During the drain phase, the client can read the previously cached audio data frames from the specified buffer and send them to the server at the first transmission rate.

[0058] Furthermore, as can be seen from the above, the audio data frames stored in the specified buffer can be divided into two types: one is the original data frame collected by the client, and the other is the compressed audio data frame after the original data frame collected by the client has been compressed.

[0059] Therefore, when the client sends audio data frames stored in the specified buffer to the server at the first transmission rate, it can directly encapsulate the compressed audio data frames stored in the specified buffer into transport protocol data packets and send them to the server without further encoding. For the raw audio data frames stored in the specified buffer, encoding is required to obtain their corresponding compressed audio data frames, which are then encapsulated into transport protocol data packets and sent to the server.

[0060] The first transmission rate mentioned above can be used to indicate the unit time and / or the number of audio data frames read from the specified buffer per unit time.

[0061] For example, the first transmission rate mentioned above can indicate the unit time for reading audio data frames from a specified buffer, such as reading once every 20ms or once every 10ms; in addition, the first transmission rate mentioned above can also indicate the number of audio data frames read from a specified buffer per unit time, such as reading two audio data frames from a specified buffer every 10ms.

[0062] In practical applications, since the underlying real-time communication protocols (such as RTC) typically use a fixed packet interval (e.g., 20ms) to encapsulate audio data frames to balance latency and protocol overhead, the unit time for reading audio data frames from the specified buffer can be set to perform a read every 20ms to maintain consistency with the underlying real-time communication protocol.

[0063] Based on this, the first transmission rate can be used to indicate the number of audio data frames read from the specified buffer per unit time, that is, the number of audio data frames read from the specified buffer each time.

[0064] In practical applications, the aforementioned first transmission rate can be determined based on the available transmission bandwidth of the current network environment.

[0065] Specifically, when the client detects that the session has been established, it can obtain the available transmission bandwidth of the current network environment. If the available transmission bandwidth is less than the preset first bandwidth threshold, it indicates that the current network condition is extremely poor and the available bandwidth is insufficient to support the minimum audio data transmission (i.e., it is in the silent zone). At this time, it can pause reading data from the specified buffer, that is, the first transmission rate is 20ms to read 0 frames of audio data from the specified buffer.

[0066] If the available transmission bandwidth is not less than the first bandwidth threshold and less than the preset second bandwidth threshold, it indicates that the current network condition is average, and the available bandwidth can only maintain normal real-time audio transmission. There is not enough spare bandwidth to accelerate the emptying (i.e., it is in the minimum range). At this time, it can be determined that the first transmission rate is equal to the real-time acquisition rate of audio data.

[0067] In addition, it should be emphasized that, as explained above, the audio data frames stored in the specified buffer can be divided into two types: one is the raw data frame captured by the client, and the other is the compressed audio data frame after the raw data frame captured by the client has been compressed.

[0068] Therefore, if the audio data frames stored in the specified buffer are raw audio data frames, and the real-time acquisition rate of the audio data is one audio data frame generated every 10ms, then when the available transmission bandwidth is not less than the first bandwidth threshold and less than the preset second bandwidth threshold, the determined first transmission rate can be to read 2 raw audio data frames from the specified buffer every 20ms.

[0069] If the audio data frame stored in the specified buffer is a compressed audio data frame, and the real-time acquisition rate of the audio data is one audio data frame generated every 10ms, and during the compression process, the client merges the original audio data frames acquired every 2 acquisition cycles and performs Opus encoding as a coding unit to generate one compressed audio data frame, then when the available transmission bandwidth is not less than the first bandwidth threshold and less than the preset second bandwidth threshold, the determined first transmission rate can be one compressed audio data frame read from the specified buffer every 20ms.

[0070] If the available transmission bandwidth is not less than the second bandwidth threshold, then the target transmission rate is determined based on the difference between the available transmission bandwidth and the second bandwidth threshold, and is used as the first transmission rate. The target transmission rate is greater than the real-time acquisition rate and is positively correlated with the difference between the available transmission bandwidth and the second bandwidth threshold, as shown in the following formula:

[0071] In the above formula, That is, the speedup factor relative to the real-time acquisition rate; Indicates available transmission bandwidth. Indicates the second bandwidth threshold This is the maximum reference bandwidth threshold, used to limit the upper limit of bandwidth when calculating the speedup factor relative to the real-time acquisition rate; The maximum speedup factor (e.g., 5.0) is used to prevent excessively high transmission rates from causing network jitter or excessive server processing pressure. That is, the target transmission rate. This is a rounding function used to ensure that integer frames of data are read each time.

[0072] As can be seen from the formula above, once the available transmission bandwidth exceeds the second bandwidth threshold, the acceleration factor increases with the increase in available transmission bandwidth. This means that the smoother the network, the more data frames the client reads from the buffer each time, thus enabling it to clear the audio data frames backed up during session establishment more quickly, minimizing server-side reception latency and improving the perceived smoothness of interaction for the user.

[0073] It should be noted that the first bandwidth threshold mentioned above can be the minimum bandwidth required to maintain the minimum transmission of audio data frames (e.g., the sum of Opus encoding bitrate and RTP / UDP / IP protocol header overhead).

[0074] The aforementioned second bandwidth threshold can be a baseline bandwidth value that supports the normal transmission of real-time audio data frames. When the available transmission bandwidth exceeds this bandwidth threshold, it indicates that the network has spare capacity that can be used to accelerate the emptying of audio data frames stored in the specified buffer.

[0075] In addition, the client can also use a segmented emptying strategy based on a specific bandwidth range to determine the number of audio data frames read from the specified buffer each time, as shown in Table 1.

[0076] Table 1:

[0077] As can be seen from Table 1, as the available transmission bandwidth increases, the acceleration factor gradually increases, resulting in an increase in the number of audio data frames read from the specified buffer in each transmission cycle (20ms), thereby improving the emptying speed.

[0078] For example, when the available transmission bandwidth reaches 300kbps or more, the client can send the buffered audio data frames at a speed 5 times faster than the real-time acquisition rate, thereby quickly clearing the audio data frames backed up during session establishment and significantly reducing cloud reception latency.

[0079] In practical applications, the available network bandwidth is not constant, but fluctuates dynamically with factors such as network environment, signal strength, and background traffic. Therefore, it is necessary to dynamically adjust the first transmission rate according to the changes in available network bandwidth.

[0080] Specifically, the client can use the available transmission bandwidth of the current network environment obtained when the session is established as the initial available transmission bandwidth, determine the first transmission rate based on the initial available transmission bandwidth, and send the audio data frames stored in the specified buffer to the server at the first transmission rate. At the same time, instantaneous bandwidth sampling values ​​are collected at a preset time period, and the current available transmission bandwidth is updated based on the instantaneous bandwidth sampling values. The first transmission rate is then adjusted based on the updated available transmission bandwidth.

[0081] For example, the client can set a preset time period of 100ms. During the drain phase, every 100ms, the client obtains a sample value of the current instantaneous available transmission bandwidth from the underlying network protocol stack (such as the RTC module). Subsequently, the client can directly use this latest obtained instantaneous bandwidth sample value as the current available transmission bandwidth to update the current available transmission bandwidth. Based on the updated current available transmission bandwidth, the client can recalculate the first transmission rate and send audio data frames based on the recalculated first transmission rate.

[0082] It is important to emphasize that although during the emptying phase, audio data frames can be read from the designated buffer and sent to the server every 20ms at the first sending rate, network bandwidth changes are usually slow, and the underlying bandwidth detection mechanism itself has a long update cycle (e.g., 100ms-200ms). The drastic fluctuations in the instantaneous bandwidth value in a short period of time are often noise rather than a true trend. Therefore, the update frequency of the available bandwidth value does not need to be consistent with it, but is lower than the data reading frequency.

[0083] Therefore, the aforementioned preset time period can be longer than the interval period for reading audio data frames from the specified buffer at the first transmission rate. This can accurately reflect the long-term trend of network bandwidth while avoiding the waste of computing resources caused by frequent adjustments to the transmission rate.

[0084] Furthermore, since the instantaneous bandwidth value provided by the underlying bandwidth probing mechanism (such as bandwidth estimation based on feedback from the Real-time Transmission Control Protocol (RTCP)) may have short-term jitter, noise, or sudden changes, directly using the instantaneous value of a single sample as the basis for decision-making may cause frequent and drastic changes in the first transmission rate, which may lead to problems such as audio playback stuttering, network congestion, or out-of-order data packets.

[0085] Therefore, the client can also select at least a portion of the instantaneous bandwidth sample values ​​from each instantaneous bandwidth sample value based on a preset sliding time window, perform weighted fusion on the selected instantaneous bandwidth sample values, determine the fused bandwidth value, and then update the currently available transmission bandwidth based on the fused bandwidth value.

[0086] Among them, the closer the sampling time is to the current time, the greater the corresponding weight of the instantaneous bandwidth sample value.

[0087] For example, the client can set the sliding time window length to 2000ms (i.e., 2 seconds) and the sampling period to 100ms, then the window can contain a maximum of 20 instantaneous bandwidth sample values. When calculating the fused bandwidth value, the client can sort the instantaneous bandwidth sample values ​​in chronological order and select the 20 most recent instantaneous bandwidth sample values, for example, at the current moment, as the selected instantaneous bandwidth sample values. A weight can be assigned to each selected instantaneous bandwidth sample value. Finally, based on the weight of each selected instantaneous bandwidth sample value, the selected instantaneous bandwidth sample values ​​are weighted and fused to obtain the fused bandwidth value.

[0088] S204: After the audio data frames in the specified buffer have been sent, the newly acquired audio data frames are sent to the server in real time at the second sending rate; wherein, the second sending rate is equal to the real-time acquisition rate.

[0089] In this specification, once the client detects that the audio data in the designated buffer has been completely transmitted, it can switch to the pass-through phase. At this time, newly acquired raw audio data frames are no longer stored in the designated buffer, but are sent to the server in real time at the second transmission rate.

[0090] The second transmission rate mentioned above can be equal to the real-time acquisition rate of the audio data frames. For example, as emphasized above, the client can merge the raw audio data frames acquired every two acquisition cycles into a single encoding unit for Opus encoding, generating a compressed audio data frame. In this case, the client performs encoding once for every two raw audio data frames acquired, resulting in a compressed audio data frame, and then sends this compressed audio data frame to the server.

[0091] For ease of understanding, the following is combined with Figure 5 The process of voice interaction using the above method will be explained in detail.

[0092] Figure 5 This is a timing diagram of a voice interaction process provided in an exemplary embodiment.

[0093] Combination Figure 5 As can be seen, the client responds to the user's voice interaction trigger command by concurrently executing two processes: first, it initiates an RTC session establishment request to the server to establish a session with the server; second, it immediately starts the local audio acquisition process. In the audio acquisition process, the audio acquisition module acquires one frame of raw audio data at a fixed acquisition period (e.g., every 10ms) and sends the acquired raw audio data frame to the Opus encoder for encoding processing to generate a smaller compressed audio data frame.

[0094] Because establishing a session requires a certain amount of network transmission time, the encoded compressed audio data frames cannot be sent directly before the session is established. Therefore, the client sequentially writes these compressed audio data frames into a designated buffer for temporary storage. During this process, the client continuously monitors the status of the designated buffer and the session establishment status. If the accumulated audio duration cached in the local audio buffer reaches the preset maximum cache duration threshold, or if there is no remaining storage space in the designated buffer, it indicates that the current network environment is abnormal or the connection has timed out. In this case, the client can stop subsequent processing and display an error message to the user.

[0095] When the client detects that the session has been established, it enters the emptying phase. At this time, the client can determine a first transmission rate based on the available transmission bandwidth of the current network environment, and read the backlogged audio data frames from the designated buffer based on the first transmission rate. Specifically, the client can read the corresponding number of audio data frames from the designated buffer according to a preset transmission period (e.g., every 20ms) and send them to the server through the established session.

[0096] As the emptying process proceeds, once the audio data frames in the designated buffer are completely cleared, the voice interaction process enters the pass-through phase. In this phase, newly acquired and encoded audio data frames no longer wait in the designated buffer but are directly sent to the server in real-time via the session. After receiving the audio data frames from the client, the server performs speech recognition and semantic understanding using the cloud-based AS service, and returns the processing results to the client, which then presents them to the user, thus completing the entire voice interaction process.

[0097] Figure 6 This is a schematic structural diagram of a device provided in an exemplary embodiment. For example... Figure 6 As shown, device 600 mainly consists of a communication interface 602, a user interface 604, a processor 606, and a data storage 608. These components are interconnected and communicate with each other via a system bus, network, or other connection mechanism 610. The communication interface 602 enables device 600 to communicate with other devices, access networks, and transmission networks via analog or digital modulation. For example, the communication interface 602 may include a chipset and antenna for wireless communication with a radio access network or access point. Furthermore, the communication interface 602 can be a wired interface such as Ethernet, Token Ring, or a USB port, or a wireless interface such as Wi-Fi, Bluetooth, Global Positioning System (GPS), or a wide-area wireless interface (e.g., WiMAX or LTE). Of course, the communication interface 602 can also support other forms of physical layer interfaces and standard or proprietary communication protocols. The communication interface 602 may also include multiple physical communication interfaces, such as Wi-Fi, Bluetooth, and wide-area wireless interfaces.

[0098] User interface 604 includes receiving user input and providing output to the user. Therefore, user interface 604 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and video camera, and output components such as a display screen (which may be combined with a touch-sensitive panel), CRT, LCD, LED, display using DLP technology, printer, and other similar devices known or developed in the future. User interface 604 may also generate auditory output via speakers, speaker jacks, audio output ports, audio output devices, headphones, and other similar devices known or developed in the future. In some embodiments, user interface 604 may include software, circuitry, or other forms of logic capable of transmitting and receiving data from external user input / output devices. Additionally or alternatively, device 600 may support remote access from other devices via communication interface 602 or another physical interface (not shown). User interface 604 may be configured to receive user input, the position and movement of which may be indicated by indicators or cursors described herein. User interface 604 may also be configured as a display device for rendering or displaying text fragments.

[0099] Processor 606 may contain one or more general-purpose processors and / or special-purpose processors.

[0100] Data storage 608 may include one or more volatile and / or non-volatile storage components and may be integrated wholly or partially with processor 606. Data storage 608 may include removable and non-removable components.

[0101] Processor 606 is capable of executing program instructions 618 (e.g., compiled or uncompiled program logic and / or machine code) stored in data storage 608 to perform the various functions described herein. Data storage 608 may contain a non-transitory computer-readable medium on which program instructions are stored, which, when executed by device 600, enable device 600 to perform any methods, processes, or functions disclosed in this specification and / or the accompanying drawings. Execution of program instructions 618 by processor 606 may result in processor 606 using data 612.

[0102] For example, program instructions 618 may include an operating system 622 (e.g., an operating system kernel, device drivers, and / or other modules) installed on device 600 and one or more applications 620 (e.g., a browser, social application, or game application). Similarly, data 612 may include operating system data 616 and application data 614. Operating system data 616 is primarily accessible to the operating system 622, while application data 614 is primarily accessible to one or more applications 620. Application data 614 may reside in a file system visible or hidden from the user of device 600.

[0103] Application 620 can communicate with operating system 622 through one or more application programming interfaces (APIs). These APIs help application 620 read and / or write application data 614, transmit or receive information via communication interface 602, receive or display information on user interface 604, etc.

[0104] In some terminology, application 620 may be simply referred to as "app". Furthermore, application 620 can be downloaded to device 600 through one or more online app stores or app markets. However, applications can also be installed on device 600 in other ways, such as through a web browser or a physical interface on device 600 (e.g., a USB port).

[0105] Please refer to Figure 7 Voice interaction devices can be applied to, for example Figure 6 The device shown is used to implement the technical solution of this specification. The voice interaction device may include: The execution module 701 is used to trigger instructions based on the user's voice interaction and concurrently execute the session establishment process with the server and the local audio acquisition process; wherein, in the local audio acquisition process, the acquired audio data frames are stored in a specified buffer. The first sending module 702 is used to send the audio data frames stored in the specified buffer to the server at a first sending rate when the session establishment is detected to be completed; wherein, the first sending rate is greater than the real-time acquisition rate of the audio data frames; The second sending module 703 is used to send newly acquired audio data frames to the server in real time at a second sending rate after the audio data frames in the designated buffer have been sent; wherein the second sending rate is equal to the real-time acquisition rate.

[0106] Optionally, the execution module 701 is specifically used to encode at least a portion of the acquired original audio data frames to obtain compressed audio data frames; and to store the compressed audio data frames and / or the original audio data frames into a specified buffer.

[0107] Optionally, the first sending module 702 is specifically used to encapsulate the compressed audio data frames stored in the designated buffer into transport protocol data packets and send them to the server; and to encode the original audio data frames stored in the designated buffer into transport protocol data packets and send them to the server.

[0108] Optionally, the first sending module 702 is specifically used to: when the session is detected to be established, obtain the available sending bandwidth of the current network environment; determine a first sending rate based on the available sending bandwidth, and send the audio data frames stored in the specified buffer to the server at the first sending rate; the first sending rate is used to indicate the number of compressed audio data frames read from the specified buffer per unit time.

[0109] Optionally, the first sending module 702 is specifically configured to: if the available sending bandwidth is less than a preset first bandwidth threshold, then pause reading data from the designated buffer; if the available sending bandwidth is not less than the first bandwidth threshold and is less than a preset second bandwidth threshold, then determine that the first sending rate is equal to the real-time acquisition rate of the audio data; if the available sending bandwidth is not less than the second bandwidth threshold, then determine a target sending rate as the first sending rate based on the difference between the available sending bandwidth and the second bandwidth threshold; the target sending rate is greater than the real-time acquisition rate, and the target sending rate is positively correlated with the difference.

[0110] Optionally, the first sending module 702 is specifically configured to: when the session is detected to be established, obtain the initial available sending bandwidth of the current network environment; determine a first sending rate based on the initial available sending bandwidth, and send the audio data frames stored in the specified buffer to the server at the first sending rate; and collect instantaneous bandwidth sampling values ​​at a preset time period, update the current available sending bandwidth based on the instantaneous bandwidth sampling values, and adjust the first sending rate based on the updated available sending bandwidth.

[0111] Optionally, the preset time period is greater than the interval period for reading audio data frames from the specified buffer at the first transmission rate.

[0112] Optionally, the first sending module 702 is specifically used to: select at least a portion of the instantaneous bandwidth sample values ​​from each instantaneous bandwidth sample value based on a preset sliding time window; perform weighted fusion on the selected instantaneous bandwidth sample values ​​to determine the fused bandwidth value; wherein, the instantaneous bandwidth sample value whose sampling time is closer to the current time has a larger corresponding weight; and update the currently available sending bandwidth based on the fused bandwidth value.

[0113] Optionally, the designated buffer is a circular buffer; the size of the circular buffer is determined based on the preset maximum session establishment time, the preset network fluctuation protection time, and the real-time acquisition rate.

[0114] For ease of description, the above devices are described by dividing them into various modules or units based on their functions. Of course, when implementing one or more of these specifications, the functions of each module or unit can be implemented in the same or different software and / or hardware, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0115] Based on the same concept as the methods described above, this specification also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor performs the steps of the method as described in any of the above embodiments by executing the executable instructions.

[0116] Based on the same concept as the methods described above, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0117] Based on the same concept as the methods described above, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0118] What those skilled in the art will understand is: In this specification, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded.

[0119] In this specification, “a,” “an,” and “the” do not specifically refer to the singular, but may also include the plural.

[0120] In this specification, ordinal numbers such as "first," "second," etc., do not necessarily indicate order; they are often used to distinguish between objects. For example, "first server" and "second server" usually refer to two servers. To differentiate between these two servers, they are described as "first server" and "second server." Of course, sometimes these two servers may be the same server.

[0121] In this specification, unless explicitly stated otherwise, "receiving and sending data" does not necessarily mean direct receiving and sending; it can also mean indirect receiving and sending. For example, A receiving data sent by B can be understood as A directly receiving the data sent by B, or it can be understood as A indirectly receiving the data sent by B through other entities such as C. Similarly, B sending data to A can be understood as B sending the data directly to A, or it can be understood as B indirectly sending the data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0122] In this specification, unless explicitly stated otherwise, the relationships between structures can be direct or indirect. For example, when describing "A is connected to B," unless it is explicitly stated that A and B are directly connected, it should be understood that A can be directly connected to B or indirectly connected to B. Similarly, when describing "A is on top of B," unless it is explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements, and A is above B). And so on.

[0123] This specification uses specific terms to describe embodiments thereof. Terms such as "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of those different embodiments or examples, without contradiction.

[0124] Although one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is only one of many possible execution orders and does not represent the only execution order. Therefore, when the claims involve method steps, any changes or adjustments to the order of such steps, or the parallelism between steps, are also within the scope of protection of the claims.

Claims

1. A voice interaction method, comprising: Based on the user's voice interaction trigger command, the session establishment process with the server and the local audio acquisition process are executed concurrently; wherein, in the local audio acquisition process, the acquired audio data frames are stored in a designated buffer. When the session is detected to be established, the audio data frames stored in the specified buffer are sent to the server at a first sending rate; wherein, the first sending rate is greater than the real-time acquisition rate of the audio data frames; After the audio data frames in the designated buffer have been sent, the newly acquired audio data frames are sent to the server in real time at a second sending rate; wherein the second sending rate is equal to the real-time acquisition rate.

2. The method as described in claim 1, wherein storing the acquired audio data frames into a designated buffer, specifically includes: At least a portion of the acquired raw audio data frames are encoded to obtain compressed audio data frames. Store the compressed audio data frame and / or the original audio data frame into a designated buffer.

3. The method as described in claim 2, wherein the audio data frames stored in the designated buffer are sent to the server at a first transmission rate, specifically includes: For the compressed audio data frames stored in the specified buffer, they are encapsulated into transport protocol data packets and sent to the server; The original audio data frames stored in the specified buffer are encoded and then encapsulated into transport protocol data packets and sent to the server.

4. The method as described in claim 1, wherein when the session establishment is detected to be complete, the audio data frames stored in the designated buffer are sent to the server at a first sending rate, specifically including: When the session is detected to be established, the available transmission bandwidth of the current network environment is obtained; A first transmission rate is determined based on the available transmission bandwidth, and audio data frames stored in the designated buffer are transmitted to the server at the first transmission rate; the first transmission rate is used to indicate the number of compressed audio data frames read from the designated buffer per unit time.

5. The method of claim 4, wherein determining the first transmission rate based on the available transmission bandwidth specifically includes: If the available transmission bandwidth is less than a preset first bandwidth threshold, then reading data from the specified buffer is paused; If the available transmission bandwidth is not less than the first bandwidth threshold and is less than the preset second bandwidth threshold, then the first transmission rate is determined to be equal to the real-time acquisition rate of the audio data. If the available transmission bandwidth is not less than the second bandwidth threshold, then a target transmission rate is determined based on the difference between the available transmission bandwidth and the second bandwidth threshold, which is used as the first transmission rate; the target transmission rate is greater than the real-time acquisition rate, and the target transmission rate is positively correlated with the difference.

6. The method as described in claim 4, wherein when the session is detected to be established, the available transmission bandwidth of the current network environment is obtained, specifically including: When the session is detected to be established, the initial available transmission bandwidth of the current network environment is obtained; Based on the available transmission bandwidth, a first transmission rate is determined, and the audio data frames stored in the specified buffer are transmitted to the server at the first transmission rate. Specifically, this includes: Based on the initial available transmission bandwidth, a first transmission rate is determined, and the audio data frames stored in the specified buffer are sent to the server at the first transmission rate. as well as Instantaneous bandwidth sampling values ​​are collected at a preset time period, and the available transmission bandwidth is updated based on the instantaneous bandwidth sampling values. The first transmission rate is adjusted based on the updated available transmission bandwidth. The preset time period is greater than the interval period for reading audio data frames from the specified buffer at the first transmission rate.

7. The method as described in claim 6, wherein updating the currently available transmission bandwidth based on the instantaneous bandwidth sample value specifically includes: Based on a preset sliding time window, at least a portion of the instantaneous bandwidth sample values ​​are selected from each instantaneous bandwidth sample value. The selected instantaneous bandwidth sample values ​​are then weighted and fused to determine the fused bandwidth value. Among them, the instantaneous bandwidth sample value whose sampling time is closer to the current time has a larger corresponding weight. Based on the merged bandwidth value, the currently available transmission bandwidth is updated.

8. The method according to any one of claims 1 to 7, wherein the designated buffer is a circular buffer; the size of the circular buffer is determined based on a preset maximum session establishment time, a preset network fluctuation protection time, and the real-time acquisition rate.

9. A voice interaction device, comprising: The execution module is used to trigger commands based on user voice interaction and concurrently execute the session establishment process with the server and the local audio acquisition process; wherein, in the local audio acquisition process, the acquired audio data frames are stored in a specified buffer. A first sending module is configured to send the audio data frames stored in the specified buffer to the server at a first sending rate when the session is detected to be established; wherein the first sending rate is greater than the real-time acquisition rate of the audio data frames. The second sending module is used to send newly acquired audio data frames to the server in real time at a second sending rate after the audio data frames in the designated buffer have been sent. The second sending rate is equal to the real-time acquisition rate.

10. An electronic device, comprising: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method as described in any one of claims 1-8 by executing the executable instructions.

11. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1-8.

12. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1-8.

Images