Audio data processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses an audio data processing method and device, electronic equipment and a storage medium, which can be applied to various scenes such as cloud technology, artificial intelligence, intelligent traffic and auxiliary driving, and the method comprises the following steps: after obtaining playing audio data to be played, storing the obtained playing audio data to a set storage area, playing the obtained playing audio data, and collecting audio data through an audio collection device to obtain collected audio data; and then, according to the sequence of the corresponding storage time from morning to evening, extracting the playing audio data of which the corresponding playing time length is matched with the playing time length of the collected audio data from the storage area, thereby carrying out echo cancellation processing on the collected audio data according to the extracted playing audio data. According to the technical scheme of the embodiment of the invention, the accuracy of echo cancellation processing can be improved, and the quality of audio data is improved.

Description

Technical Field

[0001] The present application relates to the field of computers, and more specifically, to an audio data processing method and device, electronic equipment, storage medium, and program product. Background Art

[0002] When an audio acquisition device (e.g., a microphone) and an audio playback device (e.g., a speaker) are operating simultaneously, the sound played by the audio playback device propagates through the air and enters the audio acquisition device, where it is picked up, resulting in an echo. To ensure that the audio data collected by the audio acquisition device does not contain the sound played by the audio playback device, echo cancellation can be performed on the audio data collected by the audio acquisition device.

[0003] In related technologies, echo cancellation is usually achieved by directly predicting the echo signal and subtracting the audio signal collected by the audio acquisition device from the predicted echo signal. However, a large amount of echo remains in the audio data after being processed by this echo cancellation method, which reduces the accuracy of the echo cancellation processing and the quality of the audio data. Summary of the Invention

[0004] The embodiments of the present application provide an audio data processing method and apparatus, an electronic device, a storage medium, and a program product, which can improve the accuracy of echo cancellation processing and the quality of audio data.

[0005] According to one aspect of an embodiment of the present application, a method for processing audio data is provided, the method comprising:

[0006] Acquire the audio data to be played, and store the acquired audio data in a set storage area;

[0007] Play the acquired playback audio data, and collect audio data through the audio collection device to obtain collected audio data;

[0008] Extracting, from the storage area, the playback audio data whose corresponding playback duration matches the playback duration of the collected audio data in order of the corresponding storage time from earliest to latest;

[0009] According to the extracted playback audio data, echo cancellation processing is performed on the collected audio data.

[0010] According to one aspect of an embodiment of the present application, there is provided an audio data processing apparatus, the apparatus comprising:

[0011] An acquisition module is configured to acquire the audio data to be played and store the acquired audio data in a set storage area;

[0012] A processing module is configured to play the acquired playback audio data and collect audio data through an audio collection device to obtain collected audio data;

[0013] An extraction module configured to extract, from the storage area, the playback audio data whose corresponding playback duration matches the playback duration of the collected audio data in order of the corresponding storage time from earliest to latest;

[0014] The cancellation module is configured to perform echo cancellation processing on the collected audio data according to the extracted playback audio data.

[0015] In an exemplary embodiment, based on the above solution, the apparatus further includes a first deletion module configured to:

[0016] Detecting the total playback duration corresponding to the playback audio data contained in the storage area;

[0017] If the total playback time is greater than or equal to the playback time threshold, extracting the target playback audio data to be deleted from the storage area in order of the corresponding storage time from earliest to latest;

[0018] The target playback audio data is deleted from the storage area.

[0019] In an exemplary embodiment, based on the above solution, the apparatus further includes a computing module configured to:

[0020] Calculate the time difference between the acquisition time and the playback time of the audio data to obtain a first delay;

[0021] The playback duration threshold is calculated according to the first time delay; wherein the playback duration threshold is positively correlated with the first time delay.

[0022] In an exemplary embodiment, based on the above solution, the calculation module is specifically configured as follows:

[0023] Calculating the time difference between the time when the audio data is collected and the time when the echo cancellation processing is performed to obtain a second delay;

[0024] The playback time threshold is calculated according to the total delay of the first delay and the second delay; wherein the playback time threshold is positively correlated with the total delay.

[0025] In an exemplary embodiment, based on the above solution, the apparatus further includes a second deletion module configured to:

[0026] Searching for the earliest stored audio data from the multiple frames of audio data stored in the storage area;

[0027] If the storage time of the playback audio data with the earliest storage time in the storage area is greater than or equal to the storage time threshold, the playback audio data with the earliest storage time is deleted from the storage area.

[0028] In an exemplary embodiment, based on the above solution, the acquisition module is specifically configured as follows:

[0029] Detecting the working status of the audio acquisition device to obtain a status detection result;

[0030] If the status detection result indicates that the audio acquisition device is acquiring audio data, the acquired playback audio data is stored in the storage area.

[0031] In an exemplary embodiment, based on the above solution, the apparatus further includes a generating module configured to:

[0032] If the storage area does not contain the playback audio data, generating silent audio data whose corresponding playback duration matches the playback duration of the collected audio data;

[0033] Perform echo cancellation processing on the collected audio data according to the silent audio data.

[0034] In an exemplary embodiment, based on the above solution, the extraction module is specifically configured as follows:

[0035] If a set number of frames of audio data are collected, extracting the set number of frames of audio data for playback from the storage area in the order of the corresponding storage time from earliest to latest;

[0036] The step of performing echo cancellation processing on the collected audio data according to the extracted playback audio data includes:

[0037] The audio data is played according to the extracted set number of frames, and the echo cancellation processing is performed on the collected audio data of the set number of frames.

[0038] In an exemplary embodiment, based on the above solution, under the condition that the storage area includes a message queue, the extraction module is specifically configured as follows:

[0039] The playback audio data whose corresponding playback duration matches the playback duration of the collected audio data is extracted from the head of the message queue.

[0040] In an exemplary embodiment, based on the aforementioned solution, under the condition that the played audio data includes audio data sent by the external communication terminal during voice communication with the external communication terminal, and the collected audio data includes audio data collected during the voice communication, the apparatus further includes a sending module configured to:

[0041] The collected audio data after the echo cancellation process is sent to the external communication terminal.

[0042] In an exemplary embodiment, based on the above solution, the acquisition module is specifically configured as follows:

[0043] If the storage area does not contain the playback audio data, calculating the storage time corresponding to the acquired playback audio data according to the waiting time and the acquisition time corresponding to the acquired playback audio data; wherein the storage time is later than the acquisition time;

[0044] According to the storage time, the acquired playback audio data is stored in the storage area.

[0045] According to one aspect of an embodiment of the present application, an electronic device is provided, including:

[0046] one or more processors;

[0047] The storage device is used to store one or more computer programs, and when the one or more computer programs are executed by the one or more processors, the electronic device implements the method as described above.

[0048] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor of an electronic device, the electronic device implements the method as described above.

[0049] According to one aspect of an embodiment of the present application, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the method as described above is implemented.

[0050] In the technical solution provided by the embodiment of the present application, after the playback audio data to be played is obtained, the obtained playback audio data is stored in a set storage area, the obtained playback audio data is played, and the audio data is collected by the audio collection device to obtain collected audio data; then, in the order of the corresponding storage time from early to late, the playback audio data whose corresponding playback duration matches the playback duration of the collected audio data is extracted from the storage area, so that the collected audio data is echo-cancelled based on the extracted playback audio data. That is, using the playback audio data as reference data for echo-cancelling the collected audio data can improve the accuracy of the echo cancellation processing. In addition, the playback audio data is first stored in the storage area, and then after the collected audio data is obtained, the playback audio data for echo cancellation is extracted from the storage area, thereby ensuring that there is sufficient playback audio data for echo cancellation. Furthermore, in the process of extracting the playback audio data, it is extracted in order from early to late according to the corresponding storage time, and the playback duration of the extracted playback audio data matches the playback duration of the collected audio data, thereby improving the matching degree between the extracted playback audio data and the collected audio data in time and duration, improving the accuracy of the playback audio data used for echo cancellation, thereby improving the accuracy of the echo cancellation processing, and further improving the quality of the audio data.

[0051] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a schematic diagram of an implementation environment shown in an exemplary embodiment of the present application;

[0053] Figure 2 is a flowchart of an audio data processing method shown in an exemplary embodiment of the present application;

[0054] Figure 3 is a flowchart of an audio data processing method shown in another exemplary embodiment of the present application;

[0055] Figure 4 is a flowchart of an audio data processing method shown in another exemplary embodiment of the present application;

[0056] Figure 5 is a flowchart of an audio data processing method shown in another exemplary embodiment of the present application;

[0057] Figure 6 is a flowchart of an audio data processing method shown in another exemplary embodiment of the present application;

[0058] Figure 7is a flowchart of an audio data processing method shown in another exemplary embodiment of the present application;

[0059] Figure 8 is a flowchart of an audio data processing method shown in another exemplary embodiment of the present application;

[0060] Figure 9 is a flowchart of an audio data processing method shown in another exemplary embodiment of the present application;

[0061] Figure 10 is a flowchart of an audio data processing method shown in another exemplary embodiment of the present application;

[0062] Figure 11 is a flowchart of an audio data processing method shown in another exemplary embodiment of the present application;

[0063] Figure 12 is a flowchart of an audio data processing method shown in another exemplary embodiment of the present application;

[0064] Figure 13 is a schematic diagram of an implementation environment shown in another exemplary embodiment of the present application;

[0065] Figure 14 is a schematic diagram of a data feeding module shown in an exemplary embodiment of the present application;

[0066] Figure 15 is a flowchart of a voice data caching process shown in an exemplary embodiment of the present application;

[0067] Figure 16 This is a block diagram of a data feeding process shown in an exemplary embodiment of the present application;

[0068] Figure 17 This is a flow chart of a data feeding process shown in an exemplary embodiment of the present application;

[0069] Figure 18 is a structural diagram of an audio data processing device shown in an exemplary embodiment of the present application;

[0070] Figure 19 A schematic diagram of the structure of a computer system suitable for implementing an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0071] Example embodiments will now be described in a more complete manner with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to these examples; rather, these embodiments are provided to make this application more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art.

[0072] In addition, the features, structures or characteristics described in the present application may be combined in one or more embodiments in any suitable manner. In the following description, there are many specific details so that the embodiments of the present application can be fully understood. However, it will be appreciated by those skilled in the art that when implementing the technical solution of the present application, it is not necessary to use all the detailed features in the embodiments, one or more specific details may be omitted, or other methods, elements, devices, steps, etc. may be adopted.

[0073] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0074] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0075] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0076] It should also be noted that the term "plurality" used in this application refers to two or more. "And / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0077] The following is a detailed introduction to the technical solutions of the embodiments of this application:

[0078] In related technologies, echo cancellation processing has low accuracy, which reduces the quality of audio data. Based on this, embodiments of the present application provide an audio data processing method and device, electronic device, storage medium, and program product that can improve the accuracy of echo cancellation processing and the quality of audio data.

[0079] See also Figure 1 , Figure 1 1 is a schematic diagram of an implementation environment involved in this application, which includes a terminal device 110 and a server 120. The terminal device 110 and the server 120 communicate with each other via a wired or wireless network, and the terminal device 110 can upload its own data to the server 120 or obtain data from the server 120.

[0080] Among them, the terminal device 110 may include but is not limited to mobile phones, tablets, laptops, computers, voice interaction devices, home appliances, vehicle-mounted terminals, aircraft, remote driving terminals, payment devices (for example, palm-swiping devices, etc.), etc.; the server 120 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. This section does not limit the specific forms of terminal devices and servers.

[0081] It should be noted that Figure 1 The number of terminal devices 110 and servers 120 in the figure is only for reference. Any number of terminal devices 110 and servers 120 may be provided according to actual needs.

[0082] In an exemplary embodiment, the audio data processing method provided by the embodiments of the present application may be executed by the terminal device 110 . Exemplarily, after the terminal device 110 obtains the playback audio data to be played, it can store the obtained playback audio data in a set storage area, play the obtained playback audio data, and collect audio data through the audio collection device to obtain collected audio data; then, according to the corresponding storage time from early to late, the playback audio data whose corresponding playback duration matches the playback duration of the collected audio data is extracted from the storage area, so that the collected audio data is echo-cancelled according to the extracted playback audio data. That is, using the playback audio data as reference data for echo-cancelling the collected audio data can improve the accuracy of the echo cancellation processing. In addition, the playback audio data is first stored in the storage area, and then after the collected audio data is obtained, the playback audio data for echo cancellation is extracted from the storage area, thereby ensuring that there is sufficient playback audio data for echo cancellation, so that the playback audio data whose corresponding playback duration matches the playback duration of the collected audio data can be extracted according to the corresponding storage time from early to late. Compared with directly performing the playback and collection processing according to the obtained playback and collection data after the playback audio data and the collected audio data are obtained, When echo cancellation is performed on the collected audio data using the playback audio data, the playback duration of the acquired playback audio data and the collected audio data may not match, making the alignment of the collected audio data with the playback audio data prone to errors. Furthermore, if the difference between the playback duration of the acquired playback audio data and the playback duration of the playback audio data is greater than a certain threshold, some of the playback audio data will be deleted, resulting in a lack of reference data for the subsequently acquired collected audio data, reducing the accuracy of the echo cancellation solution. (For example, at a certain moment, 7 frames of playback audio data and 3 frames of collected audio data may be acquired. The 3 frames of playback audio data extracted from the 7 frames of playback audio data and used for echo cancellation of the collected audio data may be unrelated to the collected audio data. Moreover, after the remaining 4 frames of playback audio data are deleted, the subsequently acquired collected audio data will lack the playback audio data that serves as reference data for echo cancellation.) The solution provided by the embodiments of the present application can improve the time and duration matching between the extracted playback audio data and the collected audio data, improve the accuracy of the playback audio data used for echo cancellation, thereby improving the accuracy of the echo cancellation processing and, in turn, improving the quality of the audio data.

[0083] In another exemplary embodiment, the server 120 may have similar functions to the terminal device 110, thereby executing the audio data processing method provided in the embodiment of the present application. Exemplarily, after acquiring the playback audio data to be played, the server 120 may store the acquired playback audio data in a set storage area, play the acquired playback audio data, and collect the audio data through an audio collection device to obtain collected audio data; then, in the order of the corresponding storage time from earliest to latest, extract the playback audio data whose corresponding playback duration matches the playback duration of the collected audio data from the storage area, thereby performing echo cancellation processing on the collected audio data based on the extracted playback audio data.

[0084] In another exemplary embodiment, the terminal device 110 and the server 120 can also jointly execute the audio data processing method provided in the embodiment of the present application. Exemplarily, the terminal device 110 can obtain the playback audio data to be played, play the obtained playback audio data, and send the obtained playback audio data to the server 120, so that the server 120 stores the playback audio data from the terminal device 110 in a set storage area; the terminal device 110 can also collect audio data through an audio collection device to obtain collected audio data, and transmit the collected audio data to the server 120. After the server 120 receives the collected audio data from the terminal device 110, it can extract the playback audio data whose corresponding playback duration matches the playback duration of the collected audio data from the storage area in the order of the corresponding storage time from early to late, thereby performing echo cancellation processing on the collected audio data based on the extracted playback audio data.

[0085] The audio data processing method in the embodiments of the present application can be applied to various scenarios requiring echo cancellation, such as voice calls, video calls, live interactions, etc. The embodiments of the present application involve playing audio data, collecting audio data, and other user-related data. When the method of the present application is applied to a specific product or technology, it is necessary to obtain the user's permission or consent, and the extraction, use, and processing of the relevant data comply with local safety standards and local laws and regulations.

[0086] See also Figure 2 , Figure 2 This is a flowchart of an audio data processing method shown in an exemplary embodiment of the present application. This method can be applied to Figure 1 The implementation environment shown, which can be Figure 1 The terminal device 110 in the implementation environment shown in the figure may also be executed by Figure 1 The server 120 in the embodiment shown in the embodiment is executed, or it can also be executed by Figure 1 The terminal device 110 and the server 120 in the illustrated implementation environment execute together.

[0087] like Figure 2 As shown, in an exemplary embodiment, the audio data processing method may include steps S210 to S240, which are described in detail as follows:

[0088] Step S210: Acquire the audio data to be played, and store the acquired audio data in a set storage area.

[0089] It should be noted that playing audio data refers to audio data that needs to be played, and its types include but are not limited to audio data in the voice communication process, music, audio contained in the video, etc., among which voice communication includes but is not limited to voice calls, live broadcast interactions and other communication scenarios.

[0090] The set storage area is used to store the playback audio data for subsequent echo cancellation processing, wherein the specific form of the storage area can be flexibly set according to actual needs, including but not limited to cache, message queue, etc. It should be noted that Acoustic Echo Cancelling (AEC) refers to the elimination of noise generated by the feedback path (feedback path) generated by the audio acquisition device and the audio playback device due to air propagation, that is, in the audio collected by the audio acquisition device, the audio played by the audio playback device is eliminated. Among them, the audio acquisition device is used to collect sounds in the environment, and its types include but are not limited to microphones, etc., and the audio playback device is used to play audio, and its types include but are not limited to speakers, horns, etc.

[0091] During the playback of the acquired audio data, if the audio acquisition device is currently collecting audio data, the audio acquisition device may directly or indirectly collect the acquired audio data. For example, the acquired sound may be indirectly collected after being reflected by the room. As a result, the audio data collected by the audio acquisition device may include not only the sounds in the surrounding environment but also the sounds corresponding to the acquired audio data. In order to remove the sounds corresponding to the acquired audio data from the audio data collected by the audio acquisition device, after acquiring the acquired audio data to be played, the acquired audio data may be stored in a storage area to serve as reference data for subsequent echo cancellation processing.

[0092] Step S220 , playing the acquired audio data, and collecting audio data through an audio collection device to obtain collected audio data.

[0093] After obtaining the audio data to be played, the audio data can be played through an audio playback device such as a speaker. Optionally, the storage and playback of the audio data can be performed simultaneously, or the audio data can be played first and then stored, or the audio data can be stored first and then played.

[0094] Furthermore, the sound of the surrounding environment can be collected by an audio collection device such as a microphone to obtain collected audio data. The playing of the audio data and the collection of the audio data can be started synchronously or asynchronously.

[0095] Step S230 , extracting the playback audio data whose corresponding playback duration matches the playback duration of the collected audio data from the storage area in the order of the corresponding storage time from earliest to latest.

[0096] The storage time refers to the time when the played audio data is stored in the storage area, and can represent the storage order of the played audio data.

[0097] After the collected audio data is acquired, since the collected audio data may contain sounds corresponding to the played audio data, the played audio data may be acquired from the storage area to serve as reference data for recycling and elimination processing. Considering that the earlier the playback audio data is stored in the storage area, the earlier it is played, and thus the earlier it is collected by the audio collection device, the higher the probability that the collected audio data contains the sound corresponding to the playback audio data, and the longer the playback duration of the collected audio data, the longer the playback duration of the playback audio data contained therein. Therefore, the playback audio data is extracted from the storage area in order of the corresponding storage time from early to late until the playback duration of the extracted playback audio data matches the playback duration of the collected audio data. For example, assuming that the storage area contains 3 frames of playback audio data, the storage time of the first frame of playback audio data is earlier than the storage time of the second frame of playback audio data, and the storage time of the second frame of playback audio data is earlier than the storage time of the third frame of playback audio data, that is, the first frame of playback audio data is stored first, then the second frame of playback audio data is stored, and then the third frame of playback audio data is stored. If 1 frame of collected audio data is collected, the first frame of playback audio data is extracted from the storage area. If 2 frames of collected audio data are collected, the first and second frames of playback audio data are extracted from the storage area. Here, one frame of audio data refers to audio data of a short period of time (ie, unit playback duration), wherein the unit playback duration may be 10 milliseconds, 20 milliseconds, etc.

[0098] Step S240: performing echo cancellation processing on the collected audio data according to the extracted played audio data.

[0099] After extracting the playback audio data from the storage area, echo cancellation processing can be performed on the collected audio data based on the extracted playback audio data. The specific method of the echo cancellation processing can be flexibly set according to actual needs. In an optional embodiment, the size of the echo signal contained in the collected audio data can be predicted based on the extracted playback audio data, and the echo signal can be subtracted from the audio signal corresponding to the collected audio data to obtain the collected audio data after echo cancellation processing.

[0100] exist Figure 2 In the illustrated embodiment, the playback audio data is used as reference data for echo cancellation processing of the collected audio data, which can improve the accuracy of the echo cancellation processing. In addition, the playback audio data is first stored in a storage area, and then after the collected audio data is obtained, the playback audio data for echo cancellation is extracted from the storage area, thereby ensuring that there is sufficient playback audio data for echo cancellation. Furthermore, in the process of extracting the playback audio data, it is extracted in order from early to late according to the corresponding storage time, and the playback duration of the extracted playback audio data matches the playback duration of the collected audio data, thereby improving the matching degree between the extracted playback audio data and the collected audio data in time and duration, improving the accuracy of the playback audio data used for echo cancellation, thereby improving the accuracy of the echo cancellation processing, and further improving the quality of the audio data.

[0101] In an exemplary embodiment, see Figure 3 , Figure 3 is Figure 2 The flowchart of an audio data processing method is shown in FIG. Figure 1 The implementation environment shown, which can be Figure 1 The terminal device 110 in the implementation environment shown in the figure may also be executed by Figure 1 The server 120 in the embodiment shown in the embodiment is executed, or it can also be executed by Figure 1 The terminal device 110 and the server 120 in the illustrated implementation environment execute together.

[0102] like Figure 3 As shown, the audio data processing method further includes steps S310 to S330, which are described in detail as follows:

[0103] Step S310: Detect the total playing time corresponding to the playing audio data contained in the storage area.

[0104] The storage area contains playback audio data added to the storage area at different storage times, and the total playback time of the playback audio data contained in the storage area can be detected in real time or periodically, wherein the total playback time can be calculated based on the number of frames of the playback audio data contained in the storage area and the unit playback time corresponding to one frame of audio data. For example, assuming that the storage area contains 10 frames of playback audio data and the playback time of one frame of audio data is 10 milliseconds, the total playback time is 100 milliseconds.

[0105] Step S320: If the total playback time is greater than or equal to the playback time threshold, the target playback audio data to be deleted is extracted from the storage area in the order of the corresponding storage time from earliest to latest.

[0106] It should be noted that, if the total playback time corresponding to the playback audio data contained in the storage area is too long, it indicates that the time difference between the playback time of some playback audio data in the storage area and the current time is too long, and the probability that the playback audio data collected subsequently contains the playback audio data is low. For example, assuming that the total playback time of the playback audio data contained in the storage area is greater than 1 minute, it indicates that some playback audio data was played 1 minute ago, and the probability that the subsequently collected audio data contains the playback audio data played 1 minute ago is low. Therefore, in order to save storage resources and improve the accuracy of echo cancellation, a playback time threshold is set to delete useless playback audio data in the storage area. If it is detected that the total playback time is greater than or equal to the playback time threshold, the target playback audio data is extracted from the storage area in order of the corresponding storage time from early to late, wherein the target playback audio data refers to the playback audio data that needs to be deleted.

[0107] Among them, the specific value of the playback time threshold can be flexibly set according to actual needs. In an optional example, it can be set to 30 seconds, 20 seconds, etc. based on experience; in another optional example, there will be a time difference between obtaining the playback audio data and playing the playback audio data, that is, there is a playback link delay, which is recorded as the first delay. If the total playback time exceeds the first delay, it indicates that the time difference between the playback time of part of the playback audio data in the storage area and the current time exceeds the first delay, and the probability of the subsequently collected audio data containing the playback audio data is small. Therefore, the playback time threshold can be calculated based on the first delay, wherein the playback time threshold is positively correlated with the first delay, that is, the larger the first delay, the larger the playback time threshold, wherein the first delay can be directly Alternatively, the product of the first delay and the set coefficient may be used as the playback delay threshold, and the set coefficient may be a value greater than 1; in another optional example, there may be a time difference between collecting the playback audio data and recycling and eliminating the playback audio data, that is, the storage acquisition link delay, recorded as the second delay. If the total playback time exceeds the second delay, it indicates that the time difference between the playback time of part of the playback audio data in the storage area and the current time exceeds the second delay, and the probability that the subsequently collected audio data contains the playback audio data is small. Therefore, the playback time threshold can be calculated based on the second delay, wherein the second delay can be directly used as the playback time threshold, or the product of the second delay and the set coefficient can be used as the playback delay threshold. In another optional example, a playback duration threshold may be determined based on the first delay and the second delay, wherein the playback duration threshold is positively correlated with the first delay and positively correlated with the second delay. The playback duration threshold may be calculated based on the total delay of the first delay and the second delay. For example, the total delay may be directly used as the playback duration threshold, or the product of the total delay and a set coefficient may be used as the playback duration threshold. Thus, in this embodiment, the playback duration threshold may be determined based on at least one of the first delay and the second delay.

[0108] The specific method of extracting the target playback audio data can be flexibly set according to actual needs. In an optional example, the playback audio data can be extracted from the storage area in sequence according to the corresponding storage time from early to late, until the playback duration of the extracted playback audio data matches the target playback duration, and the extracted playback audio data is used as the target playback audio data, wherein the target playback duration = total playback duration - playback duration threshold, that is, the playback duration of the target playback audio data = total playback duration - playback duration threshold, so that after the target playback audio data is deleted, the total playback duration of the playback audio data contained in the storage area matches the playback duration threshold; in another optional example, a specified number of frames of playback audio data can be extracted from the storage area in sequence according to the corresponding storage time from early to late as the target playback audio data, wherein the specified number can be 1, 2, 3, etc.

[0109] Step S330: Delete the target playback audio data from the storage area.

[0110] After the target playback audio data is found from the storage area, the target playback audio data is deleted from the storage area so that the target playback audio data is not used for echo cancellation processing of subsequent collected audio data.

[0111] It should be noted that Figure 3 The specific implementation details of steps S210 to S240 shown in FIG. Figure 2 Steps S210 to S240 are shown and will not be described again here.

[0112] exist Figure 3 In the embodiment shown, if the total playback time corresponding to the playback audio data contained in the storage area is greater than or equal to the playback time threshold, the target playback audio data to be deleted is extracted from the storage area in order of the corresponding storage time from early to late, and the target playback audio data is deleted from the storage area, which can save storage resources and improve the accuracy of echo cancellation processing.

[0113] In an exemplary embodiment, see Figure 4 , Figure 4 is Figure 3 The flowchart of an audio data processing method is shown in FIG. Figure 1 The implementation environment shown, which can be Figure 1 The terminal device 110 in the implementation environment shown in the figure may also be executed by Figure 1 The server 120 in the embodiment shown in the embodiment is executed, or it can also be executed by Figure 1 The terminal device 110 and the server 120 in the illustrated implementation environment execute together.

[0114] like Figure 4 As shown, the audio data processing method may further include steps S410 to S420, which are described in detail as follows:

[0115] Step S410: Calculate the time difference between the acquisition time and the playback time of the audio data to obtain a first delay.

[0116] The first delay refers to the playback link delay, that is, the delay from acquiring the playback audio data to playing the playback audio data, which can represent the time difference between the acquisition time and the playback time of the playback audio data.

[0117] Among them, the specific method of calculating the first delay can be flexibly set according to actual needs. When calculating the first delay, it is considered that the audio data is played by an audio playback device, and the audio playback device is located in the playback link. The audio playback device will affect the first delay. Therefore, the first delay can be calculated based on the attribute parameters of the audio playback device; in another optional embodiment, the playback audio data can be marked with an acquisition timestamp and a playback timestamp, and the difference between the acquisition timestamp and the playback timestamp is calculated to obtain the first delay.

[0118] Optionally, in order to improve accuracy, the first delay may be recalculated and updated every set period.

[0119] Step S420: Calculate a playback duration threshold according to the first time delay; wherein the playback duration threshold is positively correlated with the first time delay.

[0120] After determining the first delay, a playback delay threshold may be calculated, wherein the longer the first delay is, the longer the playback delay threshold is, and the shorter the first delay is, the shorter the playback delay threshold is.

[0121] Optionally, the first delay can be directly used as the playback delay threshold, or the product of the first delay and the set coefficient can be used as the playback delay threshold.

[0122] It should be noted that Figure 4 The specific implementation details of steps S210 to S240 shown in FIG. Figure 2 Steps S210 to S240 are shown, Figure 4 The specific implementation details of steps S310 to S330 shown in FIG. Figure 3 Steps S310 to S330 are shown and will not be described again here.

[0123] exist Figure 4 In the illustrated embodiment, the playback duration threshold is determined based on the time difference between the acquisition time and the playback time of the played audio data, which can improve the accuracy of the echo cancellation process.

[0124] In an exemplary embodiment, see Figure 5 , Figure 5 is Figure 4 The flowchart of an audio data processing method is shown in FIG. Figure 1 The implementation environment shown, which can be Figure 1 The terminal device 110 in the implementation environment shown in the figure may also be executed by Figure 1 The server 120 in the embodiment shown in the embodiment is executed, or it can also be executed by Figure 1 The terminal device 110 and the server 120 in the illustrated implementation environment execute together.

[0125] like Figure 5 As shown, step S420 may include steps S510 to S520, which are described in detail as follows:

[0126] Step S510: Calculate the time difference between the time of collecting the audio data and the time of performing the echo cancellation process to obtain a second delay.

[0127] The second delay refers to the acquisition link delay, that is, the delay from the acquisition of the acquired audio data to the echo cancellation processing of the played audio data, which can represent the time difference between the acquisition time of the acquired audio data and the time of echo cancellation processing.

[0128] Among them, the specific method of calculating the second delay can be flexibly set according to actual needs. When calculating the second delay, it is considered that the collected audio data is collected by the audio collection device, and the audio collection device is located in the collection link. The audio collection device will affect the second delay. Therefore, the second delay can be calculated based on the attribute parameters of the audio collection device; in another optional embodiment, the collected audio data can be marked with an acquisition timestamp and a processing timestamp (that is, the time for echo cancellation processing), and the difference between the acquisition timestamp and the processing timestamp is calculated to obtain the second delay.

[0129] Optionally, in order to improve accuracy, the second delay may be recalculated and updated every set period.

[0130] Step S520: Calculate a playback duration threshold according to the total delay of the first delay and the second delay; wherein the playback duration threshold is positively correlated with the total delay.

[0131] If the total playback time corresponding to the playback audio data contained in the storage area exceeds the total delay of the first delay and the second delay (i.e., the sum of the first delay and the second delay), it indicates that the time difference between the playback time of some of the playback audio data in the storage area and the current time exceeds the total delay, and the probability that the subsequently collected audio data contains the playback audio data is low. For example, assuming that the first delay is 3 seconds, the second delay is 5 seconds, and the current time is 11:00:30, due to the existence of the second delay, the subsequent collected audio data that needs to be echo canceled is collected after 11:00:25. In the example above, the collected audio data will only include the audio data played after 11:00:25. Due to the first delay, the collected audio data will only include the audio data played after 11:00:22. That is, the time difference between the acquisition time of the useful audio data and the current time is less than or equal to 8 seconds (i.e., the total delay). Therefore, if the total playback time of the audio data contained in the storage area exceeds 8 seconds, it indicates that the time difference between the acquisition time of some audio data and the current time is greater than 8 seconds, and the storage area contains useless audio data. Therefore, after calculating the first delay and the second delay, the playback time threshold can be calculated based on the total delay of the first delay and the second delay, wherein the longer the total delay, the larger the playback time threshold, and the smaller the total delay, the smaller the playback time threshold.

[0132] Optionally, the total delay may be directly used as the playback duration threshold, or the product of the total delay and a set coefficient may be used as the playback delay threshold.

[0133] It should be noted that Figure 5 The specific implementation details of steps S210 to S240 shown in FIG. Figure 2 Steps S210 to S240 are shown, Figure 5 The specific implementation details of steps S310 to S330 shown in FIG. Figure 3 Steps S310 to S330 are shown. Figure 5 The specific implementation details of step S410 can be referred to Figure 4 Step S410 shown is not repeated here.

[0134] exist Figure 5 In the illustrated embodiment, the playback duration threshold is determined based on the time difference between the acquisition time and the playback time of the played audio data, and the time difference between the acquisition time of the collected audio data and the time of performing the echo cancellation processing, which can improve the accuracy of the echo cancellation processing.

[0135] In an exemplary embodiment, see Figure 6 , Figure 6 is Figure 2The flowchart of an audio data processing method is shown in FIG. Figure 1 The implementation environment shown, which can be Figure 1 The terminal device 110 in the implementation environment shown in the figure may also be executed by Figure 1 The server 120 in the embodiment shown in the embodiment is executed, or it can also be executed by Figure 1 The terminal device 110 and the server 120 in the illustrated implementation environment execute together.

[0136] like Figure 6 As shown, the audio data processing method further includes steps S610 to S620, which are described in detail as follows:

[0137] Step S610: searching for the earliest stored audio data from the multiple frames of audio data stored in the storage area.

[0138] The storage area contains multiple frames of playback audio data, and the frame of playback audio data with the earliest storage time can be searched therefrom, that is, the playback audio data stored earliest in the storage area among the multiple frames of playback audio data contained in the storage area. For example, assuming that the storage area contains 3 frames of playback audio data, which are stored in the storage area at 11:00:01, 11:00:02, and 11:00:03 respectively, then the playback audio data stored in the storage area at 11:00:01 is the playback audio data with the earliest storage time.

[0139] Step S620: If the storage duration of the audio data with the earliest storage time in the storage area is greater than or equal to the storage duration threshold, the audio data with the earliest storage time is deleted from the storage area.

[0140] The storage duration of the played audio data in the storage area represents the length of time from the time the played audio data is added to the storage area to the current time. If the storage duration of the played audio data is too long, it means that the time difference between the playback time of the played audio data and the current time is too long. The probability that the collected audio data that needs to be echo eliminated later contains the played audio data is low. Therefore, a storage duration threshold is set. If the storage duration of the earliest played audio data in the storage area is greater than or equal to the storage duration threshold, the played audio data is deleted.

[0141] The specific value of the storage duration threshold can be flexibly set according to actual needs. For example, it can be set to 10 milliseconds, 20 milliseconds, etc. based on experience; or it can be determined based on at least one of the first delay and the second delay. The specific method of determining the storage duration threshold based on at least one of the first delay and the second delay is similar to the method of determining the playback duration threshold based on at least one of the first delay and the second delay, and will not be repeated here. In an optional example, the storage duration threshold is the same as the playback duration threshold.

[0142] It should be noted that Figure 6 The specific implementation details of steps S210 to S240 shown in FIG. Figure 2 Steps S210 to S240 are shown and will not be described again here.

[0143] exist Figure 6 In the illustrated embodiment, the corresponding played audio data with a longer storage duration is deleted from the storage area, which can save storage resources and improve the accuracy of echo cancellation.

[0144] In an exemplary embodiment, see Figure 7 , Figure 7 is Figure 2 The flowchart of an audio data processing method is shown in FIG. Figure 1 The implementation environment shown, which can be Figure 1 The terminal device 110 in the implementation environment shown in the figure may also be executed by Figure 1 The server 120 in the embodiment shown in the embodiment is executed, or it can also be executed by Figure 1 The terminal device 110 and the server 120 in the illustrated implementation environment execute together.

[0145] like Figure 7 As shown, step S210 may include steps S710 to S730, which are described in detail as follows:

[0146] Step S710: Acquire audio data to be played.

[0147] It should be noted that the type of playback audio data is different, and the method of obtaining the playback audio data is different. For example, if the playback audio data is music stored locally, the playback audio data is searched from the local storage. If the playback audio data is audio data collected by the target communication terminal during voice communication with the target communication terminal, the playback audio data from the target communication terminal is received during the voice communication process.

[0148] Step S720: Detect the working status of the audio acquisition device and obtain a status detection result.

[0149] If the audio acquisition device does not collect audio data during the playback of the audio data, there is no need to store the played audio data in the storage area. Therefore, after obtaining the played audio data, it is necessary to detect the working status of the audio acquisition device to obtain a status detection result indicating whether the audio acquisition device is collecting audio data.

[0150] Step S730: If the status detection result indicates that the audio acquisition device is acquiring audio data, the acquired playback audio data is stored in the storage area.

[0151] If the status detection result indicates that the audio acquisition device is collecting audio data, it indicates that in the subsequent playback process of the playback audio data, the collected audio data collected by the audio acquisition device may include the playback audio data, so the playback audio data is stored in the storage area.

[0152] Optionally, since there is a first delay in the playback of the playback audio data, there is a high probability that the collected audio data collected within a period of time after the playback audio data is obtained contains the playback audio data. For example, assuming that the first delay is 9 seconds and the time of obtaining the playback audio data to be played is 12:00:00, the playback time of the playback audio data is 12:00:09, and the collected audio data collected at 12:00:09 will contain the playback audio data. Therefore, if the status detection result indicates that the audio collection device is not currently collecting audio data, it can be detected whether the audio collection device collects audio data within a period of time starting from the acquisition time of the playback audio data and lasting for a set time. If so, the playback audio data is stored in the storage area; if not, the playback audio data is not stored in the storage area. The set time can be positively correlated with the first delay. The first delay can be directly used as the set time, or the sum of the first delay and the obtained playback time of the playback audio data can be used as the set time.

[0153] It should be noted that Figure 7 The specific implementation details of steps S220 to S240 shown in FIG. Figure 2 Steps S220 to S240 shown are not repeated here.

[0154] exist Figure 7 In the illustrated embodiment, the acquired audio data to be played is stored in the storage area only when the audio acquisition device is acquiring audio data, which not only saves storage resources but also improves the accuracy of echo cancellation processing.

[0155] In an exemplary embodiment, see Figure 8 , Figure 8 is Figure 2The flowchart of an audio data processing method is shown in FIG. Figure 1 The implementation environment shown, which can be Figure 1 The terminal device 110 in the implementation environment shown in the figure may also be executed by Figure 1 The server 120 in the embodiment shown in the embodiment is executed, or it can also be executed by Figure 1 The terminal device 110 and the server 120 in the illustrated implementation environment execute together.

[0156] like Figure 8 As shown, after the audio data is collected by the audio collection device and the collected audio data is obtained, steps S810 to S820 may be further included, which are described in detail as follows:

[0157] Step S810: If the storage area does not contain the playback audio data, generate silent audio data whose corresponding playback duration matches the playback duration of the collected audio data.

[0158] After obtaining the collected audio data, if the storage area is empty, that is, the storage area does not contain the played audio data, in order to adapt the echo cancellation processing process, silent audio data can be generated whose playback duration matches the playback duration of the collected audio data. In other words, the silent audio data does not contain any sound, and the playback duration of the silent audio data matches the playback duration of the collected audio data.

[0159] Step S820: performing echo cancellation processing on the collected audio data according to the silent audio data.

[0160] Using silent audio data as reference data, echo cancellation processing is performed on the collected audio data.

[0161] It should be noted that Figure 8 The specific implementation details of steps S210 to S240 shown in FIG. Figure 2 Steps S210 to S240 are shown and will not be described again here.

[0162] exist Figure 8 In the embodiment shown, after obtaining the collected audio data, if the storage area does not contain the played audio data, silent audio data with a corresponding playback duration that matches the playback duration of the collected audio data is generated as reference data for echo cancellation processing of the collected audio data, thereby eliminating the need to change the echo cancellation processing process and saving processing resources.

[0163] In an exemplary embodiment, see Figure 9 , Figure 9 is Figure 2 The flowchart of an audio data processing method is shown in FIG. Figure 1 The implementation environment shown, which can be Figure 1 The terminal device 110 in the implementation environment shown in the figure may also be executed by Figure 1 The server 120 in the embodiment shown in the embodiment is executed, or it can also be executed by Figure 1 The terminal device 110 and the server 120 in the illustrated implementation environment execute together.

[0164] like Figure 9 As shown, step S230 may include step S910, which is described in detail as follows:

[0165] Step S910: If a set number of frames of audio data are collected, the set number of frames of audio data are extracted from the storage area in the order of the corresponding storage time from earliest to latest.

[0166] In order to improve the real-time performance of playing audio data, a set number can be set, where the set number is an integer greater than or equal to 1, for example, it can be set to 1, 2, 3, etc., and its specific value can be negatively correlated with the real-time performance requirement of playing audio data. That is, if the real-time performance requirement of playing audio data is high, the set number is lower; if the real-time performance requirement of playing audio data is low, the set number can be higher.

[0167] In the process of collecting audio data through the audio collection device, if a set number of frames of collected audio data are collected, the echo cancellation processing flow of the collected audio data can be triggered. During the echo cancellation processing, multiple frames of playback audio data matching the set number are first extracted from the storage area in the order of the corresponding storage time from earliest to latest. That is to say, the set number of frames of playback audio data are extracted. For example, assuming the set number is 1, after collecting one frame of collected audio data, the frame of playback audio data with the earliest storage time can be extracted from the storage area.

[0168] Under this condition, step S240 may include step S920, which is described in detail as follows:

[0169] Step S920 , playing audio data according to the extracted set number of frames, and performing echo cancellation processing on the collected audio data of the set number of frames.

[0170] After extracting a set number of frames of playback audio data, echo cancellation processing is performed on the set number of frames of collected audio data according to the set number of frames of playback audio data. That is, the amount of playback audio data and collected audio data subjected to echo cancellation processing each time is the set number of frames, thereby avoiding the accumulation of playback audio data that needs to be processed by echo cancellation, reducing the efficiency of echo cancellation processing and the real-time performance of playback audio data.

[0171] It should be noted that Figure 9 The specific implementation details of steps S210 to S220 shown in FIG. Figure 2 Steps S210 to S220 shown are not repeated here.

[0172] exist Figure 9 In the illustrated embodiment, the amount of audio data played and audio data collected for each echo cancellation process is a set number of frames, thereby avoiding accumulation of audio data that requires echo cancellation processing, improving the efficiency of echo cancellation processing and the real-time performance of audio data played.

[0173] In an exemplary embodiment, see Figure 10 , Figure 10 is Figure 2 The flowchart of an audio data processing method is shown in FIG. Figure 1 The implementation environment shown, which can be Figure 1 The terminal device 110 in the implementation environment shown in the figure may also be executed by Figure 1 The server 120 in the embodiment shown in the embodiment is executed, or it can also be executed by Figure 1 The terminal device 110 and the server 120 in the illustrated implementation environment execute together.

[0174] like Figure 10 As shown, under the condition that the storage area includes a message queue, step S230 may include step S1010, which is described in detail as follows:

[0175] Step S1010: extracting, from the head of the message queue, the playback audio data whose corresponding playback duration matches the playback duration of the collected audio data.

[0176] It should be noted that the message queue adopts a first-in-first-out mode, which only allows data to be inserted from the end of the queue and extracted from the head of the queue. Correspondingly, under the condition that the storage area contains a message queue, the playback audio data is stored in the message queue, and the storage time of the playback audio data at the head of the queue is earlier than the storage time of the playback audio data at the end of the queue. Therefore, after obtaining the collected audio data, in order to extract the playback audio data with an earlier storage time, the playback audio data can be extracted in sequence starting from the head of the message queue until the playback duration of the extracted playback audio data matches the playback duration of the collected audio data, thereby realizing the extraction of playback audio data from the storage area in the order of the corresponding storage time from early to late.

[0177] It should be noted that Figure 10 The specific implementation details of steps S210-S220 and S240 can be referred to Figure 2Steps S210 to S220 and step S240 are shown and will not be described in detail here.

[0178] exist Figure 10 In the illustrated embodiment, storing the played audio data in the message queue can improve the efficiency of extracting the played audio data from the message queue.

[0179] In an exemplary embodiment, see Figure 11 , Figure 11 is Figure 2 The flowchart of an audio data processing method is shown in FIG. Figure 1 The implementation environment shown, which can be Figure 1 The terminal device 110 in the implementation environment shown in the figure may also be executed by Figure 1 The server 110 in the embodiment shown in the embodiment is executed, or it can be executed by Figure 1 The terminal device 110 and the server 110 in the illustrated implementation environment execute together.

[0180] like Figure 11 As shown, under the condition that the playback audio data includes audio data sent by the external communication terminal during voice communication with the external communication terminal, and the collected audio data includes audio data collected during the voice communication, after performing echo cancellation processing on the collected audio data based on the extracted playback audio data, the audio data processing method may further include step S1110, which is described in detail as follows:

[0181] Step S1110 , sending the collected audio data after the echo cancellation processing to an external communication terminal.

[0182] It should be noted that the external communication terminal refers to a communication terminal outside the local terminal, which can conduct voice communication with the local terminal, and its types include but are not limited to terminal devices, servers, etc. For the specific introduction of terminal devices and servers, please refer to the above records and will not be repeated here. In the process of voice communication with the external communication terminal, the audio acquisition device of the local terminal will play the sound from the external communication terminal during the process of collecting sound, so that the sound collected by the local terminal contains the sound from the external communication terminal. If the sound collected by the local terminal is directly sent to the external communication terminal for playback, it will cause the user of the external communication terminal to hear his own voice, that is, echo. Among them, voice communication includes but is not limited to instant voice communication and non-instant voice communication, among which instant voice communication includes but is not limited to voice calls, video calls, live broadcasts, etc.

[0183] The audio data processing method of this embodiment can be applied to voice communication scenarios to eliminate echoes in common voice scenarios. In a voice communication scenario, the played audio data includes audio data sent by an external communication terminal during the voice communication process, which may be audio data collected by an audio collection device corresponding to the external communication terminal; the collected audio data includes audio data collected by the audio collection device of the local terminal during the voice communication process, which needs to be sent to the external communication terminal. Therefore, after performing echo cancellation processing on the collected audio data, the collected audio data after echo cancellation processing can be sent to the external communication terminal.

[0184] Optionally, the played audio data may further include other audio data played by the local end during the voice communication process, except for the audio data from the external communication end.

[0185] It should be noted that Figure 11 The specific implementation details of steps S210 to S240 shown in FIG. Figure 2 Steps S210 to S240 are shown and will not be described again here. In addition to being applied to voice communication scenarios, the audio data processing method can also be applied to other scenarios that require echo cancellation. For example, if a user records their own voice while playing music, the recording contains not only the user's voice but also the music being played, resulting in a relatively unclear voice in the recording. Therefore, the played music can be used as the played audio data and the recording as the collected audio data. Echo cancellation can be performed using the audio data processing method, thereby removing the music contained in the recording and improving the clarity of the user's voice in the recording.

[0186] exist Figure 11 In the illustrated embodiment, the audio data processing method is used to perform echo cancellation processing on the audio data that needs to be sent to the external communication terminal during the voice communication process, thereby improving the voice communication quality.

[0187] In an exemplary embodiment, see Figure 12 , Figure 12 is Figure 2 The flowchart of an audio data processing method is shown in FIG. Figure 1 The implementation environment shown, which can be Figure 1 The terminal device 110 in the implementation environment shown in the figure may also be executed by Figure 1 The server 120 in the embodiment shown in the embodiment is executed, or it can also be executed by Figure 1 The terminal device 110 and the server 120 in the illustrated implementation environment execute together.

[0188] like Figure 12As shown, step S210 may include steps S1210 to S1230, which are described in detail as follows:

[0189] Step S1210: Acquire audio data to be played.

[0190] The specific method for obtaining the played audio data can be found in the description of the aforementioned embodiment and will not be repeated here.

[0191] Step S1220: If the storage area does not contain the playback audio data, the storage time corresponding to the acquired playback audio data is calculated according to the waiting time and the acquisition time corresponding to the acquired playback audio data; wherein the storage time is later than the acquisition time.

[0192] After obtaining the playback audio data to be played, if the storage area does not contain the playback audio data, if the playback audio data is immediately added to the storage area, the playback audio data will be used for echo cancellation processing of the currently obtained collected audio data. Due to the existence of the first delay and the second delay, the probability that the currently obtained collected audio data contains the playback audio data is small. For example, assuming that the first delay is 1 second and the second delay is 2 seconds, the storage area does not contain the playback audio data, and the playback audio data is obtained at 11:30:20, then the playback audio data will be played at 11:30:21, and the collected audio data that needs to be echo cancelled at 11:30:20 is collected at 11:30:18. If the playback audio data is immediately added to the storage area, the playback audio data played at 11:30:21 will be used for echo cancellation of the collected audio data collected at 11:30:18, so that the playback audio data and the collected audio data are not aligned, which reduces the echo cancellation effect. Therefore, in order to avoid this situation, a waiting time can be set. After obtaining the audio data to be played, it can be checked whether the storage area contains the audio data. If not, the storage time of the audio data can be calculated based on the waiting time and the acquisition time of the audio data, where storage time = acquisition time + waiting time.

[0193] Among them, the waiting period can be determined based on at least one of the first delay and the second delay. The waiting period can be positively correlated with the first delay, positively correlated with the second delay, or positively correlated with the total delay of the first delay and the second delay. For example, any one of the first delay, the second delay, and the total delay (the sum of the first delay and the second delay) can be used as the waiting period.

[0194] Step S1230: storing the acquired playback audio data in a storage area according to the storage time.

[0195] According to the calculated storage time, the obtained playback audio data is stored in the storage area. That is to say, after obtaining the playback audio data to be played, if the storage area does not contain the playback audio data, the playback audio data can be added to the storage area after waiting for a certain period of time.

[0196] Optionally, after the audio data to be played is acquired, if the storage area contains the audio data, the audio data can be directly added to the storage area.

[0197] It should be noted that Figure 12 The specific implementation details of steps S220 to S240 shown in FIG. Figure 2 Steps S220 to S240 shown are not repeated here.

[0198] exist Figure 12 In the embodiment shown, after obtaining the playback audio data to be played, if the storage area does not contain the playback audio data, the playback audio data is added to the storage area after waiting for a certain period of time, so that the playback audio data is aligned with the collected audio data, thereby improving the accuracy of the echo cancellation processing.

[0199] In order to better understand the present invention, the audio data processing method is applied to the instant voice communication scenario as an example for explanation. Figure 13 , Figure 13This is a schematic diagram of an implementation environment involved in the present application, which includes a first terminal device and a second terminal device. The audio data processing method is executed by the first terminal device. The second terminal device is an external terminal device for instant voice communication with the first terminal device. The first terminal device includes a speaker, a microphone, a data feeding module, and an echo cancellation module. During the instant voice communication with the second terminal device, the first terminal device will receive external voice data from the second terminal device, wherein the external voice data refers to the voice data collected by the second terminal device during the instant voice communication. After receiving the external voice data, the first terminal device can play the external voice data through the speaker contained in itself, that is, The external voice data is the audio data played by the first terminal device, and the first terminal device can send the external voice data to the data feeding module. At the same time, during the instant voice communication, the microphone of the first terminal device will also collect the user's voice to obtain local voice data. Since the speaker is playing the external voice data during the collection of the user's voice, the local voice data may contain external voice data. The local voice data can be transmitted to the data feeding module. The data feeding module feeds the local voice data and the external voice data to the echo cancellation module. The echo cancellation module performs echo cancellation processing on the local voice data according to the fed external voice data to obtain the outgoing voice data, and sends the outgoing voice data to the second terminal device.

[0200] See also Figure 14 As shown, the data feeding module includes a message queue and a data feeding decision module, wherein the message queue is used to cache external voice data, and the data feeding decision module is used to determine how to feed external voice data and local voice data to the echo cancellation module.

[0201] After the data feeding module obtains external voice data, see Figure 15As shown, it can be determined whether the echo cancellation function is turned on. If the echo cancellation function is not turned on, it indicates that the external voice data does not need to be used for echo cancellation, so the external voice data is deleted; if the echo cancellation function is turned on, it indicates that the external voice data needs to be cached as reference data for subsequent echo cancellation, so the external voice data is added to the message queue; since the duration of the external voice data contained in the message queue is too long, it indicates that the playback time of some of the external voice data is too early, and the probability of the local voice data that needs to be processed by the echo message later containing the external voice data is low. Therefore, the data feeding module can also obtain the total delay and determine whether the total playback time of the multiple frames of external voice data contained in the message queue is greater than the total delay. If the total playback time is greater than the total delay, the frame of external voice data at the head of the message queue is deleted, that is, the frame of external voice data with the earliest storage time in the message queue is deleted; if the total playback time is less than or equal to the total delay, it is re-determined whether the total playback time of the multiple frames of external voice data contained in the message queue is greater than the total delay.

[0202] Among them, the total delay refers to the sum of the playback link delay and the acquisition link delay, wherein the playback link refers to the transmission link of external voice data from the receipt of external voice data to the playback of external voice through the speaker. The link includes a speaker, and the delay of this link is the playback link delay. That is, the playback link delay refers to the delay from the receipt of external voice data to the playback of external voice data. The acquisition link refers to the transmission link of local voice data from the collection of local voice data to the echo cancellation processing of local voice data. The link includes a microphone. Optionally, the acquisition link may include the transmission link of local voice data from the input end of the microphone to the input end of the data feeding module. The delay of this link is the acquisition link delay. That is, the acquisition link delay refers to the delay from the collection of local voice data to the echo cancellation processing of local voice data.

[0203] After the data feeding module obtains a frame of local voice data from the microphone, Figure 16 As shown, the data feeding decision module can obtain a frame of reference voice data and feed the reference voice data and a frame of local voice data to the echo cancellation module for processing. The reference voice data can be silent voice data or from the message queue. For details, see Figure 17As shown, after the data feeding decision module obtains a frame of local voice data, it can determine whether the message queue contains external voice data. If so, a frame of external voice data is extracted from the head of the message queue, that is, the frame of external voice data with the earliest storage time is extracted, and the external voice data is used as reference voice data; if not, that is, the message queue does not contain external voice data, a frame of silent voice data can be generated, the silent voice data is used as reference voice data, and the reference voice data and local voice data are fed to the echo message module. The echo cancellation module performs echo cancellation processing on the local voice data through the reference voice data to obtain the outgoing voice data output by the echo cancellation module. In this way, the outgoing voice data will not contain the sound from the second terminal device, and the outgoing voice data will be sent to the second terminal device for playback, and the user of the second terminal device will not hear his own voice.

[0204] It should be noted that the specific implementation details of each step in this embodiment have been described in detail in the above embodiments and will not be repeated here.

[0205] In this embodiment, it can be ensured that in the process of feeding the echo cancellation module, the feeding is strictly carried out according to one frame of playback audio data and one frame of acquisition audio data, thereby achieving alignment of the acquisition audio data and the playback audio data, and improving the effect of echo cancellation. In addition, the acquisition link delay and the playback link delay are used to manage the playback audio data contained in the message queue, so that too old playback audio data can be safely deleted without worrying about the problem of echo leakage, saving storage resources and further improving the effect of echo cancellation.

[0206] See also Figure 18 , Figure 18 FIG. 1 is a block diagram of an audio data processing device according to an exemplary embodiment of the present application. Figure 18 As shown, the device includes:

[0207] The acquisition module 1801 is configured to acquire the audio data to be played and store the acquired audio data in a set storage area;

[0208] The processing module 1802 is configured to play the acquired playback audio data and collect audio data through an audio collection device to obtain collected audio data;

[0209] The extraction module 1803 is configured to extract the playback audio data whose corresponding playback duration matches the playback duration of the collected audio data from the storage area in the order of the corresponding storage time from earliest to latest;

[0210] The cancellation module 1804 is configured to perform echo cancellation processing on the collected audio data according to the extracted played audio data.

[0211] In an exemplary embodiment, based on the above solution, the apparatus further includes a first deletion module configured to:

[0212] Detecting the total playback time corresponding to the playback audio data contained in the storage area;

[0213] If the total playback time is greater than or equal to the playback time threshold, the target playback audio data to be deleted is extracted from the storage area in the order of the corresponding storage time from earliest to latest;

[0214] Delete the target playback audio data from the storage area.

[0215] In an exemplary embodiment, based on the above solution, the apparatus further includes a computing module configured to:

[0216] Calculate the time difference between the acquisition time and the playback time of the audio data to obtain a first delay;

[0217] A playback duration threshold is calculated based on the first time delay; wherein the playback duration threshold is positively correlated with the first time delay.

[0218] In an exemplary embodiment, based on the above solution, the calculation module is specifically configured as follows:

[0219] Calculating the time difference between the time when the audio data is collected and the time when the echo cancellation processing is performed to obtain a second delay;

[0220] A playback time threshold is calculated according to the total delay of the first delay and the second delay; wherein the playback time threshold is positively correlated with the total delay.

[0221] In an exemplary embodiment, based on the above solution, the apparatus further includes a second deletion module configured to:

[0222] Searching for the earliest stored audio data from the multiple frames of played audio data contained in the storage area;

[0223] If the storage time of the playback audio data with the earliest storage time in the storage area is greater than or equal to the storage time threshold, the playback audio data with the earliest storage time is deleted from the storage area.

[0224] In an exemplary embodiment, based on the above solution, the acquisition module 1801 is specifically configured as follows:

[0225] Detecting the working status of the audio acquisition device and obtaining a status detection result;

[0226] If the status detection result indicates that the audio acquisition device is acquiring audio data, the acquired playback audio data is stored in the storage area.

[0227] In an exemplary embodiment, based on the above solution, the apparatus further includes a generating module configured to:

[0228] If the storage area does not contain the playback audio data, generating silent audio data whose corresponding playback duration matches the playback duration of the collected audio data;

[0229] Perform echo cancellation processing on the collected audio data according to the silent audio data.

[0230] In an exemplary embodiment, based on the above solution, the extraction module 1803 is specifically configured as follows:

[0231] If a set number of frames of audio data are collected, the set number of frames of audio data are extracted from the storage area in the order of the corresponding storage time from earliest to latest, and the audio data is played;

[0232] Based on the extracted playback audio data, echo cancellation processing is performed on the collected audio data, including:

[0233] The audio data is played according to the extracted set number of frames, and the echo cancellation processing is performed on the collected audio data of the set number of frames.

[0234] In an exemplary embodiment, based on the above solution, under the condition that the storage area includes a message queue, the extraction module 1803 is specifically configured as follows:

[0235] The playback audio data whose corresponding playback duration matches the playback duration of the collected audio data is extracted from the head of the message queue.

[0236] In an exemplary embodiment, based on the aforementioned solution, under the condition that the played audio data includes audio data sent by the external communication terminal during voice communication with the external communication terminal, and the collected audio data includes audio data collected during the voice communication, the apparatus further includes a sending module configured to:

[0237] The collected audio data after echo cancellation processing is sent to the external communication terminal.

[0238] In an exemplary embodiment, based on the above solution, the acquisition module 1801 is specifically configured as follows:

[0239] If the storage area does not contain the playback audio data, then calculating the storage time corresponding to the acquired playback audio data according to the waiting time and the acquisition time corresponding to the acquired playback audio data; wherein the storage time is later than the acquisition time;

[0240] According to the storage time, the acquired playback audio data is stored in the storage area.

[0241] It should be noted that Figure 18The provided audio data processing device and the audio data processing method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs operations has been described in detail in the method embodiment and will not be repeated here.

[0242] An embodiment of the present application also provides an electronic device, comprising: one or more processors; a storage device for storing one or more computer programs, wherein when the one or more computer programs are executed by one or more processors, the electronic device implements the audio data processing method provided in the above-mentioned embodiments.

[0243] Figure 19 A schematic diagram showing the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. The electronic device may be Figure 1 The terminal device 110 or the server 120 is shown.

[0244] It should be noted that Figure 19 The computer system 1900 of the electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present application.

[0245] like Figure 19 As shown, computer system 1900 includes a central processing unit (CPU) 1901, which can perform various appropriate actions and processes according to computer programs stored in read-only memory (ROM) 1902 or computer programs loaded from storage unit 1908 into random access memory (RAM) 1903, such as executing the audio data processing method described in the above embodiment. Various computer programs and data required for system operation are also stored in RAM 1903. CPU 1901, ROM 1902, and RAM 1903 are connected to each other via bus 1904. Input / output (I / O) interface 1905 is also connected to bus 1904.

[0246] In some embodiments, the following components are connected to the I / O interface 1905: an input section 1906 including a keyboard, a mouse, and the like; an output section 1907 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage section 1908 including a hard disk; and a communication section 1909 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 1909 performs communication processing via a network such as the Internet. A drive 1910 is also connected to the I / O interface 1905 as needed. Removable media 1911, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1910 as needed, so that computer programs read from the removable media can be installed in the storage section 1908 as needed.

[0247] In particular, according to an embodiment of the present application, a computer program that implements the audio data processing method can be carried on a computer-readable medium, and the computer program can be downloaded and installed from the network through the communication part 1909, and / or installed from the removable medium 1911.

[0248] It should be noted that the computer-readable medium shown in the embodiments of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a computer program that can be used by or in combination with an instruction execution system, device or device. A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable computer program, and such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer program embodied in the computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0249] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and a computer program.

[0250] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. In some cases, the names of these units do not constitute limitations on the units themselves.

[0251] Another aspect of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor of an electronic device, the electronic device implements the aforementioned audio data processing method. The computer-readable storage medium may be included in the electronic device described in the above embodiments, or may exist independently and not be incorporated into the electronic device.

[0252] Another aspect of the present application provides a computer program product, comprising a computer program that, when executed by a processor, implements the audio data processing methods provided in each of the above-described embodiments. The computer program may be stored in a computer-readable storage medium. The computer program product may be a computer program as a product, such as an APP (Application, mobile application software), a webpage, or a mini-program; or, the computer program product may be a storage medium, device, terminal, or virtual machine containing the computer program.

[0253] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. Ordinary technicians in this field can easily make corresponding changes or modifications based on the main ideas and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. A method for processing audio data, characterized in that: The method comprises: Acquire the audio data to be played and store the acquired audio data in a set storage area; Play the acquired playback audio data, and collect audio data through the audio collection device to obtain collected audio data; Extracting, from the storage area, the playback audio data whose corresponding playback duration matches the playback duration of the collected audio data in order of the corresponding storage time from earliest to latest; According to the extracted playback audio data, echo cancellation processing is performed on the collected audio data.

2. The method according to claim 1, wherein The method further comprises: Detecting the total playback duration corresponding to the playback audio data contained in the storage area; If the total playback time is greater than or equal to the playback time threshold, extracting the target playback audio data to be deleted from the storage area in order of the corresponding storage time from earliest to latest; The target playback audio data is deleted from the storage area.

3. The method according to claim 2, wherein The method further comprises: Calculate the time difference between the acquisition time and the playback time of the audio data to obtain a first delay; The playback duration threshold is calculated according to the first time delay; wherein the playback duration threshold is positively correlated with the first time delay.

4. The method according to claim 3, wherein Calculating the playback duration threshold according to the first time delay includes: Calculating the time difference between the time when the audio data is collected and the time when the echo cancellation processing is performed to obtain a second delay; The playback time threshold is calculated according to the total delay of the first delay and the second delay; wherein the playback time threshold is positively correlated with the total delay.

5. The method according to claim 1, wherein The method further comprises: Searching for the earliest stored audio data from the multiple frames of audio data stored in the storage area; If the storage time of the playback audio data with the earliest storage time in the storage area is greater than or equal to the storage time threshold, the playback audio data with the earliest storage time is deleted from the storage area.

6. The method according to claim 1, wherein The step of storing the acquired playback audio data in a set storage area includes: Detecting the working status of the audio acquisition device to obtain a status detection result; If the status detection result indicates that the audio acquisition device is acquiring audio data, the acquired playback audio data is stored in the storage area.

7. The method according to claim 1, wherein After the audio data is collected by the audio collection device to obtain the collected audio data, the method further includes: If the storage area does not contain the playback audio data, generating silent audio data whose corresponding playback duration matches the playback duration of the collected audio data; Perform echo cancellation processing on the collected audio data according to the silent audio data.

8. The method according to claim 1, wherein The extracting, from the storage area in the order of the corresponding storage time from earliest to latest, the playback audio data whose corresponding playback duration matches the playback duration of the collected audio data, comprises: If a set number of frames of audio data are collected, extracting the set number of frames of audio data for playback from the storage area in the order of the corresponding storage time from earliest to latest; The step of performing echo cancellation processing on the collected audio data according to the extracted playback audio data includes: The audio data is played according to the extracted set number of frames, and the echo cancellation processing is performed on the collected audio data of the set number of frames.

9. The method according to claim 1, wherein The storage area includes a message queue; The extracting, from the storage area in the order of the corresponding storage time from earliest to latest, the playback audio data whose corresponding playback duration matches the playback duration of the collected audio data, comprises: The playback audio data whose corresponding playback duration matches the playback duration of the collected audio data is extracted from the head of the message queue.

10. The method according to claim 1, wherein The played audio data includes audio data sent by the external communication terminal during voice communication with the external communication terminal, and the collected audio data includes audio data collected during the voice communication; After performing echo cancellation processing on the collected audio data according to the extracted playback audio data, the method further includes: The collected audio data after the echo cancellation process is sent to the external communication terminal.

11. The method according to claim 1, wherein The step of storing the acquired playback audio data in a set storage area includes: If the storage area does not contain the playback audio data, calculating the storage time corresponding to the acquired playback audio data according to the waiting time and the acquisition time corresponding to the acquired playback audio data; wherein the storage time is later than the acquisition time; According to the storage time, the acquired playback audio data is stored in the storage area.

12. An audio data processing device, characterized in that: The device comprises: An acquisition module is configured to acquire the audio data to be played and store the acquired audio data in a set storage area; A processing module is configured to play the acquired playback audio data and collect audio data through an audio collection device to obtain collected audio data; An extraction module configured to extract, from the storage area, the playback audio data whose corresponding playback duration matches the playback duration of the collected audio data in order of the corresponding storage time from earliest to latest; The cancellation module is configured to perform echo cancellation processing on the collected audio data according to the extracted playback audio data.

13. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more computer programs, which, when executed by the one or more processors, enables the electronic device to implement the method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor of an electronic device, the electronic device is enabled to implement the method according to any one of claims 1 to 11.

15. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 11 when the computer program is executed by a processor.

Images