Audio data processing method and device, equipment and storage medium

Abstract

This application provides an audio data processing method, apparatus, device, and storage medium, applicable to technical fields such as web conferencing. The method includes: acquiring a list of audio features of interest for a target object, the list including audio features of at least one object of interest; acquiring an audio data frame to be played and extracting its audio features; and playing the audio data frame if its audio features match one of the audio features in the list of audio features of interest. This application's embodiments can accurately identify the audio data of an object of interest, thereby improving the target object's readiness to listen to the audio data of that object.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an audio data processing method, apparatus, device, and storage medium. Background Technology

[0002] With the rapid development of internet technology, online meetings have become increasingly common, facilitating message transmission and greatly simplifying daily work. However, in real-world work, meetings are often frequent. If, while participating in a meeting, one is only interested in the speeches of certain individuals (such as users), and these speeches are scattered throughout the meeting, one would have to listen to all the speakers in order to hear everything.

[0003] Existing conferencing software offers several methods for this scenario, such as subscribing to only the remote streams of interested participants or providing meeting reminders based on keywords. However, these methods all have limitations. For instance, subscribing to only the remote streams of interested participants is only suitable for scenarios where each remote participant corresponds to a single remote device. In practice, many meetings utilize a single remote device in a conference room, with multiple people participating together. In such cases, this method cannot achieve the goal of listening only to the speech of interested participants. Keyword-based subscriptions also have limitations because they cannot accurately predict the content of the speech of interested participants. Therefore, current methods cannot achieve the goal of preparing to listen only to the audio data of interested participants. Summary of the Invention

[0004] This application provides an audio data processing method, apparatus, device, and storage medium, which can enable the preparation and listening of audio data of an object of interest.

[0005] In a first aspect, this application provides an audio data processing method, including:

[0006] Obtain a list of audio features of interest for the target object, wherein the list of audio features of interest includes audio feature information of at least one object of interest for the target object;

[0007] Obtain the audio data frame to be played and extract the audio feature information of the audio data frame to be played. The audio data frame to be played is the mixed data of audio data sent by M remote devices. One remote device corresponds to one or more remote objects. The object of interest is the remote object. M is a positive integer.

[0008] If the audio feature information of the audio data frame to be played matches one of the audio feature information in the list of audio features of interest, then the audio data frame to be played is played.

[0009] Secondly, this application provides an audio data processing apparatus, comprising:

[0010] The interest list acquisition unit is used to acquire an interest audio feature information list of the target object, wherein the interest audio feature information list includes audio feature information of at least one interest object of the target object;

[0011] The feature extraction unit is used to acquire the audio data frame to be played and extract the audio feature information of the audio data frame to be played. The audio data frame to be played is mixed data of audio data sent by M remote devices. One remote device corresponds to one or more remote objects. The object of interest is the remote object. M is a positive integer.

[0012] The processing unit is configured to play the audio data frame to be played if the audio feature information of the audio data frame to be played matches one of the audio feature information in the list of audio features of interest.

[0013] In some embodiments, the interest list acquisition unit is specifically used to acquire first audio data, which includes mixed data of audio data sent by the M remote devices within an initial time period; and to determine the interest audio feature information list based on the first audio data.

[0014] In some embodiments, the interest list acquisition unit is specifically used to play the first audio data; during the playback of the first audio data, it receives a first interest instruction input by the target object at N different times, the first interest instruction being used to indicate that the target object is interested in the remote object of the audio data played by the terminal device at the N different times; for the i-th time among the N different times, it determines the audio data corresponding to the i-th time in the first audio data, where i is a positive integer from 1 to N; it extracts the audio feature information of the audio data corresponding to the i-th time; and based on the audio feature information of the audio data corresponding to each of the N different times, it obtains the list of audio feature information of interest.

[0015] In some embodiments, the interest list acquisition unit is specifically used to determine the audio data in the first audio data that is located within a preset time period before the i-th time and the i-th time as the audio data corresponding to the i-th time.

[0016] In some embodiments, the interest list acquisition unit is specifically used to receive a setting completion instruction input by the target object, the setting completion instruction being used to indicate that the target object's interest object setting is complete; based on the setting completion instruction, the audio feature information of the audio data corresponding to each of the N different times is added to the interest audio feature information list.

[0017] In some embodiments, the interest list acquisition unit is further configured to receive object identification information of the audio data corresponding to the i-th time moment input by the target object; add the object identification information and audio feature information of the audio data corresponding to the i-th time moment to the interest audio feature information list, wherein the object identification information of the audio data corresponding to the i-th time moment corresponds to the audio feature information of the audio data corresponding to the i-th time moment.

[0018] In some embodiments, the interest list acquisition unit is further configured to, in response to a deletion instruction input by the target object, delete the audio feature information corresponding to the first object identifier information in the interest audio feature information list, wherein the deletion instruction is used to instruct the deletion of the audio feature information corresponding to the first object identifier information in the interest audio feature information list.

[0019] In some embodiments, the interest list acquisition unit is further configured to, in response to an interest object query operation input by the target object, display the interest audio feature information list, wherein each object identifier in the interest audio feature information list corresponds to a deletion option; and, in response to the target object triggering an operation on the deletion option corresponding to the first object identifier in the interest audio feature information list, delete the audio feature information corresponding to the first object identifier in the interest audio feature information list.

[0020] In some embodiments, before extracting the audio feature information of the audio data frame to be played, the feature extraction unit is further configured to determine whether the audio data frame to be played contains human voice information; if the audio data frame to be played contains the human voice information, the audio feature information of the audio data frame to be played is extracted.

[0021] In some embodiments, the processing unit is further configured to play the audio data frame to be played if the audio data frame to be played does not contain the human voice information.

[0022] In some embodiments, the processing unit is further configured to adjust the audio data frame to be played as silent data and play it if the audio feature information of the audio data frame to be played does not match any of the audio feature information in the list of audio features of interest.

[0023] The processing unit is further configured to: display a remote object corresponding to the audio data frame to be played when playing the silence data corresponding to the audio data frame to be played; receive a playback instruction input by the target object, the playback instruction being input by the target object based on the remote object corresponding to the displayed audio data frame to be played; and play the audio data frame to be played based on the playback instruction.

[0024] In some embodiments, if the playback instruction is a second interest instruction, the second interest instruction is used to indicate that the target object is interested in the remote object corresponding to the audio data frame to be played, and the interest list acquisition unit is further used to add the audio feature information of the audio data frame to be played to the target object's list of audio feature information of interest.

[0025] In some embodiments, the processing unit is further configured to receive a disinterest instruction input by the target object when playing the audio data frame to be played, the disinterest instruction being used to indicate that the target object is not interested in the remote object corresponding to the audio data frame to be played; based on the disinterest instruction, adjust the audio data frame to be played to silent data for playback, and delete the audio feature information of the audio data frame to be played from the list of interested audio feature information.

[0026] In some embodiments, the processing unit is specifically configured to display an "uninterested" option when playing the audio data frame to be played; and to receive a trigger operation from the object on the "uninterested" option as the "uninterested" instruction.

[0027] Thirdly, an electronic device is provided, including a processor and a memory. The memory is used to store a computer program, and the processor is used to invoke and run the computer program stored in the memory to perform the methods described in the first aspect and its various implementations.

[0028] Fourthly, a chip is provided for implementing the methods of any one of the first aspects or their respective implementations. Specifically, the chip includes a processor for calling and running a computer program from a memory, causing a device on which the chip is installed to perform the methods of the first aspect and its respective implementations as described above.

[0029] Fifthly, a computer-readable storage medium is provided for storing a computer program that causes a computer to perform the methods described in the first aspect and its various implementations.

[0030] In a sixth aspect, a computer program product is provided, including computer program instructions that cause a computer to perform the methods described in the first aspect and its various implementations.

[0031] In a seventh aspect, a computer program is provided that, when run on a computer, causes the computer to perform the methods described in the first aspect and its various implementations.

[0032] In summary, this embodiment first obtains a list of audio features of interest for the target object, which includes audio feature information of at least one object of interest for the target object. During audio data playback, the audio data frame to be played is filtered based on this list of audio features of interest. Specifically, the audio feature information of the audio data frame to be played is extracted and compared with each audio feature in the aforementioned list of audio features of interest for the target object. If the audio feature information of the audio data frame to be played matches one of the audio feature in the aforementioned list of audio features of interest, then the audio data frame to be played is played. Therefore, this embodiment filters the audio data frame to be played based on the target object's list of audio features of interest. Regardless of whether the remote objects share a single remote device or each corresponds to a separate remote device, the target object can listen only to the audio data of the object of interest, enriching the usage scenarios. In addition, by comparing the audio feature information of the audio data frame to be played with each audio feature information in the target object's list of audio feature information of interest, this embodiment of the application can accurately identify whether the remote device of the audio data frame to be played is the target object of interest. Compared with the method of guessing the speech keywords of the object of interest, this embodiment of the application can achieve accurate identification of the audio data of the object of interest, thereby improving the readiness to listen to the audio data of the object of interest. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 A schematic diagram of an implementation environment for the audio data processing method provided in this application embodiment;

[0035] Figure 2 This is a schematic flowchart of an audio data processing method provided in an embodiment of this application;

[0036] Figure 3 This is a schematic diagram illustrating an application scenario according to an embodiment of this application;

[0037] Figure 4 This is a schematic diagram of a data caching method;

[0038] Figure 5 This is a schematic diagram of one type of input for the first instruction of interest;

[0039] Figure 6 This is a schematic diagram illustrating an input method where the first instruction of interest is entered multiple times.

[0040] Figure 7 A diagram illustrating the creation of a list of audio features of interest;

[0041] Figure 8 A diagram illustrating the process of deleting audio feature information from a list of audio features of interest.

[0042] Figure 9 A schematic diagram for extracting audio feature information;

[0043] Figure 10 This is a diagram illustrating how audio data of interest can be adjusted to data of no interest.

[0044] Figure 11 This is a diagram illustrating how to adjust uninteresting audio data for playback.

[0045] Figure 12 This is a diagram illustrating how to adjust uninteresting audio data to interesting audio data;

[0046] Figure 13 This is a schematic flowchart of an audio data processing method provided in an embodiment of this application;

[0047] Figure 14 This is a schematic block diagram of an audio data processing apparatus provided in an embodiment of this application;

[0048] Figure 15 This is a schematic block diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0049] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0050] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those illustrated or described herein. In embodiments of the invention, "B corresponding to A" means that B is associated with A. In one implementation, B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0051] The technical solution proposed in this application can be applied to the fields of network conferencing, audio processing and other technologies, and can enable the listening of audio data of the object of interest in scenarios where multiple remote objects are speaking.

[0052] The relevant technologies involved in the embodiments of this application are described below.

[0053] Pulse Code Modulation (PCM) is audio information converted from digital signals. It is also the most primitive audio format, essentially a series of audio signals converted from digital values.

[0054] Gaussian Mixture Model - Universal Background Model (GMM-UBM) is a method that can model any continuous probability distribution. By treating speech patterns as a probability distribution model, GMM can model speech patterns, and different parameters can represent the speech patterns of different users. Given the requirement for a large amount of target user data for training Gaussian Mixture Models, GMM-UBM can first build a general model based on various speech types, and then adjust the mean parameters of each Gaussian component only for the target user's speech pattern. This allows training to be completed with a smaller amount of data from the target user.

[0055] The identity vector (i-vector) is used because the GMM model has many parameters, which makes training and recognition more complicated. Therefore, the same hypervector subspace is used to model speech and channel. In this way, each user's voiceprint can be mapped to a vector with only 400-600 elements, which is more convenient for training and recognition.

[0056] With the rapid development of internet technology, online meetings have become increasingly common, greatly facilitating daily work and enabling message transmission. In reality, meetings are frequent, and if one is only interested in the speeches of certain individuals (e.g., users), and their speeches are scattered throughout the meeting, listening to all participants' speeches is necessary to hear everything. Existing meeting software offers methods to address this, such as subscribing to the remote streams of interested individuals or providing meeting reminders based on keywords. However, these methods have limitations. For example, subscribing to the remote streams of interested individuals is only suitable for scenarios where each remote individual corresponds to a single remote device. In many work scenarios, meetings utilize a single remote device in a conference room, with multiple people participating together. In such cases, this method cannot achieve selective listening to the speeches of only interested individuals. Keyword-based subscriptions also have limitations because they cannot accurately predict the content of the speeches of interested individuals. Therefore, current methods cannot effectively listen to the audio data of only interested individuals in scenarios with multiple remote participants.

[0057] To address the aforementioned technical problems, this application provides an audio data processing method. First, a list of audio features of interest for a target object is obtained. This list includes audio feature information of at least one object of interest for the target object. During audio data playback, the audio data frame to be played is filtered based on this list of audio features of interest. Specifically, the audio feature information of the audio data frame to be played is extracted and compared with each audio feature in the list of audio features of interest for the target object. If the audio feature information of the audio data frame to be played matches one of the audio features in the list of audio features of interest, then the audio data frame to be played is played. Therefore, this application, by filtering audio data frames to be played based on the list of audio features of interest for the target object, allows the target object to listen only to the audio data of the object of interest, regardless of whether the remote objects share a single remote device or each corresponds to a separate remote device, thus enriching the usage scenarios. In addition, by comparing the audio feature information of the audio data frame to be played with each audio feature information in the target object's list of audio feature information of interest, this embodiment of the application can accurately identify whether the remote device of the audio data frame to be played is the target object of interest. Compared with the method of guessing the speech keywords of the object of interest, this embodiment of the application can achieve accurate identification of the audio data of the object of interest, thereby improving the readiness to listen to the audio data of the object of interest.

[0058] The implementation environment of the embodiments of this application is described below.

[0059] Figure 1 This is a schematic diagram of an implementation environment for the audio data processing method provided in this application, including a terminal device 101 and one or more remote devices 102. Figure 1 As shown, one remote device 102 corresponds to one or more remote objects, and a terminal device corresponds to one object. The terminal device 101 is communicatively connected to each remote device 102. In some embodiments, such as... Figure 2 As shown, both the terminal device 102 and the remote device 102 are connected to the server 103 for communication. The audio data sent by the remote device 102 to the terminal device 102 is forwarded by the server 103.

[0060] In this embodiment, the terminal device 101 and each remote device 102 have a client for a certain program installed, such as a client for an instant messaging application. In some embodiments, the server 102 can be understood as the background of the program, such as the server of the instant messaging application.

[0061] In this embodiment, one remote device 102 corresponds to one or more remote objects (also referred to as remote users). In this embodiment, each remote device 102 receives audio data input from the corresponding remote object, encodes the audio data, and sends it to the terminal device 101. For example, each remote device 102 sends the encoded audio data to the server 102, which then sends the encoded audio data to the terminal device 101. When the terminal device 101 receives encoded audio data from multiple remote devices 102, it decodes the encoded audio data sent by each remote device 102 to obtain the decoded audio data corresponding to each remote device 102. Then, the terminal device 101 mixes the decoded audio data corresponding to each remote device 102 to obtain mixed audio data.

[0062] In this embodiment, terminal device 101 obtains a list of audio features of interest for a target object, which includes audio feature information of at least one object of interest for the target object. Thus, for an acquired audio data frame to be played (i.e., the mixed data of audio data sent by M remote devices obtained through decoding and mixing), terminal device 101 extracts the audio feature information of the audio data frame to be played. Then, the audio feature information of the audio data frame to be played is compared with each audio feature in the list of audio features of interest for the target object. If the audio feature information of the audio data frame to be played matches one audio feature in the list of audio features of interest for the target object, the audio data frame to be played is then played. In one example, if the audio feature information of the audio data frame to be played does not match any audio feature in the list of audio features of interest for the target object, the audio data frame to be played is converted to silent data for playback.

[0063] Therefore, this application embodiment filters the audio data frames to be played based on the target object's list of audio features of interest. Regardless of whether the remote objects share a single remote device or each corresponds to a separate remote device, the target object can listen only to the audio data of the objects of interest, enriching the usage scenarios. Furthermore, by comparing the audio feature information of the audio data frame to be played with each audio feature in the target object's list of audio features of interest, this application embodiment can accurately identify whether the remote device of the audio data frame to be played is an object of interest for the target object. Compared to methods that guess the spoken keywords of the objects of interest, this application embodiment can accurately identify the audio data of the objects of interest, thereby improving the readiness for listening to the audio data of the objects of interest.

[0064] In some embodiments, the terminal device 101 and remote device 102 described above include, but are not limited to, desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, etc. Portable wearable devices may include smartwatches, smart bracelets, head-mounted devices, etc. Terminal devices are often equipped with a display device, which may also be a monitor, display screen, touchscreen, etc., and the touchscreen may also be a touch screen, touch panel, etc.

[0065] In some embodiments, the server 103 may be one or more servers. When there are multiple servers, at least two servers may be used to provide different services, and / or at least two servers may be used to provide the same service, such as providing the same service in a load-balanced manner. This application embodiment does not limit this. The server may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The server may also be a node in a blockchain.

[0066] It should be noted that the implementation environment of this application embodiment includes, but is not limited to, Figure 1 As shown.

[0067] The technical solutions of the embodiments of this application will be described in detail below through some examples. The following embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0068] Figure 2 This is a schematic flowchart of an audio data processing method provided in an embodiment of this application. The executing entity in this embodiment is a device capable of processing audio data, such as an audio data processing device. This audio data processing device can be part of the aforementioned terminal device 101, part of the aforementioned server 103, or a system composed of terminal device 101 and server 103. For ease of description, the example given here uses a terminal device as the executing entity.

[0069] like Figure 2 As shown, the audio data processing method of this application embodiment includes the following steps:

[0070] S101. Obtain a list of audio features of interest for the target object.

[0071] The list of audio features of interest includes audio feature information of at least one object of interest of the target object.

[0072] The audio data processing method of this application can be applied to scenarios such as web conferencing where multiple different objects participate in audio data interaction. The target object of this application is any one of these multiple different objects. For ease of description, this application uses a single object, namely the target object, as an example to describe the audio data processing method of this application.

[0073] In this embodiment, both the terminal device and the remote device are equipped with an audio data interaction application, such as a client for a web conferencing application. The target object can input audio data through the client on the terminal device or receive audio data from a remote object. Similarly, the remote object can input audio data and listen to audio data from other objects through the client on the remote device.

[0074] like Figure 3 As shown in this embodiment, in an audio data interaction scenario, such as a web conference, multiple objects participate. The target object accesses the web conference through a single terminal device, while the other objects access the web conference through M terminal devices. For ease of description, these M terminal devices are referred to as the remote devices of the target object, and the objects corresponding to these M remote devices are referred to as the remote objects of the target object. Each of the M remote devices corresponds to one or more remote devices. That is, one or more remote objects access the web conference through a single remote device. During the web conference, for each of the M remote devices, the remote device receives audio data input from one or more corresponding remote objects, encodes the audio data, and obtains an audio stream. Then, the audio stream is sent to the target object's terminal device. In this way, the target object's terminal device can receive the audio stream sent by each of the M remote devices. Then, the target object's terminal device decodes each audio stream to obtain the audio data sent by each remote device. Then, the audio data sent by each remote device is mixed to obtain the audio data to be played. In this embodiment, the remote device sends audio data in units of audio data frames, and the corresponding target terminal device receives audio data in units of audio data frames, where an audio data frame can be understood as audio data within a certain period of time. In this way, the target terminal device can use the audio data to be played obtained from the above steps as an audio data frame, i.e., the audio data frame to be played.

[0075] In this embodiment of the application, after the target terminal device obtains the audio data frame to be played based on the above steps, as follows: Figure 3 As shown, the audio data frame to be played is filtered to determine whether it is the audio data of the target object's object of interest, so as to determine whether to play the audio data frame to be played.

[0076] In this embodiment of the application, the decision on whether to play the audio data frame to be played is made by obtaining a list of audio features of interest to the target object.

[0077] This application does not limit the specific method by which the terminal device obtains the list of audio feature information of interest for the target object.

[0078] In some embodiments, the list of audio features of interest for the target object can be determined by the data uploaded by the target object. For example, the target object can collect audio data of objects of interest and then send the audio data of the objects of interest to the terminal device. In this way, the terminal device performs feature extraction on the audio data of each object of interest of the target object to obtain the audio feature information of each object of interest, and then obtains the list of audio features of interest for the target object based on the audio feature information of each object of interest.

[0079] In some embodiments, the terminal device may obtain a list of audio feature information of interest for the target object through the following steps S101-A and S101-B:

[0080] S101-A1. Obtain first audio data, which includes mixed data of audio data sent by M remote devices within an initial time period;

[0081] S101-B1. Based on the first audio data, determine the list of audio feature information of interest.

[0082] In this implementation, the terminal device determines a list of audio features of interest for the target object based on the audio data received from M remote devices within an initial time period. For ease of description, the mixed data of the audio data sent by the M remote devices within the initial time period is denoted as the first audio data.

[0083] In one example, such as Figure 4 As shown, after the terminal device receives audio data sent by M remote devices, it caches the audio data in a cache module. Optionally, the cache module stores the data in a queue, with the queue length corresponding to a threshold L of the audio data duration. When the stored audio data duration reaches the threshold L, the first frame of audio data written into the queue is removed from the queue in chronological order. This allows the cache module to store the latest audio data of duration L.

[0084] In the initial stage, while the terminal device sends the first audio data cached by the caching module to the player for playback, it also processes the first audio data to determine a list of audio feature information of interest to the target object. It should be noted that the duration of the first audio data can be equal to or less than L.

[0085] For example, such as Figure 4 As shown, this caching module can cache audio data from three audio data frames. In the initial stage, the target terminal device initially receives the bitstream of the first audio data frame 1 sent by M remote devices. After decoding and mixing this bitstream, the audio data frame 1 to be played is obtained and added to the caching module. Next, the target terminal device receives the bitstream of the second audio data frame 2 sent by M remote devices. After decoding and mixing this bitstream, the audio data frame 2 to be played is obtained and added to the caching module. Then, the target terminal device receives the bitstream of the third audio data frame 3 sent by M remote devices. After decoding and mixing this bitstream, the audio data frame 3 to be played is obtained and added to the caching module. In one example, the terminal device sends one or more of the three audio data frames to be played to the player for playback, and at the same time makes a copy of the one or more audio data frames as the first audio data to determine the list of audio feature information of interest to the target object.

[0086] This application does not limit the specific method by which the terminal device determines the list of audio feature information of interest for the target object based on the first audio data.

[0087] In some embodiments, the target object can send audio data of objects of interest to a terminal device. The terminal device divides the first audio data into multiple audio segments. For example, the first audio data can be evenly divided into multiple audio segments according to a preset time length. Alternatively, the first audio data can be divided into multiple audio segments according to the characteristics of a remote object, with each audio segment corresponding to a remote object; that is, the audio segment is the speech data output by the remote object. Next, the terminal device compares each of the aforementioned audio segments with the audio data of the target object's objects of interest. For example, if an audio segment matches the audio data of one of the target object's objects of interest, the remote object corresponding to that audio segment is determined to be the target object's object of interest. The terminal device then extracts the audio feature information of that audio segment and adds it to the target object's list of audio feature information of interest.

[0088] In some embodiments, the terminal device may determine a list of audio feature information of interest for the target object through steps S101-B11 and S101-B15:

[0089] S101-B11, The terminal device plays the first audio data;

[0090] S101-B12. During the playback of the first audio data, the terminal device receives a first interest instruction input by the target object at N different times. The first interest instruction is used to indicate that the target object is interested in the remote object of the audio data played by the terminal device at N different times.

[0091] S101-B13. For the i-th time among N different times, the terminal device determines the audio data corresponding to the i-th time in the first audio data, where i is a positive integer from 1 to N;

[0092] S101-B14. The terminal device extracts the audio feature information of the audio data corresponding to the i-th time moment;

[0093] S101-B15. The terminal device obtains a list of audio feature information of interest based on the audio feature information of the audio data corresponding to each of N different times.

[0094] In this implementation, the terminal device plays the first audio data. During the playback of the first audio data, when the target object encounters the speech of an object of interest, it can input a first interest command to indicate that the target object is interested in the distant object in the audio data being played by the terminal device. Thus, based on the first interest command input by the target object, the terminal device can obtain the audio data played by the terminal device at the time the first interest command was input from the first audio data, and further extract the audio feature information of that audio data. In this way, the terminal device can construct a list of audio feature information of interest to the target object based on the first interest command input by the target object.

[0095] This application does not limit the specific method by which the target object inputs the first interest instruction.

[0096] In one example, the target object can input a first command of interest via voice. For instance, when the terminal device is playing audio data from remote object 1, if the target object is interested in the speech of remote object 1, it can input a voice control command such as "interested".

[0097] In one example, in an embodiment of this application, such as Figure 5As shown, when the client of the terminal device displays the first audio data, the display interface includes an interest option. If the remote object corresponding to the audio data currently being played by the terminal device is interested in the target object, the interest option can be triggered to input the first interest command.

[0098] In this embodiment of the application, when the target object encounters the speech of the object of interest during the playback of the first audio data on the terminal device, it can input a first interest instruction. One or more first interest instructions can be input, which are denoted as N first interest instructions.

[0099] For example, such as Figure 6 As shown, when the terminal device is playing the first audio data, if the target object is interested in the speech of the remote object 1 at a certain moment (e.g., triggering the interest option in the display interface), the target object inputs a first interest command. Responding to the first interest command input by the target object, the terminal device obtains the audio data corresponding to the first moment from the first audio data, and then uses this audio data as the audio data of the remote object 1, extracting the audio feature information of the audio data corresponding to the first moment. At a certain moment (e.g., when the terminal device is playing the audio data of the remote object 2), if the target object is interested in the speech of the remote object 2, the target object inputs the first interest command again (e.g., triggering the interest option in the display interface again). Responding to the first interest command input by the target object, the terminal device obtains the audio data corresponding to the second moment from the first audio data, and then uses this audio data as the audio data of the remote object 2, extracting the audio feature information of the audio data corresponding to the second moment. Similarly, when the terminal device is playing audio data of remote object n at time N, and the target object is interested in the speech of remote object n, the target object inputs the first interest command again (e.g., triggering the interest option in the display interface). Responding to the first interest command input by the target object, the terminal device obtains the audio data corresponding to time n from the first audio data, and then uses this audio data corresponding to time n as the audio data of remote object n, extracting the audio feature information of this audio data. In this way, audio feature information corresponding to audio data at each of N different times can be obtained, and based on the audio feature information corresponding to each of these N different times, a list of audio feature information of interest for the target object can be obtained.

[0100] The embodiments of this application do not limit the specific method by which the terminal device determines the audio data corresponding to the i-th moment in the first audio data.

[0101] In one possible implementation, the terminal device determines the audio data located between time it and time i+t in the first audio data as the audio data corresponding to time i.

[0102] In one possible implementation, the terminal device determines the audio data within a preset time period prior to the i-th moment from the first audio data as the audio data corresponding to the i-th moment.

[0103] This application does not limit the specific method by which the terminal device obtains the list of audio features of interest for the target object based on the audio feature information of the audio data corresponding to each of the N different times.

[0104] In some embodiments, if the terminal device does not receive a first interest instruction from the target object within a preset time period, it adds the audio feature information of the audio data corresponding to each of the N different times to the target object's list of audio feature information of interest, and ends the construction operation of the target object's list of audio feature information of interest.

[0105] In some embodiments, when the target object determines that the settings for the object of interest have been completed, it inputs a setting completion command to the terminal device. The terminal device receives the setting completion command input by the target object, which indicates that the setting of the object of interest for the target object is complete. Based on the setting completion command, the terminal device adds the audio feature information of the audio data corresponding to each of the N different times to the list of audio features of interest.

[0106] This application does not limit the specific method by which the target object inputs the setting completion instruction.

[0107] In one example, the target object can input a setting completion command via voice. For instance, after the target object inputs the first command of interest at time N, it can input a voice control command such as "complete" as the setting completion command.

[0108] In one example, in an embodiment of this application, such as Figure 7 As shown, when the client of the terminal device displays the first audio data, the display interface includes a "Complete" option. If the target object determines that it has completed the settings for the object of interest, it can trigger this "Complete" option. At this time, the terminal device can determine the target object's triggering operation on this "Complete" option as a setting completion command input by the target object.

[0109] In this implementation, the list of audio features of interest for the target object is shown in Table 1:

[0110] Table 1

[0111] index Audio features 1 Audio Feature Information 1 2 Audio Feature Information 2 3 Audio Feature Information 3 …… …… N Audio feature information N

[0112] As shown in Table 1, the list of audio features of interest for the target object includes N audio features, which are the audio features of audio data from N different remote objects. Because different objects have differences in their oral cavity and vocal tract, the same sound will sound different, meaning their timbre will be different. In the frequency domain, this means that the frequency distribution corresponding to the same sound produced by different objects will be different, and the frequency distribution can be viewed as a continuous probability distribution function. In this embodiment, an identity vector (i-vector) model can be used to model the voiceprint of the object. Since speech can be viewed as a continuous probability distribution function described by multiple Gaussian functions, and the parameters of the probability distribution function will be different for different objects, the identity vector can represent all the probability distribution function parameters as a single vector with a size of only 400-600. Furthermore, even when the same object speaks different languages, this vector remains essentially unchanged, thus using this vector to uniquely identify the voiceprint of an object. In this embodiment, the identity vector of the object is recorded as the audio feature information of the object. In other words, in this embodiment of the application, the audio feature information of the object of interest is used as the identity vector of the object of interest to uniquely identify the object of interest, so that in subsequent audio filtering, the audio data of the object of interest can be accurately filtered out based on the audio feature information of the object of interest.

[0113] In some embodiments, for each of the N different times, such as time i, the target object also inputs object identification information for the audio data corresponding to time i, to identify the remote object to which the audio data corresponding to time i belongs. The terminal device receives the object identification information of the audio data corresponding to time i input by the target object, and then adds the object identification information and audio feature information of the audio data corresponding to time i to the target object's list of audio features of interest, wherein the object identification information of the audio data corresponding to time i corresponds to the audio feature information of the audio data corresponding to time i.

[0114] At this point, the list of audio features of interest for the target object is shown in Table 2:

[0115] Table 2

[0116] index logo Audio features 1 Object A1 Audio Feature Information 1 2 Object A2 Audio Feature Information 2 3 Object A3 Audio Feature Information 3 …… …… …… N object AN Audio feature information N

[0117] This application does not limit the specific form of the object identification information of the audio data corresponding to the i-th time moment input by the target object. For example, if the target object knows that the audio data corresponding to the i-th time moment is the audio data of a remote object 1, the target object can input the name or nickname of the remote object 1 as the object identification information of the audio data corresponding to the i-th time moment. As another example, if the target object does not know that the audio data corresponding to the i-th time moment is the audio data of a remote object 1, the target object can name the remote object 1 and use that name as the object identification information of the audio data corresponding to the i-th time moment. In this way, the target object can obtain the identification information of the object of its interest through the list of audio feature information of interest.

[0118] In some embodiments, if the target object is not interested in a particular object in the constructed list of audio features of interest, it can input a deletion command. This deletion command instructs the audio feature information corresponding to the first object identifier in the list of audio features of interest to be deleted. Thus, in response to the deletion command input by the target object, the terminal device deletes the audio feature information corresponding to the first object identifier in the list of audio features of interest.

[0119] The specific method by which the target object inputs the deletion command in the embodiments of this application is not limited.

[0120] In some embodiments, the target object can input the first object identification information to be deleted via voice. For example, if the first object identification information is object A3 in Table 2 above, the target object can input the voice deletion command as "I am no longer interested in object A3". At this time, the terminal device parses the voice deletion command and then deletes the audio feature information 3 corresponding to object A3 in the list of interested audio feature information shown in Table 2.

[0121] In some embodiments, when a target object needs audio feature information of a specific object of interest, it can input an object of interest query operation. For example, the terminal device's display interface includes a query option, which the object can trigger. In response to the target object's input of the object of interest query operation, the terminal device displays a list of audio feature information of interest for the target object. Figure 8As shown, each object identifier in the list of audio features of interest corresponds to a deletion option. If the target object needs to delete the audio feature information corresponding to the first object identifier, the target object triggers the deletion option corresponding to the first object identifier. For example, if the first object identifier is object A3, the target object triggers the deletion option corresponding to object A3. In response to the triggering operation of the deletion option corresponding to the first object identifier in the list of audio features of interest for the target object, the terminal device deletes the audio feature information corresponding to the first object identifier in the list of audio features of interest, resulting in the list of audio features of interest shown in Table 3.

[0122] Table 3

[0123] index logo Audio features 1 Object A1 Audio Feature Information 1 2 Object A2 Audio Feature Information 2 4 Object A4 Audio feature information 4 …… …… …… N object AN Audio feature information N

[0124] In other words, in this embodiment of the application, the target object can modify the constructed list of audio features of interest to obtain a satisfactory list of audio features of interest for audio data filtering.

[0125] In this embodiment of the application, after the terminal device obtains the list of audio features of interest of the target object and the audio data frames to be played based on the above steps, it executes the following step S102.

[0126] S102. Obtain the audio data frame to be played and extract the audio feature information of the audio data frame to be played.

[0127] The audio data frame to be played is a mixed stream of audio data sent by M remote devices. One remote device corresponds to one or more remote objects. The object of interest is the remote object, and M is a positive integer.

[0128] In this embodiment of the application, the audio data frame to be played can be understood as the audio data frame to be played obtained by the terminal device after obtaining the list of audio features of interest of the target object.

[0129] As described above, the list of audio features of interest for the target object in this application embodiment includes audio feature information of each of at least one object of interest in the target object. Therefore, when the terminal device acquires an audio data frame to be played, in order to facilitate comparison between the audio data frame to be played and each audio feature in the list of audio features of interest for the target object, the audio feature information of the audio data frame to be played is extracted.

[0130] In this embodiment, the specific method by which the terminal device extracts the audio feature information of the audio data frame to be played is basically the same as the specific method by which it extracts the audio feature information of at least one object of interest of the target object.

[0131] For example, such as Figure 9 As shown, the terminal device extracts audio feature information of the audio data frame to be played using a Gaussian Mixture Model-Global Background Model (GMM-UBM). Specifically, the terminal device inputs the audio data frame to be played into the Gaussian Mixture Model-Global Background Model for audio feature extraction, thereby obtaining the audio feature information of the audio data frame to be played.

[0132] In some embodiments, the terminal device determines whether the audio data frame to be played contains human voice information before extracting the audio feature information of the audio data frame to be played. This application embodiment does not limit the specific method by which the terminal device determines whether the audio data frame to be played contains human voice information.

[0133] In one possible implementation method, the terminal device invokes a human voice detection model, which can detect whether the input audio data contains human voice information. Thus, the terminal device inputs the audio data frame to be played into the human voice detection model for detection, obtaining the human voice detection result.

[0134] In some embodiments of this application, the audio data frame to be played is typically a 20ms or 10ms data frame. The terminal device can analyze the audio data frame to be played, for example, using Voice Activity Detection (VAD) technology. For instance, the audio data frame to be played can be converted into a frequency domain signal, and then the proportion of energy below 2kHz to the total energy of the frame data can be calculated. If it exceeds a certain threshold, the audio data frame to be played is considered to contain human voice information. If the audio data frame to be played is detected to contain the human voice information, then the audio feature information of the audio data frame to be played is extracted.

[0135] In some embodiments, if the audio data frame to be played does not contain the human voice information, the audio data frame to be played is played.

[0136] S103. If the audio feature information of the audio data frame to be played matches one of the audio feature information in the list of audio features of interest, then the audio data frame to be played is played.

[0137] In this embodiment, after extracting the audio feature information of the audio data frame to be played based on the above steps, the terminal device compares the audio feature information of the audio data frame to be played with each audio feature information in the list of audio features of interest one by one. For example, for each audio feature information in the list of audio features of interest, the terminal device calculates the similarity between the audio feature information and the audio feature information of the audio data frame to be played. An exemplary terminal device uses a distance calculation formula (e.g., cosine distance) to calculate the similarity between the audio feature information and the audio feature information of the audio data frame to be played. If the similarity between the audio feature information and the audio feature information of the audio data frame to be played is greater than or equal to a certain threshold, then it is determined that the audio feature information matches the audio feature information of the audio data frame to be played. If the similarity between the audio feature information and the audio feature information of the audio data frame to be played is greater than the threshold, then it is determined that the audio feature information does not match the audio feature information of the audio data frame to be played. By referring to this method, the terminal device can determine whether there is any audio feature information in the list of audio feature information of interest that matches the audio feature information of the audio data frame to be played.

[0138] If the terminal device determines that the audio feature information of the audio data frame to be played matches one of the audio feature information in the list of audio features of interest, then the terminal device plays the audio data frame to be played.

[0139] In some embodiments, when playing the audio data frame to be played, if the target object is not interested in the audio data frame to be played, it can input a "not interested" instruction. This "not interested" instruction instructs the target object not to be interested in the remote object corresponding to the audio data frame to be played. When the terminal device receives the "not interested" instruction from the target object, based on this instruction, the audio data frame to be played is adjusted to silent data for playback, and the audio feature information of the audio data frame to be played is deleted from the target object's list of audio feature information of interest.

[0140] This application does not limit the specific method by which the target object inputs an instruction indicating that it is not interested.

[0141] In one possible implementation, the target object inputs the playback command via voice. For example, when the terminal device is playing the audio data frame to be played, if the target object inputs a command such as "not interested," the terminal device, based on this voice command, switches the playback of the audio data frame from normal to silent mode. That is, it changes the audio data frame from a audible state to a silent state. Simultaneously, the audio feature information of the audio data frame to be played is removed from the target object's list of audio features of interest.

[0142] In one possible implementation, such as Figure 10 As shown, when the terminal device plays the audio data frame to be played, it displays an "Not Interested" option on the display interface. This way, if the target user does not want to listen to the audio data frame to be played, the "Not Interested" option is triggered. In response to the target user's triggering of the "Not Interested" option, the terminal device switches the normal playback of the audio data frame to silent playback. That is, it changes the audio data frame from a sound state to a silent state. Simultaneously, it removes the audio feature information of the audio data frame to be played from the target user's list of audio features of interest.

[0143] In some embodiments, if the terminal device determines that the audio feature information of the audio data frame to be played does not match any of the audio feature information in the list of audio features of interest, then the audio data frame to be played is adjusted to be silent data for playback.

[0144] In some embodiments, when a terminal device plays an audio data frame to be played in a muted state, it may display a remote object corresponding to the audio data frame to be played. If the target object determines to answer the audio data frame to be played based on the remote object corresponding to the audio data frame displayed by the terminal device, it inputs a playback command. The terminal device receives the playback command input by the target object, which is input by the target object based on the remote object corresponding to the audio data frame to be played displayed by the terminal device, and the playback command is used to instruct the playback of the audio data frame to be played. Based on the playback command, the terminal device plays the audio data frame to be played normally.

[0145] This application does not limit the specific method by which the target object inputs playback instructions.

[0146] In one possible implementation, the target object inputs the playback command via voice. For example, when the terminal device plays the silence data corresponding to the audio data frame to be played, if the target object inputs a command such as "play normally," the terminal device will play the audio data frame normally based on the voice playback command. That is, the audio data frame to be played is changed from a silent state to a sound state.

[0147] In one possible implementation, such as Figure 11 As shown, when the terminal device plays the silence data corresponding to the audio data frame to be played, it displays a playback option on the display interface. When the target object needs to answer the audio data frame to be played, the playback option is triggered. In response to the target object's triggering of the playback option, the terminal device plays the audio data frame to be played normally.

[0148] In one possible implementation, the playback instruction input by the target object is a second interest instruction, which indicates that the target object is interested in the remote object corresponding to the audio data frame to be played. In this case, when the terminal device plays the silence data corresponding to the audio data frame to be played, it receives the second interest instruction input by the target object. Based on the second interest instruction, it plays the audio data frame to be played and simultaneously adds the audio feature information of the audio data frame to be played to the target object's list of audio feature information of interest.

[0149] This application does not limit the specific method by which the target object inputs the second interest instruction.

[0150] In one example, the target object can input a second interest command via voice. For instance, when the terminal device is playing the silence data corresponding to the audio data frame to be played, if the target object is interested in the remote object corresponding to the audio data frame to be played, it can input a voice control command such as "interested". At this time, based on the second interest command input by voice, the terminal device plays the audio data frame to be played and adds the audio feature information of the audio data frame to be played to the target object's list of audio feature information of interest.

[0151] In one example, in an embodiment of this application, such as Figure 12 As shown, when the terminal device plays the silence data corresponding to the audio data frame to be played, the display interface includes an "interest" option. If the target object is interested in the remote object corresponding to the audio data frame to be played, it can trigger the "interest" option to input a second "interest" command. At this time, based on the triggering operation of the "interest" option, the terminal device plays the audio data frame to be played and adds the audio feature information of the audio data frame to be played to the target object's list of audio feature information of interest.

[0152] As can be seen from the above, the user-friendly interactive interface provided in this application embodiment allows the target object to add and delete audio feature information of objects of interest, so as to make real-time adjustments to the objects of interest. In multi-person meeting scenarios such as conference room meetings, online interviews, and online annual meetings, the target object can accurately listen to the audio data of remote objects of interest, reducing the number of times the target object is disturbed and improving the user experience.

[0153] The audio data processing method provided in this application first obtains a list of audio features of interest for the target object. This list includes audio feature information of at least one object of interest for the target object. During audio data playback, the audio data frame to be played is filtered based on this list of audio features of interest. Specifically, the audio feature information of the audio data frame to be played is extracted and compared with each audio feature in the list of audio features of interest for the target object. If the audio feature information of the audio data frame to be played matches one of the audio features in the list of audio features of interest, then the audio data frame to be played is played. Therefore, this application's method, by filtering audio data frames to be played based on the list of audio features of interest for the target object, allows the target object to listen only to the audio data of the object of interest, regardless of whether the remote objects share a single remote device or each corresponds to a separate remote device, thus enriching the usage scenarios. In addition, by comparing the audio feature information of the audio data frame to be played with each audio feature information in the target object's list of audio feature information of interest, this embodiment of the application can accurately identify whether the remote device of the audio data frame to be played is the target object of interest. Compared with the method of guessing the speech keywords of the object of interest, this embodiment of the application can achieve accurate identification of the audio data of the object of interest, thereby improving the readiness to listen to the audio data of the object of interest.

[0154] The above provides an overall overview of the audio data processing method provided in the embodiments of this application. The following section will combine... Figure 13 The audio data processing method of the embodiments of this application will be further described.

[0155] Figure 13 This is a schematic flowchart of an audio data processing method provided in an embodiment of this application. The executing entity in this embodiment is a device capable of processing audio data, such as an audio data processing device. This audio data processing device can be part of the aforementioned terminal device 101, part of the aforementioned server 103, or a system composed of terminal device 101 and server 103. For ease of description, the example given here uses a terminal device as the executing entity.

[0156] like Figure 13 As shown, the audio data processing method of this application embodiment includes the following steps:

[0157] S201, The terminal device acquires the first audio data.

[0158] The first audio data includes a mixed stream of audio data sent by M remote devices within an initial time period.

[0159] In the initial interaction phase of audio information, for each of the M remote devices, the remote device receives audio data input from one or more corresponding remote objects, encodes the audio data to obtain an audio bitstream, and then sends the audio bitstream to the target object's terminal device. Thus, the target object's terminal device can receive the audio bitstream sent by each of the M remote devices. Next, the target object's terminal device decodes each audio bitstream to obtain the audio data sent by each remote device. Then, the audio data sent by each remote device is mixed to obtain the first audio data. In this embodiment, the remote devices send audio data in units of audio data frames, and the corresponding target object's terminal device receives audio data in units of audio data frames. The first audio data includes one or more audio data frames.

[0160] In some embodiments, when the terminal device obtains the first audio data, it stores the first audio data in the cache module.

[0161] S202, The terminal device plays the first audio data.

[0162] S203. During the playback of the first audio data, the terminal device receives the first interest instruction input by the target object at N different times.

[0163] The first interest instruction is used to indicate that the target object is interested in the remote object of the audio data played by the terminal device at N different times.

[0164] The specific execution process of S202 to S203 can be referred to the relevant descriptions of S101-B11 and S101-B12 above, and will not be repeated here.

[0165] S204. For the i-th time among N different times, the terminal device determines the audio data corresponding to the i-th time from the first audio data.

[0166] Where i is a positive integer from 1 to N.

[0167] For example, the terminal device determines the audio data located between the i-th time point and the preset time period preceding the i-th time point from the first audio data as the audio data corresponding to the i-th time point.

[0168] S205. The terminal device extracts the audio feature information of the audio data corresponding to the i-th time moment.

[0169] S206. The terminal device obtains a list of audio feature information of interest for the target object based on the audio feature information of the audio data corresponding to each of the N different times.

[0170] For example, the terminal device receives a setting completion instruction from the target object, which indicates that the target object's object of interest has been set. Based on the setting completion instruction, the audio feature information of the audio data corresponding to each of the N different times is added to the list of audio features of interest.

[0171] In some embodiments, the terminal device further receives object identification information of the audio data corresponding to the i-th time input by the target object; adds the object identification information and audio feature information of the audio data corresponding to the i-th time to the list of audio features of interest, wherein the object identification information of the audio data corresponding to the i-th time corresponds to the audio feature information of the audio data corresponding to the i-th time.

[0172] In some embodiments, the terminal device further responds to a deletion instruction input by the target object by deleting the audio feature information corresponding to the first object identifier in the list of audio feature information of interest. The deletion instruction is used to instruct the deletion of the audio feature information corresponding to the first object identifier in the list of audio feature information of interest. For example, in response to a query operation of the target object, the terminal device displays a list of audio feature information of interest, where each object identifier in the list of audio feature information of interest corresponds to a deletion option; in response to the target object's triggering operation on the deletion option corresponding to the first object identifier in the list of audio feature information of interest, the terminal device deletes the audio feature information corresponding to the first object identifier in the list of audio feature information of interest. This results in an updated list of audio feature information of interest.

[0173] The above S201 to S206 can be understood as the process by which the terminal device constructs a list of audio feature information of interest for the target object.

[0174] S207. The terminal device obtains the audio data frame to be played.

[0175] The audio data frame to be played is a mixed stream of audio data sent by M remote devices, where one remote device corresponds to one or more remote objects. The objects of interest mentioned above are remote objects, and M is a positive integer.

[0176] S208. The terminal device determines whether the audio data frame to be played contains human voice information.

[0177] If it is determined that the audio data frame to be played contains human voice information, then the following step S209 is executed.

[0178] If it is determined that the audio data frame to be played does not contain human voice information, then the following step S213 is executed.

[0179] The specific execution process of S208 can be referred to the relevant description of S102 above, and will not be repeated here.

[0180] S209. The terminal device extracts the audio feature information of the audio data frame to be played.

[0181] For example, the terminal device uses Gaussian Mixture Model-Global Background Model (GMM-UBM) to extract audio feature information from the audio data frames to be played.

[0182] S210, The terminal device compares the audio feature information of the audio data frame to be played with each audio feature information in the list of audio features of interest.

[0183] S211. If the audio feature information of the audio data frame to be played matches one of the audio feature information in the list of audio features of interest, then the terminal device plays the audio data frame to be played.

[0184] In some embodiments, when playing an audio data frame to be played, the terminal device receives a disinterest instruction input by the target object. The disinterest instruction is used to indicate that the target object is not interested in the remote object corresponding to the audio data frame to be played. Based on the disinterest instruction, the audio data frame to be played is adjusted to be silent data for playback, and the audio feature information of the audio data frame to be played is deleted from the list of audio feature information of interest.

[0185] For example, when a terminal device plays an audio data frame to be played, it displays an option that it is not interested in; the receiving object triggers the operation of the option that it is not interested in as a disinterest instruction.

[0186] The specific execution process of S210 and S211 can be referred to the relevant description of S103 above, and will not be repeated here.

[0187] S212. If the audio feature information of the audio data frame to be played does not match any of the audio feature information in the list of audio features of interest, the terminal device will adjust the audio data frame to be played as silent data for playback.

[0188] In some embodiments, when the terminal device plays the silence data corresponding to the audio data frame to be played, it displays the remote object corresponding to the audio data frame to be played; receives a playback instruction input by the target object, which is input by the target object based on the remote object corresponding to the displayed audio data frame to be played; and plays the audio data frame to be played based on the playback instruction.

[0189] In some embodiments, if the playback instruction is a second interest instruction, the second interest instruction is used to indicate that the target object is interested in the remote object corresponding to the audio data frame to be played. In this case, the terminal device will also add the audio feature information of the audio data frame to be played to the target object's list of audio feature information of interest.

[0190] S213. If the audio data frame to be played does not contain human voice information, the terminal device plays the audio data frame to be played.

[0191] The audio data processing method provided in this application firstly involves a terminal device acquiring first audio data, which includes mixed data of audio data sent by M remote devices within an initial time period. The terminal device plays the first audio data, and during this playback, it receives a first interest instruction input by a target object at N different times. For the i-th time among the N different times, the terminal device determines the audio data corresponding to the i-th time from the first audio data. The terminal device extracts the audio feature information of the audio data corresponding to the i-th time, and based on the audio feature information of the audio data corresponding to each of the N different times, obtains a list of audio feature information of interest for the target object. Next, the terminal device acquires the audio feature information of an audio data frame to be played, and determines whether the audio data frame to be played contains human voice information. If it is determined that the audio data frame to be played does not contain human voice information, then the audio data frame to be played is played. When it is determined that the audio data frame to be played contains human voice information, the audio feature information of the audio data frame to be played is extracted, and the audio feature information of the audio data frame to be played is compared with each audio feature information in the list of audio features of interest. If the audio feature information of the audio data frame to be played matches one audio feature information in the list of audio features of interest, the terminal device plays the audio data frame to be played. If the audio feature information of the audio data frame to be played does not match any audio feature information in the list of audio features of interest, the terminal device adjusts the audio data frame to be played to silent data before playback. Therefore, it can be seen that, regardless of whether the remote objects share a single remote device or each corresponds to a separate remote device, the target object can listen to only the audio data of the object of interest, enriching the usage scenarios. In addition, by comparing the audio feature information of the audio data frame to be played with each audio feature information in the target object's list of audio feature information of interest, this embodiment of the application can accurately identify whether the remote device of the audio data frame to be played is the target object of interest. Compared with the method of guessing the speech keywords of the object of interest, this embodiment of the application can achieve accurate identification of the audio data of the object of interest, thereby improving the readiness to listen to the audio data of the object of interest.

[0192] The preferred embodiments of this application have been described in detail above with reference to the accompanying drawings. However, this application is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this application, various simple modifications can be made to the technical solutions of this application, and these simple modifications all fall within the protection scope of this application. For example, the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this application will not describe the various possible combinations separately. Furthermore, various different embodiments of this application can also be arbitrarily combined, as long as they do not violate the spirit of this application, they should also be considered as the content disclosed in this application.

[0193] It should also be understood that, in the various method embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0194] The above text combined Figures 3 to 13 The following describes in detail the embodiments of the task processing method of this application, in conjunction with... Figure 14 The following describes in detail the device embodiments of this application.

[0195] Figure 14 This is a schematic block diagram of an audio data processing apparatus provided in one embodiment of this application. The apparatus 10 can be applied to a server.

[0196] like Figure 14 As shown, the audio data processing device 10 includes:

[0197] Interest list acquisition unit 11 is used to acquire an interest audio feature information list of the target object, the interest audio feature information list including audio feature information of at least one interest object of the target object;

[0198] The feature extraction unit 12 is used to acquire the audio data frame to be played and extract the audio feature information of the audio data frame to be played. The audio data frame to be played is mixed data of audio data sent by M remote devices. One remote device corresponds to one or more remote objects. The object of interest is the remote object. M is a positive integer.

[0199] The processing unit 13 is configured to play the audio data frame to be played if the audio feature information of the audio data frame to be played matches one of the audio feature information in the list of audio features of interest.

[0200] In some embodiments, the interest list acquisition unit 11 is specifically used to acquire first audio data, which includes mixed data of audio data sent by the M remote devices within an initial time period; and to determine the interest audio feature information list based on the first audio data.

[0201] In some embodiments, the interest list acquisition unit 11 is specifically used to play the first audio data; during the playback of the first audio data, it receives a first interest instruction input by the target object at N different times, the first interest instruction being used to indicate that the target object is interested in the remote object of the audio data played by the terminal device at the N different times; for the i-th time among the N different times, it determines the audio data corresponding to the i-th time in the first audio data, where i is a positive integer from 1 to N; it extracts the audio feature information of the audio data corresponding to the i-th time; and based on the audio feature information of the audio data corresponding to each of the N different times, it obtains the list of audio feature information of interest.

[0202] In some embodiments, the interest list acquisition unit 11 is specifically used to determine the audio data in the first audio data that is located within a preset time period before the i-th time and the i-th time as the audio data corresponding to the i-th time.

[0203] In some embodiments, the interest list acquisition unit 11 is specifically used to receive a setting completion instruction input by the target object, the setting completion instruction being used to indicate that the target object's interest object setting is complete; based on the setting completion instruction, the audio feature information of the audio data corresponding to each of the N different times is added to the interest audio feature information list.

[0204] In some embodiments, the interest list acquisition unit 11 is further configured to receive object identification information of the audio data corresponding to the i-th time moment input by the target object; add the object identification information and audio feature information of the audio data corresponding to the i-th time moment to the interest audio feature information list, wherein the object identification information of the audio data corresponding to the i-th time moment corresponds to the audio feature information of the audio data corresponding to the i-th time moment.

[0205] In some embodiments, the interest list acquisition unit 11 is further configured to, in response to a deletion instruction input by the target object, delete the audio feature information corresponding to the first object identifier information in the interest audio feature information list, wherein the deletion instruction is used to instruct the deletion of the audio feature information corresponding to the first object identifier information in the interest audio feature information list.

[0206] In some embodiments, the interest list acquisition unit 11 is further configured to, in response to an interest object query operation input by the target object, display the interest audio feature information list, wherein each object identifier in the interest audio feature information list corresponds to a deletion option; and, in response to the target object triggering an operation on the deletion option corresponding to the first object identifier in the interest audio feature information list, delete the audio feature information corresponding to the first object identifier in the interest audio feature information list.

[0207] In some embodiments, before extracting the audio feature information of the audio data frame to be played, the feature extraction unit 12 is further configured to determine whether the audio data frame to be played contains human voice information; if the audio data frame to be played contains the human voice information, the audio feature information of the audio data frame to be played is extracted.

[0208] In some embodiments, the processing unit 13 is further configured to play the audio data frame to be played if the audio data frame to be played does not contain the human voice information.

[0209] In some embodiments, the processing unit 13 is further configured to adjust the audio data frame to be played as silent data for playback if the audio feature information of the audio data frame to be played does not match any of the audio feature information in the list of audio features of interest.

[0210] The processing unit 13 is further configured to: display a remote object corresponding to the audio data frame to be played when playing the silence data corresponding to the audio data frame to be played; receive a playback instruction input by the target object, the playback instruction being input by the target object based on the remote object corresponding to the displayed audio data frame to be played; and play the audio data frame to be played based on the playback instruction.

[0211] In some embodiments, if the playback instruction is a second interest instruction, the second interest instruction is used to indicate that the target object is interested in the remote object corresponding to the audio data frame to be played, and the interest list acquisition unit 11 is further used to add the audio feature information of the audio data frame to be played to the target object's list of audio feature information of interest.

[0212] In some embodiments, the processing unit 13 is further configured to receive a disinterest instruction input by the target object when playing the audio data frame to be played, the disinterest instruction being used to indicate that the target object is not interested in the remote object corresponding to the audio data frame to be played; based on the disinterest instruction, adjust the audio data frame to be played to silent data for playback, and delete the audio feature information of the audio data frame to be played from the list of interested audio feature information.

[0213] In some embodiments, the processing unit 13 is specifically configured to display an "uninterested" option when playing the audio data frame to be played; and to receive a trigger operation from the object on the "uninterested" option as the "uninterested" instruction.

[0214] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here. Specifically, Figure 14 The apparatus shown can perform the embodiments of the above-described method, and the foregoing and other operations and / or functions of each module in the apparatus are respectively for implementing the method embodiments, which will not be described in detail here for the sake of brevity.

[0215] The apparatus of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this application embodiment can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0216] Figure 15 This is a schematic block diagram of the electronic device provided in the embodiments of this application. Figure 15 The electronic device can be the aforementioned task controller, node, or terminal device.

[0217] like Figure 15 As shown, the electronic device 30 may include:

[0218] The system includes a memory 31 and a processor 32. The memory 31 stores a computer program 33 and transfers the program code 33 to the processor 32. In other words, the processor 32 can retrieve and run the computer program 33 from the memory 31 to implement the methods described in the embodiments of this application.

[0219] For example, the processor 32 can be used to execute the steps in the above method according to the instructions in the computer program 33.

[0220] In some embodiments of this application, the processor 32 may include, but is not limited to:

[0221] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0222] In some embodiments of this application, the memory 31 includes, but is not limited to:

[0223] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROP), programmable ROP (PROP), erasable PROP (EPROP), electrically erasable EPROP (EEPROP), or flash memory. Volatile memory can be random access memory (RAP), used as an external cache. By way of example, but not limitation, many forms of RAP are available, such as Static RAP (SRAP), Dynamic RAP (DRAP), Synchronous DRAP (SDRAP), Double Data Rate SDRAP (DDR SDRAP), Enhanced SDRAP (ESDRAP), Synchronous Link DRAP (SLDRAP), and Direct Rambus RAP (DR RAP).

[0224] In some embodiments of this application, the computer program 33 may be divided into one or more modules, which are stored in the memory 31 and executed by the processor 32 to complete the page recording method provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 33 in the electronic device.

[0225] like Figure 15 As shown, the electronic device 30 may further include:

[0226] Transceiver 34, which can be connected to processor 32 or memory 31.

[0227] The processor 32 can control the transceiver 34 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 34 may include a transmitter and a receiver. The transceiver 34 may further include antennas, and the number of antennas may be one or more.

[0228] It should be understood that the various components in the electronic device 30 are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0229] According to one aspect of this application, a computer storage medium is provided that stores a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, embodiments of this application also provide a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.

[0230] According to another aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method described in the above-described method embodiments.

[0231] In other words, when implemented using software, it can be implemented entirely or partially as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0232] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0233] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0234] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0235] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An audio data processing method, characterized by, include: Obtain a list of audio features of interest for the target object, wherein the list of audio features of interest includes audio feature information of at least one object of interest for the target object; Obtain the audio data frame to be played and extract the audio feature information of the audio data frame to be played. The audio data frame to be played is the mixed data of audio data sent by M remote devices. One remote device corresponds to one or more remote objects. The object of interest is the remote object. M is a positive integer. If the audio feature information of the audio data frame to be played matches one of the audio feature information in the list of audio features of interest, then the audio data frame to be played is played.

2. The method of claim 1, wherein, The list of audio features of interest for the target object includes: Acquire first audio data, which includes mixed data of audio data sent by the M remote devices within an initial time period; Based on the first audio data, a list of audio features of interest is determined.

3. The method of claim 2, wherein, The step of determining the list of audio features of interest based on the first audio data includes: Play the first audio data; During the playback of the first audio data, a first interest instruction is received from the target object at N different times. The first interest instruction is used to indicate that the target object is interested in the remote object of the audio data played by the terminal device at the N different times. For the i-th time among the N different times, determine the audio data corresponding to the i-th time in the first audio data, where i is a positive integer from 1 to N; Extract the audio feature information of the audio data corresponding to the i-th time moment; Based on the audio feature information of the audio data corresponding to each of the N different times, the list of audio features of interest is obtained.

4. The method of claim 3, wherein, Determining the audio data corresponding to the i-th time moment from the first audio data includes: The audio data located within a preset time period preceding the i-th time point in the first audio data is determined as the audio data corresponding to the i-th time point.

5. The method of claim 3, wherein, The audio feature information list of interest is obtained based on the audio feature information corresponding to each of the N different times, including: The system receives a setting completion instruction from the target object, which indicates that the target object's object of interest has been set. Based on the setting completion instruction, the audio feature information of the audio data corresponding to each of the N different times is added to the list of audio features of interest.

6. The method of claim 5, wherein, The method further includes: The object identifier information of the audio data corresponding to the i-th time moment input by the target object; The object identifier information and audio feature information of the audio data corresponding to the i-th time are added to the list of audio features of interest, wherein the object identifier information of the audio data corresponding to the i-th time corresponds to the audio feature information of the audio data corresponding to the i-th time.

7. The method of claim 6, wherein, The method further includes: In response to a deletion instruction input by the target object, the audio feature information corresponding to the first object identifier information in the list of audio features of interest is deleted. The deletion instruction is used to instruct the deletion of the audio feature information corresponding to the first object identifier information in the list of audio features of interest.

8. The method of claim 7, wherein, The step of deleting the audio feature information corresponding to the first object identifier information from the list of audio feature information of interest in response to the deletion instruction input by the target object includes: In response to the target object's input of an object of interest query operation, the list of audio features of interest is displayed, and each object identifier in the list of audio features of interest corresponds to a delete option; In response to the target object's triggering operation on the deletion option corresponding to the first object identifier in the list of audio features of interest, the audio feature information corresponding to the first object identifier in the list of audio features of interest is deleted.

9. The method of claim 1, wherein, Before extracting the audio feature information of the audio data frame to be played, the method further includes: Determine whether the audio data frame to be played contains human voice information; The extraction of audio feature information from the audio data frame to be played includes: If the audio data frame to be played contains the human voice information, the audio feature information of the audio data frame to be played is extracted.

10. The method of claim 9, wherein, The method further includes: If the audio data frame to be played does not contain the human voice information, then the audio data frame to be played is played.

11. The method of claim 1, wherein, The method further includes: If the audio feature information of the audio data frame to be played does not match any of the audio feature information in the list of audio features of interest, then the audio data frame to be played will be adjusted to silent data for playback.

12. The method of claim 11, wherein, The method further includes: When playing the silence data corresponding to the audio data frame to be played, display the remote object corresponding to the audio data frame to be played; Receive a playback instruction input by the target object, wherein the playback instruction is input by the target object based on the remote object corresponding to the displayed audio data frame to be played; Based on the playback command, the audio data frame to be played is played.

13. The method of claim 12, wherein, If the playback instruction is a second interest instruction, the second interest instruction being used to indicate that the target object is interested in the far-end object corresponding to the audio data frame to be played, the method further includes: The audio feature information of the audio data frame to be played is added to the list of audio features of interest of the target object.

14. The method of claim 1, wherein, The method further includes: When playing the audio data frame to be played, a disinterest instruction is received from the target object. The disinterest instruction is used to indicate that the target object is not interested in the remote object corresponding to the audio data frame to be played. Based on the "not interested" instruction, the audio data frame to be played is adjusted to be silent data and played, and the audio feature information of the audio data frame to be played is deleted from the list of audio feature information of interest.

15. The method of claim 14, wherein, The receipt of the "not interested" instruction from the target object includes: When playing the audio data frame to be played, display the "not interested" option; The object's trigger operation for the option it is not interested in is received as the instruction of disinterest.

16. An audio data processing apparatus, characterized by comprising: include: The interest list acquisition unit is used to acquire an interest audio feature information list of the target object, wherein the interest audio feature information list includes audio feature information of at least one interest object of the target object; The feature extraction unit is used to acquire the audio data frame to be played and extract the audio feature information of the audio data frame to be played. The audio data frame to be played is mixed data of audio data sent by M remote devices. One remote device corresponds to one or more remote objects. The object of interest is the remote object. M is a positive integer. The processing unit is configured to play the audio data frame to be played if the audio feature information of the audio data frame to be played matches one of the audio feature information in the list of audio features of interest.

17. An electronic device, comprising: Including processor and memory; The memory is used to store computer programs; The processor is configured to execute the computer program to implement the method as described in any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that, Used to store computer programs; The computer program causes the computer to perform the method as described in any one of claims 1 to 15.

Images