Audio processing method, device, computer equipment, storage medium and program product

Abstract

The present application discloses an audio processing method, device, computer equipment, storage medium and program product, which belongs to the field of artificial intelligence technology. The method includes: extracting features from the first target audio to obtain a first target feature map; performing feature encoding based on the first target feature map to obtain a first target object feature and a first target age feature, the first target object feature is used to characterize the object information of the first target audio corresponding to the first target speaking object, and the first target age feature is used to characterize the age information of the first target speaking object; based on the first target age feature, separating the first target identity feature from the first target object feature, the first target identity feature is used to characterize the identity information of the first target speaking object, and the object information at least includes identity information and age information. This method can extract identity features that do not contain age information, avoid the influence of age information on identity features, and improve the accuracy of identity feature extraction.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an audio processing method, device, computer equipment, storage medium and program product. Background Art

[0002] With the rapid development of deep learning, deep learning has been widely applied and developed in various fields. For example, in the field of speaker recognition, the identity of the speaker in the audio is identified through the speaker recognition model.

[0003] In the related art, during the speaker recognition process, audio of variable length is input into the speaker recognition model, and a speaker feature vector containing information related to the speaker is extracted through a feature extraction layer, an encoding layer, and an output layer. The speaker feature vector is then input into a speaker recognition classifier to identify the identity of the speaker corresponding to the audio.

[0004] However, as the speaker ages, the speaker features corresponding to the same speaker also change, thereby widening the intra-class gap in the data. The age factor will also affect the accuracy of speaker identity recognition. Summary of the invention

[0005] The present application provides an audio processing method, apparatus, computer equipment, storage medium and program product. The technical solution is as follows:

[0006] According to one aspect of the present application, there is provided an audio processing method, the method comprising:

[0007] Extracting features of the first target audio to obtain a first target feature map;

[0008] Performing feature encoding based on the first target feature map to obtain a first target object feature and a first target age feature, wherein the first target object feature is used to characterize object information of a first target speaker corresponding to the first target audio, and the first target age feature is used to characterize age information of the first target speaker;

[0009] Based on the first target age feature, a first target identity feature is separated from the first target object feature, wherein the first target identity feature is used to characterize identity information of the first target speaking object, and the object information at least includes the identity information and the age information.

[0010] According to another aspect of the present application, there is provided an audio processing method, the method comprising:

[0011] Extract features from the sample audio through a first feature extraction network to obtain a first sample feature map;

[0012] Performing feature encoding on the first sample feature map through a second feature extraction network to obtain a sample object feature and a sample age feature, wherein the sample object feature is used to characterize object information of a sample speaker corresponding to the sample audio, and the sample age feature is used to characterize age information of the sample speaker;

[0013] Based on the sample age feature, separating the sample identity feature from the sample object feature;

[0014] Performing identity recognition based on the sample identity feature to obtain a sample identity label;

[0015] Perform age recognition based on the sample age feature to obtain a first sample age label;

[0016] The first feature extraction network and the second feature extraction network are trained based on the sample identity label, the annotated identity label corresponding to the sample audio, the first sample age label, and the annotated age label corresponding to the sample audio.

[0017] According to another aspect of the present application, there is provided an audio processing device, the device comprising:

[0018] A feature extraction module, used to extract features from the first target audio to obtain a first target feature map;

[0019] a feature encoding module, configured to perform feature encoding based on the first target feature map to obtain a first target object feature and a first target age feature, wherein the first target object feature is used to characterize object information of a first target speaker corresponding to the first target audio, and the first target age feature is used to characterize age information of the first target speaker;

[0020] The feature separation module is used to separate a first target identity feature from the first target object feature based on the first target age feature, wherein the first target identity feature is used to represent the identity information of the first target speaking object, and the object information at least includes the identity information and the age information.

[0021] According to another aspect of the present application, there is provided an audio processing device, the device comprising:

[0022] A feature extraction module, used to extract features from the sample audio through a first feature extraction network to obtain a first sample feature map;

[0023] a feature encoding module, configured to perform feature encoding on the first sample feature map through a second feature extraction network to obtain a sample object feature and a sample age feature, wherein the sample object feature is used to characterize object information of a sample speaker corresponding to the sample audio, and the sample age feature is used to characterize age information of the sample speaker;

[0024] A feature separation module, used to separate the sample identity feature from the sample object feature based on the sample age feature;

[0025] An identity recognition module, used to perform identity recognition based on the sample identity feature to obtain a sample identity label;

[0026] An age recognition module, used to perform age recognition based on the sample age feature to obtain a first sample age label;

[0027] A training module is used to train the first feature extraction network and the second feature extraction network based on the sample identity label, the labeled identity label corresponding to the sample audio, the first sample age label and the labeled age label corresponding to the sample audio.

[0028] According to another aspect of the present application, a computer device is provided, the computer device comprising: a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the audio processing method as described in the above aspects.

[0029] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is loaded and executed by a processor to implement the audio processing method as described above.

[0030] According to another aspect of the present application, a computer program product is provided, the computer program product comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned audio processing method.

[0031] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0032] A method for identifying a speaking object is provided: by performing feature extraction and feature encoding on a target audio of a speaking object, object information (object feature) corresponding to the speaking object and age information (age feature) of the speaking object are extracted from the target audio; since the object information includes age information and identity information, identity features can be separated from object features based on the extracted age features, so that the separated identity features do not have age information, thereby avoiding the influence of age information on identity features and improving the extraction accuracy of identity features; moreover, in a scenario of identifying a speaking object based on identity features, the influence of age information on the recognition accuracy of the same speaking object can also be avoided, thereby further improving the recognition accuracy of the speaking object. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0034] Figure 1 It is a recognition framework diagram of a speaking object in the related art;

[0035] Figure 2 is a schematic diagram of a process of identifying a speaking object shown in an exemplary embodiment of the present application;

[0036] Figure 3 is a schematic diagram of a computer system shown in an exemplary embodiment of the present application;

[0037] Figure 4 A flowchart of an audio processing method provided by an exemplary embodiment of the present application is shown;

[0038] Figure 5 A flowchart of an audio processing method provided by another exemplary embodiment of the present application is shown;

[0039] Figure 6 A schematic diagram of a model training process shown in an exemplary embodiment of the present application is shown;

[0040] Figure 7 A flowchart of an audio processing method provided by another exemplary embodiment of the present application is shown;

[0041] Figure 8 A schematic diagram of a model training process shown in another exemplary embodiment of the present application is shown;

[0042] Fig. 9 A schematic diagram of a model training process shown in another exemplary embodiment of the present application is shown;

[0043] Fig.10 A flowchart of an audio processing method provided by another exemplary embodiment of the present application is shown;

[0044] Fig.11 A schematic diagram of a model training process shown in another exemplary embodiment of the present application is shown;

[0045] Fig.12 A flowchart of an audio processing method provided by another exemplary embodiment of the present application is shown;

[0046] Fig.13 A schematic diagram of an identity feature extraction process shown in an exemplary embodiment of the present application is shown;

[0047] Fig.14 A schematic diagram of the extraction process of identity features shown in another exemplary embodiment of the present application is shown;

[0048] Fig.15 A flowchart of an audio processing method provided by another exemplary embodiment of the present application is shown;

[0049] Fig.16 is a structural block diagram of an audio processing device provided by an exemplary embodiment of the present application;

[0050] Fig.17 is a structural block diagram of an audio processing device provided by another exemplary embodiment of the present application;

[0051] Fig.18 The diagram is a schematic structural diagram of a computer device according to an exemplary embodiment. DETAILED DESCRIPTION

[0052] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0053] In the field of speaker recognition based on speech or audio, the recognition framework of speaker recognition is as follows: Figure 1As shown. The speaker recognition model 100 is composed of a feature extraction layer 101, a feature encoding layer 102, an output layer 103, and a speaker recognition classifier 104. The feature extraction layer 101 convolves the spectral signal of the input audio 10 and extracts information related to the speaker to obtain a set of high-dimensional variable-length feature maps 105; the feature encoding layer 102 performs a pooling operation on the feature map 105 to convert it into a fixed-length feature vector 106; the output layer 103 performs feature mapping on the feature vector 106 and outputs a speaker representation vector 107; and then the speaker representation vector 107 is input into the speaker recognition classifier 104 to predict the speaker, and a speaker prediction result 108 is obtained.

[0054] Obviously, the related art directly performs speaker identification based on the speaker representation vector extracted from the input audio, without considering the influence of age information on the speaker representation vector corresponding to the same speaker. The speaker-related information of the same speaker object will change at different age stages. For example, the pitch and timbre of the same speaker will be significantly different before and after the voice change period. Therefore, it is obviously inaccurate to directly use the speaker representation vector for subsequent speaker identification.

[0055] In order to improve the accuracy of subsequent speaker recognition and avoid the influence of age information on speaker recognition, this application will focus on how to generate a speaker representation vector that does not contain age information, thereby eliminating the influence of age information on speaker recognition. Figure 2 As shown, it is a schematic diagram of the process of speaker recognition shown in an exemplary embodiment of the present application. Feature extraction is performed on the target audio 201 to extract information related to the speaker, and a target feature map 202 is obtained; further, feature encoding is performed based on the target feature map 202 to obtain object features 203 representing object information corresponding to the speaker, and age features 204 representing age information corresponding to the speaker; by subtracting the age feature 204 from the object feature 203 to obtain an identity feature 205 that does not contain age information, the identity feature 205 is used for subsequent speaker recognition, which can avoid the influence of age information on the accuracy of speaker recognition.

[0056] Figure 3 FIG. 1 is a schematic diagram of a computer system shown in an exemplary embodiment of the present application. Figure 3 As shown, the computer system includes a first device 310 and a second device 320 .

[0057] The first device 310 is a training device for training a feature extraction network for extracting identity features that do not contain age information. After the feature extraction network training is completed, the first device 310 can send the trained feature extraction network to the second device 320, so as to deploy the feature extraction network in the second device 320. The second device 320 is a device that uses the feature extraction network to extract identity features.

[0058] Optionally, the first device 310 and the second device 320 may be computer devices with machine learning capabilities, for example, the computer device may be a terminal or a server.

[0059] Optionally, the first device 310 and the second device 320 may be the same computer device, or the first device 310 and the second device 320 may be different computer devices. Furthermore, when the first device 310 and the second device 320 are different devices, the first device 310 and the second device 320 may be devices of the same type, such as the first device 310 and the second device 320 may both be servers; or the first device 310 and the second device 320 may also be devices of different types. The server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal may be a smart phone, a vehicle-mounted terminal, a smart TV, a wearable device, a tablet computer, a laptop, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, and this application is not limited thereto.

[0060] Please refer to Figure 4 , which shows a flowchart of an audio processing method provided by an exemplary embodiment of the present application, and the method is applied to Figure 3 Taking the second device 320 as an example, the method includes:

[0061] Step 401, extract features from the first target audio to obtain a first target feature map.

[0062] The first target audio is audio containing text information, such as voice, dubbing, song, etc. When there is a need to extract identity features corresponding to the speaker in the first target audio, in a possible implementation, feature extraction can be performed on the first target audio through a feature extraction network, so as to extract a first target feature map containing information related to the speaker from the first target audio.

[0063] Optionally, the feature extraction network for extracting features from the first target audio may be a commonly used feature extraction network such as TDNN, ResNet, or ECAPA-TDNN.

[0064] Step 402, feature encoding is performed based on the first target feature map to obtain a first target object feature and a first target age feature, wherein the first target object feature is used to characterize object information of the first target audio corresponding to the first target speaking object, and the first target age feature is used to characterize age information of the first target speaking object.

[0065] Different from the related art, only the extracted feature map is feature-encoded to obtain the object feature. In the embodiment of the present application, in order to avoid the influence of age information on the speaking object feature, in the process of feature encoding based on the first target feature map, not only the first target object feature containing all the object information is extracted from the first target feature map, but also the first target age feature containing the age information is extracted, so that the age information can be removed from all the object information later.

[0066] Step 403: Separate a first target identity feature from the first target object feature based on the first target age feature, wherein the first target identity feature is used to characterize the identity information of the first target speaking object, and the object information at least includes the identity information and the age information.

[0067] The object information includes at least identity information and age information, and the age information is information in the object information that changes as the speaker grows older, and is variable information; then the variable information is removed from the object information, and the identity information that does not change with age can be obtained, so that the identity information can uniquely represent the first target speaker. It should be noted that in the field of audio processing, the identity information is information related to audio features such as the speaker's timbre, pitch, fundamental frequency, speech speed, etc.

[0068] In a possible implementation, after obtaining the first target age feature and the first target object feature, the first target age feature can be subtracted from the first target object feature to obtain the first target identity feature, so that the first target identity feature does not contain the variable factor of age, so that the extracted first target identity feature is the speaking object feature that is not affected by the age factor.

[0069] Optionally, after obtaining the speaking object feature (first target identity feature) that is not affected by age factors, the first target identity feature can be applied to other audio processing scenarios, for example, directly identifying the first target speaking object corresponding to the first target audio based on the first target identity feature; it is also possible to determine whether the two target audios correspond to the same speaking object by comparing whether the identity features of the two target audios are similar; and speech recognition (speech to text) can be performed based on the processed first target identity feature to avoid the problem of unclear pronunciation due to age factors.

[0070] In summary, an embodiment of the present application provides a method for identifying a speaking object: by performing feature extraction and feature encoding on a target audio of a speaking object, object information (object feature) corresponding to the speaking object and age information (age feature) of the speaking object are extracted from the target audio; since the object information includes age information and identity information, the identity feature can be separated from the object feature based on the extracted age feature, so that the separated identity feature does not have age information, thereby avoiding the influence of age information on identity feature and improving the extraction accuracy of identity feature; moreover, in a scenario of identifying a speaking object based on identity feature, the influence of age information on the recognition accuracy of the same speaking object can also be avoided, thereby further improving the recognition accuracy of the speaking object.

[0071] The identity feature is obtained by separating the age feature from the object feature, so how to accurately extract the age feature is the key to extracting the identity feature. The following embodiment provides a model training framework for extracting identity features that do not contain age information. The model training framework uses two classification tasks to train a feature extraction network to accurately extract age features and identity features.

[0072] Please refer to Figure 5 , which shows a flowchart of an audio processing method provided by another exemplary embodiment of the present application, in which the method is applied to Figure 3 Taking the first device 310 as an example, the method includes:

[0073] Step 501: extract features from the sample audio through a first feature extraction network to obtain a first sample feature map.

[0074] Before the model training begins, training samples need to be prepared in advance. Since the purpose of the model training provided in the embodiment of the present application is to accurately extract age features and identity features, the corresponding model training tasks include: age classification tasks and identity classification tasks, and it is necessary to pre-label the age label and identity label of each sample audio corresponding to the sample speaker, so that training can be performed based on the labeling results and prediction results in the subsequent model training process. The corresponding training sample set includes several training sample pairs, each training sample pair includes: sample audio, the labeled identity label corresponding to the sample audio, and the labeled age label corresponding to the sample audio.

[0075] Optionally, since different speaking objects correspond to different identity object features, different speaking objects should be assigned different identity tags.

[0076] Optionally, since the speaker characteristics of the same speaker in a certain age period remain basically unchanged, that is, the age characteristics are unchanged in a certain age period, in order to conform to the changing trend of age characteristics in real application scenarios, an age group classification method is used to set the age label, that is, a certain age period is set as an age label. Indicatively, the age can be divided into 7 groups, namely: 0-20, 21-30, 31-40, 41-50, 51-60, 61-70 and 70-100, and each age group corresponds to a marked age label, for example, 0-20 corresponds to marked age label 1, 21-30 corresponds to marked age label 2, 31-40 corresponds to marked age label 3, 41-50 corresponds to marked age label 4, 51-60 corresponds to marked age label 5, 61-70 corresponds to marked age label 6, and 70-100 corresponds to marked age label 7.

[0077] In the process of extracting age features and identity features in this embodiment, the feature extraction process mainly includes two stages. The first stage is to first extract information related to the speaking object from the sample audio through the first feature extraction network to obtain a first sample feature map; the second stage is to perform feature encoding on the first sample feature map through the second feature extraction network to obtain sample object features and sample age features. Subsequently, the sample identity features can be separated based on the sample object features and the sample age features.

[0078] Step 502, feature encoding is performed on the first sample feature map through a second feature extraction network to obtain sample object features and sample age features, wherein the sample object features are used to characterize object information of the sample audio corresponding to the sample speaking object, and the sample age features are used to characterize age information of the sample speaking object.

[0079] In a possible implementation, a second feature extraction network is provided to extract sample object features including all object information corresponding to the sample speaking object and sample age features including age information corresponding to the sample speaking object from the first sample feature map.

[0080] The specific manner in which the second feature extraction network extracts sample age features and sample object features is described in detail in the following embodiments, and will not be described in detail in this embodiment.

[0081] Step 503: Separate the sample identity feature from the sample object feature based on the sample age feature.

[0082] The relationship between the sample identity feature, the sample age feature and the sample object feature can be shown as formula (1):

[0083] Sample identity characteristics = sample subject characteristics - sample age characteristics (1)

[0084] It can be seen from formula (1) that in order to obtain the speaking object features that do not contain age information, the sample age features can be subtracted from the sample object features, that is, the sample identity features that do not contain age information can be obtained.

[0085] It should be noted that in order to implement the subtraction operation, it is necessary to ensure that the extracted sample object features and sample age features are feature vectors of the same dimension.

[0086] Step 504: perform identity recognition based on the sample identity feature to obtain a sample identity label.

[0087] In order to judge whether the identity features extracted through the above steps are valid, in a possible implementation, an identity recognition task is set up to perform identity recognition based on the extracted sample identity features and predict the corresponding sample identity label, so that the validity of the extracted identity features can be determined by comparing the difference between the sample identity label and the annotated identity label.

[0088] Step 505: Perform age recognition based on the sample age feature to obtain a first sample age label.

[0089] Since the accuracy of extracting the sample identity feature depends on the accuracy of extracting the sample age feature, it is necessary to judge whether not only the identity feature is valid, but also whether the age feature is valid. Therefore, in a possible implementation, not only an identity recognition task but also an age recognition task is provided. Age recognition is performed based on the extracted sample age feature, and the first sample age label of the sample speaker corresponding to the sample audio is predicted, so that the validity of the extracted age feature can be determined by comparing the difference between the first sample age label and the marked age label, so as to indirectly determine whether the extracted identity feature is valid.

[0090] It should be noted that step 504 and step 505 may be performed simultaneously, or step 504 may be performed first and then step 505; or step 505 may be performed first and then step 504. This embodiment does not limit the order of performing step 504 and step 505.

[0091] Step 506: Train a first feature extraction network and a second feature extraction network based on the sample identity label, the annotated identity label corresponding to the sample audio, the first sample age label, and the annotated age label corresponding to the sample audio.

[0092] During the model training process, in order to improve the feature extraction accuracy of the first feature extraction network and the second feature extraction network, in a possible implementation, after obtaining the sample identity label identified based on the sample identity feature and the first sample age label identified based on the sample age feature, the first feature extraction network and the second feature extraction network can be trained by comparing the loss between the prediction result (sample identity label and first sample age label) and the labeled label (labeled identity label and labeled age label) so that the first feature extraction network and the second feature extraction network learn in the direction of feature extraction accuracy. Thus, through multiple rounds of iterative learning, the first feature extraction network and the second feature extraction network have the ability to accurately extract age features and identity features, so that the feature extraction network can be used to extract identity features that do not contain age information in subsequent applications.

[0093] like Figure 6As shown, it shows a schematic diagram of the model training process shown in an exemplary embodiment of the present application. Sample audio 601 is input into the first feature extraction network 602 for feature extraction to obtain a sample feature map 603; the sample feature map 603 is feature encoded by the second feature extraction network 604 to obtain a sample object feature 605 and a sample age feature 606; the sample age feature 606 is subtracted from the sample object feature 605 to obtain a sample identity feature 607 with age information removed; the sample identity feature 607 is identified to obtain a sample identity label 608; and the sample age feature 606 is identified to obtain a sample age label 609; so as to train the first feature extraction network 602 and the second feature extraction network 604 according to the sample identity label 608, the annotated identity label 610, and the sample age label 609 and the annotated age label 611.

[0094] To summarize, in the embodiments of the present application, feature extraction is performed on the sample audio through the first feature extraction network and the second feature extraction network to extract sample age features and sample identity features, and age recognition is performed on the sample age features, and identity recognition is performed on the sample identity features. The age recognition results and the identity recognition results are used together with the labeling results to train the first feature extraction network and the second feature extraction network, so that the first feature extraction network and the second feature extraction network have the ability to accurately extract age features and identity features, so that the feature extraction network can be used to extract identity features that do not contain age information in subsequent applications.

[0095] Since both the first sample feature graph and the sample object feature contain all the object information of the sample speaking object, there are two feature extraction methods in the process of extracting the sample age feature. The age feature can be directly extracted from the first sample feature graph, or the sample age feature can be extracted from the sample object feature. This embodiment focuses on describing the extraction process of the sample age feature.

[0096] Please refer to Figure 7 , which shows a flowchart of an audio processing method provided by another exemplary embodiment of the present application, in which the method is applied to Figure 3 Taking the first device 310 as an example, the method includes:

[0097] Step 701: extract features from the sample audio through a first feature extraction network to obtain a first sample feature map.

[0098] The implementation of step 701 may refer to the above embodiment, and will not be described in detail in this embodiment.

[0099] Step 702: feature encoding is performed on the first sample feature map through a first sub-feature extraction network to obtain sample object features.

[0100] Since the first sample feature map contains all object information corresponding to the speaking object, in a possible implementation manner, feature extraction can be directly performed on the first sample feature map to obtain sample object features and sample age features respectively.

[0101] Optionally, since the information contained in the sample object feature and the sample age feature is different, two sub-feature extraction networks are set up to respectively extract features from the first sample feature map. Correspondingly, the second feature extraction network includes a first sub-feature extraction network and a second sub-feature extraction network, wherein the first sub-feature extraction network is used to perform feature encoding on the first sample feature map to obtain sample object features containing all object information of the speaking object.

[0102] Since the first sample feature map to sample object feature is only a process of converting a multi-dimensional feature map into a one-dimensional feature vector, the corresponding network structure of the first sub-feature extraction network includes: a pooling layer and a fully connected layer, wherein the pooling layer is used to perform pooling operations on the first sample feature map to obtain a feature vector, and the fully connected layer is used to perform feature mapping processing on the feature vector to obtain the sample object feature.

[0103] Optionally, the pooling layer in the first sub-feature extraction network may adopt a global average pooling method or a global maximum pooling method.

[0104] Step 703: extract age features from the first sample feature graph through a second sub-feature extraction network to obtain sample age features.

[0105] In a possible implementation, the second sub-feature extraction network is used to extract age features from the first sample feature graph to obtain sample age features containing more age information. Compared with the first sub-feature extraction network, the second sub-feature extraction network needs to pay more attention to age information during feature extraction.

[0106] In order for the second sub-feature extraction network to extract the age information in the first sample feature map, the second sub-feature extraction network needs to have the ability to distinguish age features. Correspondingly, in an exemplary example, step 703 may include steps 703A to 703C.

[0107] Step 703A, the first sample feature map is processed through the attention mechanism layer to obtain a second sample feature map, and the attention score of the age feature in the second sample feature map is higher than the attention score of the non-age feature.

[0108] Unlike the first sub-feature extraction network, the second sub-feature extraction network includes an attention mechanism layer, a pooling layer, and a fully connected layer. By adding an additional attention mechanism layer, the second sub-feature extraction network can pay more attention to age-related features during feature extraction. Optionally, the pooling layer and the fully connected layer can both be the same as those of the first sub-feature extraction network; or different from those of the first sub-feature extraction network.

[0109] In order to enable the second sub-feature extraction network to have the ability to pay attention to the age information in the first sample feature map, in a possible implementation, an attention mechanism layer is provided in the second sub-feature extraction network, and attention scores are assigned to different features in the first sample feature map through the attention mechanism layer, and the attention score of the age feature is set higher than the attention score of the non-age feature, so that more attention can be paid to age-related features in the subsequent feature extraction process; after the attention score processing, the second sample feature map can be obtained, and the age feature in the second sample feature map has a higher feature value.

[0110] Step 703B, performing a pooling operation on the second sample feature map through a pooling layer to obtain an intermediate sample feature.

[0111] In a possible implementation, after the second sample feature map is obtained, a pooling operation may be performed on the second sample feature map through a pooling layer to transform the multi-dimensional feature map into a one-dimensional intermediate sample feature.

[0112] Step 703C, perform feature mapping processing on the intermediate sample features through the fully connected layer to obtain the sample age features.

[0113] The intermediate sample features output by the pooling layer are input into the fully connected layer for feature mapping processing, and the sample age features containing age information can be obtained.

[0114] like Figure 8As shown, it shows a schematic diagram of the model training process shown in another exemplary embodiment of the present application. The sample audio 801 is input into the first feature extraction network 802 for feature extraction to obtain a first sample feature map 803; after obtaining the first sample feature map 803, on the one hand, the first sample feature map 803 is input into the global statistical pooling layer 804 and the fully connected layer 805, and the sample object feature 806 is output; on the other hand, the first sample feature map 803 is input into the attention mechanism layer 807 to obtain a second sample feature map 808; the second sample feature map 808 is subjected to feature extraction through the global statistical pooling layer 809 and the fully connected layer 810 to obtain the sample age feature 811; from the sample object feature By subtracting the sample age feature 811 from feature 806, a sample identity feature 812 with age information removed can be obtained; the sample identity feature 812 is identified through the identity classification network to obtain a sample identity label 813; and the sample age feature 811 is identified through the first age classification network to obtain a sample age label 814; thereby determining the first classification loss based on the sample identity label 813 and the labeled identity label 815, and determining the second classification loss based on the sample age label 814 and the labeled age label 816, so as to perform model training based on the first classification loss and the second classification loss.

[0115] Step 704: feature encode the first sample feature map through the third sub-feature extraction network to obtain sample object features.

[0116] In another possible implementation, age features can be extracted from sample object features, and the corresponding second feature extraction network may include a third sub-feature extraction network and a fourth sub-feature extraction network, wherein the third sub-feature extraction network is used to perform feature encoding on the first sample feature map so as to extract sample object features from the first sample feature map, and the fourth sub-feature extraction network is used to extract sample age features from the sample object features.

[0117] Optionally, the third sub-feature extraction network and the first sub-feature extraction network are both used to extract sample object features from the first sample feature map, and the network structure of the third sub-feature extraction network can be the same as the network structure of the first sub-feature extraction network.

[0118] Step 705: extract age features from sample object features through the fourth sub-feature extraction network to obtain sample age features.

[0119] Since the sample object features include not only age information but also other speaking object information, in a possible implementation, it is necessary to extract age features from the sample object features through a fourth sub-feature extraction network to obtain sample age features including age information.

[0120] Optionally, the purpose of the fourth sub-feature extraction network is to pay more attention to age-related features during feature extraction. Therefore, the fourth sub-feature extraction network can also use an attention mechanism layer to assign a higher attention score to age-related features.

[0121] like Fig. 9 As shown, it shows a schematic diagram of the model training process shown in another exemplary embodiment of the present application. Sample audio 901 is input into the first feature extraction network 902 for feature extraction to obtain a first sample feature map 903; after obtaining the first sample feature map 903, the first sample feature map 903 is input into the global statistical pooling layer 904 and the fully connected layer 905, and the sample object feature 906 is output; the sample object feature 906 is input into the fourth sub-feature extraction network 907 to obtain a sample age feature 908; the sample age feature 908 is subtracted from the sample object feature 906 to obtain a sample identity feature 909 with age information removed; the sample identity feature 909 is identified through the identity classification network to obtain a sample identity label 910; and the sample age feature 908 is identified through the first age classification network to obtain a sample age label 911; thereby determining a first classification loss based on the sample identity label 910 and the labeled identity label 912, and determining a second classification loss based on the sample age label 911 and the labeled age label 913, so as to perform model training based on the first classification loss and the second classification loss.

[0122] Step 706: Separate the sample identity feature from the sample object feature based on the sample age feature.

[0123] The implementation of step 706 may refer to the above embodiment, and this embodiment will not be described in detail here.

[0124] Step 707, input the sample identity feature into the identity classification network to obtain the sample identity label output by the identity classification network.

[0125] In order to judge the accuracy of the extracted sample identity features, an identity classification network is set up. The extracted sample identity features are input into the identity classification network. The identity classification network performs identity classification based on the sample identity features and predicts the sample identity label corresponding to the sample identity features, so as to calculate the loss of the identity classification task based on the sample identity label.

[0126] In an illustrative example, step 707 may include step 707A and step 707B.

[0127] Step 707A, determining the sample similarity between the sample identity feature and the candidate identity feature corresponding to the candidate identity tag through the identity classification network.

[0128] Regarding the way in which the identity classification network predicts sample identity labels, in one possible implementation, if the sample identity feature corresponding to the sample audio is similar to the candidate identity feature corresponding to a candidate identity label, it can be determined that the sample speaking object corresponding to the sample audio corresponds to the candidate identity label. Therefore, the identity classification network determines the sample identity feature and the candidate identity feature corresponding to the candidate identity label by determining the sample similarity between the sample identity feature and the candidate identity feature corresponding to the candidate identity label, and then determines the sample identity label corresponding to the sample identity feature through the sample similarity.

[0129] Step 707B: determine the candidate identity label corresponding to the candidate identity feature with the highest sample similarity as the sample identity label.

[0130] Optionally, the higher the sample similarity, the more similar the candidate identity feature is to the sample identity feature, which means that the sample speaking object corresponding to the sample audio corresponds to the candidate identity label corresponding to the candidate identity feature. Therefore, in a possible implementation, the candidate identity label corresponding to the candidate identity feature with the highest sample similarity can be determined as the identity recognition result of the identity classification network based on the sample identity feature - the sample identity label.

[0131] Step 708: Input the sample age feature into the first age classification network to obtain a first sample age label output by the first age classification network.

[0132] In order to judge the accuracy of the extracted sample age features, a first age classification network is set up. The extracted sample age features are input into the first age classification network. The first age network performs age classification based on the sample age features, predicts the age stage of the sample speaker corresponding to the sample audio, and outputs a first sample age label so that the loss calculation of the age classification task can be performed based on the first sample age label.

[0133] In the process of predicting the first sample age label by the first age classification network, the first age classification network predicts the probability of the sample being in each candidate age label based on the sample age feature, and determines the candidate age label with the highest probability as the first sample age label.

[0134] Step 709: Determine the first classification loss corresponding to the identity network based on the sample identity label and the annotated identity label.

[0135] In order to determine whether the sample identity feature is accurate, it is necessary to calculate the loss of the predicted result and the labeled result of the sample identity feature according to the identity classification network. In a possible implementation, based on the predicted result sample identity label and the labeled identity label, the first classification loss corresponding to the identity classification network can be determined. The first classification loss can use the cross entropy loss.

[0136] In an illustrative example, the calculation process of the first classification loss can be shown as formula (2):

[0137]

[0138] in, represents the first classification loss, L ce represents the cross entropy loss, z id represents the sample identity feature, C(z id ) represents the sample identity label (i.e., the prediction result of the identity classification network based on the sample identity features), y id Indicates the annotated identity label corresponding to the sample audio, y id The value of can be {0, 1, 2, …, N}.

[0139] Step 710: Determine a second classification loss corresponding to the first age classification network based on the first sample age label and the annotated age label.

[0140] In order to determine whether the sample age feature is accurate, it is necessary to calculate the loss of the prediction result and the labeling result of the sample age feature according to the first age classification network. In a possible implementation, based on the predicted sample age label and the labeled age label, the second classification loss corresponding to the age classification network can be determined. The second classification loss can also use cross entropy loss.

[0141] In an exemplary example, the calculation process of the second classification loss can be shown as formula (3):

[0142]

[0143] in, represents the second classification loss, L ce represents the cross entropy loss, z age represents the sample age characteristics, A(z age ) represents the first sample age label (i.e., the prediction result of the first age classification network based on the sample age feature), y age Indicates the age label, y age The value of can be {0, 1, 2, ..., 6}, indicating that it is divided into 7 age groups.

[0144] Step 711: Based on the first classification loss and the second classification loss, train the first feature extraction network, the second feature extraction network, the first age classification network, and the identity classification network.

[0145] In order to enable the model to learn to more accurately extract sample age features and sample identity features, in a possible implementation, after determining the first classification loss and the second classification loss, the first feature extraction network, the second feature extraction network, the first age classification network, and the identity classification network can be jointly trained based on the sum of the first classification loss and the second classification loss.

[0146] In an illustrative example, the calculation process of the total loss of the model can be shown as formula (4):

[0147]

[0148] in, represents the total loss of the model, represents the first classification loss, represents the second classification loss, λ age Represents the loss weight of the second classification loss.

[0149] Optionally, during model training, developers can adjust the weight ratio between the first classification loss and the second classification loss as needed, that is, adjust λ age .

[0150] In this embodiment, the age-related information in the high-dimensional feature graph is captured by setting an attention mechanism layer, thereby obtaining a fixed-length sample age feature. In addition, the second classification loss (age classification loss) is set to control the extracted sample age features to contain age information, and the first classification loss (identity classification loss) is set to control the extracted sample identity features to contain accurate identity information, so that the trained model has the ability to accurately extract sample age features and sample identity features.

[0151] In another possible application scenario, in order to further eliminate age information from the sample identity features, a second age classification network is provided for age recognition of the sample identity features. The greater the loss of the second age classification network, the less age information is contained in the sample identity features, so as to reduce the age information in the sample identity features through adversarial learning.

[0152] Please refer to Fig.10 , which shows a flowchart of an audio processing method provided by another exemplary embodiment of the present application, in which the method is applied to Figure 3 Taking the first device 310 as an example, the method includes:

[0153] Step 1001: extract features from sample audio through a first feature extraction network to obtain a first sample feature map.

[0154] Step 1002: feature encoding is performed on the first sample feature map through a second feature extraction network to obtain sample object features and sample age features.

[0155] Step 1003: Separate the sample identity feature from the sample object feature based on the sample age feature.

[0156] Step 1004, input the sample identity feature into the identity classification network to obtain the sample identity label output by the identity classification network.

[0157] Step 1005: input the sample age feature into the first age classification network to obtain the first sample age label output by the first age classification network.

[0158] The implementation of steps 1001 to 1005 may refer to the above embodiments, and will not be described in detail in this embodiment.

[0159] Step 1006, inputting the sample identity feature into the second age classification network to obtain a second sample age label output by the second age classification network, wherein the second age classification network has a gradient reversal layer.

[0160] In order to further reduce the age information contained in the sample identity features, in a possible implementation, a second age classification network with a gradient reversal layer (GRL) is added, and the sample identity features are input into the second age classification network, and the second age classification network performs age classification to obtain a second sample age label output by the second age classification network.

[0161] Step 1007: Determine the first classification loss corresponding to the identity network based on the sample identity label and the annotated identity label.

[0162] Step 1008: Determine a second classification loss corresponding to the first age classification network based on the first sample age label and the annotated age label.

[0163] The implementation of step 1007 and step 1008 can refer to the above embodiment, and this embodiment will not be described in detail here.

[0164] Step 1009: Determine the third classification loss corresponding to the second age classification network based on the second sample age label and the annotated age label.

[0165] Age recognition is performed based on sample identity features through the second age classification network. If the sample identity features are extracted more accurately and do not contain age information, the larger the third classification loss of the second age classification network, the more the age information in the sample identity features can be further reduced by gradient reversal. Therefore, in a possible implementation, in addition to the identity classification task and the age classification task, an age adversarial learning task is also provided. The third classification loss corresponding to the second age classification network is determined by the second sample age label determined based on the sample identity features and the annotated age label, so as to achieve the purpose of adversarial learning.

[0166] Step 1010: Based on the first classification loss, the second classification loss, and the third classification loss, train the first feature extraction network, the second feature extraction network, the first age classification network, the second age classification network, and the identity classification network.

[0167] In one possible implementation, after the age object learning task is introduced, the total loss of the model consists of three parts: the first classification loss (the loss of the identity classification task), the second classification loss (the loss of the age classification task), and the third classification loss (the loss of the age adversarial learning task). During the model training process, the sum of the first classification loss, the second classification loss, and the third classification loss can be used to train the first feature extraction network, the second feature extraction network, the first age classification network, the second age classification network, and the identity classification network.

[0168] Optionally, during the model training process, the loss ratio between different classification losses can be adjusted according to the training purpose requirements, and different loss weights can correspond to each classification loss. During the loss calculation process, the total loss of the model can be determined based on the first classification loss and the first loss weight, the second classification loss and the second loss weight, and the third classification loss and the third loss weight, and then the first feature extraction network, the second feature extraction network, the first age classification network, the second age classification network, and the identity classification network are trained according to the total loss.

[0169] In an illustrative example, after introducing the age adversarial learning task, the calculation process of the total loss of the model can be shown as formula (5):

[0170]

[0171] in, represents the total loss of the model, represents the first classification loss, represents the second classification loss, λ age represents the loss weight of the second classification loss, represents the third classification loss, λ grl Represents the loss weight of the third classification loss.

[0172] like Fig.11 As shown, it shows a schematic diagram of the model training process shown in another exemplary embodiment of the present application. The sample audio 1101 is input into the first feature extraction network 1102 for feature extraction to obtain a first sample feature map 1103; after obtaining the first sample feature map 1103, on the one hand, the first sample feature map 1103 is input into the global statistical pooling layer 1104 and the fully connected layer 1105, and the sample object feature 1106 is output; on the other hand, the first sample feature map 1103 is input into the attention mechanism layer 1107 to obtain a second sample feature map 1108; the second sample feature map 1108 is subjected to feature extraction through the global statistical pooling layer 1109 and the fully connected layer 1110 to obtain a sample age feature 1111; the sample age feature 1111 is subtracted from the sample object feature 1106 to obtain a sample identity feature 1112 with age information removed; The sample identity feature 1112 is identified through the identity classification network to obtain the sample identity label 1113; the sample age feature 1111 is identified through the first age classification network to obtain the first sample age label 1114; the sample identity feature 1112 is identified through the second age classification network to obtain the second sample age label 1117; thereby determining the first classification loss based on the sample identity label 1113 and the labeled identity label 1115, determining the second classification loss based on the sample age label 1114 and the labeled age label 1116, and determining the third classification loss based on the second sample age label 1117 and the labeled age label 1116, so as to perform model training based on the first classification loss, the second classification loss and the third classification loss.

[0173] In this embodiment, a second age classification network with a gradient reversal layer is additionally added to perform age classification on the sample identity features, thereby achieving the adversarial learning goal that the greater the loss, the less age information contained in the sample identity features, thereby further reducing the age information in the sample identity features.

[0174] In order to verify the performance of the model trained in this embodiment, the developers conducted a verification comparison on a large data set. The results of training and testing using VoxCeleb data are shown in Table 1.

[0175] Table 1

[0176]

[0177]

[0178] ARE represents the method proposed in this embodiment to use the attention mechanism to operate at the high-dimensional feature map level to capture the features of age-related information, then obtain the age information vector (age feature) through the statistical pooling layer and the fully connected layer, and then subtract the age information vector from the speaker vector (object feature) to obtain the identity feature vector. ADAL is a method for using the ARE module and the gradient reversal layer together. The results show that the ARE structure and ADAL structure proposed in this embodiment do not deteriorate on Vox-E and Vox-H, and perform better than the Baseline, GRL and Age Residual methods on the Vox cross-age test set. As the time span increases, the improvement becomes more obvious, especially on the test set with an age span of 20 years (Vox-20), with an improvement of 5% and 10% respectively.

[0179] From the extraction process of sample age features during model training in the above embodiment, it can be seen that there are two extraction processes for sample age features: directly extracting from the first sample feature map, or extracting from sample object features. In the model application process, there are also two corresponding age feature extraction methods.

[0180] exist Figure 4 On the basis of Fig.12 As shown, step 402 may include step 402A and step 402B, or step 402 may include step 402C and step 402D.

[0181] Step 402A, feature encoding is performed on the first target feature map to obtain first target object features.

[0182] The model trained in the above embodiment mainly includes a first feature extraction network, a second feature extraction network, a first age classification network, a second age classification network, and an identity classification network. Since the purpose of model training is mainly to extract identity features that do not contain age information, in the model application stage, only the first feature extraction network and the second feature extraction network need to be deployed.

[0183] In a possible implementation, the first target audio is input into a first feature extraction network to extract a high-dimensional first target feature map, and then the first target feature map can be feature encoded through a second feature extraction network to obtain first target object features and first target age features.

[0184] According to different age feature extraction methods, the second feature extraction network may include a first sub-feature extraction network and a second sub-feature extraction network, or the second feature extraction network may include a third sub-feature extraction network and a fourth sub-feature extraction network.

[0185] For a scenario where the second feature extraction network includes a first sub-feature extraction network and a second sub-feature extraction network, the first sub-feature extraction network can be used to perform feature encoding on the first target feature map to obtain the first target object feature.

[0186] Step 402B, extracting age features from the first target feature graph to obtain first target age features.

[0187] At the same time, the second sub-feature extraction network is used to extract the age feature of the first target feature map to obtain the second target age feature.

[0188] Optionally, the second sub-feature extraction network is composed of an attention mechanism layer, a pooling layer and a fully connected layer. Correspondingly, in an exemplary example, step 402B may include steps 1 to 3.

[0189] Step 1: Process the first target feature map through the attention mechanism layer to obtain a second target feature map, in which the attention score of the age feature is higher than the attention score of the non-age feature.

[0190] In a possible implementation, the first target feature map input to the second sub-feature extraction network is first processed by an attention mechanism layer, and attention scores are assigned to different features in the first target feature map through the attention mechanism layer, and the attention score of the age feature is set higher than the attention score of the non-age feature, so that more attention can be paid to age-related features in the subsequent feature extraction process; after the attention score processing, the second target feature map can be obtained, and the age feature in the second target feature map has a higher feature value.

[0191] Step 2: Perform a pooling operation on the second target feature map through a pooling layer to obtain intermediate sample features.

[0192] After the second target feature map is obtained, a pooling operation can be performed on the second target feature map through a pooling layer to convert the high-dimensional feature map into a one-dimensional feature vector to obtain an intermediate sample feature.

[0193] Step 3: Perform feature mapping on the intermediate sample features through the fully connected layer to obtain the first target age feature.

[0194] The intermediate sample features output by the pooling layer are input into the fully connected layer for feature mapping processing, so that the first target age features containing age information can be obtained.

[0195] like Fig.13As shown, it shows a schematic diagram of the extraction process of identity features shown in an exemplary embodiment of the present application. The target audio 1301 is input into the first feature extraction network 1302 for feature extraction to obtain a first target feature map 1303; after obtaining the first target feature map 1303, on the one hand, the first target feature map 1303 is input into the global statistical pooling layer 1304 and the fully connected layer 1305, and the target object feature 1306 is output; on the other hand, the first target feature map 1303 is input into the attention mechanism layer 1307 to obtain a second target feature map 1308; the second target feature map 1308 is subjected to feature extraction through the global statistical pooling layer 1309 and the fully connected layer 1310 to obtain a target age feature 1311; the target age feature 1311 is subtracted from the target object feature 1306 to obtain a target identity feature 1312 with age information removed.

[0196] Step 402C, feature encoding the first target feature map to obtain the first target object feature.

[0197] For a scenario where the second feature extraction network includes a third sub-feature extraction network and a fourth sub-feature extraction network, the third sub-feature extraction network can be used to perform feature encoding on the first target feature map to obtain the first target object feature.

[0198] Step 402D: extract age features from the first target object features to obtain first target age features.

[0199] Furthermore, the age feature of the first target object feature is extracted through the fourth sub-feature extraction network to obtain the first target age feature.

[0200] like Fig.14 As shown, it shows a schematic diagram of the extraction process of identity features shown in another exemplary embodiment of the present application. The target audio 1401 is input into the first feature extraction network 1402 for feature extraction to obtain a first target feature map 1403; after obtaining the first target feature map 1403, the first target feature map 1403 is input into the global statistical pooling layer 1404 and the fully connected layer 1405, and the target object feature 1406 is output; the target object feature 1406 is input into the fourth sub-feature extraction network 1407 to obtain a target age feature 1408; the target age feature 1408 is subtracted from the target object feature 1406 to obtain a target identity feature 1409 with age information removed.

[0201] In this embodiment, the first target age feature is extracted from the first target feature map by using two different age feature extraction methods, so that the first target identity feature can be subsequently separated from the first target object feature based on the first target age feature, thereby realizing the extraction of identity features that do not contain age information.

[0202] In a possible application scenario, the audio processing method provided in the embodiment of the present application can be used to establish a relationship between an identity tag and an identity feature (vector) to construct an identity feature library, so that when the speaker of the audio needs to be identified later, the identity features of the audio can be obtained and compared with the identity features in the identity feature library for similarity, thereby accurately determining the speaker corresponding to the audio.

[0203] Please refer to Fig.15 , which shows a flowchart of an audio processing method provided by another exemplary embodiment of the present application, in which the method is applied to Figure 3 Taking the second device 320 as an example, the method includes:

[0204] Step 1501, extract features from the first target audio to obtain a first target feature map.

[0205] Step 1502, feature encoding is performed based on the first target feature map to obtain a first target object feature and a first target age feature.

[0206] Step 1503: Separate the first target identity feature from the first target object feature based on the first target age feature.

[0207] The implementation of steps 1501 to 1503 may refer to the above embodiment, and will not be described in detail in this embodiment.

[0208] Step 1504: Acquire a second target identity feature corresponding to the second target audio, where the second target identity feature is used to represent identity information of a second target speaker corresponding to the second target audio.

[0209] In a possible application scenario, the similarity between the identity features corresponding to the two target audios can be compared to determine whether the speakers of the two target audios are the same speaker. A first target identity feature corresponding to the first target audio and a second target identity feature corresponding to the second target audio are obtained respectively, and the second target identity feature does not contain age information either.

[0210] Optionally, the second target identity feature may have been extracted before the first target identity feature is extracted, or may be extracted simultaneously with the first target identity feature. In an illustrative example, step 1504 may also include steps 1504A to 1504C.

[0211] Step 1504A, extract features from the second target audio to obtain a second target feature map.

[0212] Step 1504B, feature encoding is performed based on the second target feature map to obtain a second target object feature and a second target age feature, wherein the second target object feature is used to characterize the object information of the second target speaking object, and the second target age feature is used to characterize the age information of the second target speaking object.

[0213] Step 1504C, based on the second target age feature, separate the second target identity feature from the second target object feature.

[0214] The method for extracting the second target identity feature is the same as the method for extracting the first target identity feature. The specific extraction process of the second target identity feature can refer to the extraction process of the first target identity feature, which will not be described in detail in this embodiment.

[0215] Step 1505 , determining the target similarity between the first target identity feature and the second target identity feature.

[0216] If the first target speaking object corresponding to the first target audio is the same as the second target speaking object corresponding to the second target audio, the higher the similarity of the target identity features between the first target audio and the second target audio, the higher the similarity. Therefore, in a possible implementation, by determining the target similarity between the first target identity feature and the second target identity feature, it is determined whether the first target audio and the second target audio correspond to the same speaking object.

[0217] The target similarity between the first target identity feature and the second target identity feature can be determined by directly calculating the cosine similarity between the first target identity feature and the second target identity feature. In an exemplary example, the target similarity determination process can be expressed as:

[0218] Target similarity = Z1Z2 / (||Z1||*||Z2||) (6)

[0219] Among them, Z1 represents the first target identity feature, and Z2 represents the second target identity feature. It can be seen that if the difference between the first target identity feature and the second target identity feature is smaller, the target similarity between the first target identity feature and the second target identity feature is higher.

[0220] Step 1506: When the target similarity is higher than the similarity threshold, determine that the first target speaking object and the second target speaking object are the same speaking object.

[0221] In a possible implementation, a similarity threshold is set. When it is determined that the target similarity between the first target identity feature and the second target identity feature is higher than the similarity threshold, it means that the smaller the difference between the first target identity feature and the second target identity feature, the more likely it is that the first target speaking object corresponding to the first target audio and the second target speaking object corresponding to the second target audio are the same speaking object.

[0222] The similarity threshold can be set by the user, for example, the similarity threshold is 90%.

[0223] Step 1507: when the target similarity is lower than the similarity threshold, determine that the first target speaking object and the second target speaking object are not the same speaking object.

[0224] On the contrary, if the target similarity between the first target identity feature and the second target identity feature is low, lower than the similarity threshold, it means that the difference between the first target identity feature and the second target identity feature is large, and it can be further determined that the first target speaking object corresponding to the first target audio is not the same as the second target speaking object corresponding to the second target audio.

[0225] In this embodiment, by comparing the similarity between the target identity features corresponding to the two target audios, it is possible to determine whether the two target audios are the audios of the same speaker based on the degree of similarity. Since the target identity features do not contain age information, the influence of age information on the judgment of the speaker can be avoided, and the accuracy of comparing audio recognition of the same speaker in different age groups can be improved.

[0226] In another possible application scenario, an identity classification network can also be deployed during the model application process, and the first target identity feature is input into the identity classification network. The extracted first target identity feature is identified through the identity classification network to obtain a target identity label output by the identity classification network, thereby determining the first target speaking object corresponding to the target audio based on the target identity label.

[0227] In the process of identifying the first target identity feature through the identity classification network, the identity classification network determines the feature similarity between the first target identity feature and the candidate identity feature corresponding to the candidate identity tag, thereby determining the candidate identity tag corresponding to the candidate identity feature with the highest feature similarity as the target identity tag of the first target speaking object, thereby identifying the first target speaking object corresponding to the first target audio.

[0228] Optionally, in other possible application scenarios, an age classification network may be deployed during the model application process to perform age recognition on the extracted target age features to determine the target age label of the target speaker corresponding to the target audio.

[0229] Fig.16 : is a structural block diagram of an audio processing device provided by an exemplary embodiment of the present application, the device comprising:

[0230] The feature extraction module 1601 is used to extract features from the first target audio to obtain a first target feature map;

[0231] A feature encoding module 1602 is used to perform feature encoding based on the first target feature map to obtain a first target object feature and a first target age feature, wherein the first target object feature is used to characterize object information of a first target speaker corresponding to the first target audio, and the first target age feature is used to characterize age information of the first target speaker;

[0232] The feature separation module 1603 is used to separate a first target identity feature from the first target object feature based on the first target age feature, wherein the first target identity feature is used to represent the identity information of the first target speaking object, and the object information at least includes the identity information and the age information.

[0233] Optionally, the feature encoding module 1602 is further used to:

[0234] Performing feature encoding on the first target feature map to obtain the first target object feature;

[0235] Age features are extracted from the first target feature graph to obtain the first target age features.

[0236] Optionally, the feature encoding module 1602 is further used to:

[0237] Processing the first target feature map through an attention mechanism layer to obtain a second target feature map, wherein an attention score of the age feature in the second target feature map is higher than an attention score of the non-age feature;

[0238] Performing a pooling operation on the second target feature map through a pooling layer to obtain an intermediate sample feature;

[0239] The intermediate sample features are subjected to feature mapping processing through a fully connected layer to obtain the first target age feature.

[0240] Optionally, the feature encoding module 1602 is further used to:

[0241] Performing feature encoding on the first target feature map to obtain the first target object feature;

[0242] An age feature is extracted from the first target object feature to obtain the first target age feature.

[0243] Optionally, the device further comprises:

[0244] An acquisition module, used to acquire a second target identity feature corresponding to the second target audio, where the second target identity feature is used to represent identity information of a second target speaker corresponding to the second target audio;

[0245] a determination module, configured to determine a target similarity between the first target identity feature and the second target identity feature;

[0246] The determination module is further configured to determine that the first target speaking object and the second target speaking object are the same speaking object when the target similarity is higher than a similarity threshold;

[0247] The determination module is further configured to determine, when the target similarity is lower than the similarity threshold, that the first target speaking object and the second target speaking object are not the same speaking object.

[0248] Optionally, the acquisition module is further used to:

[0249] Extracting features of the second target audio to obtain a second target feature map;

[0250] Performing feature encoding based on the second target feature graph to obtain a second target object feature and a second target age feature, wherein the second target object feature is used to characterize object information of the second target speaker, and the second target age feature is used to characterize age information of the second target speaker;

[0251] Based on the second target age feature, the second target identity feature is separated from the second target object feature.

[0252] Optionally, the device further comprises:

[0253] A classification module, configured to determine, through an identity classification network, a feature similarity between the first target identity feature and a candidate identity feature corresponding to the candidate identity tag;

[0254] The classification module is further used to determine the candidate identity tag corresponding to the candidate identity feature with the highest feature similarity as a target identity tag, and the target identity tag is used to indicate the first target speaking object corresponding to the first target audio.

[0255] In summary, the embodiments of the present application provide a method for identifying a speaking object: by performing feature extraction and feature encoding on a target audio of a speaking object, object information (object feature) corresponding to the speaking object and age information (age feature) of the speaking object are extracted from the target audio; since the object information includes age information and identity information, the identity feature can be separated from the object feature based on the extracted age feature, so that the separated identity feature does not have age information, thereby avoiding the influence of age information on identity feature and improving the extraction accuracy of identity feature; moreover, in a scenario of identifying a speaking object based on identity feature, the influence of age information on the recognition accuracy of the same speaking object can also be avoided, thereby further improving the recognition accuracy of the speaking object.

[0256] Fig.17 is a structural block diagram of an audio processing device provided by another exemplary embodiment of the present application, the device comprising:

[0257] A feature extraction module 1701 is used to extract features from the sample audio through a first feature extraction network to obtain a first sample feature map;

[0258] A feature encoding module 1702 is used to perform feature encoding on the first sample feature map through a second feature extraction network to obtain a sample object feature and a sample age feature, wherein the sample object feature is used to characterize the object information of the sample audio corresponding to the sample speaking object, and the sample age feature is used to characterize the age information of the sample speaking object;

[0259] A feature separation module 1703 is used to separate the sample identity feature from the sample object feature based on the sample age feature;

[0260] An identity recognition module 1704 is used to perform identity recognition based on the sample identity feature to obtain a sample identity tag;

[0261] An age recognition module 1705 is used to perform age recognition based on the sample age feature to obtain a first sample age label;

[0262] The training module 1706 is used to train the first feature extraction network and the second feature extraction network based on the sample identity label, the labeled identity label corresponding to the sample audio, the first sample age label and the labeled age label corresponding to the sample audio.

[0263] Optionally, the second feature extraction network includes a first sub-feature extraction network and a second sub-feature extraction network;

[0264] The feature encoding module 1702 is further used for:

[0265] Performing feature encoding on the first sample feature map through the first sub-feature extraction network to obtain the sample object feature;

[0266] The age feature of the sample is extracted by using the second sub-feature extraction network.

[0267] Optionally, the second sub-feature extraction network includes an attention mechanism layer, a pooling layer and a fully connected layer;

[0268] The feature encoding module 1702 is further used for:

[0269] Processing the first sample feature map through the attention mechanism layer to obtain a second sample feature map, wherein the attention score of the age feature in the second sample feature map is higher than the attention score of the non-age feature;

[0270] Performing a pooling operation on the second sample feature map through the pooling layer to obtain an intermediate sample feature;

[0271] The intermediate sample features are subjected to feature mapping processing through the fully connected layer to obtain the sample age features.

[0272] Optionally, the second feature extraction network includes a third sub-feature extraction network and a fourth sub-feature extraction network;

[0273] The feature encoding module 1702 is further used for:

[0274] Performing feature encoding on the first sample feature map through the third sub-feature extraction network to obtain the sample object feature;

[0275] The fourth sub-feature extraction network is used to extract age features from the sample object features to obtain the sample age features.

[0276] Optionally, the identity recognition module 1704 is further used to:

[0277] Inputting the sample identity feature into an identity classification network to obtain the sample identity label output by the identity classification network;

[0278] The age recognition module 1705 is also used for:

[0279] Inputting the sample age feature into a first age classification network to obtain the first sample age label output by the first age classification network;

[0280] The training module is also used for:

[0281] Determining a first classification loss corresponding to the identity network based on the sample identity label and the annotated identity label;

[0282] Determining a second classification loss corresponding to the first age classification network based on the first sample age label and the annotated age label;

[0283] Based on the first classification loss and the second classification loss, the first feature extraction network, the second feature extraction network, the first age classification network, and the identity classification network are trained.

[0284] Optionally, the device further comprises:

[0285] The age recognition module 1705 is further used to input the sample identity feature into a second age classification network to obtain a second sample age label output by the second age classification network, wherein the second age classification network has a gradient reversal layer;

[0286] A determination module, configured to determine a third classification loss corresponding to the second age classification network based on the second sample age label and the annotated age label;

[0287] The training module is also used for:

[0288] The first feature extraction network, the second feature extraction network, the first age classification network, the second age classification network, and the identity classification network are trained based on the first classification loss, the second classification loss, and the third classification loss.

[0289] Optionally, the training module is further used to:

[0290] Based on the first classification loss and the first loss weight, the second classification loss and the second loss weight, and the third classification loss and the third loss weight, the first feature extraction network, the second feature extraction network, the first age classification network, the second age classification network, and the identity classification network are trained.

[0291] Optionally, the identity recognition module 1704 is further used to:

[0292] Determining, by means of the identity classification network, a sample similarity between the sample identity feature and a candidate identity feature corresponding to a candidate identity tag;

[0293] The candidate identity label corresponding to the candidate identity feature with the highest sample similarity is determined as the sample identity label.

[0294] In summary, the embodiments of the present application provide a method for identifying a speaking object: by performing feature extraction and feature encoding on a target audio of a speaking object, object information (object feature) corresponding to the speaking object and age information (age feature) of the speaking object are extracted from the target audio; since the object information includes age information and identity information, the identity feature can be separated from the object feature based on the extracted age feature, so that the separated identity feature does not have age information, thereby avoiding the influence of age information on identity feature and improving the extraction accuracy of identity feature; moreover, in a scenario of identifying a speaking object based on identity feature, the influence of age information on the recognition accuracy of the same speaking object can also be avoided, thereby further improving the recognition accuracy of the speaking object.

[0295] Fig.18 is a schematic diagram of a computer device according to an exemplary embodiment. The computer device 1800 may be Figure 3 The first device 310 may also be Figure 3 The computer device 1800 includes a central processing unit (CPU) 1801, a system memory 1804 including a random access memory (RAM) 1802 and a read-only memory (ROM) 1803, and a system bus 1805 connecting the system memory 1804 and the central processing unit 1801. The computer device 1800 also includes a basic input / output system (I / O system) 1806 for facilitating information transmission between various components in the computer device, and a large-capacity storage device 1807 for storing an operating system 1813, application programs 1814 and other program modules 1815.

[0296] The basic input / output system 1806 includes a display 1808 for displaying information and an input device 1809 such as a mouse and a keyboard for user inputting information. The display 1808 and the input device 1809 are connected to the central processing unit 1801 through an input / output controller 1810 connected to the system bus 1805. The basic input / output system 1806 may also include an input / output controller 1810 for receiving and processing inputs from a plurality of other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 1810 also provides output to a display screen, a printer, or other types of output devices.

[0297] The mass storage device 1807 is connected to the central processing unit 1801 through a mass storage controller (not shown) connected to the system bus 1805. The mass storage device 1807 and its associated computer device readable medium provide non-volatile storage for the computer device 1800. That is, the mass storage device 1807 may include a computer device readable medium (not shown) such as a hard disk or a Compact Disc Read-Only Memory (CD-ROM) drive.

[0298] Without loss of generality, the computer device readable medium may include computer device storage media and communication media. Computer device storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer device readable instructions, data structures, program modules or other data. Computer device storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), CD-ROM, Digital Video Disc (DVD) or other optical storage, cassette, tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will know that the computer device storage medium is not limited to the above. The above-mentioned system memory 1804 and mass storage device 1807 can be collectively referred to as memory.

[0299] According to various embodiments of the present disclosure, the computer device 1800 can also be connected to a remote computer device on a network through a network such as the Internet. That is, the computer device 1800 can be connected to the network 1811 through the network interface unit 1812 connected to the system bus 1805, or the network interface unit 1812 can be used to connect to other types of networks or remote computer device systems (not shown).

[0300] The memory also includes one or more programs, which are stored in the memory. The central processing unit 1801 implements all or part of the steps of the above audio processing method by executing the one or more programs.

[0301] The present application also provides a computer-readable storage medium, in which at least one instruction, at least one program, a code set or an instruction set is stored. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the audio processing method provided by the above method embodiment.

[0302] The present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the audio processing method provided by the above method embodiment.

[0303] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0304] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0305] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An audio processing method, characterized in that: The method comprises: Performing feature extraction on the first target audio to obtain a first target feature map, where the first target feature map is a high-dimensional feature map; Performing feature encoding on the first target feature map to obtain a first target object feature, where the first target object feature is used to represent object information of the first target audio corresponding to a first target speaking object; Processing the first target feature map through an attention mechanism layer to obtain a second target feature map, wherein an attention score of the age feature in the second target feature map is higher than an attention score of the non-age feature; Performing a pooling operation on the second target feature map through a pooling layer to obtain an intermediate sample feature; Performing feature mapping processing on the intermediate sample features through a fully connected layer to obtain a first target age feature, where the first target age feature is used to characterize age information of the first target speaker, and the first target speaker feature and the first target age feature are fixed-length feature vectors; Based on the first target age feature, a first target identity feature is separated from the first target object feature, wherein the first target identity feature is used to characterize the identity information of the first target speaking object, and the object information at least includes the identity information and the age information.

2. The method according to claim 1, characterized in that The method further comprises: Acquire a second target identity feature corresponding to the second target audio, where the second target identity feature is used to represent identity information of a second target speaker corresponding to the second target audio; determining a target similarity between the first target identity feature and the second target identity feature; When the target similarity is higher than a similarity threshold, determining that the first target speaking object and the second target speaking object are the same speaking object; When the target similarity is lower than the similarity threshold, it is determined that the first target speaking object and the second target speaking object are not the same speaking object.

3. The method according to claim 2, characterized in that The obtaining of the second target identity feature corresponding to the second target audio includes: Extracting features of the second target audio to obtain a second target feature map; Performing feature encoding based on the second target feature graph to obtain a second target object feature and a second target age feature, wherein the second target object feature is used to characterize object information of the second target speaker, and the second target age feature is used to characterize age information of the second target speaker; Based on the second target age feature, the second target identity feature is separated from the second target object feature.

4. The method according to claim 1, characterized in that: The method further comprises: Determining, through an identity classification network, a feature similarity between the first target identity feature and a candidate identity feature corresponding to the candidate identity tag; The candidate identity tag corresponding to the candidate identity feature with the highest feature similarity is determined as a target identity tag, and the target identity tag is used to indicate the first target speaking object corresponding to the first target audio.

5. An audio processing method, characterized in that: The method comprises: Extract features from the sample audio through a first feature extraction network to obtain a first sample feature map, where the first sample feature map is a high-dimensional feature map; Performing feature encoding on the first sample feature map through a first sub-feature extraction network of a second feature extraction network to obtain a sample object feature, wherein the sample object feature is used to characterize object information of a sample speaker corresponding to the sample audio; Processing the first sample feature map through the attention mechanism layer of the second sub-feature extraction network in the second feature extraction network to obtain a second sample feature map, wherein the attention score of the age feature in the second sample feature map is higher than the attention score of the non-age feature; Performing a pooling operation on the second sample feature map through a pooling layer to obtain an intermediate sample feature; Performing feature mapping processing on the intermediate sample features through a fully connected layer to obtain a sample age feature, wherein the sample age feature is used to characterize the age information of the sample speaker, and the sample object feature and the sample age feature are fixed-length feature vectors; Based on the sample age feature, separating the sample identity feature from the sample object feature; Performing identity recognition based on the sample identity feature to obtain a sample identity label; Perform age recognition based on the sample age feature to obtain a first sample age label; The first feature extraction network and the second feature extraction network are trained based on the sample identity label, the annotated identity label corresponding to the sample audio, the first sample age label, and the annotated age label corresponding to the sample audio.

6. The method according to claim 5, characterized in that The performing identity recognition based on the sample identity feature to obtain the sample identity label includes: Inputting the sample identity feature into an identity classification network to obtain the sample identity label output by the identity classification network; The performing age recognition based on the sample age feature to obtain a first sample age label includes: Inputting the sample age feature into a first age classification network to obtain the first sample age label output by the first age classification network; The training of the first feature extraction network and the second feature extraction network based on the sample identity label, the annotated identity label corresponding to the sample audio, the first sample age label, and the annotated age label corresponding to the sample audio includes: Determining a first classification loss corresponding to the identity network based on the sample identity label and the annotated identity label; Determining a second classification loss corresponding to the first age classification network based on the first sample age label and the annotated age label; Based on the first classification loss and the second classification loss, the first feature extraction network, the second feature extraction network, the first age classification network, and the identity classification network are trained.

7. The method according to claim 6, characterized in that The method further comprises: Inputting the sample identity feature into a second age classification network to obtain a second sample age label output by the second age classification network, wherein the second age classification network has a gradient reversal layer; Determining a third classification loss corresponding to the second age classification network based on the second sample age label and the annotated age label; The step of training the first feature extraction network, the second feature extraction network, the first age classification network, and the identity classification network based on the first classification loss and the second classification loss includes: The first feature extraction network, the second feature extraction network, the first age classification network, the second age classification network, and the identity classification network are trained based on the first classification loss, the second classification loss, and the third classification loss.

8. The method according to claim 7, characterized in that The training of the first feature extraction network, the second feature extraction network, the first age classification network, the second age classification network, and the identity classification network based on the first classification loss, the second classification loss, and the third classification loss includes: Based on the first classification loss and the first loss weight, the second classification loss and the second loss weight, and the third classification loss and the third loss weight, the first feature extraction network, the second feature extraction network, the first age classification network, the second age classification network, and the identity classification network are trained.

9. The method according to claim 6, characterized in that The step of inputting the sample identity feature into an identity classification network to obtain the sample identity label output by the identity classification network includes: Determining, by means of the identity classification network, a sample similarity between the sample identity feature and a candidate identity feature corresponding to a candidate identity tag; The candidate identity label corresponding to the candidate identity feature with the highest sample similarity is determined as the sample identity label.

10. An audio processing device, characterized in that: The device comprises: A feature extraction module, used to extract features from the first target audio to obtain a first target feature map, where the first target feature map is a high-dimensional feature map; a feature encoding module, configured to perform feature encoding on the first target feature map to obtain a first target object feature, wherein the first target object feature is used to characterize object information of the first target audio corresponding to a first target speaking object; Processing the first target feature map through an attention mechanism layer to obtain a second target feature map, wherein an attention score of the age feature in the second target feature map is higher than an attention score of the non-age feature; Performing a pooling operation on the second target feature map through a pooling layer to obtain an intermediate sample feature; Performing feature mapping processing on the intermediate sample features through a fully connected layer to obtain a first target age feature, where the first target age feature is used to characterize age information of the first target speaker, and the first target speaker feature and the first target age feature are fixed-length feature vectors; The feature separation module is used to separate a first target identity feature from the first target object feature based on the first target age feature, wherein the first target identity feature is used to represent the identity information of the first target speaking object, and the object information at least includes the identity information and the age information.

11. An audio processing device, characterized in that: The device comprises: A feature extraction module, configured to extract features from the sample audio through a first feature extraction network to obtain a first sample feature map, wherein the first sample feature map is a high-dimensional feature map; a feature encoding module, configured to perform feature encoding on the first sample feature map through a first sub-feature extraction network of a second feature extraction network to obtain a sample object feature, wherein the sample object feature is used to characterize object information of a sample speaker corresponding to the sample audio; Processing the first sample feature map through the attention mechanism layer of the second sub-feature extraction network in the second feature extraction network to obtain a second sample feature map, wherein the attention score of the age feature in the second sample feature map is higher than the attention score of the non-age feature; Performing a pooling operation on the second sample feature map through a pooling layer to obtain an intermediate sample feature; Performing feature mapping processing on the intermediate sample features through a fully connected layer to obtain a sample age feature, wherein the sample age feature is used to characterize the age information of the sample speaker, and the sample object feature and the sample age feature are fixed-length feature vectors; A feature separation module, used to separate the sample identity feature from the sample object feature based on the sample age feature; An identity recognition module, used to perform identity recognition based on the sample identity feature to obtain a sample identity label; An age recognition module, used to perform age recognition based on the sample age feature to obtain a first sample age label; A training module is used to train the first feature extraction network and the second feature extraction network based on the sample identity label, the labeled identity label corresponding to the sample audio, the first sample age label and the labeled age label corresponding to the sample audio.

12. A computer device, characterized in that: The computer device comprises: a processor and a memory, the memory stores a computer program, and the computer program is loaded and executed by the processor to implement the audio processing method as described in any one of claims 1 to 4, or to implement the audio processing method as described in any one of claims 5 to 9.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is loaded and executed by a processor to implement the audio processing method as described in any one of claims 1 to 4, or to implement the audio processing method as described in any one of claims 5 to 9.

14. A computer program product, characterized in that The computer program product stores a computer program, and the computer program is loaded and executed by a processor to implement the audio processing method as described in any one of claims 1 to 4, or to implement the audio processing method as described in any one of claims 5 to 9.

Images