Auditory model evaluation method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to the technical field of computers and the field of artificial intelligence, and provides an evaluation method and device of an auditory model, electronic equipment, a storage medium and a program product, and the evaluation method comprises the steps: obtaining a first acoustic feature and an auditory response signal for target audio data; according to each brain region corresponding to the auditory response signal, dividing the first acoustic feature to obtain a second acoustic feature corresponding to the auditory response signal; and based on the second acoustic feature and the auditory response signal, evaluating brain-like characteristics of the auditory model. According to the invention, the problem that there is no evaluation scheme for the relation and difference between the auditory model and the human auditory system can be solved, and the acoustic features of the model and the auditory response signals of the human brain auditory system can be analyzed and compared more meticulously by analyzing the correspondence between the acoustic features and the brain regions.

Description

Technical Field

[0001] The present disclosure relates to the fields of computer technology and artificial intelligence, and in particular to an evaluation method and device for an auditory model, an electronic device, a storage medium, and a program product. Background Art

[0002] Advances in machine learning have led to rapid development of deep neural networks (DNNs) in the past decade. At the same time, the performance of audio models (also called "acoustic models") dedicated to processing sound signals such as speech and music has also made great progress, demonstrating powerful sound processing capabilities in tasks such as sound recognition, enhancement, and synthesis.

[0003] However, like most DNNs, as black box models, auditory models also lack interpretability. In particular, there is a lack of evaluation schemes for the relationship and differences between auditory models and the human auditory system, which makes it difficult to provide valuable guidance for further optimization and improvement of auditory models. Summary of the invention

[0004] The present disclosure provides an evaluation method and device for an auditory model, an electronic device, a storage medium, and a program product, so as to at least solve the problem in the related art that there is a lack of an evaluation scheme for the relationship and difference between an auditory model and a human auditory system. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of the present disclosure, a method for evaluating an auditory model is provided, the method comprising: obtaining a first acoustic feature and an auditory response signal for target audio data, wherein the first acoustic feature is obtained by inputting the target audio data into a target auditory model, and the auditory response signal is a response signal of the human brain auditory system to the target audio data; dividing the first acoustic feature according to brain regions corresponding to the auditory response signal to obtain a second acoustic feature corresponding to the auditory response signal; and evaluating the brain-like characteristics of the auditory model based on the second acoustic feature and the auditory response signal.

[0006] Optionally, the first acoustic feature includes layer features output by each network layer of the target auditory model, wherein the second acoustic feature is obtained by: dividing the features of each layer based on each brain area corresponding to the auditory response signal to determine the correspondence between the features of each layer and each brain area; encoding the features of each layer according to the division result of the features of each layer to obtain the second acoustic feature, wherein the second acoustic feature has the same information form as the auditory response signal.

[0007] Optionally, the second acoustic feature is obtained in the following manner: inputting each layer feature of the first acoustic feature into a pre-trained coding model, wherein the pre-trained coding model includes a router and multiple expert models, each expert model corresponds to at least one brain region, and different expert models correspond to different brain regions; using the router to predict the prediction probability of each layer feature corresponding to each expert model; using each expert model to encode each layer feature, and using the prediction probability to weight each encoding result to obtain a weighted layer feature; based on the weighted layer feature, obtaining the second acoustic feature.

[0008] Optionally, the coding model is trained in the following manner: obtaining training samples and sample annotation information, wherein the training samples include first sample acoustic features output by the target auditory model, each first sample acoustic feature includes sample layer features output by each network layer of the target auditory model, and the sample annotation information represents the second sample acoustic features corresponding to each first sample acoustic feature; inputting the training samples into the coding model, and using the router to predict the prediction probability of each sample layer feature corresponding to each expert model; using each expert model to encode the corresponding sample layer features, and using the prediction probability to weight each encoding result to obtain a weighted sample layer feature; based on the weighted sample layer feature, obtaining a predicted second acoustic feature; and determining a prediction loss by comparing the predicted second acoustic feature with the second sample acoustic feature to train the coding model, wherein the prediction loss includes a first loss and a second loss, the first loss represents the difference between the predicted second acoustic feature and the second sample acoustic feature, and the second loss is determined by statistically analyzing the prediction probabilities corresponding to each sample layer feature and each expert model.

[0009] Optionally, the brain-like characteristics of the auditory model are evaluated in the following manner: the second acoustic feature is compared with the auditory response signal to obtain multiple evaluation indicators, wherein each evaluation indicator represents the signal difference between the second acoustic feature and the auditory response signal, and different evaluation indicators measure the signal difference in different measurement methods; the multiple evaluation indicators are merged to obtain an evaluation result of the brain-like characteristics of the auditory model.

[0010] Optionally, the target audio data includes two audio data, and the first acoustic feature includes dimensional features of multiple dimensions, wherein the second acoustic feature is obtained by: determining the characteristic distance between the dimensional features of two first acoustic features corresponding to the two audio data in each dimension; determining the division weights for dividing the dimensional features in each dimension into each brain region, wherein the division weights represent the contribution of each dimension of the first acoustic feature to the formation of information of each brain region; based on the division weights, weighting the characteristic distances in each dimension to obtain the second acoustic feature, wherein the second acoustic feature includes the weighted characteristic distance.

[0011] Optionally, the brain-like characteristics of the auditory model are evaluated in the following manner: determining the signal distance between two auditory response signals corresponding to the two audio data in each dimension; and obtaining an evaluation result of the brain-like characteristics of the auditory model by comparing the weighted feature distance with the signal distance.

[0012] According to a second aspect of the present disclosure, an evaluation device for an auditory model is provided, the evaluation device comprising: an acquisition unit, configured to acquire a first acoustic feature and an auditory response signal for target audio data, wherein the first acoustic feature is obtained by inputting the target audio data into a target auditory model, and the auditory response signal is a response signal of a human brain auditory system to the target audio data; a division unit, configured to divide the first acoustic feature according to brain regions corresponding to the auditory response signal, to obtain a second acoustic feature corresponding to the auditory response signal; and an evaluation unit, configured to evaluate the brain-like characteristics of the auditory model based on the second acoustic feature and the auditory response signal.

[0013] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor, wherein the instructions executable by the processor, when executed by the processor, cause the processor to execute the method for evaluating the auditory model according to the present disclosure.

[0014] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the auditory model evaluation method according to the present disclosure.

[0015] According to a fifth aspect of the present disclosure, a computer program product is provided, which includes computer executable instructions, and when the computer executable instructions are executed by at least one processor, the method for evaluating the auditory model according to the present disclosure is implemented.

[0016] The technical solution provided by the present disclosure brings at least the following beneficial effects:

[0017] By adopting the evaluation scheme of the auditory model disclosed in the present invention, the first acoustic feature obtained by inputting the target audio data into the target auditory model can be divided according to the brain areas corresponding to the auditory response signals of the human brain auditory system to the target audio data, and the second acoustic feature corresponding to the auditory response signal can be obtained. Therefore, the brain-like characteristics of the auditory model can be evaluated based on the second acoustic feature and the auditory response signal. In this way, the brain areas related to human hearing can be taken into consideration, the acoustic features output by the model can be divided, and the correspondence between the acoustic features and the brain areas can be analyzed, so that the acoustic features and the auditory response signals are more comparable, which is conducive to a more detailed analysis and comparison of the acoustic features of the model with the auditory response signals of the human brain auditory system, and the evaluation of the brain-like characteristics of the auditory model.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0020] Figure 1 is a schematic diagram of an application of a method for evaluating a hearing model according to an exemplary embodiment of the present disclosure.

[0021] Figure 2 is a schematic flowchart of a method for evaluating an auditory model according to an exemplary embodiment of the present disclosure.

[0022] Figure 3 is a schematic flowchart of an example of obtaining a second acoustic feature in a method for evaluating an auditory model according to an exemplary embodiment of the present disclosure.

[0023] Figure 4 It is a schematic flowchart of evaluating brain-like characteristics by encoding a first acoustic feature in a method for evaluating an auditory model according to an exemplary embodiment of the present disclosure.

[0024] Figure 5 It is a schematic flowchart of obtaining a second acoustic feature by using a coding model in a method for evaluating an auditory model according to an exemplary embodiment of the present disclosure.

[0025] Figure 6 is a schematic diagram of encoding layer features by using a coding model in a method for evaluating an auditory model according to an exemplary embodiment of the present disclosure.

[0026] Figure 7It is a schematic flowchart of training a coding model in a method for evaluating an auditory model according to an exemplary embodiment of the present disclosure.

[0027] Figure 8 is a schematic flowchart of another example of obtaining a second acoustic feature in a method for evaluating an auditory model according to an exemplary embodiment of the present disclosure.

[0028] Fig. 9 2 is a schematic diagram of determining a feature distance in a method for evaluating an auditory model according to an exemplary embodiment of the present disclosure.

[0029] Fig.10 is a schematic block diagram of an evaluation apparatus for an auditory model according to an exemplary embodiment of the present disclosure.

[0030] Fig.11 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the present application will clearly and completely describe the technical solutions in the embodiments of the present disclosure in conjunction with the accompanying drawings.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0033] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0034] As mentioned above, in the related art, there is a lack of evaluation schemes for the relationship and differences between auditory models and the human auditory system.

[0035] Specifically, by evaluating the brain-like properties of auditory models to explore the connection between auditory models and the human auditory system, such research has two-way significance in computer science and neuroscience. First, with the goal of simulating and reproducing the information processing method of the brain, we can better understand and explain the functions of the sensory system by building more accurate computational models, and further improve the performance of auditory models. Such models can not only be used for theoretical verification, but also be applied in multiple fields.

[0036] For example, in the field of human-computer interaction technology such as brain-computer interface, such models can be guided by specific brain stimulation patterns to achieve precise control of consciousness and movement, which provides important help for people with disabilities and enables them to restore their mobility. In addition, such models can also be applied in the medical field, by revealing the brain's response patterns to specific stimuli, providing doctors with more accurate data so that doctors can make more accurate diagnoses and develop personalized treatment plans.

[0037] In addition, from the perspective of auditory neuroscience, analyzing the network structure of brain-like auditory models can also provide in-depth research on brain functions, thereby promoting the understanding of cognitive and psychological processes. By simulating and analyzing the brain's internal representations and information flows, key mechanisms and neural circuits in the sensory system can be revealed, which provides a new perspective for cognitive science research and is expected to promote the understanding of the cognitive mechanisms of abnormal performance and diseases.

[0038] In view of the above, exemplary embodiments of the present disclosure provide an auditory model evaluation method, an auditory model evaluation device, an electronic device, a computer-readable storage medium, and a computer program product, which can solve or at least alleviate the above problems.

[0039] In a first aspect of an exemplary embodiment of the present disclosure, a method for evaluating an auditory model is provided.

[0040] The evaluation method of the auditory model according to the exemplary embodiment of the present disclosure can be applied to scenarios where users interact with software. For example, the software can be loaded on the user terminal, and the user can input auditory model evaluation instructions on the user terminal, such as evaluating the brain-like characteristics of a certain auditory model. The user terminal can obtain the final evaluation result by executing the evaluation method of the auditory model according to the exemplary embodiment of the present disclosure.

[0041] Specifically, the user terminal can obtain a first acoustic feature and an auditory response signal for the target audio data, wherein the first acoustic feature is obtained by inputting the target audio data into the target auditory model, and the auditory response signal is a response signal of the human brain auditory system to the target audio data. The user terminal can also divide the first acoustic feature according to the brain regions corresponding to the auditory response signal to obtain a second acoustic feature corresponding to the auditory response signal. The user terminal can also evaluate the brain-like characteristics of the auditory model based on the second acoustic feature and the auditory response signal.

[0042] Figure 1 An example application scenario of executing the evaluation method of the auditory model according to the exemplary embodiment of the present disclosure is shown. Figure 1 As shown, the user terminal can evaluate the auditory model from the model library by executing the evaluation method and utilizing the data from the database (such as the target audio data mentioned above, the auditory response signal, etc.), and the obtained evaluation result can be used to achieve at least one of the following items: comparing the brain-likeness of the auditory model and the artificial acoustic feature; comparing the brain-likeness of each auditory model; exploring the alignment relationship between the hierarchical structure of the auditory model and the brain regions of the human brain; exploring the influence of the model training method and the training task on the brain-like characteristics; exploring the selectivity of the auditory model and the human brain for the audio types, etc.

[0043] The above-mentioned user terminal can be, for example, a tablet computer, a laptop computer, a digital assistant, a wearable device, etc. However, the implementation scenario of the above method is only an example scenario. The evaluation method of the auditory model according to the exemplary embodiment of the present disclosure can also be applied to other application scenarios. For example, the user can request the server to evaluate the auditory model through the network on the user terminal (for example, a mobile phone, a desktop computer, a tablet computer, etc.). The server can complete the request by executing the evaluation method of the auditory model according to the exemplary embodiment of the present disclosure. Here, the server can be an independent server, a server cluster, a cloud computing platform or a virtualization center.

[0044] According to the evaluation method of the auditory model of the exemplary embodiment of the present disclosure, the brain areas related to human hearing can be taken into consideration, the acoustic features of the model output can be divided, and the correspondence between the acoustic features and each brain area can be analyzed to make the acoustic features and the auditory response signals more comparable, thereby facilitating a more detailed analysis and comparison of the acoustic features of the model with the auditory response signals of the human brain auditory system, evaluating the brain-like characteristics of the auditory model, and providing guidance for the optimization of the auditory model.

[0045] The following will refer to Figures 2 to 11 An example of a method for evaluating an auditory model according to an embodiment of the present disclosure is described. Figure 2 As shown, the evaluation method of the auditory model may include the following steps:

[0046] In step S210 , a first acoustic feature and an auditory response signal for target audio data may be acquired.

[0047] Here, the target audio data may be audio data of any format and content, and the target audio data may convey certain information, which may be obtained by receiving and understanding the content of the audio data.

[0048] As an example, the target audio data may come from a preset database. In the database, a large number of audio clips of different categories may be collected, such as but not limited to human speech sounds, human non-speech vocal sounds, animal cries, musical instruments, scenes from nature, tool sounds, etc.

[0049] The first acoustic feature can be obtained by inputting the target audio data into the target auditory model. Here, the target auditory model can be an auditory model to be evaluated, which can be a trained auditory model.

[0050] As an example, the target auditory model can come from a preset auditory model library. In the auditory model library, deep auditory models with convolutional neural networks (CNN) and Transformer as the main structure and self-supervision as the main training method commonly used in speech processing tasks can be collected and sorted, and other models can also be added for comparison, such as supervised models. The target auditory model includes, for example, but is not limited to: Wav2Vec2, Wav2Vec2-Conformer, HuBERT (Hidden-Unit BERT), WavLM, SEW (Squeezed and Efficient Wav2Vec), SEW-D (Squeezed and Efficient Wav2Vec with Disentangledattention), UniSpeech-SAT, AST (Audio Spectrogram Transformer) trained in a supervised manner, and UniSpeech, which is involved in both training methods.

[0051] For the model to be evaluated, in the related art, generally only the mainstream models used in various auditory tasks can be evaluated, and it is difficult to evaluate the deep auditory model. The evaluation method according to the embodiment of the present disclosure is universal for various auditory models. Not only can the mainstream models be evaluated, but also the deep auditory models such as the Wav2vec2 model with CNN-Transformer as the main structure and self-supervision as the main training method can be evaluated.

[0052] The first acoustic feature may be any feature capable of characterizing the acoustic characteristics of the audio, where the acoustic characteristics include, but are not limited to, timbre, pitch, volume, etc. For a deep learning auditory model, the first acoustic feature may include features output by each layer of the model obtained by inputting audio data into the auditory model.

[0053] The auditory response signal can be a response signal of the human brain auditory system to the target audio data. Here, the human brain auditory system can be a real human brain auditory system, or it can be a simulation system that can simulate human brain hearing, as long as it can feedback the auditory response signals from different brain regions when receiving audio data. Here, the brain regions of the human brain can cover primary auditory brain regions and advanced auditory brain regions, for example, including but not limited to Heschl's gyrus (HG), planum polare (PP), planum temporale (PT), anterior superior temporal gyrus (aSTG), middle STG (mSTG), posterior STG (pSTG), etc. According to the evaluation method of the embodiment of the present disclosure, it is allowed to finely divide the brain regions for evaluation and analysis.

[0054] The auditory response signal may be, for example but not limited to, a functional magnetic resonance imaging (fMRI) signal.

[0055] In addition, as an example, the auditory response signal can come from the same database as the above-mentioned target audio data. As mentioned above, the database contains a large number of audio clips of different categories, which can be provided to the subject's human brain auditory system as audio stimuli, and the auditory response signals when the human brain auditory system receives the stimulation are collected. These auditory response signals can form paired data with the corresponding audio stimuli.

[0056] In addition, in this step, the first acoustic feature and the auditory response signal may be obtained in advance, for example, both from a database; or one or both of them may be obtained when executing the evaluation method.

[0057] For example, the target audio data provided by the database can be sent to the target auditory model and the human brain auditory system respectively. In the target auditory model pathway, the target audio data obtains acoustic representation through the target auditory model to be evaluated from the model library. In the human brain auditory system pathway, during the database preparation stage, the target audio data can be provided as a stimulus to the subject's human brain auditory system, and the auditory response signal of the human brain auditory system when receiving the stimulus is collected as paired data with the target audio data.

[0058] In the evaluation method of the auditory model according to the exemplary embodiment of the present disclosure, two stages may be included: the model library and database establishment stage and the evaluation stage. In the model library and database establishment stage, in terms of the auditory model library, a deep auditory model commonly used in speech processing tasks is included, and commonly used artificial acoustic features are added as a comparison; in terms of the database, auditory response signals such as fMRI signal data with fine brain area division and stimulus audio sample pairs with rich categories may be included. In this way, a comprehensive and fine auditory model library and an auditory response signal-audio database can be established, thereby supporting the drawing of universal evaluation conclusions.

[0059] In step S220, the first acoustic feature may be divided according to the brain regions corresponding to the auditory response signal to obtain a second acoustic feature corresponding to the auditory response signal.

[0060] In this step, it can be considered that different feature parts in the first acoustic feature correspond to different brain area signals, and the first acoustic feature can be divided. The division dimension can be based on the depth of feature processing, according to the network level of the auditory model, for example, each layer or every few layers as a division dimension; it can also be based on the width of feature processing, according to the dimension of each layer output feature itself. For example, each layer output feature includes multiple feature dimensions, and the features of each layer on each feature dimension are used as a division dimension. However, the division method is not limited to the above example, and other methods can also be used for division, such as using the feature dimension as the division dimension, that is, using the features of all layers on each feature dimension as a division dimension.

[0061] In one example, the first acoustic feature can be divided according to each brain region, and the first acoustic feature can be encoded using the result of the division to obtain the second acoustic feature. Here, in the first acoustic feature, the feature parts divided into corresponding to different brain regions can be processed differently during encoding, so that different feature parts can be encoded more reasonably according to the characteristics of different brain regions. For example, the encoding method of the feature parts corresponding to the advanced brain regions can be more complex than the encoding method of the feature parts corresponding to the primary brain regions, such as higher encoding granularity.

[0062] As an example, Figure 3 As shown, the first acoustic feature may include layer features output by each network layer of the target auditory model, wherein the second acoustic feature may be obtained in the following manner: in step S310, the features of each layer may be divided based on each brain region corresponding to the auditory response signal to determine the correspondence between the features of each layer and each brain region; in step S320, the features of each layer may be encoded according to the division results of the features of each layer to obtain the second acoustic feature.

[0063] Here, the second acoustic feature may have the same information form as the auditory response signal so as to be able to be used for comparison with the auditory response signal.

[0064] Specifically, if Figure 4 As shown, multiple encoders can be set, such as encoders E1, E2, E3 and E4 based on expert models, which correspond to different brain regions respectively. Before encoding each layer of features of the first acoustic feature, the correspondence between each layer of features and each brain region can be predicted, so that each encoder can be used to encode the current layer features respectively, and according to the degree of correspondence between the current layer features and each brain region, the corresponding encoding results are weighted to obtain the second acoustic feature, which can be compared with an auditory response signal such as an fMRI signal.

[0065] Based on the above method, the idea of ​​encoding can be used to construct a mapping from the acoustic representation of the auditory model to the auditory response signal such as fMRI brain signal. Considering that each network layer of the auditory model extracts the characteristics of different levels of sound, corresponding to different activation levels of different brain regions, in this way, starting from the internal structure of the auditory model, the correspondence between different levels of the auditory model and the brain regions can be analyzed to evaluate the degree of fit of the acoustic representation of the auditory model to the neural response of the human brain.

[0066] For the above process, for example, a pre-trained classifier can be used to classify layer features, for example, different categories correspond to different brain regions, and encoders can be trained separately for different categories, and the encoding results of each encoder can be weighted according to the classification results of the classifier. The classifier and encoder can be trained separately or together, for example, both can be trained using supervised learning. As an example, an embodiment of the present disclosure provides a coding model design that can achieve faster and more accurate feature division and encoding.

[0067] Specifically, if Figure 5 As shown, the second acoustic feature can be obtained by:

[0068] In step S510, each layer feature of the first acoustic feature may be input into a pre-trained encoding model.

[0069] Here, if Figure 6 As shown, the pre-trained encoding model may include a router and multiple expert models, each expert model may correspond to at least one brain region, and different expert models may correspond to different brain regions.

[0070] The first acoustic feature may, for example, include layer features of each layer output of the auditory model. For example, the auditory model may include N layers (e.g., CNN layers or Transformer layers). Accordingly, the first acoustic feature may include N layer features, where N is a positive integer.

[0071] In step S520, the router may be used to predict the prediction probability of each layer feature corresponding to each expert model.

[0072] like Figure 6 As shown, the encoding model may include a selection feedforward layer, and the selection feedforward layer may include the above-mentioned router and expert model, wherein the selection feedforward layer may include multiple expert models, for example, Figure 6 There are 4 expert models included Here, preferably, the number of network layers of the auditory model can be greater than the number of expert models, so that the acoustic features of multiple layers can share the same expert model, thereby reducing the number of parameters.

[0073] In selecting the feedforward layer, each layer feature of the first acoustic feature first passes through a router, which is used to predict the probability p(x) that the layer feature of the current input corresponds to the expert model, that is, the probability that the layer feature corresponds to each brain region. For example, the router can be a classifier.

[0074] In addition, the coding model may further include a projection layer. When the auditory model includes N layers, the projection layer may include N projection modules. (For example, Figure 6 Since different layer features may have different dimensions, these projection modules can project different layer features into the same feature space P l (A l (x)), and then encoded through a selection feed-forward layer.

[0075] In this example, for the layer feature of the lth layer, the router can calculate the probability p(x) of routing the layer feature to each expert model by the following formula (1):

[0076] p(x)=Softmax[W r ·P l (A l (x))](1)

[0077] Among them, Wr A represents the weight of the router, which can be in the form of a vector, each element of which corresponds to an expert model; l (x) represents the layer feature of the lth layer; P l (·) represents the lth projection module; Softmax[·] means that the output probability is normalized by the Softmax function.

[0078] In step S530, each layer feature may be encoded using each expert model, and each encoding result may be weighted using the prediction probability to obtain a weighted layer feature.

[0079] In this step, each layer feature can be passed through each expert model, so that multiple encoding results can be obtained for each layer feature, such as Figure 6 As shown, 4 encoding results can be output from 4 expert models.

[0080] After encoding, the prediction probability output by the router can be used to determine the weighted layer features. Specifically, the final output of the feedforward layer can be the weighted sum of the outputs of each expert model, which can be expressed by the following formula (2):

[0081]

[0082] Among them, R l (x) represents the final output of the selected feed-forward layer; p i (x) represents the prediction probability corresponding to the i-th expert model; E i (P l (A l (x))) represents the output of the i-th expert model.

[0083] In step S540, a second acoustic feature may be obtained based on the weighted layer feature.

[0084] In this step, the weighted summed layer features may be post-processed, such as residual processing, normalization, and concatenation, to obtain the second acoustic features.

[0085] As an example, Figure 6 As shown, the encoding model may further include a residual normalization layer (Add+Normalize) and a final feed-forward network (FFN).

[0086] Specifically, the features of each layer encoded by the selected feedforward layer (such as FFN1, FFN2, FFN3 and FFN4) can be normalized to a residual layer and then concatenated. For example, it can be expressed by the following equations (3) and (4):

[0087] Nl (x) = Norm(R l (x)+P l (A l (x)))(3)

[0088]

[0089] Among them, N(x) represents the concatenated features, Norm(·) represents normalization, and concat[·] represents concatenation.

[0090] Based on the concatenated features, a second acoustic feature may be output through a final feed-forward layer, and the second acoustic feature may have the same signal form as the auditory response signal.

[0091] Based on the above method, the classifier and encoder can be taken as a whole to obtain an encoding model based on a hybrid expert model. Different expert models will predict the second acoustic features at different levels, thereby enhancing the degree of fit of the acoustic model to the neural response.

[0092] An example of the training process of the encoding model is described below. Figure 7 As shown, the encoding model can be trained in the following way:

[0093] In step S710, training samples and sample annotation information may be obtained.

[0094] Here, the training sample may include the first sample acoustic features output by the target auditory model, each first sample acoustic feature may include the sample layer features output by each network layer of the target auditory model, and the sample annotation information may characterize the second sample acoustic features corresponding to each first sample acoustic feature.

[0095] In step S720, the training samples may be input into the coding model, and the router may be used to predict the prediction probability corresponding to each sample layer feature and each expert model.

[0096] In step S730, the corresponding sample-level features may be encoded using each expert model, and each encoding result may be weighted using the prediction probability to obtain a weighted sample-level feature.

[0097] In step S740, a predicted second acoustic feature may be obtained based on the weighted sample-level feature.

[0098] Here, in steps S710 to S740, the process of predicting the second acoustic feature for the encoding model for the training sample is the same as that described above with reference to Figure 5 and Figure 6 The encoding process described is similar, so it will not be repeated here.

[0099] In step S750, the prediction loss may be determined by comparing the predicted second acoustic feature with the second sample acoustic feature to train the encoding model.

[0100] In this step, after obtaining the predicted second acoustic feature, the prediction loss can be determined to adjust the parameters of the encoding model to achieve training of the model.

[0101] As an example, the prediction loss can be determined by an existing loss calculation method. As another example, considering that different expert models need to learn the encoding capabilities for different brain regions, and during the training process, the router may tend to use one or several expert models for encoding, resulting in an imbalance in the learning between the expert models. According to an embodiment of the present disclosure, the final prediction loss can be calculated based on both a general prediction loss and a balance loss for balancing the learning between the expert models.

[0102] Specifically, the prediction loss according to an embodiment of the present disclosure may include a first loss and a second loss, the first loss may represent the difference between the predicted second acoustic feature and the second sample acoustic feature, and the second loss may be determined by statistically analyzing the prediction probability of the router, and the prediction probability may refer to the probability that the router predicts that each sample layer feature corresponds to each expert model.

[0103] The first loss can be obtained by using an existing loss calculation method, for example, it can be determined by the following formula (5):

[0104]

[0105] in, Indicates the first loss; represents the predicted second acoustic feature, Figure 6 In the example shown, N(x) represents the concatenated features; y represents the acoustic features of the second sample.

[0106] Although formula (5) here gives an example function for calculating the first loss, it is not limited thereto, and other calculation methods may also be used as long as they can reflect the difference between the predicted second acoustic feature and the second sample acoustic feature.

[0107] As an example, the second loss may be determined based on a coefficient of variation (CV) of the router output probability, for example, by the following formula (6):

[0108]

[0109] in, represents the second loss; std represents the standard deviation; mean represents the mean; α is a hyperparameter; p(x) represents the predicted probability of the router output.

[0110] Based on the above prediction loss, the probability of router output imbalance can be prevented, avoiding the bias of using certain expert models for encoding, and improving the performance of the trained encoding model.

[0111] The above describes an example of obtaining the second acoustic feature by encoding based on layer features. In another example, the feature parts of the first acoustic feature can be recombined according to the weights of different feature parts of the first acoustic feature in each brain region to obtain the second acoustic feature.

[0112] As an example, the target audio data may include two audio data, and the first acoustic feature may include dimensional features of multiple dimensions, such as Figure 8 As shown, the second acoustic feature can be obtained by:

[0113] In step S810, a feature distance between dimensional features of two first acoustic features respectively corresponding to two audio data in each dimension may be determined.

[0114] Specifically, the auditory model can obtain a first acoustic feature for each audio data, and each first acoustic feature can include dimensional features of multiple dimensions, where the dimension mentioned here can be a characteristic dimension of the acoustic feature itself, for example, each layer output of the auditory model can include 512-dimensional features. Two first acoustic features can determine the distance between the two dimensional features in each dimension.

[0115] For example, Fig. 9 As shown, similar to the coding-based method, the target audio data provided by the database passes through the auditory model and the human brain auditory system respectively to obtain a pair of first acoustic features and auditory response signals. Here, the number of target audio data can be multiple, Figure 8 The process of obtaining the second acoustic feature shown can be performed on any two of the multiple target audio data.

[0116] Here, the feature distance in each dimension can be represented by a distance matrix. For example, the k-th dimension feature A of the two first acoustic features can be represented by k (x i ) and A k (x j ), the distance matrix M of the model k ∈R m×m, where k = {1, 2, ... D}, D is the number of dimensions, m is the number of target audio data, and here, the distance matrix can be a similarity matrix (Representational Similarity Matrix, RSM) or a dissimilarity matrix (Representational Dissimilarity Matrix, RDM). Each element in the matrix represents the distance between two target audio samples in the k-th dimension representation space of the auditory model, that is, the distance between the dimensional features of the two first acoustic features on the k-th dimension. The elements in the distance matrix can be represented, for example, by the following formula (7):

[0117]

[0118] in, represents the distance between two dimensional features in the kth dimension; d(·) represents the distance function of two samples, which may be, for example but not limited to, Euclidean distance or Manhattan distance. In this way, D model distance matrices (Model RDM) can be obtained, which respectively represent the dissimilarity between the acoustic features output by the D-dimensional features of the auditory model under different audio stimuli.

[0119] In addition, the formula (7) is only an example, and the similarity or dissimilarity can also be constructed by using cosine similarity or Pearson Correlation Coefficient (PCC) and other methods.

[0120] In step S820, the division weights for dividing the dimensional features on each dimension into each brain region may be determined.

[0121] Here, the division weight may represent the contribution of each dimension of the first acoustic feature to the formation of each brain region information. As an example, the division weight may be determined before executing the evaluation method, and may be directly called when executing step S820, however, it is not limited thereto, and the division weight may also be determined during the execution of the evaluation method.

[0122] As an example, the partition weight may be set according to, for example, an empirical method, a statistical method, etc. As another example, an embodiment of the present disclosure provides a method of optimizing regression to determine the partition weight.

[0123] For example, the weight can be determined in the following manner: obtaining an optimized sample and sample annotation information, wherein the optimized sample includes a sample first acoustic feature and a sample auditory response signal for the same audio data; optimizing the weight of the sample first acoustic feature, and using the weight that minimizes the difference between the sample first acoustic feature weighted by the weight and the sample auditory response signal as the partitioning weight.

[0124] Specifically, the determination of the partition weights can be regarded as a regression problem, and a regression method such as ridge regression is used to determine the weight of each dimension of the first acoustic feature relative to each brain region (ROI). For example, based on a given sample first acoustic feature and a sample auditory response signal, an optimization target as shown in the following formula (8) can be set:

[0125]

[0126] in, represents the weight of the kth dimension for the brain region ROI; ROI represents the brain region, for example, it can be HG, PP, pSTG, etc.; B ROI Represents the sample auditory response signal in the brain area ROI; M k represents the k-th dimension model distance matrix obtained based on the first acoustic feature of the sample; μ is the regularization coefficient. Based on formula (8), the optimization is performed to minimize as the partition weight.

[0127] Although the process of determining the partition weights is described above using ridge regression as an example, it is not limited to this, and other regression methods may also be used or the partition weights may be obtained by training a machine learning model.

[0128] In step S830, the feature distances in each dimension may be weighted based on the partition weights to obtain a second acoustic feature.

[0129] Here, the second acoustic feature may include a weighted feature distance. Fig. 9 For example, in the above step S810, D model distance matrices can be obtained, which respectively represent the representation dissimilarity of the D-dimensional features of the auditory model under different audio stimuli. The final second acoustic feature M can be the weighted sum of the model distance matrices, which can be expressed by the following formula (9):

[0130]

[0131] Among them, α k represents the weight of the kth dimension; M k represents the distance matrix of the kth dimension. For example, Fig. 9 As shown, for the same brain region, a weight can be set for each model distance matrix. For example, for brain region HG, D distance matrices M 1 、M 2 To M D The weights can be to

[0132] In the above process, by calculating the feature distance of similarity or dissimilarity, the relationship between different neural or cognitive representations can be characterized. This is different from the traditional distance calculation method. In the embodiment of the present disclosure, it is believed that the weights of each dimension of the first acoustic feature are inconsistent. By assigning different weights to each dimension of the acoustic feature, an attempt is made to make the acoustic representation similarity of the model more brain-like, which is more convenient for comparing the signal distances of different brain regions in the auditory response signal in subsequent processing, and deeply analyzing the correspondence between the model and each brain region.

[0133] It should be noted here that the two examples of obtaining the second acoustic feature mentioned above can be used either one or at the same time, so as to obtain two second acoustic features in two ways respectively, so that the auditory model can be evaluated from different dimensions, and the two evaluation results can be compared and cross-validated.

[0134] Return to reference Figure 2 In step S230, the brain-like characteristics of the auditory model can be evaluated based on the second acoustic feature and the auditory response signal.

[0135] In this step, the evaluation result can be obtained by comparing the second acoustic feature with the corresponding auditory response signal.

[0136] As an example, the second acoustic feature and the corresponding auditory response signal can be compared by multiple evaluation indicators. Specifically, the brain-like characteristics of the auditory model can be evaluated in the following manner: the second acoustic feature is compared with the auditory response signal to obtain multiple evaluation indicators; and the multiple evaluation indicators are fused to obtain an evaluation result of the brain-like characteristics of the auditory model.

[0137] Here, each evaluation index can represent the signal difference between the second acoustic feature and the auditory response signal, and different evaluation indexes measure the signal difference in different measurement methods. Based on this evaluation method, a more comprehensive evaluation result can be obtained by fusing multiple evaluation indexes, thereby improving the accuracy of the evaluation.

[0138] For example, in the above example of obtaining the second acoustic feature by encoding, the total sum of squares SST, the regression sum of squares SSR and the residual sum of squares SSE of the signal difference between the second acoustic feature and the auditory response signal can be calculated respectively:

[0139]

[0140]

[0141]

[0142] in, represents the mean of the auditory response signal, that is, the mean of the true value, m represents the number of target audio data, that is, m second acoustic features and m auditory response signals can be obtained, and m can be a positive integer; y i represents the auditory response signal corresponding to the i-th target audio data; Represents the second acoustic feature, which is the predicted value of the model.

[0143] In order to evaluate the degree of coding fit, that is, the regression accuracy, the total sum of squares SST, the regression sum of squares SSR and the residual sum of squares SSE can be fused based on the above equations (10), (11) and (12) to obtain the determination coefficient R 2 , to evaluate the model. For example, the coefficient of determination R can be expressed by the following formula (13): 2 :

[0144]

[0145] For another example, the covariance and standard deviation of the signal difference between the second acoustic feature and the auditory response signal may be calculated respectively, and the two indicators of covariance and standard deviation may be fused to obtain the Pearson correlation coefficient PCC. For example, the Pearson correlation coefficient PCC may be expressed by the following formula (14):

[0146]

[0147] in, represents the covariance of variables X and Y, σ(X) represents the standard deviation of variable X, represents a set of m second acoustic features, and y represents a set of m auditory response signals.

[0148] In the above calculations, the coefficient of determination and the Pearson correlation coefficient can both measure the linear relationship between variables. The value of the coefficient of determination is [0,1]. The closer it is to "1", the better the acoustic representation of the auditory model fits the human brain, or the higher the degree of brain-likeness; conversely, the lower the degree of fit. The value of the Pearson correlation coefficient is [-1,1]. The closer it is to "1", the better the acoustic representation of the auditory model fits the human brain; conversely, the lower the degree of fit.

[0149] Although the determination coefficient and the Pearson correlation coefficient are used as examples for description here, it is not limited thereto, and other difference analysis methods or other calculation dimensions may also be used for evaluation and calculation.

[0150] For example, in the above evaluation process, the difference between the second acoustic feature and the overall auditory response signal was compared. In addition, the analysis can be performed from a more fine-grained perspective. For example, the comparison can be made at the voxel level of each brain region. Specifically, the second acoustic feature matrix predicted during the encoding process can be The PCC is calculated on the voxel dimension of each brain region with the real auditory response signal matrix Y to obtain the correlation between each voxel and the acoustic feature, which is recorded as PCC v Similarly, we can compare the PCC of each voxel with which layer of network features v The highest, and can clearly show the mapping relationship between voxels and the best network layer through visualization means, showing the human brain's processing process of sound from low-level to high-level, and can analyze the correspondence between brain areas and network layers by analyzing the correspondence between voxels and network layers.

[0151] As another example, in the above example of obtaining the second acoustic feature by calculating the feature distance, the brain-like characteristics of the auditory model can be evaluated in the following way: determine the signal distance between the two auditory response signals corresponding to the two audio data in each dimension; and obtain the evaluation result of the brain-like characteristics of the auditory model by comparing the weighted feature distance with the signal distance.

[0152] Specifically, the signal distance of the human brain auditory system can also be expressed in the form of a distance matrix, for example, as the distance matrix B ROI ∈R m×m , each element in the distance matrix represents the distance between two target audio data in the representation space of the auditory response signal. Based on the above method of calculating the feature distance for the first acoustic feature, the signal distance between the two auditory response signals in each dimension can be calculated, which can be expressed by the following formula (15):

[0153]

[0154] in, represents the distance between two auditory response signals on the brain region ROI, where ROI represents the brain region, such as HG, PP, pSTG, etc.; and Represents the signals of two auditory response signals in the brain area ROI respectively.

[0155] In a known brain region, the model distance matrix (Model RDM) M ROI and the distance matrix of the human brain auditory system (Brain RDM) B ROI In this case, the brain-like characteristics of the model can be evaluated by calculating the correlation coefficient between the two.

[0156] As an example, Representational Similarity Analysis (RSA) can be used for evaluation. RSA is a method for analyzing neural and cognitive data. It can be used to compare and quantify the similarities between different representations in the brain or cognitive system. It can characterize the relationship between different neural or cognitive representations by calculating similarity or dissimilarity matrices.

[0157] In this example, each element in the distance matrix represents the acoustic features of the auditory model or the dissimilarity of the auditory response signals under two audio stimulation conditions. By analyzing the correlation between the two types of distance matrices, the connection between the auditory model and the human brain auditory system can be established.

[0158] For example, when calculating the correlation coefficient between the two, PCC or Spearman Correlation Coefficient (SCC) can be used. Specifically, the model distance matrix M ROI The distance matrix B from the human auditory system ROI Stretch them into one dimension respectively to obtain two m 2 Sequence of dimensions and In the example of using PCC as the evaluation indicator, it can be calculated using the following formula (16):

[0159]

[0160] Among them, score ROI Represents the evaluation score for the brain region ROI, for example Fig. 9 As shown, score ROI Can be score HG 、score PP 、score pSTG wait.

[0161] In the example using SCC as the evaluation metric, and Sort and store the order in r M and r B , and use the following formula (17) to calculate:

[0162]

[0163] Among them, d i Represents the difference in the order of the i-th element, i.e. d i =r i M -r i B, r i M and r i B Respectively represent r M and r B The i-th element in .

[0164] Here, the values ​​of PCC and SCC are [-1, 1], where the closer to "1" the better the acoustic representation of the auditory model fits the human brain; conversely, the lower the fit.

[0165] The final evaluation score can be the weighted sum of the evaluation scores of all brain regions, which can be expressed by the following formula (18):

[0166] score=∑ ROI β ROI score ROI (18)

[0167] Among them, ∑ ROI β ROI =1, which represents the score weight of each brain region, which can be set according to experience or actual needs, for example Fig. 9 As shown, β ROI Can be β HG , β PP , β pSTG wait.

[0168] Although the Pearson correlation coefficient and the Spearman correlation coefficient are used as examples for description here, it is not limited thereto, and other correlation analysis methods may also be used for evaluation and calculation.

[0169] Through the above evaluation method, by assigning different weights to each dimension of the first acoustic feature and fitting the auditory response signal, we attempt to make the representation similarity of the auditory model more brain-like, so as to obtain the upper limit of the degree of brain-likeness of the acoustic model. For the signal distance of the auditory response signal, different weights can be assigned to each brain region, so that important brain regions occupy a larger weight in the final evaluation score.

[0170] The above describes the evaluation method of the auditory model according to the embodiment of the present disclosure. The method can be adapted to a variety of evaluation scenarios. Several evaluation items will be described in detail below in an illustrative manner.

[0171] As an example, the evaluation method according to the present disclosure may further include: based on the evaluation result of the brain-like characteristics of the auditory model, comparing the brain-likeness of the auditory model with that of the artificial acoustic feature.

[0172] Specifically, for the same "audio data-auditory response signal" sample pair, such as an audio-fMRI signal sample pair, the audio representation can be calculated using a DNN-based deep auditory model and a commonly used artificial acoustic feature formula, respectively. The first acoustic feature can be encoded according to the evaluation method described above to obtain the second acoustic feature, and the determination coefficient and Pearson correlation coefficient of the deep auditory model can be calculated; or, the feature distance of the first acoustic feature in each dimension is weighted to obtain the second acoustic feature, and the Pearson correlation coefficient and Spearman correlation coefficient of the deep auditory model are calculated, so as to compare the correlation between the auditory model representation and the auditory response signal of the human brain and the correlation between the artificial acoustic feature and the auditory response signal of the human brain, and compare the degree of fit of the two to the human brain, that is, the degree of brain-likeness. As an example, such acoustic features can be characterized by existing artificial acoustic feature quantities, such as but not limited to spectrograms, mel spectrograms, mel frequency cepstral coefficients MFCC, linear frequency cepstral coefficients LFCC, and fundamental frequency Pitch.

[0173] As an example, the evaluation method according to the present disclosure may further include: based on the evaluation results of the brain-like characteristics of multiple auditory models, comparing the brain-likeness between different auditory models.

[0174] Specifically, for the same "audio data-auditory response signal" sample pair, such as the audio-fMRI signal sample pair, different deep auditory models can be used to extract acoustic representations, and then the correlation between the acoustic representations and the auditory response signals of the human brain can be calculated according to the evaluation method described above, and then the degree of fit of different auditory models to the human brain, that is, the degree of brain-likeness of the model, can be compared. After evaluating a large number of deep auditory models, the connection between the model network structure and the degree of brain-likeness can be established, and a more brain-like network framework can be screened out as a reference for selecting and designing models under various auditory tasks. Considering that the human brain still outperforms deep models in scenarios such as complex tasks and multimodal tasks, models with a high degree of brain-likeness may perform better in these tasks.

[0175] As an example, the evaluation method according to the present disclosure may further include: based on the evaluation result of the brain-like characteristics of the auditory model, analyzing the alignment relationship between the hierarchical structure in the auditory model and different brain regions.

[0176] For example, the signal corresponding to any brain region in the auditory response signal can be extracted, and the layer features of each network layer output in the auditory model can be extracted; by comparing the signal of the brain region with the features of each layer, the network layer with the highest correlation with the brain region is determined. Here, the comparison of the signal of the brain region with the layer features can include, for example: encoding the layer features (for example, using Figure 6The encoding model shown in the figure is used to obtain the encoded layer features; the encoded layer features are compared with the signals of the brain areas.

[0177] Specifically, the network structure of the auditory model, such as the deep auditory model, processes sounds from low to high from shallow to deep, which is similar to the processing mechanism of the human brain auditory system. Analyze at the brain region level and extract the auditory response signal of a certain brain region ROI in the database And extract the network features of a certain layer l of the auditory model (for example, a CNN layer or a Transformer layer) as the audio feature data A l (x i ), using the coded evaluation method, constructing l (x i )arrive , and obtain correlation indicators such as the Pearson correlation coefficient and the Spearman correlation coefficient. By analogy, the correlation between the auditory response signals of all brain regions and the acoustic features of all network layers can be calculated, and relevant conclusions can be obtained, such as which layer of network features has the highest correlation with a certain brain region. If the alignment relationship between the hierarchical structure of the auditory model and the brain regions of the human brain is established, the low-level auditory brain regions will have a higher correlation with the shallow model features, and the high-level auditory brain regions will have a higher correlation with the deep model features.

[0178] As an example, the evaluation method according to the present disclosure may further include: based on the evaluation result of the brain-like characteristics of the auditory model, analyzing the influence of the training method and / or training task of the auditory model on the brain-like characteristics of the auditory model.

[0179] For example, multiple auditory models trained under different training methods and / or training tasks can be obtained; based on the evaluation results of the brain-like characteristics of each auditory model, the training effects or training performances of the auditory models are compared to compare the training methods and / or training tasks of the auditory models.

[0180] Specifically, in the process of training the auditory model, the supervision method, training tasks, and data sets are different. Will this affect the brain-like characteristics of the model? For example, by adopting the evaluation method of the present disclosure, it can be found that the correlation between the Wav2Vec2 model trained in a self-supervised manner and the human brain is stronger than the correlation between the Wav2Vec2 model trained in a supervised manner and the human brain. Based on the evaluation method of the embodiment of the present disclosure, the brain-like performance of the same auditory model under different training tasks or even different training sets can also be compared. Here, the training tasks may include but are not limited to audio classification, phoneme classification, speech recognition, etc., and the data sets may include but are not limited to sound data sets (AudioSet), speech data sets LS-960 (including 960 hours of speech data), LV-60k (including 53.2k hours of speech data), etc. By comparing different training methods and training tasks, the connection between brain-like characteristics and model training can be deeply explored to guide the training optimization of the model.

[0181] As an example, the evaluation method according to the present disclosure may further include: analyzing the selectivity of the auditory model and the human brain for audio types based on the evaluation result of the brain-like characteristics of the auditory model.

[0182] For example, the evaluation results of the brain-like characteristics of the auditory model under audio data of different audio types can be obtained; by comparing the evaluation results under audio data of different audio types, the audio type corresponding to the evaluation result with the highest brain-like degree is determined to obtain the selectivity of the auditory model for audio types.

[0183] Specifically, the human brain has a gradient in its perception of various sounds, from natural sounds to human speech. Different sounds have different activation locations and activation intensities in the brain, and the auditory model also has similar characteristics. For example, there is a big difference between native language audio and non-native language audio in the auditory model. This evaluation method can use the gradient of audio stimulation to explore the selectivity of the human brain and the auditory model for audio types, and compare whether the selectivity of the two is consistent.

[0184] As an example, four types of audio stimuli with obvious gradients, namely human speech, human non-speech, animal calls and natural sounds, can be used for experiments. The experimental task used is the ABX task, that is, randomly sampling two audio stimuli A and B belonging to different categories, and then selecting a third audio stimulus X so that the category of audio stimulus X is consistent with one of audio stimuli A and B. The auditory response signals of different brain regions of the human brain under these three audio stimuli and the acoustic features of different network layers of the auditory model are extracted respectively, and the audio stimulus X is closer to which of audio stimulus A and audio stimulus B by calculating the similarity. After multiple groups of experiments and statistical judgment accuracy, it can be determined which type of audio stimulus a certain brain region or a certain network layer has the highest judgment accuracy for, so that it can be determined that it has selectivity for this category of audio. After calculating the selectivity of the human brain and the auditory model to the audio type, the calculation results are compared with the alignment relationship between the model hierarchical structure and the human brain region described above to observe whether there is a similar alignment relationship.

[0185] According to the evaluation method of the auditory model of the exemplary embodiment of the present disclosure, it is possible to make up for the defects of the model's interpretability, improve the interpretability of the auditory model, and gain more insights into the human auditory system. By analyzing the internal characteristics and activation patterns of the model and comparing them with the neural activity of the human auditory system, the auditory model's processing of audio data can be revealed. Based on such an evaluation method, it is possible to provide higher credibility and understanding while meeting the requirements for the black box model. This evaluation method can open up new prospects for cross-research between auditory science and artificial intelligence, and promote the application and development of intelligent sound processing systems in various fields.

[0186] In a second aspect of the exemplary embodiments of the present disclosure, there is provided an evaluation device for an auditory model. Fig.10 is a schematic block diagram of an evaluation device for an auditory model according to an exemplary embodiment of the present disclosure. Fig.10 As shown, the evaluation device of the auditory model may include an acquisition unit 1010 , a division unit 1020 and an evaluation unit 1030 .

[0187] The acquisition unit 1010 can be configured to acquire a first acoustic feature and an auditory response signal for the target audio data, wherein the first acoustic feature is obtained by inputting the target audio data into a target auditory model, and the auditory response signal is a response signal of the human brain auditory system to the target audio data.

[0188] The dividing unit 1020 may be configured to divide the first acoustic feature according to the brain regions corresponding to the auditory response signal to obtain a second acoustic feature corresponding to the auditory response signal.

[0189] The evaluation unit 1030 may be configured to evaluate the brain-like characteristics of the auditory model based on the second acoustic feature and the auditory response signal.

[0190] As an example, the first acoustic feature includes layer features output by each network layer of the target auditory model, wherein the division unit 1020 can be configured to obtain the second acoustic feature in the following manner: based on each brain region corresponding to the auditory response signal, the features of each layer are divided to determine the correspondence between the features of each layer and each brain region; according to the division results of the features of each layer, the features of each layer are encoded to obtain the second acoustic feature, wherein the second acoustic feature has the same information form as the auditory response signal.

[0191] As an example, the division unit 1020 can be configured to obtain the second acoustic feature in the following manner: input each layer feature of the first acoustic feature into a pre-trained coding model, wherein the pre-trained coding model includes a router and multiple expert models, each expert model corresponds to at least one brain region, and different expert models correspond to different brain regions; use the router to predict the prediction probability of each layer feature corresponding to each expert model; use each expert model to encode each layer feature, and use the prediction probability to weight each coding result to obtain a weighted layer feature; based on the weighted layer feature, obtain the second acoustic feature.

[0192] As an example, the coding model can be trained in the following manner: obtaining training samples and sample annotation information, wherein the training samples include a first sample acoustic feature output by a target auditory model, each first sample acoustic feature includes a sample layer feature output by each network layer of the target auditory model, and the sample annotation information represents a second sample acoustic feature corresponding to each first sample acoustic feature; inputting the training samples into the coding model, and using a router to predict the prediction probability of each sample layer feature corresponding to each expert model; using each expert model to encode the corresponding sample layer feature, and using the prediction probability to weight each encoding result to obtain a weighted sample layer feature; based on the weighted sample layer feature, obtaining a predicted second acoustic feature; determining a prediction loss by comparing the predicted second acoustic feature with the second sample acoustic feature to train the coding model, wherein the prediction loss includes a first loss and a second loss, the first loss represents the difference between the predicted second acoustic feature and the second sample acoustic feature, and the second loss is determined by statistically analyzing the prediction probabilities corresponding to each sample layer feature and each expert model.

[0193] As an example, the evaluation unit 1030 can be configured to evaluate the brain-like characteristics of the auditory model in the following manner: compare the second acoustic feature with the auditory response signal to obtain multiple evaluation indicators, wherein each evaluation indicator represents the signal difference between the second acoustic feature and the auditory response signal, and different evaluation indicators measure the signal difference in different measurement methods; and fuse the multiple evaluation indicators to obtain an evaluation result of the brain-like characteristics of the auditory model.

[0194] As an example, the target audio data includes two audio data, and the first acoustic feature includes dimensional features of multiple dimensions, wherein the division unit 1020 can be configured to obtain the second acoustic feature in the following manner: determine the characteristic distance between the dimensional features of the two first acoustic features corresponding to the two audio data in each dimension; determine the division weights for dividing the dimensional features in each dimension into each brain region, wherein the division weights represent the contribution of each dimension of the first acoustic feature to the formation of information of each brain region; based on the division weights, weight the characteristic distances in each dimension to obtain the second acoustic feature, wherein the second acoustic feature includes the weighted characteristic distance.

[0195] As an example, the evaluation unit 1030 can be configured to evaluate the brain-like characteristics of the auditory model in the following manner: determine the signal distance between two auditory response signals corresponding to two audio data in each dimension; and obtain an evaluation result of the brain-like characteristics of the auditory model by comparing the weighted feature distance with the signal distance.

[0196] Regarding the device in the above-mentioned embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method. Each unit in the auditory model evaluation device can execute the corresponding steps in the method according to the auditory model evaluation method in the method embodiment of the first aspect above, which will not be elaborated here.

[0197] Fig.11 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. Fig.11 As shown, the electronic device 10 includes a processor 101 and a memory 102 for storing processor executable instructions. Here, when the processor executable instructions are executed by the processor, the processor is prompted to execute the auditory model evaluation method as described in the above exemplary embodiment.

[0198] As an example, the electronic device 10 is not necessarily a single device, but may be any collection of devices or circuits that can execute the above instructions (or instruction sets) individually or in combination. The electronic device 10 may also be part of an integrated control system or system manager, or may be configured as a server that is interconnected with a local or remote (e.g., via wireless transmission) interface.

[0199] In the electronic device 10, the processor 101 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not a limitation, the processor 101 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0200] The processor 101 may execute instructions or codes stored in the memory 102, wherein the memory 102 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0201] The memory 102 may be integrated with the processor 101, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 102 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 102 and the processor 101 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 101 can read files stored in the memory 102.

[0202] In addition, the electronic device 10 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 10 may be connected to each other via a bus and / or a network.

[0203] In an exemplary embodiment, a computer-readable storage medium may also be provided, and when the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the evaluation method of the auditory model as described in the above exemplary embodiment. The computer-readable storage medium may be, for example, a memory including instructions. Optionally, the computer-readable storage medium may be: read-only memory (ROM), random access memory (RAM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0204] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided. The computer program product includes computer executable instructions. When the computer executable instructions are executed by at least one processor, the method for evaluating the auditory model according to the exemplary embodiment of the present disclosure is implemented.

[0205] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0206] In addition, it should be noted that although several examples of each step are described above with reference to specific drawings, it should be understood that the embodiments of the present disclosure are not limited to the combinations given in the examples, and the steps appearing in different drawings may be combined, and the order of execution of the steps may be changed, which is not exhaustive here.

[0207] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for evaluating an auditory model, characterized in that: The evaluation methods include: Acquire a first acoustic feature and an auditory response signal for target audio data, wherein the first acoustic feature is obtained by inputting the target audio data into a target auditory model, and the auditory response signal is a response signal of a human brain auditory system to the target audio data; Dividing the first acoustic feature according to the brain regions corresponding to the auditory response signal to obtain a second acoustic feature corresponding to the auditory response signal; Based on the second acoustic feature and the auditory response signal, the brain-like characteristics of the auditory model are evaluated.

2. The evaluation method according to claim 1, characterized in that: The first acoustic feature includes layer features output by each network layer of the target auditory model, wherein the second acoustic feature is obtained by: Based on the brain regions corresponding to the auditory response signals, the features of each layer are divided to determine the correspondence between the features of each layer and each brain region; According to the division result of each layer feature, each layer feature is encoded to obtain the second acoustic feature, The second acoustic feature has the same information form as the auditory response signal.

3. The evaluation method according to claim 1, characterized in that: The second acoustic feature is obtained by: Inputting each layer feature of the first acoustic feature into a pre-trained coding model, wherein the pre-trained coding model includes a router and a plurality of expert models, each expert model corresponds to at least one brain region, and different expert models correspond to different brain regions; Using the router to predict the prediction probability of each layer feature corresponding to each expert model; Encoding each layer feature using each expert model, and weighting each encoding result using the prediction probability to obtain a weighted layer feature; The second acoustic feature is obtained based on the weighted layer feature.

4. The evaluation method according to claim 3, characterized in that: The encoding model is trained in the following way: Acquire training samples and sample annotation information, wherein the training samples include first sample acoustic features output by the target auditory model, each first sample acoustic feature includes sample layer features output by each network layer of the target auditory model, and the sample annotation information represents a second sample acoustic feature corresponding to each first sample acoustic feature; Input the training samples into the coding model, and use the router to predict the prediction probability corresponding to each sample layer feature and each expert model; Using each expert model to encode the corresponding sample-level features, and using the prediction probability to weight each encoding result to obtain a weighted sample-level feature; Based on the weighted sample layer features, obtaining a predicted second acoustic feature; determining a prediction loss by comparing the predicted second acoustic feature with the second sample acoustic feature to train the encoding model, The prediction loss includes a first loss and a second loss, the first loss represents the difference between the predicted second acoustic feature and the second sample acoustic feature, and the second loss is determined by statistically analyzing the prediction probabilities corresponding to each sample layer feature and each expert model.

5. The evaluation method according to claim 4, characterized in that: The brain-like properties of the auditory model were evaluated by: Comparing the second acoustic feature with the auditory response signal to obtain a plurality of evaluation indicators, wherein each evaluation indicator represents a signal difference between the second acoustic feature and the auditory response signal, and different evaluation indicators measure the signal difference in different measurement methods; The multiple evaluation indicators are fused to obtain an evaluation result of the brain-like characteristics of the auditory model.

6. The evaluation method according to claim 1, characterized in that: The target audio data includes two audio data, the first acoustic feature includes dimensional features of multiple dimensions, and the second acoustic feature is obtained by: Determine a feature distance between dimensional features of two first acoustic features corresponding to the two audio data respectively in each dimension; Determining a division weight for dividing the dimensional features on each dimension into each brain region, wherein the division weight represents a contribution degree of each dimension of the first acoustic feature to forming information of each brain region; Based on the division weights, the feature distances in each dimension are weighted to obtain the second acoustic feature, wherein the second acoustic feature includes the weighted feature distances.

7. The evaluation method according to claim 6, characterized in that: The brain-like properties of the auditory model were evaluated by: Determine a signal distance between two auditory response signals corresponding to the two audio data respectively in each dimension; By comparing the weighted feature distance with the signal distance, an evaluation result of the brain-like characteristics of the auditory model is obtained.

8. An evaluation device for an auditory model, characterized in that: The evaluation device comprises: an acquisition unit, configured to acquire a first acoustic feature and an auditory response signal for target audio data, wherein the first acoustic feature is obtained by inputting the target audio data into a target auditory model, and the auditory response signal is a response signal of a human brain auditory system to the target audio data; a dividing unit configured to divide the first acoustic feature according to each brain region corresponding to the auditory response signal to obtain a second acoustic feature corresponding to the auditory response signal; An evaluation unit is configured to evaluate the brain-like characteristics of the auditory model based on the second acoustic feature and the auditory response signal.

9. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor, When the processor executable instructions are executed by the processor, they prompt the processor to execute the auditory model evaluation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method for evaluating the auditory model according to any one of claims 1 to 7.

11. A computer program product comprising computer executable instructions, characterized in that: When the computer executable instructions are executed by at least one processor, the method for evaluating the auditory model according to any one of claims 1 to 7 is implemented.

Images