A method and apparatus for evaluating a spatial audio playback system based on multi-modal information

Abstract

The application discloses a kind of based on multi-modal information's space audio playing system evaluation method and device, belong to space audio understanding field.The training set under multiple scenes is acquired, acoustic and spatial clues are extracted using spatial audio encoder, fine-grained perception mode is captured to realize space audio representation, and coded conditional embedding features are generated Space audio embedding features;Complete first stage training based on space audio embedding features;Then by sound-visual collaborative encoder, the sound-visual collaborative coding of playing environment panorama image space feature, space audio embedding feature and coded conditional embedding feature is carried out, and visual audio joint embedding feature is obtained;Complete second stage training based on visual audio joint embedding feature.The present application can be used to effectively guide the quality optimization of space audio playing system in indoor scene, solve the problem of insufficient sound field, sound image perception dimension evaluation of space audio playing system in prior art.

Description

Technical Field

[0001] This invention relates to the field of spatial audio understanding, and more particularly to a method and apparatus for evaluating spatial audio playback systems based on multimodal information. Background Technology

[0002] With the continuous development of spatial audio technology, its applications are becoming increasingly widespread, ranging from virtual reality and augmented reality systems to in-vehicle entertainment platforms, enabling users to obtain immersive auditory experiences. This trend has created an urgent need for comprehensive evaluation methods of the spatial audio playback capabilities of audio systems. However, traditional subjective listening tests are not only costly but also difficult to meet the needs of rapid testing of audio systems. Existing audio playback quality evaluation schemes are mostly limited to mono audio, ignoring human needs for sound field and spatial perception.

[0003] Existing evaluation paradigms for spatial audio playback systems can be divided into three categories: 1) traditional human listening methods, 2) evaluation schemes based on specific audio features, and 3) evaluation schemes based on deep learning.

[0004] Traditional human listening methods rely on subjective evaluations from a large number of subjects. While they can reflect human perception well, they suffer from significant bottlenecks in cost, efficiency, and consistency, making them unsuitable for large-scale system evaluation and rapid iteration. Evaluation schemes based on specific audio features extract spatial perception-related features such as phase difference and intensity difference for analysis. These schemes have some interpretability and usability in specific scenarios, but they rely on manually designed rules or indicators, often only covering a portion of the spatial audio experience, lacking generalization ability, and failing to accurately model complex perceptual mechanisms. Deep learning-based evaluation methods attempt to model human subjective perception of spatial audio in a data-driven manner, demonstrating superior performance to specific audio feature evaluation schemes on multiple tasks. However, current schemes do not deeply explore the subjective-objective correlation in spatial audio evaluation, nor do they consider the influence of environmental factors and reverberation in the audio playback system, and still fall far short of truly reflecting human perception. Summary of the Invention

[0005] To overcome the problems of cost, efficiency, and consistency in manual listening tests, as well as the shortcomings of existing automated evaluation schemes in terms of evaluation accuracy, relevance, and diversity of evaluation dimensions, this invention proposes a spatial audio playback system evaluation method and apparatus based on multimodal information, which can realize automated, multi-dimensional evaluation of spatial audio playback systems.

[0006] The specific technical solution adopted in this invention is as follows:

[0007] In a first aspect, the present invention proposes an evaluation method for a spatial audio playback system based on multimodal information, comprising the following steps:

[0008] (1) Obtain the two-channel stereo audio recorded in the spatial audio playback environment, the panoramic image of the playback environment, the objective perception annotations and the multi-dimensional subjective evaluation scores, and construct the training dataset; the objective perception annotations include sound events, sound source location and sound source distance;

[0009] (2) Extract initial spectral features from the two-channel stereo audio, and spatially encode the initial spectral features by the spatial audio encoder. In the spatial encoding process, the spatial audio latent variables are first generated using the frequency mixing expert model, and then the initialized conditional embedding features are spliced ​​together and input into the non-autoregressive transformer to generate spatial audio embedding features and encoded conditional embedding features. The conditional embedding features contain three feature vectors.

[0010] (3) First-stage training: Based on the spatial audio embedding features, perform the spatial audio objective perception task, and use the objective perception annotation as the training target to train the spatial audio encoder;

[0011] (4) Extract the spatial features of the panoramic image of the playback environment from the panoramic image of the playback environment, and perform audio-visual co-coding on the spatial features of the panoramic image of the playback environment, the spatial audio embedding features after one stage of training and the encoded conditional embedding features by the audio-visual co-encoder to obtain the visual-audio joint embedding features.

[0012] (5) Two-stage training: Based on the visual-audio joint embedding features, a spatial audio score evaluation task is performed, and the sound-visual co-encoder is trained with multi-dimensional subjective evaluation scores as the training target.

[0013] (6) Obtain the two-channel stereo audio and panoramic image of the playback environment from the spatial audio playback system to be evaluated, and use the trained model to predict multi-dimensional subjective evaluation scores.

[0014] Furthermore, the initial spectral feature extraction process includes:

[0015] A spectrum extractor based on short-time Fourier transform is used to extract the left and right channel Mel spectrum, channel loudness difference spectrum, and channel phase difference spectrum from two-channel stereo audio. The initial spectral features are obtained by stacking the three spectra.

[0016] ;

[0017] in, For initial spectral characteristics, , denoted as Mel spectrum for the left and right channels, ILD as the channel loudness difference spectrum, IPD as the channel phase difference spectrum, and melW as the Mel filter bank weight matrix.

[0018] Furthermore, the computational process of the spatial audio encoder includes:

[0019] The initial spectral features are input into a frequency mixing expert model after one-dimensional convolution. Features are extracted by frequency interval in the frequency mixing expert model. The frequency mixing expert model consists of a frequency router and frequency experts based on a feedforward neural network. The frequency router takes the initial spectral features after one-dimensional convolution as input, assigns weights to each frequency expert according to the output probability, and obtains spatial audio latent variables based on the weighted results of the frequency experts' outputs.

[0020] The spatial audio latent variables are concatenated with conditional embedding features initialized to 0 and then positionally encoded. The conditional embedding features contain three feature vectors, each of which represents an independent condition.

[0021] The encoded result is transformed using a multilayer non-autoregressive transformer, and the transformed result is decomposed according to the positional encoding information to obtain spatial audio embedding features and encoded conditional embedding features.

[0022] Furthermore, during the first phase of training, an objective perception prediction head is introduced to perform spatial audio objective perception tasks, including sound event classification tasks, sound source localization tasks, and sound source distance estimation tasks. The sound source localization tasks and sound source distance estimation tasks are set as classification tasks based on range prediction.

[0023] Furthermore, the aforementioned sound-visual co-encoder consists of a series of co-coding blocks concatenated, and the encoding method includes:

[0024] Spatial audio embedding features are used as query features, and spatial features of panoramic images of the playback environment are used as key features. Cross attention is calculated, and the cross attention is connected with the query feature residuals to obtain spatial features with acoustic information.

[0025] Spatial features with acoustic information are input into a spatial hybrid expert model. The encoded conditional embedding features are used as guiding conditions. The hybrid embedding features output by the spatial hybrid expert model are connected with the spatial feature residuals with acoustic information to obtain the output of the current co-coding block.

[0026] The output of the previous co-coding block is used as the input to the query feature of the next co-coding block. After a series of co-coding blocks, the final co-coding block outputs a visual-audio joint embedding feature with spatial information.

[0027] Furthermore, the spatial hybrid expert model includes an event expert model and a spatial expert model;

[0028] The event expert model consists of an event router and an event expert group based on a feedforward neural network. The event router takes the first feature vector in the encoded conditional embedding feature and the spatial feature with acoustic information as input, and selects a combination of one or more event experts according to the output probability to generate an embedding feature with event information.

[0029] The spatial expert model consists of an orientation router and an orientation expert group based on a feedforward neural network. The orientation router takes the second and third feature vectors in the encoded conditional embedding features and the embedding features with event information as input, and selects a combination of one or more orientation experts to generate hybrid embedding features based on the output probability.

[0030] Furthermore, during the second-stage training process, a multi-dimensional evaluation prediction head is introduced to perform spatial audio score evaluation tasks, including an overall evaluation task and evaluation tasks for three sub-dimensions: sound quality, sound image, and sound field. The evaluation task is set as a classification task based on range prediction.

[0031] Furthermore, the steps for constructing the training dataset include:

[0032] Panoramic images of the playback environment are captured and audio is recorded under various spatial audio playback devices and scenarios. The audio is then processed by loudness normalization, silent segment removal, and segment merging to generate effective segments as two-channel stereo audio.

[0033] Objectively label audio data to generate sound event labels, sound source location labels, and sound source distance labels.

[0034] The audio is evaluated subjectively from multiple dimensions, and a consistency check is performed to generate a subjective evaluation score.

[0035] Secondly, the present invention proposes a spatial audio playback system evaluation device based on multimodal information, which is used to implement the above-mentioned spatial audio playback system evaluation method based on multimodal information.

[0036] Thirdly, the present invention proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned evaluation method for spatial audio playback systems based on multimodal information.

[0037] Fourthly, a computer electronic device, including a memory and a processor;

[0038] The memory is used to store computer programs;

[0039] The processor is configured to implement the above-described evaluation method for spatial audio playback systems based on multimodal information when executing the computer program.

[0040] Compared with the prior art, the beneficial effects of this invention are:

[0041] This invention employs a spatial audio encoder to obtain rich semantic information and acoustic features from spatial audio modalities. It also obtains spatial cues based on spectral feature input, enhancing the understanding of spatial audio in terms of sound and space, thereby achieving advanced spatial audio event detection and localization effects.

[0042] This invention integrates audio and visual information. By using a cross-attention mechanism, it combines spatial cues in the audio signal with information such as echo and reverberation of the playback environment contained in the visual information. It also uses a hybrid expert model to comprehensively extract the performance of the spatial audio playback system in terms of sound quality, sound image, and sound field, effectively improving the model's performance in terms of evaluation accuracy, evaluation relevance, and evaluation dimension diversity.

[0043] This invention can fully understand the acoustic and spatial cues of spatial audio playback systems and capture fine-grained perception patterns, thereby achieving a high-quality evaluation of spatial audio playback systems that maintains consistency between subjective and objective perspectives. Attached Figure Description

[0044] Figure 1 This is a flowchart illustrating an evaluation method for a spatial audio playback system based on multimodal information according to the present invention.

[0045] Figure 2 This is a structural block diagram of the two-stage training method of the present invention;

[0046] Figure 3 This is a structural block diagram of the spatial audio encoder of the present invention;

[0047] Figure 4 This is a structural block diagram of the frequency mixing expert model of the present invention;

[0048] Figure 5 This is a structural block diagram of the sound-vision co-encoder of the present invention;

[0049] Figure 6 This is a structural block diagram of the spatial hybrid expert model of the present invention;

[0050] Figure 7 This is a schematic diagram of the electronic device terminal structure of the present invention. Detailed Implementation

[0051] The present invention will be further described and illustrated below with reference to specific embodiments. The embodiments described are merely examples of the content of this disclosure and do not limit the scope of the invention. The technical features of each embodiment in the present invention can be combined accordingly, provided that there is no mutual conflict.

[0052] The accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0053] The flowchart shown in the attached diagram is merely an illustrative example and does not necessarily include all steps. For example, some steps may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0054] This invention proposes an evaluation method for spatial audio playback systems based on multimodal information. It primarily utilizes a spatial audio encoder and a sound-visual co-encoder. The spatial audio encoder extracts rich semantic information and acoustic features from spatial audio modalities, while the sound-visual co-encoder achieves the fusion of audio and visual information. The training phase employs a two-stage training approach, such as... Figure 2 The first training phase introduces an objective perception prediction head to perform spatial audio objective perception tasks, predicting sound events, sound source location, and sound source distance; the second training phase introduces a multi-dimensional evaluation prediction head to perform spatial audio score evaluation tasks, predicting the overall evaluation score and the evaluation scores of the three sub-dimensions of sound quality, sound image, and sound field.

[0055] like Figure 1 As shown, the evaluation method for spatial audio playback systems based on multimodal information mainly includes the following steps:

[0056] Step 1: Obtain the two-channel stereo audio recorded in the spatial audio playback environment, the panoramic image of the playback environment, objective annotations (including sound events, sound source location, and sound source distance information), and multi-dimensional subjective evaluation scores to construct the training dataset.

[0057] In this embodiment, a 50-hour recording dataset of spatial audio for playback, encompassing various recording devices and scenarios, was constructed. It provides fine-grained subjective quality scores, including sound event annotations, virtual sound source location annotations, virtual sound source distance annotations, and four dimensions (sound quality, sound image, sound field, and overall), to support multi-dimensional automated evaluation. The dataset was constructed as follows:

[0058] Audio recording: Six spatial audio playback devices and seven spatial audio playback scenarios were selected to build 42 different spatial audio playback systems. The selected spatial audio clips came from various forms such as music, songs, and movies. The recording equipment used binaural microphones and professional-grade sound cards.

[0059] Audio preprocessing and annotation: 1) Removal of silent segments. After loudness normalization, audio segments with a duration exceeding 0.2 seconds and a decibel value less than -40.0 were removed, and the audio was segmented into segments of varying lengths at the cut-off points; 2) Merging of audio segments. The audio segments were merged to ensure that the effective duration of the final spatial audio segment was between 10 and 20 seconds; 3) Audio annotation: Audio experts were organized to complete the annotation of sound events, virtual sound source location, and virtual sound source distance for the spatial audio segments; here, sound events refer to the type of sound, such as a dog barking or a motorcycle engine roaring.

[0060] Subjective Evaluation: Twenty audio professionals evaluated spatial audio segments, including overall assessment and evaluations of three sub-dimensions: sound quality, sound image, and sound field. After collecting scores from the 20 professionals, the data were tested using Pearson correlation coefficient and Kendall-W concordance coefficient consistency tests to ensure that the professionals used the same or similar evaluation criteria to evaluate the spatial audio.

[0061] Step 2: Extract the initial spectral features of the two-channel stereo audio, and process the initial spectral features using a spatial audio encoder to obtain spatial audio embedding features.

[0062] In this step, the short-time Fourier transform is used as a spectral feature extractor to extract the left and right channel Mel spectrum, channel loudness difference spectrum, and channel phase difference spectrum from the two-channel stereo audio. The three are stacked to obtain the initial spectral features. Then, the initial spectral features are integrated by the spatial audio encoder and conditional embedding features are introduced to obtain spatial audio embedding features and encoded conditional embedding features.

[0063] The initial spectral features are represented as follows:

[0064] ;

[0065] in, For initial spectral characteristics, , melW represents the Mel spectrum of the left and right channels, ILD represents the loudness difference spectrum of the channels, IPD represents the phase difference spectrum of the channels, and melW represents the weight matrix of the Mel filter bank. The linear spectrum is mapped to a frequency band based on human perception and the logarithm is taken.

[0066] In one specific embodiment of the present invention, such as Figure 3 As shown, the spatial audio encoder includes one-dimensional convolution, a frequency mixing expert model, and a non-autoregressive transformer (containing N1 transformer blocks). First, the initial spectral features mentioned above are processed by one-dimensional convolution. Then, the effective audio signals and spatial cues are extracted by frequency mixing expert model in different frequency ranges to obtain spatial audio latent variables. The spatial audio latent variables are concatenated with conditional embedding features initialized to 0 (containing three feature vectors, each with dimensions aligned with the spatial audio latent variables, each representing an independent condition corresponding to event, location, and distance, respectively). The concatenated result is then positionally encoded to allow the model to understand the difference between audio embedding and conditional embedding. The positionally encoded result is then transformed by a multi-layer non-autoregressive transformer, and the output is further split according to the encoded position to obtain encoded conditional embedding features and spatial audio embedding features. The conditional embedding features will subsequently be used as conditional embeddings to guide the sound-visual co-encoder.

[0067] like Figure 4 As shown, the frequency hybrid expert model in this embodiment consists of a frequency router and frequency experts composed of multiple feedforward neural networks. The frequency router takes the initial spectral features after one-dimensional convolution as input, assigns weights to each frequency expert according to the output probability, and generates spatial audio latent variables based on the combination of the frequency expert outputs. Here, the frequency router acts as a learnable gating network, which can use a small neural network (such as an MLP) to learn and analyze the characteristics of the input features in different frequency ranges, adaptively selecting the most suitable expert network for processing audio information in different frequency ranges, thus achieving frequency band processing. Step 3: Perform a first-stage training based on spatial audio embedding features.

[0068] In addition to the recorded dataset, the training data used in this phase also includes the AudioSet open-source audio dataset. An objective perception prediction head is introduced to perform a spatial audio objective perception task after linear projection of the encoded conditional embedding features, predicting sound events, sound source location, and sound source distance. The spatial audio encoder and the objective perception prediction head are trained using real objective annotations containing sound events, sound source location, and sound source distance as training targets.

[0069] In one specific embodiment of this invention, during the first-stage training, considering the limitations of human perception in determining the location and distance of a sound source, the sound source location and distance estimation tasks are changed from continuous regression tasks to classification tasks predicting angular and distance ranges. Specifically, the angle corresponding to the sound source location is divided into intervals of 5 degrees to estimate the range of the sound source location; similarly, in the distance estimation task, the range is estimated in intervals of 1 meter. Cross-entropy loss is used as the training loss function.

[0070] Step 4: Extract the spatial features of the panoramic image of the playback environment from the panoramic image of the playback environment. Then, use a sound-visual co-encoder to perform sound-visual co-encoding on the spatial features of the panoramic image of the playback environment, the spatial audio embedding features after one stage of training, and the encoded conditional embedding features to obtain the visual-audio joint embedding features.

[0071] In this embodiment, the spatial features of the panoramic image of the playback environment are extracted using a pre-trained image encoder (DINO is used in this embodiment, but CLIP or other image encoders can also be used).

[0072] like Figure 5 As shown, a sound-visual co-encoder incorporating a cross-attention mechanism and a spatial hybrid expert model first calculates cross-attention between spatial audio embedding features and spatial features of the panoramic image of the playback environment. Then, guided by the encoded conditional embedding features, a spatial hybrid expert model is used to obtain joint visual-audio embedding features with spatial information. Here, the spatial audio embedding features and the encoded conditional embedding features are generated by the spatial audio encoder after a one-stage training process.

[0073] In one specific embodiment of the present invention, the sound-vision co-encoder consists of N2 cascaded coding blocks. Taking the first coding block as an example, its coding method includes:

[0074] The spatial audio embedding features and the playback environment spatial features are used as query features and key-value features, respectively. They are fused through a cross-attention mechanism. The fusion result is connected with the query feature residual to obtain spatial features with acoustic information.

[0075] The spatial features with acoustic information and the encoded conditional embedding features output by the spatial audio encoder are input into the spatial hybrid expert model to obtain the visual-audio joint embedding features.

[0076] like Figure 6 As shown, the spatial hybrid expert model consists of an event expert model and a spatial expert model. Both hybrid expert models are composed of a task-assigning router and an expert group consisting of multiple feedforward neural networks. The event expert model uses the first feature vector from the encoded conditional embedding features and spatial features with acoustic information as the guiding input to the event router. The event router selects one event expert or a combination of multiple event experts to generate embedding features with event information. The spatial expert model uses the second and third feature vectors from the encoded conditional embedding features and the embedding features with event information output by the event expert model as the guiding input to the orientation router. The orientation router selects one orientation expert or a combination of multiple orientation experts to generate hybrid embedding features, which are then concatenated with the spatial feature residuals with acoustic information to obtain the output of the encoded block. Here, the event router and orientation router, as a learnable gated network, can be implemented using a small neural network (such as an MLP).

[0077] By concatenating multiple coding blocks, the output of the previous coding block is used as the query feature of the next coding block. After processing through N2 coding blocks, the output result is used as a visual-audio joint embedding feature with spatial information.

[0078] Step 5: Perform two-stage training based on the joint visual-audio embedding features.

[0079] The spatial audio encoder is frozen, and a multi-dimensional evaluation prediction head is introduced to perform a spatial audio score evaluation task after the visual and audio jointly embedded features are linearly projected. The real subjective evaluation scores of multiple dimensions are used as training targets, thereby achieving a high-quality spatial audio playback system evaluation that maintains consistency between subjective and objective evaluation.

[0080] In one specific embodiment of the present invention, during the two-stage training, the spatial audio score evaluation task is changed from a continuous regression task to a classification task focused on the accuracy of the prediction range, including overall classification and classification of three sub-dimensions: sound quality, sound image, and sound field. Specifically, the score is divided into intervals of 0.05 points each to predict the range to which the spatial audio evaluation score belongs.

[0081] The soft-sorting cross-entropy loss function is used as the training loss. The loss function for each classification task is as follows:

[0082]

[0083] Where L is the loss function, N represents the number of audio-image pairs in a batch, each audio-image pair is considered a sample, and s is the scaling factor. This represents the true label of the nth sample. Let | be the probability that the nth sample belongs to the i-th class. This represents the degree of difference between the predicted result and the actual value, where C represents the number of categories;

[0084] The total loss in the second stage of training is a weighted sum of the multi-dimensional soft-sorting cross-entropy loss, expressed as:

[0085]

[0086] in, , , , These represent loss functions for the overall sound, sound quality, sound field, and sound image, respectively. This is a hyperparameter.

[0087] Step 6: Evaluate the subjective assessment score of the spatial audio playback system.

[0088] In this step, during the inference phase after training, the user needs to input the two-channel stereo audio from the spatial audio playback system to be evaluated and the panoramic image of the playback environment into the trained model, and use the trained model to predict multi-dimensional subjective evaluation scores.

[0089] Specifically, initial spectral features are extracted from two-channel stereo audio, and spatial audio embedding features and encoded conditional embedding features are generated using a spatial audio encoder. Spatial features of the playback environment panoramic image are extracted from the playback environment panoramic image, and the spatial features of the playback environment panoramic image, spatial audio embedding features, and encoded conditional embedding features are co-encoded by a sound-visual co-encoder to obtain visual-audio joint embedding features. Based on the visual-audio joint embedding features, a spatial audio score evaluation task is performed to predict fine-grained prediction scores in four dimensions: sound quality, sound image, sound field, and overall sound.

[0090] To verify the effectiveness of the present invention, the following experimental results are presented in the spatial audio objective perception task, the spatial audio score evaluation task, and the ablation experiment.

[0091] Table 1: Experimental Results of the Spatial Audio Objective Perception Task

[0092]

[0093] The model performance in the first stage is shown in Table 1. SELDNet and Spatial-AST are two reference models; IPD and ILD are the spectral differences of the vocal tract phase difference and loudness difference, respectively. √ and × indicate whether the features of that dimension were used as input. The three evaluation metrics reflect the accuracy of sound event prediction, the accuracy of sound source location estimation, and the accuracy of sound source distance estimation. It can be found that the proposed scheme performs better than existing model schemes in the spatial audio objective perception task.

[0094] Table 2: Experimental Results of the Spatial Audio Score Evaluation Task

[0095]

[0096] The experimental performance results of the present invention and the comparison model on the spatial audio score evaluation task are shown in Table 2. Among them, wav2vec 2.0 is the mainstream self-supervised audio representation paradigm, SAQAM is a two-channel audio quality evaluation model based on convolutional neural networks; SELDNet and Spatial-AST refer to replacing the spatial audio encoder of the model with these two network structures used for objective perception of spatial audio. In terms of evaluation dimensions, the average error refers to the average absolute error between the true score and the predicted score, and the correlation coefficient refers to the Spearman registration correlation coefficient between the predicted score and the true score on the test set. It can be seen that the present invention achieves the best results.

[0097] Table 3: Ablation Experiment Results

[0098]

[0099] As shown in Table 3, the present invention conducted an ablation experiment on the use of the spatial hybrid expert model. It can be found that when using the spatial hybrid expert model, the model can have a more intuitive and in-depth understanding of spatial audio and visual input, thereby improving the accuracy of the model in score prediction.

[0100] Based on the same inventive concept, this embodiment also provides an evaluation device for a spatial audio playback system based on multimodal information, including:

[0101] The training data acquisition module is used to acquire two-channel stereo audio recorded in a spatial audio playback environment, panoramic images of the playback environment, objective perception annotations, and multi-dimensional subjective evaluation scores to construct a training dataset.

[0102] A spectral feature extractor module, which is used to extract initial spectral features from two-channel stereo audio;

[0103] The image encoder module is used to extract the spatial features of the panoramic image of the playback environment from the panoramic image of the playback environment.

[0104] The spatial audio encoder module is used to spatially encode the initial spectral features. In the spatial encoding process, the spatial audio latent variables are first generated using a frequency mixing expert model, and then the initialized conditional embedding features are spliced ​​together and input into a non-autoregressive transformer to generate spatial audio embedding features and encoded conditional embedding features.

[0105] An audio-visual co-encoder is used to perform audio-visual co-encoder on the spatial features of the panoramic image of the playback environment, the spatial audio embedding features after one stage of training, and the encoded conditional embedding features to obtain joint audio-visual embedding features.

[0106] The phased training module is used for training in two stages. In the first stage, a spatial audio objective perception task is performed based on spatial audio embedding features, and the spatial audio encoder is trained with the objective perception annotation as the training objective. In the second stage, a spatial audio score evaluation task is performed based on visual audio joint embedding features, and the sound-visual co-encoder is trained with the multi-dimensional subjective evaluation score as the training objective.

[0107] For the system embodiments, since they basically correspond to the method embodiments, relevant details can be found in the descriptions of the method embodiments, and the implementation methods of each module will not be repeated here. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0108] The system embodiments of the present invention can be applied to any device with data processing capabilities, such as a computer or other similar device. The system embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution.

[0109] It should also be noted that the spatial audio playback system evaluation method based on multimodal information in the above embodiments can essentially be executed by a computer program. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer electronic device corresponding to the method provided in the above embodiments, which includes a memory and a processor;

[0110] The memory is used to store computer programs;

[0111] The processor is configured to implement the spatial audio playback system evaluation method based on multimodal information in the above embodiments when executing the computer program.

[0112] From a hardware perspective, such as Figure 7The diagram shown is a hardware structure diagram provided in this embodiment. In addition to the processor, memory, network interface and non-volatile memory shown in the diagram, any device with data processing capabilities in the embodiment may also include other hardware depending on the actual function of the device with data processing capabilities, which will not be described in detail here.

[0113] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the method provided in the above embodiments. The storage medium stores a computer program, which, when executed by a processor, can realize the spatial audio playback system evaluation method based on multimodal information in the above embodiments.

[0114] It is understood that the computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0115] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A method for evaluating a spatial audio playback system based on multimodal information, characterized in that, The method comprises the following steps: (1) obtaining binaural stereo audio recorded in a spatial audio playback environment, a playback environment panorama image, objective perception annotations, and multi-dimensional subjective evaluation scores, and constructing a training data set; the objective perception annotations include sound events, sound source directions, and sound source distances; (2) extracting initial spectral features from the binaural stereo audio, and performing spatial coding on the initial spectral features by a spatial audio encoder; in the spatial coding process, first, a frequency mixing expert model is used to generate spatial audio latent variables, and then the initialized conditional embedding features are spliced and input into a non-autoregressive transformer to generate spatial audio embedding features and coded conditional embedding features; the conditional embedding features include three feature vectors; (3) one-stage training: performing a spatial audio objective perception task based on the spatial audio embedding features, taking the objective perception annotations as the training target, and training the spatial audio encoder; (4) extracting playback environment panorama image spatial features from the playback environment panorama image, and performing sound-visual collaborative coding on the playback environment panorama image spatial features, the spatial audio embedding features after one-stage training, and the coded conditional embedding features by a sound-visual collaborative encoder to obtain visual audio joint embedding features; (5) two-stage training: performing a spatial audio score evaluation task based on the visual audio joint embedding features, taking multi-dimensional subjective evaluation scores as the training target, and training the sound-visual collaborative encoder; (6) obtaining binaural stereo audio and a playback environment panorama image from a spatial audio playback system to be evaluated, and predicting multi-dimensional subjective evaluation scores by using the trained model.

2. The method of claim 1, wherein, The extraction process of the initial spectral features comprises: extracting left and right channel mel-frequency spectra, channel loudness difference spectra, and channel phase difference spectra from the binaural stereo audio by using a short-time Fourier transform-based spectrum extractor, and stacking the three to obtain the initial spectral features: ； wherein, is the initial spectral feature, , is the left and right channel mel-spectra, ILD is the inter-channel level difference spectrum, IPD is the inter-channel phase difference spectrum, and melW is the mel-filterbank weight matrix.

3. The method of claim 1, wherein, The calculation process of the spatial audio encoder comprises: inputting the initial spectral features after one-dimensional convolution into a frequency mixing expert model, extracting features in frequency intervals in the frequency mixing expert model, the frequency mixing expert model being composed of a frequency router and a frequency expert based on a feedforward neural network, the frequency router taking the initial spectral features after one-dimensional convolution as input, assigning weights to each frequency expert according to the output probability, and obtaining the spatial audio latent variables according to the weighted results output by the frequency expert; splicing the conditional embedding features, which contain three feature vectors and each feature vector represents an independent condition, with the spatial audio latent variables and performing position coding on the spliced result; converting the coded result by using a multi-layer non-autoregressive transformer, and decomposing the converted result according to the position coding information to obtain the spatial audio embedding features and the coded conditional embedding features.

4. The method of claim 1, wherein, In the one-stage training process, an objective perception prediction head is introduced to perform a spatial audio objective perception task, including a sound event classification task, a sound source direction positioning task, and a sound source distance estimation task, the sound source direction positioning task and the sound source distance estimation task being set as classification tasks based on range prediction.

5. The method of claim 1, wherein, The sound-visual cooperative encoder is composed of a series of cooperative coding blocks in series, and the coding method comprises: Taking the spatial audio embedding feature as a query feature and the playback environment panoramic image spatial feature as a key value feature, cross attention is calculated, and the cross attention is connected with the query feature residual to obtain a spatial feature with acoustic information; The spatial feature with acoustic information is input into a spatial mixing expert model, and the output of the spatial mixing expert model is connected with the spatial feature with acoustic information residual to obtain the output of the current cooperative coding block, with the encoded conditional embedding feature as a guide condition; The output of the previous cooperative coding block is taken as the query feature input into the next cooperative coding block, and after being coded through a series of cooperative coding blocks, the final cooperative coding block outputs a visual audio joint embedding feature with spatial information.

6. The method of claim 5, wherein, The spatial mixing expert model comprises an event expert model and a spatial expert model; The event expert model is composed of an event router and an event expert group based on a feedforward neural network, the event router takes the first feature vector in the encoded conditional embedding feature and the spatial feature with acoustic information as input, and generates an embedding feature with event information according to the output probability of selecting one or more combinations of event experts; The spatial expert model is composed of an orientation router and an orientation expert group based on a feedforward neural network, the orientation router takes the second and third feature vectors in the encoded conditional embedding feature and the embedding feature with event information as input, and generates a mixed embedding feature according to the output probability of selecting one or more combinations of orientation experts.

7. The method of claim 1, wherein, In the two-stage training process, a multi-dimensional evaluation prediction head is introduced to perform a spatial audio score evaluation task, including an overall evaluation task and an evaluation task of three sub-dimensions of sound quality, sound image and sound field, and the evaluation task is set as a classification task based on range prediction.

8. A spatial audio playback system evaluation apparatus based on multi-modal information, configured to implement the spatial audio playback system evaluation method based on multi-modal information according to claim 1, characterized in that, The device comprises: A training data acquisition module for acquiring binaural stereo audio recorded in a spatial audio playback environment, a playback environment panoramic image, objective perception annotations and multi-dimensional subjective evaluation scores, and constructing a training data set; A spectrum feature extractor module for extracting initial spectrum features from the binaural stereo audio; An image encoder module for extracting playback environment panoramic image spatial features from the playback environment panoramic image; A spatial audio encoder module for spatially encoding the initial spectrum features, in which a spatial audio latent variable is generated by using a frequency mixing expert model, and then the initialized conditional embedding feature is spliced and input into a non-autoregressive transformer to generate a spatial audio embedding feature and an encoded conditional embedding feature; An audio-visual cooperative encoder for sound-visual cooperative coding of the playback environment panoramic image spatial feature, the spatial audio embedding feature and the encoded conditional embedding feature after one-stage training to obtain a visual audio joint embedding feature. The phased training module is used for training in two phases, in the first phase, a spatial audio objective perception task is performed based on a spatial audio embedding feature, and a spatial audio encoder is trained with an objective perception label as a training target; in the second phase, a spatial audio score evaluation task is performed based on a visual audio joint embedding feature, and a sound-visual collaborative encoder is trained with a multi-dimensional subjective evaluation score as a training target.

9. A computer-readable storage medium, characterized in that, The storage medium has a computer program stored thereon, and when the computer program is executed by a processor, a spatial audio playing system evaluation method based on multi-modal information is realized.

10. A computer electronic device, comprising: comprising a memory and a processor; The memory is configured to store a computer program. The processor is configured to, when executing the computer program, realize the spatial audio playing system evaluation method based on multi-modal information.

Images