Audio and video joint quality evaluation method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses an audio and video joint quality evaluation method and device, equipment and a storage medium. The method comprises the following steps: determining a first audio stream and a first video stream in a to-be-evaluated audio and video stream; inputting the first audio stream into an audio quality evaluation model, outputting an audio quality score through the audio quality evaluation model, inputting the first video stream into a video quality evaluation model, and outputting a video quality score through the video quality evaluation model; inputting the first audio stream and the first video stream into a consistency evaluation model, and outputting a consistency score through the consistency evaluation model; and according to the audio quality score, the video quality score and the consistency score, determining a joint quality score of the to-be-evaluated audio and video stream. Therefore, the precision of audio and video joint quality evaluation can be improved, and the overall quality of the audio and the video can be accurately reflected.

Description

Technical Field

[0001] The present application relates to the field of audio and video technology, and in particular to a method, device, equipment and storage medium for joint audio and video quality evaluation. Background Art

[0002] With the continuous development of Internet technology, the traffic of streaming media has seen a rapid growth, and more and more consumers are willing to choose to obtain information by watching videos. In streaming media systems, most video signals are accompanied by audio signals. Therefore, when consumers enjoy audio and video, the quality of audio and video signals will directly affect consumers' perception and experience.

[0003] At present, deep learning technology has been widely used in video quality assessment and audio quality assessment. Although there are many video quality assessment frameworks and audio quality assessment frameworks in related technologies, they are relatively independent and most of them only focus on single-mode signals, which cannot accurately reflect the overall quality of audio and video. Summary of the invention

[0004] The present application proposes a method, apparatus, device and storage medium for joint audio and video quality evaluation, which can improve the accuracy of joint audio and video quality evaluation and accurately reflect the overall quality of audio and video.

[0005] The technical solution of this application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for joint audio and video quality assessment, the method comprising:

[0007] Determine a first audio stream and a first video stream in the audio and video streams to be evaluated;

[0008] Inputting the first audio stream into an audio quality assessment model, and outputting an audio quality score through the audio quality assessment model; and inputting the first video stream into a video quality assessment model, and outputting a video quality score through the video quality assessment model;

[0009] Inputting the first audio stream and the first video stream into a consistency evaluation model, and outputting a consistency score through the consistency evaluation model;

[0010] The joint quality score of the audio and video streams to be evaluated is determined based on the audio quality score, the video quality score and the consistency score.

[0011] In a second aspect, an embodiment of the present application provides an audio and video joint quality evaluation device, including a determination unit, an evaluation unit and a combination unit, wherein:

[0012] A determination unit, configured to determine a first audio stream and a first video stream in the audio and video streams to be evaluated;

[0013] an evaluation unit configured to input the first audio stream into an audio quality evaluation model, output an audio quality score through the audio quality evaluation model, and input the first video stream into a video quality evaluation model, output a video quality score through the video quality evaluation model;

[0014] The evaluation unit is further configured to input the first audio stream and the first video stream into a consistency evaluation model, and output a consistency score through the consistency evaluation model;

[0015] The combining unit is configured to determine a combined quality score of the audio and video streams to be evaluated according to the audio quality score, the video quality score and the consistency score.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein:

[0017] A memory for storing computer programs that can be run on the processor;

[0018] A processor is used to execute the method described in the first aspect when running a computer program.

[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by at least one processor, the method described in the first aspect is implemented.

[0020] The embodiment of the present application provides a method, device, equipment and storage medium for joint audio and video quality evaluation. First, the first audio stream and the first video stream in the audio and video stream to be evaluated are determined; then the first audio stream is input into the audio quality evaluation model, and the audio quality score is output through the audio quality evaluation model, and the first video stream is input into the video quality evaluation model, and the video quality score is output through the video quality evaluation model; and the first audio stream and the first video stream are input into the consistency evaluation model, and the consistency score is output through the consistency evaluation model; finally, according to the audio quality score, the video quality score and the consistency score, the joint quality score of the audio and video stream to be evaluated is determined. In this way, in the audio and video quality evaluation, not only the audio quality score output by the audio quality evaluation model and the video quality score output by the video quality evaluation model are considered, but also the consistency score of the synchronization effect between audio and video is considered, such as time consistency, content consistency and spatial consistency, etc., so that the obtained joint quality score can accurately reflect the overall quality of audio and video, improve the accuracy of audio and video joint quality evaluation, and thus improve the quality evaluation performance of audio and video. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Schematic diagram of the process of the audio and video joint quality evaluation method provided in the embodiment of the present application Figure 1 ;

[0022] Figure 2 Schematic diagram of the process of the audio and video joint quality evaluation method provided in the embodiment of the present application Figure 2 ;

[0023] Figure 3 Schematic diagram of the process of the audio and video joint quality evaluation method provided in the embodiment of the present application Figure 3 ;

[0024] Figure 4 A schematic diagram of the network structure of the first category evaluation sub-model provided in an embodiment of the present application;

[0025] Figure 5 Schematic diagram of the process of the audio and video joint quality evaluation method provided in the embodiment of the present application Figure 4 ;

[0026] Figure 6 A schematic diagram of the network structure of the second category evaluation sub-model provided in the embodiment of the present application;

[0027] Figure 7 Schematic diagram of the network structure of the visual position prediction model provided in the embodiment of the present application Figure 1 ;

[0028] Figure 8 Schematic diagram of the process of the audio and video joint quality evaluation method provided in the embodiment of the present application Figure 5 ;

[0029] Fig. 9 Schematic diagram of the network structure of the visual position prediction model provided in the embodiment of the present application Figure 2 ;

[0030] Fig.10 A schematic diagram of the overall framework of the audio and video joint quality evaluation method provided in an embodiment of the present application;

[0031] Fig.11 A schematic diagram of the structure of the audio and video joint quality assessment device provided in an embodiment of the present application;

[0032] Fig.12 A schematic diagram of the specific hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0035] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0036] It should also be pointed out that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0037] With the progress of society and the development of science and technology, the way people convey information is constantly changing. In particular, the rapid development of information technology has made streaming media represented by video and audio gradually become an indispensable way for people to convey information and communicate. In streaming media systems, most video signals are accompanied by audio signals. At the same time, vision and hearing are important ways for people to collect information from the world around them, and most of the external information enters the brain through vision and hearing. Therefore, when consumers enjoy audio and video, the quality of audio and video signals will directly affect consumers' perception and experience.

[0038] The streaming media system can generally include the stages of audio or video acquisition, compression, transmission and display. Each stage may cause distortion of the audio or video signal, thereby reducing the end-user's experience perception. For consumers, they always want to watch and listen in an audio and video system with higher resolution and fidelity. Therefore, in order to ensure that the user experience of streaming media consumers can be maintained above a certain level, an audio and video quality evaluation system is very necessary to ensure the user's final experience perception by testing the audio and video quality.

[0039] For video quality assessment (VQA), deep learning technology has been widely used in video quality assessment models in recent years. With the release of several large image video quality assessment datasets, convolutional neural networks (CNN) have been widely used in the study of video quality assessment. Kang et al. first used the CNN model for image quality assessment and specifically designed a model based on convolutional neural networks to extract image features, replacing the previous manual features. You et al. used a three-dimensional (3D) convolutional network and long short-term memory (LSTM) to perceive the spatiotemporal features of the video and achieved video quality assessment. V-MEON is a multi-task CNN framework that can predict the quality score and codec type of the video. V-MEON also uses a 3D-CNN network to extract spatiotemporal features from the video, and then jointly optimizes the codec classifier and quality predictor. The VSFA model uses a pre-trained image classification task CNN network as a feature extraction module, and then uses a gated recurrent unit (GRU) and a subjectively inspired temporal pooling layer to output the video quality score. In addition, VGG-Net is a deep convolutional network structure proposed by the Visual Geometry Group (VGG), and Yi et al. proposed an improved VGG-16 network, where the 16 in VGG-16 specifically refers to the total number of network layers, which includes non-local layers for learning the global relationship of spatial features, which can be regarded as an attention mechanism. In addition, these models combine GRU to simulate the temporal memory effect.

[0040] For Audio Quality Assessment (AQA), Mean Opinion Score (MOS) is an important indicator for measuring speech quality in communication systems. As an audio quality assessment framework, AutoMOS uses long short-term memory to obtain the temporal information of audio and realizes audio quality assessment. Quality-net is an end-to-end speech quality assessment model based on bidirectional long short-term memory (BLSTM). Mosnet uses convolutional neural networks and recurrent neural networks to extract audio features and outputs them with a fully connected layer. It has good performance in the dataset proposed at that time. Based on Mosnet, Yeunju et al. used a module called global quality token to further improve the accuracy of scoring. SESQA (Semi-supervised learning for SQA) is an audio quality assessment model proposed by Dolby Laboratories. It uses semi-supervised learning to evaluate audio quality and has better generalization than Mosnet. Mbnet (Mean-Bias Network) is an audio quality evaluation framework. It noticed that the deviation between each scorer was ignored in the previous training data, so the framework added a deviation module to obtain a more accurate audio quality MOS value. Spatial Audio Quality Assessment Metric (SAQAM) is an audio quality evaluation model proposed by Princeton University. It takes into account the spatial sense of audio, extracts audio features through feature extractors and time aggregation blocks by inputting binaural signals, and outputs audio quality and spatialization quality, thereby achieving quality evaluation of audio signals in a higher dimension.

[0041] From the above two parts, it can be seen that there are many mature video and audio quality evaluation frameworks in the relevant technology, but they are relatively independent and lack comprehensive consideration of the overall quality of audio and video. Among them, the video quality evaluation method usually only considers the temporal and spatial characteristics of the video screen, while the audio quality evaluation method only considers the characteristics of the audio. However, the final quality of audio and video is affected by both audio and video. The video quality evaluation method based on deep learning adopts an advanced network model, considers the temporal and spatial characteristics of the video, and improves the evaluation accuracy. However, this method only uses video frames as network input, and cannot reflect the overall quality of audio and video and audio quality. Similarly, the audio quality evaluation method only uses audio as network input, and cannot reflect the overall quality of audio and video and video quality.

[0042] Based on this, an embodiment of the present application provides a method for joint audio and video quality evaluation, first determining the first audio stream and the first video stream in the audio and video stream to be evaluated; then inputting the first audio stream into the audio quality evaluation model, outputting the audio quality score through the audio quality evaluation model, and inputting the first video stream into the video quality evaluation model, outputting the video quality score through the video quality evaluation model; and inputting the first audio stream and the first video stream into the consistency evaluation model, outputting the consistency score through the consistency evaluation model; finally, according to the audio quality score, the video quality score and the consistency score, determining the joint quality score of the audio and video stream to be evaluated. In this way, in the audio and video quality evaluation, not only the audio quality score output by the audio quality evaluation model and the video quality score output by the video quality evaluation model are considered, but also the consistency score of the synchronization effect between audio and video is considered, such as time consistency, content consistency and spatial consistency, etc., so that the obtained joint quality score can accurately reflect the overall quality of the audio and video, and can also improve the accuracy of the joint quality evaluation of audio and video.

[0043] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0044] In one embodiment of the present application, see Figure 1 , which shows a flow chart of a method for joint audio and video quality evaluation provided by an embodiment of the present application. Figure 1 As shown, the method may include:

[0045] S101, determining a first audio stream and a first video stream in audio and video streams to be evaluated.

[0046] It should be noted that, in the embodiment of the present application, the audio and video stream to be evaluated can be a complete audio and video stream, or a small part of the audio and video stream, which is not specifically limited here.

[0047] It should also be noted that, in the embodiment of the present application, the audio and video stream to be evaluated is split to obtain a first audio stream and a first video stream.

[0048] S102: Input the first audio stream into an audio quality evaluation model, and output an audio quality score through the audio quality evaluation model; and input the first video stream into a video quality evaluation model, and output a video quality score through the video quality evaluation model.

[0049] It should be noted that in the embodiments of the present application, the audio quality evaluation model can be any evaluation model of the related technology, such as Quality-net, Mosnet, Mbnet, etc.

[0050] For example, MeanNet in Mbnet can be selected as the basic network skeleton. Mbnet adopts a network structure based on BLSTM, which is a deep learning model commonly used for sequence data modeling. It combines the advantages of long-short term memory (LSTM) and bidirectional recurrent neural network (BiRNN or BRNN), and can perform bidirectional modeling and long-term dependency learning on sequence data.

[0051] Specifically, the MeanNet structure in Mbnet is shown in Table 1. Here, Table 1 shows a schematic Mbnet network structure, where Conv represents the convolution operation, K represents the convolution kernel size, C represents the number of output channels, and BN represents the batch normalization operation.

[0052] Table 1

[0053] Layer Name operate K C Convolutional Layer Conv×3-Dropout-BN 3 16 Convolutional Layer Conv×3-Dropout-BN 3 32 Convolutional Layer Conv×3-Dropout-BN 3 64 Convolutional Layer Conv×3-Dropout-BN 3 128 BLSTM Layer BLSTM - 128 Fully connected layer - - 1

[0054] It should also be noted that in the embodiments of the present application, the video quality evaluation model can be any evaluation model of the related technology, such as V-MEON, VSFA, VGG-16, Fast-VQA, etc.

[0055] For example, Fast-VQA can be selected as the basis of the video quality assessment model. This method uses a sampling method called "fragmentation" as the input of a deep neural network to efficiently evaluate video quality and retain quality-related information. In order to improve accuracy, Fast-VQA also designs a fragment attention network based on a Transformer structure to better utilize the characteristics of "fragments". Among them, FAST-VQA can achieve efficient end-to-end deep VQA, and has good performance for inputs of various resolutions while maintaining high efficiency.

[0056] S103: Input the first audio stream and the first video stream into a consistency evaluation model, and output a consistency score through the consistency evaluation model.

[0057] It should be noted that in the embodiments of the present application, the consistency evaluation model can be used to evaluate the temporal consistency of the first audio stream and the first video stream, or to evaluate the content consistency of the first audio stream and the first video stream, or to evaluate the spatial consistency of the first audio stream and the first video stream, or to evaluate any combination of two of the temporal consistency, content consistency and spatial consistency of the first audio stream and the first video stream, etc., and is not specifically limited here.

[0058] In some embodiments, inputting the first audio stream and the first video stream into a consistency evaluation model, and outputting a consistency score through the consistency evaluation model may include at least one of the following:

[0059] Inputting the first audio stream and the first video stream into a first category evaluation sub-model, and outputting a first category consistency score through the first category evaluation sub-model;

[0060] The first audio stream and the first video stream are input into the second category evaluation sub-model, and the second category consistency score is output through the second category evaluation sub-model.

[0061] In an embodiment of the present application, the first category evaluation submodel is used to evaluate the temporal consistency and / or content consistency of the first audio stream and the first video stream, and the second category evaluation submodel is used to evaluate the spatial consistency of the first audio stream and the first video stream.

[0062] In the embodiment of the present application, the consistency evaluation model may include a first category evaluation sub-model and / or a second category evaluation sub-model. Accordingly, the method may also include:

[0063] If the consistency model includes a first category evaluation sub-model, determining the first category consistency score output by the first category evaluation sub-model as the consistency score;

[0064] If the consistency model includes a second category evaluation sub-model, determining the second category consistency score output by the second category evaluation sub-model as the consistency score;

[0065] If the consistency model includes a first category evaluation submodel and a second category evaluation submodel, the first category consistency score output by the first category evaluation submodel and the second category consistency score output by the second category evaluation submodel are jointly determined as the consistency score.

[0066] In a possible implementation, if only time consistency and / or content consistency are considered, the consistency evaluation model may include a first category evaluation sub-model for evaluating the time consistency and / or content consistency of the first audio stream and the first video stream. In this way, when the first audio stream and the first video stream are input into the consistency evaluation model and the consistency evaluation model outputs a consistency score, it may include: inputting the first audio stream and the first video stream into the first category evaluation sub-model, outputting a first category score through the first category evaluation sub-model, and using the first category score as the consistency score.

[0067] In another possible implementation, if only spatial consistency is considered, the consistency evaluation model may include a second category evaluation sub-model for evaluating the spatial consistency of the first audio stream and the first video stream. In this way, when the first audio stream and the first video stream are input into the consistency evaluation model and the consistency evaluation model outputs a consistency score, it may include: inputting the first audio stream and the first video stream into the second category evaluation sub-model, outputting a second category score through the second category evaluation sub-model, and using the second category score as the consistency score.

[0068] In another possible implementation, if temporal consistency, content consistency and spatial consistency are considered, the consistency evaluation model may include a first category evaluation sub-model and a second category evaluation sub-model, which are used to evaluate the temporal consistency, content consistency and spatial consistency of the first audio stream and the first video stream. In this way, when the first audio stream and the first video stream are input into the consistency evaluation model and the consistency score is output through the consistency evaluation model, it may include: inputting the first audio stream and the first video stream into the first category evaluation sub-model, and outputting the first category score through the first category evaluation sub-model; inputting the first audio stream and the first video stream into the second category evaluation sub-model, and outputting the second category score through the second category evaluation sub-model; and using the first category score and the second category score together as the consistency score.

[0069] It should also be noted that in the embodiment of the present application, if the time consistency, content consistency and spatial consistency of the first audio stream and the first video stream are evaluated, there are two cases of the consistency evaluation model. In one case, the consistency evaluation model may include three sub-models, such as two first-category evaluation sub-models and one second-category evaluation sub-model, wherein the two first-category evaluation sub-models are used to evaluate the time consistency and content consistency of the first audio stream and the first video stream, respectively, and one second-category evaluation sub-model is used to evaluate the spatial consistency of the first audio stream and the first video stream; in another case, the consistency evaluation model may include two sub-models, such as one first-category evaluation sub-model and one second-category evaluation sub-model, wherein the first-category evaluation sub-model is used to evaluate the time consistency and content consistency of the first audio stream and the first video stream, and the second-category evaluation sub-model is used to evaluate the spatial consistency of the first audio stream and the first video stream; no specific limitation is made here.

[0070] In this way, after the consistency score is determined according to the first category evaluation sub-model and / or the second category evaluation sub-model, the final joint quality score can be further determined by combining the audio quality score and the video quality score.

[0071] S104: Determine a joint quality score of the audio and video streams to be evaluated according to the audio quality score, the video quality score, and the consistency score.

[0072] It should be noted that, in the embodiment of the present application, when determining the joint quality score of the audio and video stream to be evaluated, it can include: fusing the audio quality score, the video quality score and the consistency score to determine the joint quality score of the audio and video stream to be evaluated.

[0073] It should also be noted that in the embodiments of the present application, there can be many fusion methods, such as weighted fusion, fusion using a neural network model, or other fusion methods such as support vector machines and regressors, which are not specifically limited here.

[0074] In a possible implementation, a weighted calculation may be performed on the audio quality score, the video quality score, and the consistency score to determine a joint quality score of the audio and video streams to be evaluated.

[0075] Here, the audio quality score, video quality score and consistency score can be weighted according to the preset weight value to obtain the final joint quality score. In an embodiment of the present application, the weighted calculation can be a weighted summation method, or it can be a weighted product method. Exemplarily, taking the weighted summation method as an example, when the preset weight value is equal to 1 / n, it can be regarded as averaging the audio quality score, video quality score and consistency score to obtain the final joint quality score. Wherein, n represents the total number of audio quality scores, video quality scores and consistency scores.

[0076] In another possible implementation, the audio quality score, the video quality score and the consistency score may be input into a joint fitting model, and the joint fitting model may output a joint quality score of the audio and video streams to be evaluated. Here, the joint fitting model may be a pre-trained network model.

[0077] Exemplarily, the joint fitting model is a fully connected network model. Among them, the fully connected network model (Fully Connected Neural Network, FCNN) is a basic artificial neural network structure, which can also be called a multilayer perceptron (Multilayer Perceptron, MLP). In the fully connected network model, each neuron is connected to all neurons in the previous and next layers to form a dense connection structure. The fully connected neural network can learn the complex features of the input data and perform tasks such as classification and regression.

[0078] That is to say, in the embodiment of the present application, the audio quality score, the video quality score and the consistency score can be integrated using a fully connected network model to obtain a final joint quality score.

[0079] In addition, for the joint fitting model, in some embodiments, see Figure 2 , the method further comprises:

[0080] S201, collect sample data.

[0081] S202: Clean and process the collected sample data to obtain a first data set.

[0082] It should be noted that in the embodiments of the present application, data cleaning refers to processing and processing sample data to make it suitable for analysis and modeling. Data cleaning includes operations such as removing duplicate data, filling missing values, processing outliers, and converting data formats to improve the accuracy and reliability of data. Specifically, data cleaning can eliminate data errors and noise and improve the accuracy of analysis and modeling.

[0083] It should also be noted that in the embodiments of the present application, sample data is first collected, that is, the original data set needs to be read, and the format of the data set can be a text file, a CSV file, a database, etc. Then, since the original data set may contain missing values, outliers, duplicate values, etc., data cleaning is required to ensure the quality of the data. The data cleaning methods here include deleting missing values, filling missing values, deleting outliers, standardizing data, etc.; and data preprocessing is to transform and normalize the cleaned data to meet the needs of the machine learning algorithm.

[0084] S203: Determine a subjective score for each audio and video sample in the first data set.

[0085] S204, determining a second data set, wherein the joint fitting model is obtained by performing supervised training based on the second data set.

[0086] It should be noted that in the embodiment of the present application, the second data set is determined by the first data set and the corresponding subjective score. Here, the second data set can also be called an audio-video joint subjective evaluation data set; in this way, the preset network model is supervised and trained according to the audio-video joint subjective evaluation data set, and then the trained model is used as a joint fitting model.

[0087] Furthermore, in the embodiment of the present application, the audio quality score and the video quality score may be normalized. Therefore, before step S104, the method may further include: normalizing the audio quality score and the video quality score respectively to obtain normalized audio quality score and video quality score.

[0088] It should be noted that, in the embodiment of the present application, the normalized audio quality score may be within a preset normalized range, and the normalized video quality score may also be within a preset normalized range. Exemplarily, the preset normalized range is generally within the range of [0,1].

[0089] In this way, the normalized audio quality score, video quality score and consistency score can be fused to obtain a joint quality score of the audio and video stream to be evaluated.

[0090] The embodiment of the present application provides a method for joint audio and video quality evaluation, first determining the first audio stream and the first video stream in the audio and video stream to be evaluated; then determining the audio quality score of the first audio stream using an audio quality evaluation model, and determining the video quality score of the first video stream using a video quality evaluation model; and determining the consistency score of the first audio stream and the first video stream using a consistency evaluation model; finally, determining the joint quality score of the audio and video stream to be evaluated based on the audio quality score, the video quality score and the consistency score. In this way, in the audio and video quality evaluation, not only the audio quality score output by the audio quality evaluation model and the video quality score output by the video quality evaluation model are considered, but also the consistency score affected by the synchronization between audio and video, such as time consistency, content consistency and spatial consistency, etc., is considered, thereby not only improving the accuracy of the joint quality evaluation of audio and video, but also the joint quality score finally obtained can accurately reflect the overall quality of the audio and video, thereby improving the quality evaluation performance of the audio and video.

[0091] In another embodiment of the present application, based on the audio and video joint quality evaluation method described in the above embodiment, for the first category evaluation sub-model, Figure 3 A flowchart of another method for joint audio and video quality evaluation provided in an embodiment of the present application. Figure 3 As shown, the method may include:

[0092] S301, determining at least one group of first samples.

[0093] In an embodiment of the present application, each group of first samples may include a first audio sample, a second audio sample and a first video sample, and the first audio sample is consistent with the first category parameter of the first video sample, and the second audio sample is inconsistent with the first category parameter of the first video sample.

[0094] In the embodiment of the present application, the first category parameter may be a time and / or content consistency parameter.

[0095] S302: Train a first audio feature extractor and a first video feature extractor in a first preset model according to at least one group of first samples.

[0096] In the embodiment of the present application, for the first category evaluation sub-model, it can be obtained by self-supervised training based on multiple sets of triplet input parameters, wherein the triplet input parameters here are two spectrograms and a video frame sequence.

[0097] Specifically, the two spectrograms are determined by the first audio sample and the second audio sample, respectively, and the video frame sequence is determined by the first video sample. The first audio sample is consistent with the content (time) of the video frame sequence, while the second audio sample is inconsistent with the content (time) of the video frame sequence. in As an anchor sample, with f in The audio spectrogram a1 with the same content (time) is taken as a positive sample, and f in The audio spectrogram a2 with inconsistent content (time) is used as a negative sample, and each group of first samples can be recorded as (positive, anchor, negative). It should be noted that for the spectrogram, it can be obtained by performing a short-time Fourier transform (STFT) on the first audio sample or the second audio sample. Here, the horizontal axis direction (X axis) of the spectrogram represents time, in seconds (s); the vertical axis direction (Y axis) of the spectrogram represents frequency, in kilohertz (kHz).

[0098] S303: When the loss function of the first preset model satisfies a first convergence condition, the trained first audio feature extractor and the trained first video feature extractor are determined as a first category evaluation sub-model.

[0099] In the model training process, taking a group of first samples (positive, anchor, negative) as an example, (positive, anchor, negative) is input into the first preset model, and the corresponding deep feature g can be extracted by the first audio feature extractor. a (positive),g a (negative), the corresponding depth feature g can be extracted by the first video feature extractor v (anchor); where g v (•) is a 128-dimensional feature vector. Exemplarily, the first video feature extractor may use a network skeleton based on TimeSformer, and the first audio feature extractor may use a network skeleton based on ResNet-18.

[0100] In addition, calculate g a (positive) and g v The characteristic distance Dis(gv (anchor),g a (postive)), and calculate g v (anchor) and g a The characteristic distance Dis(g v (anchor),g a (negative)). For example, here, the Euclidean distance (also called "Euler distance" or "Euclidean distance") can be used as the characteristic distance between two feature vectors. The calculation formula of the Euclidean distance is as follows:

[0101]

[0102] It should also be noted that in addition to using Euclidean distance, the calculation of feature distance can also use Manhattan distance, Chebyshev distance, angle cosine distance, etc., which are not specifically limited here.

[0103] Here, we can use the triplet loss function T tri As the loss function of the first preset model. The triplet loss value is used to constrain the network model to maximize g a (positive) and g v (anchor), g v (anchor) and g a (negative) as the convergence condition of the first preset model. For example, the triple loss function T tri The definition is as follows:

[0104]

[0105] The purpose of margin is to keep the two groups as far apart as possible, and it is set based on experience. In practical applications, margin is a preset constant value, which can usually be set to 0.2, but there is no specific limit.

[0106] Further, after obtaining the trained first category evaluation sub-model, in some embodiments, inputting the first audio stream and the first video stream into the first category evaluation sub-model, and outputting the first category consistency score through the first category evaluation sub-model may include:

[0107] Using the trained first audio feature extractor to extract features from the first audio stream to obtain first audio features;

[0108] Using the trained first video feature extractor to extract features from the first video stream to obtain first video features;

[0109] A feature distance between the first audio feature and the first video feature is determined, and a first category consistency score is determined based on the feature distance.

[0110] It should be noted that in the embodiment of the present application, the first audio feature and the first video feature are both deep features extracted by the audio and video feature extractor. Here, the feature distance between the first audio feature and the first video feature can still be calculated using the Euclidean distance shown in the above formula (1). Then, the first category consistency score is determined based on the feature distance and the first preset range.

[0111] In a specific embodiment, when determining the first category consistency score based on the feature distance, it can include: if the feature distance is within a first preset range, determining the first category consistency score to be a first value; and / or, if the feature distance is outside the first preset range, determining the first category consistency score to be a second value.

[0112] It should be noted that, in the embodiment of the present application, the first category consistency score is a binomial distribution parameter, and the specific value can be 0 and 1. Among them, the value of the first value is 1, and the value of the second value is 0.

[0113] It should also be noted that in the embodiment of the present application, a first threshold α can be set here. Among them, the first preset range can be [0, α). That is, if the feature distance is less than α, then it can be considered that the content and / or time of the first audio stream and the first video stream are consistent, that is, the first category consistency score is determined to be 1; otherwise, if the feature distance is greater than or equal to α, then it can be considered that the content and / or time of the first audio stream and the first video stream are inconsistent, that is, the first category consistency score is determined to be 0.

[0114] Figure 4 A schematic diagram of the network structure of a first category evaluation sub-model provided in an embodiment of the present application. Figure 4 As shown, the network structure here may include an audio feature extractor 401, a video feature extractor 402, and an audio feature extractor 403. Among them, the audio feature extractor 401 and the audio feature extractor 403 are both convolutional network structures, the video feature extractor 402 is a network structure based on the time-space attention mechanism (Time-Space transformer, TimeSformer), and the video feature extractor 402 may include a linear embedding layer 421 and a transformer encoder 422.

[0115] exist Figure 4In the example, the input of the network structure includes audio and video 1 and audio and video 2, the audio waveform in audio and video 1 is used as the first audio sample, the video stream in audio and video 1 is used as the first video sample, and the audio waveform in audio and video 2 is used as the second audio sample. At this time, the content (time) of the first audio sample is consistent with that of the first video sample, and the content (time) of the second audio sample is inconsistent with that of the first video sample. The spectrogram corresponding to the first audio sample is input into the audio feature extractor 403 to obtain audio feature 1; the video frame sequence corresponding to the first video sample is input into the video feature extractor 402 to obtain video feature 1; the spectrogram corresponding to the second audio sample is input into the audio feature extractor 401 to obtain audio feature 2; according to the first feature distance between audio feature 1 and video feature 1 and the second feature distance between audio feature 2 and video feature 1, the triplet loss value can be determined to constrain the training of the first preset model, and finally the trained first category evaluation sub-model is obtained, which is used to determine the time and / or content consistency score between the first audio stream and the first video stream.

[0116] For the second category evaluation sub-model, Figure 5 A flowchart of another method for joint audio and video quality evaluation provided in an embodiment of the present application is shown below. Figure 5 As shown, the method may include:

[0117] S501, performing visual position prediction on a first video stream and a first audio stream based on a visual position prediction module to determine a visual position.

[0118] S502: Perform auditory position prediction on the first audio stream based on the auditory position prediction module to determine the auditory position.

[0119] S503: Calculate the visual and auditory distances of the visual position and the auditory position based on the calculation module to determine a second category consistency score.

[0120] It should be noted that, in the embodiment of the present application, the second category evaluation sub-model may include a visual position prediction module, an auditory position prediction module and a calculation module. Among them, the auditory position prediction module may be implemented using an audio position prediction algorithm of related technology, for example, using a Multiple Signal Classification (MUSIC) algorithm to estimate the auditory position, that is, the corresponding auditory position horizontal angle.

[0121] It should also be noted that, in the embodiments of the present application, the visual position prediction module may include a depth prediction model, a visual position prediction model and a splicing module. In some embodiments, when the visual position prediction is performed on the first video stream and the first audio stream based on the visual position prediction module to determine the visual position, it may include: determining the key frames corresponding to the first video stream and the depth map corresponding to the key frames; performing a splicing operation on the key frames and the depth map to obtain a splicing map; determining the spectrogram corresponding to the first audio stream, inputting the spectrogram and the splicing map into the visual position prediction model, outputting a heat map through the visual position prediction model; and determining the visual position according to the heat map.

[0122] That is to say, in the embodiment of the present application, the intermediate frame in the first video stream can be used as a key frame. Where H represents the height of the video frame and W represents the width of the video frame. Then the key frame is input into the depth prediction model, and the depth prediction model outputs the depth map corresponding to the key frame. Assume that the depth prediction model uses G Depth (.) indicates that the depth map corresponding to the key frame is Here, the depth prediction model is a pre-trained network model.

[0123] Exemplarily, Table 2 shows a schematic network structure of a deep prediction model. As shown in Table 2, K represents the size of the convolution kernel, C represents the number of output channels, S represents the step size, P represents padding, Conv represents the convolution operation, IN represents the normalization operation, ReLU and Tanh represent activation functions; in addition, E1, E2, and E3 represent the first encoding layer, the second encoding layer, and the third encoding layer, respectively; D1 and D2 represent the first decoding layer and the second decoding layer, respectively; M1 and M2 represent the first intermediate layer and the second intermediate layer, respectively; F1 and F2 represent the first fusion layer and the second fusion layer, respectively.

[0124] Table 2

[0125]

[0126] Then, the key frame and the depth map are spliced ​​through the splicing module to obtain the spliced ​​map, which is recorded as The specific expression is as follows:

[0127]

[0128] Here, the spliced ​​graph v in As one of the inputs of the visual position prediction model, the spectrogram a corresponding to the first audio stream is in As another input of the visual position prediction model; the visual position prediction model can output the heat map h of the audio and video position information to be evaluated out, the specific expression is as follows:

[0129] h out =G vision (a in ,v in ) (4)

[0130] Among them, G vision (•) denotes the visual position prediction model.

[0131] It should be noted that, in the embodiment of the present application, for the spectrogram a in For example, it can be obtained by performing STFT transformation on the first audio stream. Here, the horizontal axis direction (X axis) of the spectrogram represents time, and the unit is second (s); the vertical axis direction (Y axis) of the spectrogram represents frequency, and the unit is kilohertz (kHz).

[0132] It should also be noted that, in the embodiment of the present application, when determining the visual position based on the heat map, it can include: performing an interpolation operation on the heat map and determining the interpolated image as the visual position.

[0133] That is to say, in the embodiment of the present application, the heat map h out The size of the original video frame is restored through a preset interpolation algorithm and used as the final visual position.

[0134] Figure 6 A schematic diagram of the network structure of a second category evaluation sub-model provided in an embodiment of the present application. Figure 6 As shown, the network structure here may include a visual position prediction module 601, an auditory position prediction module 602 and a calculation module 603. Among them, the visual position prediction module 601 may include a depth prediction model 611, a splicing module 612, a visual position prediction model 613 and an interpolation module 614; the auditory position prediction module 602 may include an audio position prediction algorithm module 621.

[0135] exist Figure 6In the example, for the visual position prediction module 601, the key frame of the first video stream is first determined, and the key frame is input into the depth prediction model 611 to obtain the corresponding depth map; then the key frame and the depth map are spliced ​​by the splicing module 612 to obtain the splicing map; the spectrogram corresponding to the first audio stream (spatial audio, including the right channel R and the left channel L) is determined, and the splicing map and the spectrogram are input into the visual position prediction model 613 to obtain the corresponding heat map; finally, the heat map is interpolated by the interpolation module 614 to obtain the visual position. In addition, for the auditory position prediction module 602, the position of the first audio stream is calculated by the audio position prediction algorithm module 621 to obtain the auditory position. Then the visual position and the auditory position are input into the calculation module 603 to calculate the audiovisual position distance, so that the spatial consistency score between the first audio stream and the first video stream can be obtained.

[0136] Furthermore, for the visual position prediction model 613, the method may also include: determining at least one group of second samples; training the second audio feature extractor and the first image feature extractor in the second preset model according to the at least one group of second samples, and when the loss function of the second preset model meets the convergence condition, determining the trained second audio feature extractor and the trained first image feature extractor as the visual position prediction model.

[0137] In the embodiment of the present application, each group of second samples includes a first audio sample, a first video sample and a second video sample, and the second category parameters of the first audio sample and the first video sample are consistent, while the second category parameters of the first audio sample and the second video sample are inconsistent.

[0138] In the embodiment of the present application, the second category parameter may be a spatial consistency parameter.

[0139] It should also be noted that the training of the visual position prediction model can also be obtained by self-supervised training based on multiple sets of triplet input parameters. The triplet input parameters here are a spectrogram and two spliced ​​images; each spliced ​​image is spliced ​​based on the video frame and the corresponding depth map. In addition, the contrast loss function is used as the loss function of the second preset model. The contrast loss value is used to constrain the similarity of the network model, which is used as the convergence condition of the second preset model.

[0140] Figure 7 A schematic diagram of a network structure of a visual position prediction model provided in an embodiment of the present application. Figure 7 As shown, the network structure here may include an image feature extractor 701, an audio feature extractor 702 and an image feature extractor 703. The input of the network structure includes: video frame 1 and the corresponding depth Figure 1, video frame 2 and the corresponding depth Figure 2 , the first audio sample (cat’s meow).

[0141] exist Figure 7 In the video, frame 1 and the corresponding depth Figure 1 Splicing Gets Splicing Figure 1 , will splice Figure 1 Input the image feature extractor 701, and you can get image feature 1, size C×N×N; video frame 2 and the corresponding depth Figure 2 Splicing Gets Splicing Figure 2 , will splice Figure 2 Input the image feature extractor 703 to obtain image feature 2, which is C×N×N in size; input the spectrogram corresponding to the first audio sample into the audio feature extractor 702 to obtain audio feature 1, which is 1×C in size; content similarity matrix 1 can be determined based on image feature 1 and audio feature 1, and content similarity matrix 2 can be determined based on image feature 2 and audio feature 1; finally, the contrast loss value is determined based on content similarity matrix 1 and content similarity matrix 2 to constrain the training of the second preset model, and finally the trained visual position prediction model is obtained. Here, the size of content similarity matrix 1 and content similarity matrix 12 are both N×N.

[0142] Further, after obtaining the trained visual position prediction model, that is, obtaining the trained first image feature extractor and the trained second audio feature extractor, in some embodiments, for determining the visual position corresponding to the audio and video stream to be evaluated, see Figure 8 , the method may include:

[0143] S801: Determine key frames corresponding to a first video stream and depth maps corresponding to the key frames.

[0144] S802: Perform a stitching operation on the key frame and the depth map to obtain a stitching map.

[0145] S803: Using the trained first image feature extractor to extract features from the spliced ​​image to obtain first image features.

[0146] S804: Determine a spectrogram corresponding to the first audio and video stream.

[0147] S805: Use the trained second audio feature extractor to extract features from the spectrogram to obtain second audio features.

[0148] S806: Determine a content similarity matrix of the first image feature and the second audio feature, and determine a heat map according to the content similarity matrix.

[0149] S807: Perform an interpolation operation on the heat map to determine the visual position.

[0150] It should be noted that in an embodiment of the present application, the audio and video stream to be evaluated is first split into a first audio stream and a first video stream, and the intermediate frames in the first video stream are extracted as key frames; the depth map corresponding to the key frame is determined, and the key frame and the depth map are spliced ​​to obtain a spliced ​​map, and then the trained first image feature extractor is used to extract features from the spliced ​​map to obtain a first image feature; and the trained second audio feature extractor is used to extract features from the spectrogram corresponding to the first audio stream to obtain a second audio feature; based on the content similarity matrix between the first image feature and the second audio feature, a heat map can be determined, and then the visual position can be determined.

[0151] It should also be noted that in the embodiment of the present application, the image feature extractor g is included here. v (•) and audio feature extractor g a (·). Audio feature extractor g a (·) Extract global feature representation of audio, image feature extractor g v (·) Calculate the local representation of each position (x, y), and the audio and video are mapped into a shared latent space. Then, the extracted image features and audio features are multiplied to calculate the corresponding content similarity matrix Matrix. The content similarity matrix is ​​an N×N matrix, and each block represents the content correlation between the corresponding position of the image and the audio.

[0152] Exemplarily, the image feature extractor may also be referred to as a visual feature extractor. Table 3 shows an exemplary visual feature extractor g v (·) network structure.

[0153] Table 3

[0154]

[0155]

[0156] Fig. 9 A schematic diagram of the network structure of another visual position prediction model provided in an embodiment of the present application. Fig. 9 As shown, the network structure here may include an image feature extractor 901 and an audio feature extractor 902. Fig. 9In the embodiment, the key frame and the corresponding depth map are spliced ​​to obtain a spliced ​​map, and the spliced ​​map is input into the image feature extractor 901 to obtain a first image feature with a size of C×N×N; the spectrogram corresponding to the first audio stream is input into the audio feature extractor 902 to obtain a second audio feature with a size of 1×C; a dot product operation is performed based on the first image feature and the second audio feature to obtain a content similarity matrix, and finally a heat map can be determined based on the content similarity matrix, and the heat map is restored to the original size of the key frame using a preset interpolation algorithm to obtain a visual position.

[0157] In some embodiments, after determining the visual position and the auditory position, when calculating the visual and auditory distances of the visual position and the auditory position based on a calculation module and determining the second category consistency score, it can include: calculating the visual and auditory distances of the visual position and the auditory position based on the calculation module to determine a set of horizontal angle values; determining the minimum horizontal angle value from a set of horizontal angle values, and determining the second category consistency score based on the minimum horizontal angle value.

[0158] In a specific embodiment, when determining the second category consistency score based on the minimum horizontal angle value, it can include: if the minimum horizontal angle value is within a second preset range, determining the second category consistency score to be a first value; and / or, if the minimum horizontal angle value is outside the second preset range, determining the second category consistency score to be a second value.

[0159] It should be noted that, in the embodiment of the present application, the second category consistency score (spatial consistency score) is a binomial distribution parameter, and the specific values ​​can be 0 and 1. Among them, the value of the first value is 1, and the value of the second value is 0.

[0160] It should also be noted that in the embodiment of the present application, after obtaining the point set of the visual position and the horizontal angle of the auditory position, it is necessary to compare the visual and auditory positions to obtain the final spatial consistency score. Specifically, first, the horizontal angle range of the visual position is calculated based on the point set of the visual position, and the horizontal angle range of the visual position can be obtained; wherein the horizontal angle range of the visual position includes multiple horizontal angles of the visual position; then the minimum horizontal angle value (also called the "nearest horizontal angle") distance between the auditory position and the visual position is calculated, and the specific calculation formula is as follows:

[0161] distance=abs(min(Angle vision -Angle sound )) (5)

[0162] In a specific embodiment, considering the error of position prediction, a second threshold β can also be set here. The second preset range can be [0, β). That is, when the closest horizontal angle between the auditory position and the visual position is less than β, it can be considered that the auditory position and the visual position are coincident, that is, the audio and video are spatially consistent. The specific calculation formula is as follows:

[0163]

[0164] Here, when the distance is less than β, the spatial consistency score can be 1, that is, the spatial consistency is true; when the distance is greater than or equal to β, the spatial consistency score can be 0, that is, the spatial consistency is false.

[0165] It should also be noted that in the embodiments of the present application, the audio and video joint quality evaluation method here can further consider the content factors of the audio and video, and there is no specific limitation on this.

[0166] The embodiment of the present application provides a method for joint audio and video quality evaluation. Through the above embodiment, the specific implementation of the aforementioned embodiment is elaborated in detail. It can be seen that through the technical scheme of the aforementioned embodiment, not only the audio quality score output by the audio quality evaluation model and the video quality score output by the video quality evaluation model are considered, but also the consistency score of the synchronization effect between audio and video, such as time consistency, content consistency and spatial consistency, etc., is considered, thereby not only improving the accuracy of the joint audio and video quality evaluation, but also the final joint quality score can accurately reflect the overall quality of the audio and video, thereby improving the quality evaluation performance of the audio and video.

[0167] In another embodiment of the present application, based on the same inventive concept as the above-mentioned embodiment, Fig.10 The overall framework diagram of the audio and video joint quality evaluation method provided in the embodiment of the present application is shown in FIG. Fig.10 As shown in the figure, it can be divided into two parts: audio and video joint subjective evaluation dataset and audio and video joint quality evaluation framework.

[0168] In the audio and video joint quality evaluation framework, the audio and video to be evaluated can be split into video stream and audio stream, where the video quality score of the video stream is determined by the video quality evaluation model, and the audio quality score of the audio stream is determined by the audio quality evaluation model; and the content consistency score, time consistency score and spatial consistency score can be obtained through the consistency evaluation model (content consistency model, time consistency model and space consistency model); then the joint quality score, that is, the joint quality score, can be finally obtained through the joint fitting model.

[0169] In the audio-video joint subjective evaluation data set, the audio-video joint subjective evaluation data set may be obtained after data collection, data cleaning, data processing and subjective scoring. Then, the preset network model is supervised and trained according to the audio-video joint subjective evaluation data set, and finally the trained model is determined as the joint fitting model.

[0170] In some embodiments, the present application provides a method for evaluating the joint quality of audio and video that takes into account the temporal, content, and spatial consistency of audio and video. The method may include the following steps:

[0171] Step 1: Split the audio and video to be evaluated into audio stream and video stream.

[0172] Step 2: Send the video stream into a video quality evaluation model of the related technology to obtain a corresponding video quality score, and send the audio stream into an audio quality evaluation model of the related technology to obtain a corresponding audio quality score.

[0173] Step 3: Send the audio stream and the video stream together to the audio and video content and time consistency detection network (i.e., the first category evaluation sub-model in the aforementioned embodiment) to obtain corresponding content and time consistency scores.

[0174] Step 4: Send the audio stream and the video stream together to the spatial consistency detection network (i.e., the second category evaluation submodel in the aforementioned embodiment) to obtain the corresponding spatial consistency score.

[0175] In step 5, the video quality score, audio quality score, and content, time, and space consistency scores obtained in steps 2, 3, and 4 are fed into the joint fitting model, and finally the audio and video joint quality score can be obtained.

[0176] It should be noted that, see Fig.10 ,Here is a joint audio and video quality evaluation method that considers the consistency of audio and video time, content and space. The detailed introduction is as follows:

[0177] Step 1: Split the audio and video to be evaluated into audio stream and video stream.

[0178] Step 2: Send the video stream into a video quality evaluation model of the related technology to obtain a corresponding video quality score, and send the audio stream into an audio quality evaluation model of the related technology to obtain a corresponding audio quality score.

[0179] In the audio quality evaluation model, MeanNet in Mbnet can be selected as the basic network skeleton. Mbnet adopts a structure based on BLSTM, which is a deep learning model commonly used for sequence data modeling. It combines the advantages of long short-term memory (LSTM) and bidirectional recurrent neural network (BRNN) to perform bidirectional modeling and long-term dependency learning on sequence data. Specifically, the MeanNet structure in Mbnet is shown in Table 1 above.

[0180] In addition, Fast-VQA can also be used as the basis of the video quality assessment model. This method uses a sampling method called "fragmentation" as the input of a deep neural network to efficiently evaluate video quality and retain quality-related information. In order to improve accuracy, Fast-VQA also designs a fragmentation attention network based on the Transformer structure to better utilize the characteristics of "fragmentation".

[0181] Step 3: Send the audio stream and the video stream together to the audio and video content and time consistency detection network to obtain the corresponding content and time consistency scores.

[0182] In the examples of this application, see Figure 4 , the specific implementation of step 3 includes the following sub-steps:

[0183] Step 3.1: The input of the network model is a triplet, which contains two spectrograms and a video frame sequence, and one of the audio segments is consistent with the video frame content (time), and the other audio segment is inconsistent with the video frame content (time). in As an anchor sample, with f in The audio spectrogram a1 with the same content (time) is taken as a positive sample, and f in The audio spectrogram a2 with inconsistent content (time) is taken as a negative sample and recorded as (positive, anchor, negative).

[0184] Step 3.2: Send the triplet (positive, anchor, negative) to the corresponding audio and video feature extractor g v (·) and g a (•) and extract the corresponding deep feature g a (positive), g v (anchor), g a (negative), where g v(•) is a 128-dimensional feature vector. The video feature extractor uses a network skeleton based on TimeSformer. The network structure based on TimeSformer can better capture the temporal and spatial information in the video frame. The audio feature extractor uses a network skeleton based on ResNet-18.

[0185] Step 3.3, calculate g a (positive) and g v (anchor), g v (anchor) and g a The characteristic distance Dis(g v (anchor),g a (postive)) and Dis(g v (anchor),g a Here, the Euclidean distance is used as the feature distance between two feature vectors as an example, wherein the calculation formula of the Euclidean distance is shown in the above formula (1).

[0186] Step 3.4, use triplet loss T tri To constrain the network to maximize g a (positive) and g v (anchor), g v (anchor) and g a (negative) difference, triple loss T tri The definition is as shown in the above formula (2).

[0187] Step 4: Send the audio stream and video stream together into the spatial consistency detection network to obtain the corresponding spatial consistency score.

[0188] In the examples of this application, see Figure 6 , the specific implementation of step 4 includes the following sub-steps:

[0189] Step 4.1: Split the audio and video to be evaluated into video stream and audio stream, and extract the intermediate frames in the video stream as key frames.

[0190] In step 4.2, the key frame and the audio stream are fed into the visual position prediction module guided by the depth map to obtain the visual position.

[0191] In the examples of this application, see Fig. 9 , the specific implementation of step 4.2 includes the following sub-steps:

[0192] Step 4.2.1, for the video stream, first obtain the key frame of the video stream And input to the depth prediction model G Depth (.), the depth prediction model will output the depth map of the video frame based on the image information of the key frame Among them, the network structure of the depth prediction model is shown in the aforementioned Table 2.

[0193] Step 4.2.2, concatenate the depth map and keyframe as the visual position prediction model G vision One of the inputs is The expression is shown in the above formula (3). The spectrogram a corresponding to the audio stream in As the visual position prediction model G vision Another input of .

[0194] Step 4.2.3: The visual position prediction model calculates the heat map h of the audio and video position information based on the audio input and video input. out , the expression is shown in the above formula (4).

[0195] Here, the network structure of the visual position prediction model includes the visual feature extractor g v (·) and the audio feature extractor g a (·). Audio feature extractor g a (·) Extract global feature representation of audio, visual feature extractor g v (·) Calculate the local representation of each position (x, y), and the audio and video features are mapped into a shared latent space. Then, the extracted image features and audio features are multiplied to calculate the content similarity matrix Matrix. The content similarity matrix Matrix is ​​an N×N matrix, and each block represents the content correlation between the corresponding position of the image and the audio. For example, g v The network structure of (·) is shown in Table 3 above.

[0196] Step 4.2.4, convert the heat map h out The original video frame size is restored through the interpolation algorithm as the visual position of the image.

[0197] Step 4.3, sending the audio to the auditory position prediction module of the related art to obtain the auditory position.

[0198] In step 4.4, the distances between the visual position and the auditory position are compared to finally obtain a spatial consistency detection score.

[0199] In the embodiment of the present application, after obtaining the point set of the visual position and the horizontal angle of the auditory position, it is necessary to compare the visual and auditory positions to obtain the final spatial consistency score. The overall calculation steps are as follows:

[0200] First, the horizontal angle range of the visual position is calculated according to the point set of the visual position. After the horizontal angle range of the visual position is obtained, the closest horizontal angle distance between the auditory position and the visual position is calculated. The specific calculation formula is shown in the above formula (5).

[0201] Secondly, considering the error of position prediction, a threshold β can be set here. When the calculated closest horizontal angle distance between the auditory position and the visual position is less than β, it can be considered that the auditory position and the visual position are coincident, that is, the audio and video are spatially consistent. The specific calculation formula is shown in the above formula (6).

[0202] Step 5: The video quality score, audio quality score, and content, time, and space consistency scores obtained in steps 2, 3, and 4 are fed into the joint fitting model to finally obtain the audio and video joint quality score.

[0203] The embodiment of the present application provides a method for joint audio and video quality evaluation. Through the above embodiment, the specific implementation of the aforementioned embodiment is elaborated in detail. It can be seen that through the technical solution of the aforementioned embodiment, on the one hand, the time, space and content consistency of audio and video are taken into account when evaluating the joint quality of audio and video. It can more effectively improve the accuracy of joint audio and video quality evaluation; on the other hand, a self-supervised triple learning audio and video time and content consistency detection network is designed here, and this network model can more effectively detect the time consistency and content consistency between audio and video; on the other hand, it also involves an audio and video spatial consistency detection network based on audio-visual position comparison, which can more effectively detect the spatial consistency between audio and video, thereby improving the overall quality evaluation performance of audio and video.

[0204] In yet another embodiment of the present application, based on the same inventive concept as the above embodiment, see Fig.11 , which shows a schematic diagram of the structure of an audio and video joint quality evaluation device provided in an embodiment of the present application. Fig.11 As shown, the audio and video joint quality evaluation device 110 may include a determination unit 1101, an evaluation unit 1102 and a combination unit 1103, wherein:

[0205] The determining unit 1101 is configured to determine a first audio stream and a first video stream in the audio and video stream to be evaluated;

[0206] The evaluation unit 1102 is configured to input the first audio stream into the audio quality evaluation model, output the audio quality score through the audio quality evaluation model, and input the first video stream into the video quality evaluation model, output the video quality score through the video quality evaluation model;

[0207] The evaluation unit 1102 is further configured to input the first audio stream and the first video stream into a consistency evaluation model, and output a consistency score through the consistency evaluation model;

[0208] The combining unit 1103 is configured to determine a combined quality score of the audio and video streams to be evaluated according to the audio quality score, the video quality score and the consistency score.

[0209] In some embodiments, the evaluation unit 1102 is further configured to input the first audio stream and the first video stream into a consistency evaluation model, and output a consistency score through the consistency evaluation model, including at least one of the following:

[0210] Inputting the first audio stream and the first video stream into a first category evaluation sub-model, and outputting a first category consistency score through the first category evaluation sub-model;

[0211] Inputting the first audio stream and the first video stream into a second category evaluation sub-model, and outputting a second category consistency score through the second category evaluation sub-model;

[0212] The first category evaluation submodel is used to evaluate the temporal consistency and / or content consistency of the first audio stream and the first video stream, and the second category evaluation submodel is used to evaluate the spatial consistency of the first audio stream and the first video stream.

[0213] In some embodiments, the determination unit 1101 is further configured to, if the consistency model includes a first category evaluation sub-model, determine the first category consistency score output by the first category evaluation sub-model as the consistency score; if the consistency model includes a second category evaluation sub-model, determine the second category consistency score output by the second category evaluation sub-model as the consistency score; if the consistency model includes a first category evaluation sub-model and a second category evaluation sub-model, determine the first category consistency score output by the first category evaluation sub-model and the second category consistency score output by the second category evaluation sub-model together as the consistency score.

[0214] In some embodiments, see Fig.11 The audio and video joint quality evaluation device 110 may also include a training unit 1104, configured to determine at least one group of first samples; wherein each group of first samples includes a first audio sample, a second audio sample and a first video sample, and the first audio sample is consistent with the first category parameter of the first video sample, and the second audio sample is inconsistent with the first category parameter of the first video sample; and the first audio feature extractor and the first video feature extractor in the first preset model are trained according to at least one group of first samples, and when the loss function of the first preset model satisfies the first convergence condition, the trained first audio feature extractor and the trained first video feature extractor are determined as the first category evaluation sub-model.

[0215] In some embodiments, the evaluation unit 1102 is further configured to use the trained first audio feature extractor to perform feature extraction on the first audio stream to obtain a first audio feature; use the trained first video feature extractor to perform feature extraction on the first video stream to obtain a first video feature; and determine the feature distance between the first audio feature and the first video feature, and determine the first category consistency score based on the feature distance.

[0216] In some embodiments, the determination unit 1101 is further configured to determine the first category consistency score as a first value if the feature distance is within a first preset range; and to determine the first category consistency score as a second value if the feature distance is outside the first preset range.

[0217] In some embodiments, the second category evaluation sub-model includes a visual position prediction module, an auditory position prediction module and a calculation module; accordingly, the evaluation unit 1102 is further configured to perform visual position prediction on the first video stream and the first audio stream based on the visual position prediction module to determine the visual position; perform auditory position prediction on the first audio stream based on the auditory position prediction module to determine the auditory position; and perform audiovisual distance calculation on the visual position and the auditory position based on the calculation module to determine the second category consistency score.

[0218] In some embodiments, the determination unit 1101 is further configured to determine the key frames corresponding to the first video stream and the depth map corresponding to the key frames; perform a splicing operation on the key frames and the depth map to obtain a splicing map; determine the spectrogram corresponding to the first audio stream, input the spectrogram and the splicing map into the visual position prediction model, output a heat map through the visual position prediction model; and determine the visual position according to the heat map.

[0219] In some embodiments, the training unit 1104 is further configured to determine at least one group of second samples; wherein each group of second samples includes a first audio sample, a first video sample, and a second video sample, and the second category parameters of the first audio sample are consistent with those of the first video sample, and the second category parameters of the first audio sample are inconsistent with those of the second video sample; and the second audio feature extractor and the first image feature extractor in the second preset model are trained according to at least one group of second samples, and when the loss function of the second preset model meets the convergence condition, the trained second audio feature extractor and the trained first image feature extractor are determined as the visual position prediction model.

[0220] In some embodiments, the evaluation unit 1102 is further configured to use a trained first image feature extractor to perform feature extraction on the spliced ​​image to obtain a first image feature; use a trained second audio feature extractor to perform feature extraction on the spectrogram to obtain a second audio feature; and determine a content similarity matrix of the first image feature and the second audio feature, and determine a heat map based on the content similarity matrix.

[0221] In some embodiments, the determination unit 1101 is further configured to calculate the visual and auditory distance of the visual position and the auditory position based on the calculation module to determine a set of horizontal angle values; and determine the minimum horizontal angle value from the set of horizontal angle values, and determine the second category consistency score based on the minimum horizontal angle value.

[0222] In some embodiments, the combining unit 1103 is further configured to perform weighted calculation on the audio quality score, the video quality score and the consistency score to determine the combined quality score of the audio and video streams to be evaluated.

[0223] In some embodiments, the combining unit 1103 is further configured to input the audio quality score, the video quality score and the consistency score into a joint fitting model, and output the joint quality score of the audio and video streams to be evaluated through the joint fitting model.

[0224] In some embodiments, the jointly fitted model is a fully connected network model.

[0225] It can be understood that in this embodiment, a "unit" can be a part of a circuit, a part of a processor, a part of a program or software, etc., and of course it can also be a module, or it can be non-modular. Moreover, the components in this embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional module.

[0226] If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0227] Therefore, this embodiment provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by at least one processor, the audio and video joint quality assessment method described in any one of the aforementioned embodiments is implemented.

[0228] Based on the above-mentioned composition of the audio and video joint quality evaluation device 110 and the computer-readable storage medium, see Fig.12 , which shows a schematic diagram of the specific hardware structure of the electronic device provided in the embodiment of the present application. Fig.12 As shown, the electronic device 120 may include a communication interface 1201, a memory 1202, and a processor 1203; each component is coupled together via a bus system 1204. It is understood that the bus system 1204 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 1204 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Fig.12 Various buses are labeled as bus system 1204. Among them, the communication interface 1201 is used for receiving and sending signals in the process of sending and receiving information between other external network elements;

[0229] A memory 1202, used for storing a computer program that can be run on the processor 1203;

[0230] The processor 1203 is configured to execute, when running the computer program:

[0231] Determine a first audio stream and a first video stream in the audio and video streams to be evaluated; input the first audio stream into an audio quality evaluation model, output an audio quality score through the audio quality evaluation model, and input the first video stream into a video quality evaluation model, output a video quality score through the video quality evaluation model; input the first audio stream and the first video stream into a consistency evaluation model, output a consistency score through the consistency evaluation model; determine a joint quality score of the audio and video streams to be evaluated based on the audio quality score, the video quality score and the consistency score.

[0232] It can be understood that the memory 1202 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DRRAM). The memory 1202 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0233] The processor 1203 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 1203. The above-mentioned processor 1203 can be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor are combined and executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1202, and the processor 1203 reads the information in the memory 1202 and completes the steps of the above method in combination with its hardware.

[0234] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0235] For software implementation, the techniques described herein can be implemented by modules (e.g., procedures, functions, etc.) that perform the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0236] Optionally, as another embodiment, the processor 1203 is further configured to execute the audio and video joint quality assessment method described in any one of the aforementioned embodiments when running the computer program.

[0237] It should be noted that, in this application, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0238] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0239] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0240] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0241] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0242] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for joint audio and video quality assessment, characterized in that: The method comprises: Determine a first audio stream and a first video stream in the audio and video streams to be evaluated; Inputting the first audio stream into an audio quality assessment model, and outputting an audio quality score through the audio quality assessment model; and inputting the first video stream into a video quality assessment model, and outputting a video quality score through the video quality assessment model; Inputting the first audio stream and the first video stream into a consistency evaluation model, and outputting a consistency score through the consistency evaluation model; Determine a joint quality score of the audio and video streams to be evaluated according to the audio quality score, the video quality score and the consistency score.

2. The method according to claim 1, characterized in that The step of inputting the first audio stream and the first video stream into a consistency evaluation model and outputting a consistency score through the consistency evaluation model includes at least one of the following: Inputting the first audio stream and the first video stream into a first category evaluation sub-model, and outputting a first category consistency score through the first category evaluation sub-model; Inputting the first audio stream and the first video stream into a second category evaluation sub-model, and outputting a second category consistency score through the second category evaluation sub-model; Among them, the first category evaluation submodel is used to evaluate the temporal consistency and / or content consistency of the first audio stream and the first video stream, and the second category evaluation submodel is used to evaluate the spatial consistency of the first audio stream and the first video stream.

3. The method according to claim 2, characterized in that The method further comprises: If the consistency model includes the first category evaluation sub-model, determining the first category consistency score output by the first category evaluation sub-model as the consistency score; If the consistency model includes the second category evaluation sub-model, determining the second category consistency score output by the second category evaluation sub-model as the consistency score; If the consistency model includes the first category evaluation submodel and the second category evaluation submodel, the first category consistency score output by the first category evaluation submodel and the second category consistency score output by the second category evaluation submodel are jointly determined as the consistency score.

4. The method according to claim 2, characterized in that: The method further comprises: Determine at least one group of first samples; wherein each group of first samples includes a first audio sample, a second audio sample and a first video sample, and the first audio sample is consistent with a first category parameter of the first video sample, and the second audio sample is inconsistent with the first category parameter of the first video sample; The first audio feature extractor and the first video feature extractor in the first preset model are trained according to the at least one group of first samples, and when the loss function of the first preset model satisfies a first convergence condition, the trained first audio feature extractor and the trained first video feature extractor are determined as the first category evaluation sub-model.

5. The method according to claim 4, characterized in that The step of inputting the first audio stream and the first video stream into a first category evaluation sub-model and outputting a first category consistency score through the first category evaluation sub-model includes: Using the trained first audio feature extractor to perform feature extraction on the first audio stream to obtain a first audio feature; Using the trained first video feature extractor to extract features from the first video stream to obtain first video features; A feature distance between the first audio feature and the first video feature is determined, and the first category consistency score is determined based on the feature distance.

6. The method according to claim 5, characterized in that The determining the first category consistency score according to the feature distance includes: If the feature distance is within a first preset range, determining the first category consistency score to be a first value; If the feature distance is outside a first preset range, the first category consistency score is determined to be a second value.

7. The method according to claim 2, characterized in that The second category evaluation sub-model includes a visual position prediction module, an auditory position prediction module and a calculation module; The step of inputting the first audio stream and the first video stream into a second category evaluation sub-model, and outputting a second category consistency score through the second category evaluation sub-model includes: Performing visual position prediction on the first video stream and the first audio stream based on the visual position prediction module to determine a visual position; Performing auditory position prediction on the first audio stream based on the auditory position prediction module to determine an auditory position; The second category consistency score is determined by calculating the visual and auditory distances of the visual position and the auditory position based on the calculation module.

8. The method according to claim 7, characterized in that The performing visual position prediction on the first video stream and the first audio stream based on the visual position prediction module to determine the visual position includes: Determine a key frame corresponding to the first video stream and a depth map corresponding to the key frame; Performing a splicing operation on the key frame and the depth map to obtain a spliced ​​map; Determine a spectrogram corresponding to the first audio stream, input the spectrogram and the spliced ​​graph into a visual position prediction model, and output a heat map through the visual position prediction model; The visual position is determined according to the heat map.

9. The method according to claim 8, characterized in that The method further comprises: Determine at least one group of second samples; wherein each group of second samples includes a first audio sample, a first video sample, and a second video sample, and the first audio sample is consistent with the second category parameter of the first video sample, and the first audio sample is inconsistent with the second category parameter of the second video sample; The second audio feature extractor and the first image feature extractor in the second preset model are trained according to the at least one group of second samples, and when the loss function of the second preset model meets the convergence condition, the trained second audio feature extractor and the trained first image feature extractor are determined as the visual position prediction model.

10. The method according to claim 9, characterized in that The step of inputting the spectrogram and the spliced ​​graph into a visual position prediction model and outputting a heat map through the visual position prediction model comprises: Using the trained first image feature extractor to extract features from the spliced ​​image to obtain first image features; Using the trained second audio feature extractor to extract features from the spectrogram to obtain second audio features; Determine a content similarity matrix of the first image feature and the second audio feature, and determine the heat map based on the content similarity matrix.

11. The method according to claim 7, characterized in that The calculating the visual and auditory distances of the visual position and the auditory position based on the calculation module to determine the second category consistency score includes: Performing visual and auditory distance calculation on the visual position and the auditory position based on the calculation module to determine a set of horizontal angle values; A minimum horizontal angle value is determined from the set of horizontal angle values, and the second category consistency score is determined based on the minimum horizontal angle value.

12. The method according to any one of claims 1 to 11, characterized in that The determining, according to the audio quality score, the video quality score and the consistency score, a joint quality score of the audio and video stream to be evaluated includes: A weighted calculation is performed on the audio quality score, the video quality score, and the consistency score to determine a joint quality score of the audio and video stream to be evaluated.

13. The method according to any one of claims 1 to 11, characterized in that The determining, according to the audio quality score, the video quality score and the consistency score, a joint quality score of the audio and video stream to be evaluated includes: The audio quality score, the video quality score and the consistency score are input into a joint fitting model, and the joint quality score of the audio and video streams to be evaluated is output through the joint fitting model.

14. The method according to claim 13, characterized in that The joint fitting model is a fully connected network model.

15. An audio and video joint quality assessment device, characterized in that: It includes determination unit, evaluation unit and combination unit, among which: The determining unit is configured to determine a first audio stream and a first video stream in the audio and video stream to be evaluated; The evaluation unit is configured to input the first audio stream into an audio quality evaluation model and output an audio quality score through the audio quality evaluation model, and input the first video stream into a video quality evaluation model and output a video quality score through the video quality evaluation model; The evaluation unit is further configured to input the first audio stream and the first video stream into a consistency evaluation model, and output a consistency score through the consistency evaluation model; The combining unit is configured to determine a combined quality score of the audio and video streams to be evaluated according to the audio quality score, the video quality score and the consistency score.

16. An electronic device, characterized in that: comprising a memory and a processor, wherein: The memory is used to store a computer program that can be run on the processor; The processor is configured to execute the method according to any one of claims 1 to 14 when running the computer program.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by at least one processor, the method according to any one of claims 1 to 14 is implemented.

Images