Audio-video multitask learning, evaluation method, computer device and medium

Abstract

The embodiment of the application discloses an audio-video multi-task learning and evaluation method, a computer device and a storage medium. The method comprises the following steps: obtaining an audio-video training data set, wherein the audio-video training data set comprises video training data, audio training data and labeled data; performing feature extraction processing on the video training data to obtain a target video feature vector, and performing feature extraction processing on the audio training data to obtain a target audio feature vector; training a first deep learning neural network model by using the target audio feature vector, the target video feature vector and an audio-video quality label to obtain a first task loss function value, a second task loss function value and a third task loss function value; adjusting model parameters and retraining according to the first task loss function value, the second task loss function value and the third loss function value to obtain an audio-video multi-task evaluation model. In this way, the efficiency and accuracy of audio-video multi-task evaluation can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an audio and video multi-task learning and evaluation method, computer device and storage medium. Background Technology

[0002] Most machine learning tasks are currently single-task learning, which uses two separate models to learn the task. Single-task learning does not share the information learned across multiple tasks, which usually results in poor generalization performance of single-task learning models.

[0003] In multi-task audio and video evaluation, using a single-task model increases memory consumption and computational resources due to the single model's reliance on a single set of parameters. This also doubles the inference time, resulting in low efficiency in audio and video quality evaluation. Furthermore, audio and video quality are interrelated within a video segment; completely separating audio and video leads to inaccurate video quality evaluation. Therefore, improving the efficiency and accuracy of multi-task audio and video evaluation is crucial. Summary of the Invention

[0004] This application provides an audio and video multi-task learning and evaluation method, computer device, and storage medium, which can improve the efficiency and accuracy of audio and video multi-task evaluation.

[0005] In a first aspect, embodiments of this application provide an audio / video multi-task learning method, including:

[0006] Obtain an audio and video training dataset, which includes video training data, audio training data, and labeled data, wherein the labeled data includes audio and video quality labels;

[0007] The video training data is subjected to feature extraction processing to obtain a target video feature vector, and the audio training data is subjected to feature extraction processing to obtain a target audio feature vector;

[0008] The first deep learning neural network model is trained using the target audio feature vector, the target video feature vector, and the audio and video quality labels to obtain the target loss function value. The target loss function value includes a homoscedasticity parameter, which is associated with the multi-task weights.

[0009] The target loss function value is adjusted according to the homoscedasticity parameter, the model parameters are adjusted according to the target loss function value, and the first deep learning neural network model after adjusting the model parameters is retrained using the target audio feature vector, the target video feature vector, and the audio and video quality labels to obtain the audio and video multi-task evaluation model.

[0010] Secondly, embodiments of this application provide an audio / video multitasking evaluation method, including:

[0011] Acquire audio and video data to be evaluated, which includes audio data and video data to be evaluated;

[0012] The video data to be evaluated is subjected to feature extraction processing to obtain the video feature vector to be evaluated, and the audio data to be evaluated is subjected to feature extraction processing to obtain the audio feature vector to be evaluated.

[0013] The audio feature vector and the video feature vector to be evaluated are concatenated into a sequence, and the sequence is subjected to multimodal fusion processing to obtain the feature vector to be evaluated.

[0014] The feature vector to be evaluated is input into a pre-trained audio-visual multi-task evaluation model to obtain the evaluation data of the audio-visual data to be evaluated. The evaluation data includes comprehensive audio-visual quality level data, video quality data, and audio quality data.

[0015] Thirdly, embodiments of this application provide an audio-visual multi-task learning device, including:

[0016] The first acquisition unit is used to acquire an audio and video training dataset, which includes video training data, audio training data, and labeled data, wherein the labeled data includes audio and video quality labels.

[0017] The first extraction unit is used to perform feature extraction processing on the video training data to obtain a target video feature vector, and to perform feature extraction processing on the audio training data to obtain a target audio feature vector.

[0018] The first training unit is used to train the first deep learning neural network model using the target audio feature vector, the target video feature vector, and the audio and video quality labels to obtain the target loss function value. The target loss function value includes a homoscedasticity parameter, which is associated with the multi-task weights.

[0019] The second training unit is used to adjust the target loss function value according to the homoscedasticity parameter, adjust the model parameters according to the target loss function value, and retrain the first deep learning neural network model after adjusting the model parameters using the target audio feature vector, the target video feature vector, and the audio and video quality labels to obtain the audio and video multi-task evaluation model.

[0020] Fourthly, embodiments of this application provide an audio / video multitasking evaluation device, including:

[0021] The second acquisition unit is used to acquire audio and video data to be evaluated, wherein the audio and video data to be evaluated includes audio data to be evaluated and video data to be evaluated.

[0022] The second extraction unit is used to perform feature extraction processing on the video data to be evaluated to obtain the video feature vector to be evaluated, and to perform feature extraction processing on the audio data to be evaluated to obtain the audio feature vector to be evaluated.

[0023] The fusion unit is used to concatenate the audio feature vector to be evaluated and the video feature vector to be evaluated into a sequence, and perform multimodal fusion processing on the sequence to obtain the feature vector to be evaluated;

[0024] The evaluation unit is used to input the feature vector to be evaluated into a pre-trained audio-visual multi-task evaluation model to obtain evaluation data of the audio-visual data to be evaluated. The evaluation data includes comprehensive audio-visual quality level data, video quality data, and audio quality data.

[0025] Fifthly, embodiments of this application provide a computer device, the terminal comprising: a processor and a memory, the processor being configured to perform:

[0026] Obtain an audio and video training dataset, which includes video training data, audio training data, and labeled data, wherein the labeled data includes audio and video quality labels;

[0027] The video training data is subjected to feature extraction processing to obtain a target video feature vector, and the audio training data is subjected to feature extraction processing to obtain a target audio feature vector;

[0028] The first deep learning neural network model is trained using the target audio feature vector, the target video feature vector, and the audio and video quality labels to obtain the target loss function value. The target loss function value includes a homoscedasticity parameter, which is associated with the multi-task weights.

[0029] The target loss function value is adjusted according to the homoscedasticity parameter, the model parameters are adjusted according to the target loss function value, and the first deep learning neural network model after adjusting the model parameters is retrained using the target audio feature vector, the target video feature vector, and the audio and video quality labels to obtain the audio and video multi-task evaluation model.

[0030] Sixthly, embodiments of this application provide another computer device, the terminal comprising: a processor and a memory, the processor being configured to perform:

[0031] Acquire audio and video data to be evaluated, which includes audio data and video data to be evaluated;

[0032] The video data to be evaluated is subjected to feature extraction processing to obtain the video feature vector to be evaluated, and the audio data to be evaluated is subjected to feature extraction processing to obtain the audio feature vector to be evaluated.

[0033] The audio feature vector and the video feature vector to be evaluated are concatenated into a sequence, and the sequence is subjected to multimodal fusion processing to obtain the feature vector to be evaluated.

[0034] The feature vector to be evaluated is input into a pre-trained audio-visual multi-task evaluation model to obtain the evaluation data of the audio-visual data to be evaluated. The evaluation data includes comprehensive audio-visual quality level data, video quality data, and audio quality data.

[0035] In a seventh aspect, embodiments of this application also provide a computer-readable storage medium storing program instructions that, when executed, implement the methods described in the first or second aspect above.

[0036] This application embodiment can obtain an audio and video training dataset, which includes video training data, audio training data, and labeled data. The labeled data includes audio and video quality labels. Feature extraction processing is performed on the video training data to obtain a target video feature vector, and feature extraction processing is performed on the audio training data to obtain a target audio feature vector. The target audio feature vector, target video feature vector, and audio and video quality labels are used to train a first deep learning neural network model to obtain a target loss function value. The target loss function value includes a homoscedasticity parameter, which is related to the multi-task weights. The target loss function value is adjusted according to the homoscedasticity parameter, and the model parameters are adjusted according to the target loss function value. The first deep learning neural network model with adjusted model parameters is then retrained using the target audio feature vector, target video feature vector, and audio and video quality labels to obtain an audio and video multi-task evaluation model. This method can improve the efficiency and accuracy of audio and video multi-task learning. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart illustrating an audio-visual multi-task learning method provided in an embodiment of this application;

[0039] Figure 2This is a schematic diagram of a video image frame;

[0040] Figure 3 This is a flowchart illustrating another audio-visual multi-task learning method provided in an embodiment of this application;

[0041] Figure 4 This is a schematic diagram of a ResNet-B convolution;

[0042] Figure 5 This is a schematic diagram of a ResNet-C convolution;

[0043] Figure 6 This is a schematic diagram of a ResNet-D convolution;

[0044] Figure 7 This is a schematic diagram of an audio-visual multi-task evaluation model;

[0045] Figure 8 This is a flowchart illustrating another audio-visual multi-task learning method provided in the embodiments of this application;

[0046] Figure 9 This is a flowchart illustrating an audio / video multitasking evaluation method provided in an embodiment of this application;

[0047] Figure 10 This is a schematic diagram of the structure of an audio and video multi-task learning device provided in an embodiment of this application;

[0048] Figure 11 This is a schematic diagram of the structure of an audio / video multitasking evaluation device provided in an embodiment of this application;

[0049] Figure 12 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application;

[0050] Figure 13 This is a schematic diagram of the structure of another computer device provided in an embodiment of this application. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0052] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0053] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, large-scale image processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0054] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning / deep learning typically includes techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0055] Based on the machine learning and other technologies mentioned in the aforementioned artificial intelligence technologies, this application proposes an audio-visual multi-task evaluation scheme. By introducing multi-task learning of comprehensive audio-visual quality level data, audio quality level data, audio quality cause data, video quality level data, and video quality cause data, an audio-visual multi-task evaluation model is trained to increase model diversity. Evaluating audio-visual quality using this multi-task-learned audio-visual multi-task evaluation model helps improve the accuracy and efficiency of audio-visual multi-task evaluation. In some embodiments, the audio-visual data includes audio data and video data, wherein the audio data is the audio data of the video data.

[0056] The audio / video multitasking evaluation method provided in this application can be applied to an audio / video multitasking evaluation device, which can be installed in a computer device. In some embodiments, the computer device may include, but is not limited to, smart terminal devices such as smartphones, tablets, laptops, desktop computers, in-vehicle smart terminals, and smartwatches. In some embodiments, the computer device includes one or more databases, which can be used to store video data.

[0057] In some embodiments, the audio and video multi-task evaluation method provided in this application can be applied to scenarios such as user-generated content (UGC) and video filtering: for example, filtering high-quality videos based on evaluation data obtained from video quality assessment. Of course, the above application scenarios are merely illustrative. In other embodiments, the video quality assessment method of this application can be applied to any scenario associated with video quality assessment. In UGC scenarios, the audio and video multi-task evaluation method proposed in this application can be applied to the distribution of user-generated UGC video works. Using a model to evaluate the quality of user-generated UGC video works helps to filter out high-quality UGC videos, which has a crucial impact on subsequent business metrics such as click-through rate, playback duration, and completion rate in the distribution of these works.

[0058] The audio and video multitasking evaluation method provided in the embodiments of this application will be illustrated below with reference to the accompanying drawings.

[0059] Please see details. Figure 1 , Figure 1 This is a flowchart illustrating an audio-visual multi-task learning method provided in an embodiment of this application. The audio-visual multi-task learning method of this embodiment can be executed by an audio-visual multi-task learning device, wherein the audio-visual multi-task evaluation device is located in a terminal or computer device, as explained above. Specifically, the method of this embodiment describes the training process of the audio-visual multi-task evaluation model, specifically including the following steps.

[0060] S101: Obtain the audio and video training dataset, which includes video training data, audio training data, and labeled data, including audio and video quality labels.

[0061] In this embodiment, the computer device can acquire an audio and video training dataset, which includes video training data, audio training data, and labeled data, including audio and video quality labels. In some embodiments, the audio and video quality labels include audio quality labels, video quality labels, and a comprehensive audio and video quality level label; the audio quality labels include audio quality level labels and audio quality cause labels; and the video quality labels include video quality level labels and video quality cause labels.

[0062] S102: Perform feature extraction processing on the video training data to obtain the target video feature vector, and perform feature extraction processing on the audio training data to obtain the target audio feature vector.

[0063] In this embodiment of the application, the computer device can perform feature extraction processing on video training data to obtain a target video feature vector, and perform feature extraction processing on audio training data to obtain a target audio feature vector.

[0064] In one embodiment, when a computer device performs feature extraction processing on audio training data to obtain a target audio feature vector, it can convert the audio training data into a Mel-spectrum feature vector; segment the Mel-spectrum feature vector to obtain multiple audio feature vectors; and fuse the multiple audio feature vectors to obtain the target audio feature vector.

[0065] In one embodiment, when a computer device fuses multiple audio feature vectors to obtain a target audio feature vector, it can input the multiple audio feature vectors into the embedding module of a second deep learning neural network model to obtain multiple embedded feature vectors; and input the multiple embedded feature vectors into the fully connected module of the second deep learning neural network model to obtain the target audio feature vector.

[0066] In one embodiment, when a computer device performs feature extraction processing on video training data to obtain a target video feature vector, it may acquire at least one image frame of the video training data; perform feature extraction on the at least one image frame to obtain at least one feature vector; and perform fusion processing on the at least one feature vector to obtain the target video feature vector. In some embodiments, the at least one image frame may include some or all image frames in the video training data, wherein a portion of the image frames may include one or more image frames.

[0067] In one embodiment, when acquiring at least one image frame of video training data, the computer device may acquire at least one video segment included in the video training data; and perform frame extraction processing on the at least one video segment to obtain at least one image frame. In some embodiments, the at least one video segment may include multiple video segments in the video training data, and the at least one image frame may include multiple image frames composed of one image frame extracted from each video segment.

[0068] Furthermore, the computer device can split the video training data into multiple video segments and extract one image frame from each video segment to obtain multiple image frames. For example, the computer device can split the video training data into N video segments, where N is a positive integer, and extract one image frame from each of the N video segments to obtain N image frames.

[0069] In some embodiments, when extracting an image frame from each video segment, the computer device may randomly extract an image frame from each video segment, or it may select a specified intermediate image frame from each video segment as that image frame. Specifically, as follows... Figure 2 As shown, Figure 2 This is a schematic diagram of a video image frame.

[0070] In this embodiment of the application, by performing frame extraction processing on each video frequency band, information redundancy can be avoided, while making better use of video information.

[0071] In one embodiment, when a computer device extracts features from at least one image frame to obtain at least one video feature vector, it can input at least one image frame into the convolution module of an improved convolutional neural network model to obtain a convolution result; and perform max pooling on the convolution result to obtain at least one video feature vector.

[0072] In one embodiment, when a computer device performs fusion processing on at least one feature vector to obtain a target video feature vector, it can input at least one video feature vector into the average pooling module of an improved convolutional neural network model to obtain the mean of at least one video feature vector; and input the mean of at least one video feature vector into the fully connected module of the improved convolutional neural network model to obtain the target video feature vector.

[0073] S103: Train the first deep learning neural network model using the target audio feature vector, the target video feature vector, and the audio and video quality labels to obtain the target loss function value.

[0074] In this embodiment of the application, the computer device can train the first deep learning neural network model using the target audio feature vector, the target video feature vector, and the audio and video quality labels to obtain the target loss function value.

[0075] In one embodiment, when a computer device trains a first deep learning neural network model using a target audio feature vector, a target video feature vector, and audio / video quality labels to obtain a target loss function value, it can fuse the target audio feature vector and the target video feature vector to obtain a target feature vector; and input the target audio feature vector and audio quality labels, the target video feature vector and video quality labels, and the target feature vector and comprehensive audio / video quality level labels into the first deep learning neural network model for training to obtain the target loss function value.

[0076] In one embodiment, when a computer device trains a first deep learning neural network model by inputting the target audio feature vector and audio quality label, the target video feature vector and video quality label, and the target feature vector and comprehensive audio-visual quality level label to obtain the target loss function value, it can further train the first deep learning neural network model using the target audio feature vector and audio quality label to obtain the first task loss function value; train the first deep learning neural network model using the target video feature vector and video quality label to obtain the second task loss function value; train the first deep learning neural network model using the target feature vector and comprehensive audio-visual quality level label to obtain the third task loss function value; and determine the target loss function value based on the first task loss function value, the second task loss function value, and the third task loss function value.

[0077] In one embodiment, the audio quality label includes an audio quality level label and an audio quality cause label. When the computer device trains the first deep learning neural network model using the target audio feature vector and the audio quality label to obtain the first task loss function value, it can input the target audio feature vector, the audio quality level label, and the audio quality cause label into the first deep learning neural network model for training to obtain the first loss function value and the second loss function value. The first loss function value and the second loss function value are then weighted and summed according to a first preset weight to obtain the first task loss function value.

[0078] In one embodiment, the video quality label includes a video quality level label and a video quality cause label. When the computer device trains the first deep learning neural network model using the target video feature vector and the video quality label to obtain the second task loss function value, it can input the target video feature vector, the video quality level label, and the video quality cause label into the first deep learning neural network model for training to obtain a third loss function value and a fourth loss function value. The third loss function value and the fourth loss function value are then weighted and summed according to a second preset weight to obtain the second task loss function value.

[0079] S104: Adjust the target loss function value according to the homoscedasticity parameter, adjust the model parameters according to the target loss function value, and retrain the first deep learning neural network model after adjusting the model parameters using the target audio feature vector, the target video feature vector, and the audio and video quality labels to obtain the audio and video multi-task evaluation model.

[0080] In this embodiment, the computer device can adjust the target loss function value according to the homoscedasticity parameter, adjust the model parameters according to the target loss function value, and retrain the first deep learning neural network model after adjusting the model parameters using the target audio feature vector, the target video feature vector, and the audio and video quality labels to obtain the audio and video multi-task evaluation model.

[0081] In one embodiment, when adjusting model parameters based on the first task loss function value, the second task loss function value, and the third loss function value, the computer device can perform a weighted summation of the first task loss function value, the second task loss function value, and the third loss function value according to a third preset weight to obtain a target loss function value; when the target loss function value is greater than a function threshold, the model parameters are adjusted based on the target loss function value.

[0082] In one embodiment, the computer device can retrain a first deep learning neural network model with adjusted model parameters using the target audio feature vector, the target video feature vector, and the audio / video quality labels; when the target loss function value obtained from the retraining is less than the function threshold, the audio / video multi-task evaluation model is determined.

[0083] This application embodiment includes two multi-tasks: audio quality level data and audio quality cause data, and video quality level data and video quality cause data. These are learned through two multi-task models: one for audio quality cause and the other for video quality cause and the other for video quality cause and the other for video quality level. Because audio and video quality levels and causes are correlated, learning these two tasks together can mutually promote each other, share information, and complement each other to improve performance. Sharing a single model parameter for the audio and video quality cause and quality level tasks reduces memory usage, and since both tasks only require one forward computation in actual use, computational resources are reduced, and inference speed is increased. Furthermore, by learning adaptive parameters, the loss fusion of the audio and video quality cause and quality level tasks can be continuously adjusted during learning. This approach improves the learning performance of both tasks and saves the tedious process of manually adjusting fusion parameters.

[0084] This application embodiment uses audio and video training data and labeled data to train an audio and video multi-task evaluation model, which increases the model's performance and diversity, reduces memory usage and resource consumption, and helps improve the efficiency and accuracy of audio and video multi-task evaluation.

[0085] Please see Figure 3 , Figure 3 This is a flowchart illustrating another audio-visual multi-task learning method provided in this application embodiment. The audio-visual multi-task learning method of this application embodiment can be executed by an audio-visual multi-task learning device, which is located in a terminal or computer device, as explained above. Specifically, this application embodiment mainly describes the feature extraction process of audio and video, specifically including the following steps.

[0086] S301: Obtain the audio and video training dataset, which includes video training data, audio training data, and labeled data. The labeled data includes audio and video quality labels.

[0087] In this embodiment of the application, the computer device can acquire an audio and video training dataset, which includes video training data, audio training data, and labeled data, and the labeled data includes audio and video quality labels.

[0088] S302: Perform feature extraction processing on the video training data to obtain the target video feature vector, and convert the audio training data into Mel spectrum feature vectors, and perform segmentation processing on the Mel spectrum feature vectors to obtain multiple audio feature vectors.

[0089] In this embodiment of the application, the computer device can perform feature extraction processing on video training data to obtain target video feature vectors, and convert audio training data into Mel spectrum feature vectors, and perform segmentation processing on the Mel spectrum feature vectors to obtain multiple audio feature vectors.

[0090] In one embodiment, when a computer device converts audio training data into Mel spectrum feature vectors and segments the Mel spectrum feature vectors to obtain multiple audio feature vectors, it can convert the audio training data into a 128-dimensional Mel filter feature sequence, i.e., Mel spectrum feature vectors, and segment the Mel spectrum into N 16-dimensional × 16-dimensional audio feature vectors.

[0091] In one embodiment, when a computer device performs feature extraction processing on video training data to obtain a target video feature vector, it may acquire at least one image frame of the video training data, perform feature extraction on the at least one image frame to obtain at least one feature vector, and perform fusion processing on the at least one feature vector to obtain the target video feature vector.

[0092] In one embodiment, when a computer device extracts features from at least one image frame to obtain at least one feature vector, it can input at least one image frame into the convolution module of an improved convolutional neural network model to obtain a convolution result; and perform max pooling on the convolution result to obtain at least one video feature vector.

[0093] In some embodiments, the improved convolutional neural network model can be a ResNet50_vd model, where ResNet50_vd can refer to a ResNet-D network with 50 convolutional layers. The ResNet-D network is an improvement on the ResNet series of networks after its initial proposal through three versions: ResNet-B, ResNet-C, and ResNet-D. Figure 4 As shown, Figure 4 This is a schematic diagram of a convolution in ResNet-B. ResNet-B changes the stride of the 1*1 convolution in Path A from 2 to 1. This improvement helps to avoid information loss. Figure 5 As shown, Figure 5 This is a schematic diagram of convolution in ResNet-C. ResNet-C adjusts the first 7x7 convolution kernel into three 3x3 convolution kernels. This improvement helps reduce computation while increasing the network's non-linearity. Figure 6 As shown, Figure 6 This is a schematic diagram of a ResNet-D convolution. ResNet-D further changes the stride of the 1*1 convolution in Path B from 2 to 1 and adds an average pooling layer (i.e., the average pooling module). This improvement helps to retain more information and improves the performance of the model.

[0094] In one embodiment, when a computer device performs fusion processing on at least one video feature vector to obtain a target video feature vector, it can input the at least one video feature vector into the average pooling module of an improved convolutional neural network model to obtain the mean of at least one video feature vector; and input the mean of at least one video feature vector into the fully connected module of the improved convolutional neural network model to obtain the target video feature vector.

[0095] In one embodiment, when the computer device performs fusion processing on at least one video feature vector to obtain a target feature vector, it can also input the at least one video feature vector into the fully connected module of an improved convolutional neural network model to obtain multiple weights, and input the multiple weights and at least one video feature vector into the average pooling module of the improved convolutional neural network model to obtain the target feature vector.

[0096] This application helps to reduce video interference information and obtain higher-precision video feature vectors by fusing at least one video feature vector.

[0097] S303: Fuse multiple audio feature vectors to obtain the target audio feature vector.

[0098] In this embodiment of the application, the computer device can fuse multiple audio feature vectors to obtain a target audio feature vector.

[0099] In one embodiment, when a computer device fuses multiple audio feature vectors to obtain a target audio feature vector, it can input the multiple audio feature vectors into the embedding module of a second deep learning neural network model to obtain multiple embedded feature vectors; and input the multiple embedded feature vectors into the fully connected module of the second deep learning neural network model to obtain the target audio feature vector.

[0100] Computer devices can use a linear projection layer to flatten each 16×16-dimensional audio feature vector into a 768-dimensional one-dimensional embedded feature vector. This linear projection layer is called an embedding layer. A trainable positional embedding (also 768-dimensional) is added to each embedded feature vector to allow the model to capture the spatial structure of the two-dimensional audio spectrogram. The resulting multiple embedded feature vectors are then input into the fully connected module of a second deep learning neural network model to obtain the target audio feature vector.

[0101] This application helps to reduce audio interference information and obtain higher-precision audio feature vectors by fusing multiple audio feature vectors.

[0102] S304: Train the first deep learning neural network model using the target audio feature vector, the target video feature vector, and the audio and video quality labels to obtain the target loss function value, which includes a homoscedasticity parameter that is associated with the multi-task weights.

[0103] In this embodiment of the application, the computer device can train a first deep learning neural network model using the target audio feature vector, the target video feature vector, and the audio and video quality labels to obtain a target loss function value. The target loss function value includes a homoscedasticity parameter, which is associated with the multi-task weights.

[0104] S305: Adjust the target loss function value based on the homoscedasticity parameter, adjust the model parameters based on the target loss function value, and retrain the first deep learning neural network model after adjusting the model parameters using the target audio feature vector, target video feature vector, and audio and video quality labels to obtain the audio and video multi-task evaluation model.

[0105] In this embodiment, the computer device can adjust the target loss function value according to the homoscedasticity parameter, adjust the model parameters according to the target loss function value, and retrain the first deep learning neural network model after adjusting the model parameters using the target audio feature vector, the target video feature vector, and the audio and video quality labels to obtain the audio and video multi-task evaluation model.

[0106] In one embodiment, the training of the audio / video multi-task evaluation model is specifically as follows: Figure 7 As shown, Figure 7 This is a schematic diagram of an audio / video multi-task evaluation model, such as... Figure 7 As shown, the video training data 71 is first subjected to frame extraction processing 72 to obtain multiple image frames. These multiple image frames are then input into the convolutional module 73 of the improved convolutional neural network model to obtain convolutional results. The convolutional results are then subjected to max pooling processing 74 to obtain at least one feature vector. This at least one feature vector is then input into convolutional layer 4 through convolutional layers 1, 2, and 3 of the improved convolutional neural network model. Convolutional layer 4 includes an average pooling module and a fully connected module 75. The average pooling module obtains at least one mean value for the video feature vector. This mean value is then input into the fully connected module of the improved convolutional neural network model to obtain the target video feature vector. Finally, the target video feature vector and the video quality level label and video quality reason label from the labeled data are input into a third deep learning neural network model 76 to output video quality level data and video quality reason data 77. Next, the audio training data 78 is converted into Mel-spectrum feature vectors 79. These vectors are then segmented to obtain multiple audio feature vectors, which are then fused to obtain the target audio feature vector 710. The target audio feature vector, audio quality level label, and audio quality cause label are input into the second deep learning neural network model 711 to obtain audio quality level data and audio quality cause data 712. Further, the target video feature vector, target audio feature vector, and comprehensive audio-visual quality level label are input into the first deep learning neural network model 713 to obtain comprehensive audio-visual quality level data 714. Furthermore, a target loss function value can be determined based on the video quality level data, video quality cause data, audio quality level data, and comprehensive audio-visual quality level data. The model parameters are then adjusted and the model is retrained based on the target loss function value to obtain the audio-visual multi-task evaluation model.

[0107] This application embodiment acquires audio and video training data, performs feature extraction on the video training data to obtain a target video feature vector, and converts the audio training data into Mel-spectrum feature vectors. These Mel-spectrum feature vectors are then segmented to obtain multiple audio feature vectors. Finally, these multiple audio feature vectors are fused to obtain a target audio feature vector. This process helps reduce video and audio interference, resulting in more accurate target video and audio feature vectors. Furthermore, the target video and audio feature vectors, along with labeled data, are used to train an audio-visual multi-task evaluation model. This increases the model's performance and diversity, reduces memory usage and resource consumption, and improves the efficiency and accuracy of audio-visual multi-task evaluation.

[0108] Please see Figure 8 , Figure 8 This is a flowchart illustrating another audio-visual multi-task learning method provided in this application embodiment. The audio-visual multi-task learning method of this application embodiment can be executed by an audio-visual multi-task learning device, wherein the audio-visual multi-task learning device is located in a terminal or computer device, and the specific explanation of the terminal or computer device is as above. Specifically, this application embodiment mainly describes the training process of the audio-visual multi-task evaluation model, which specifically includes the following steps.

[0109] S801: Obtain the audio and video training dataset, which includes video training data, audio training data, and labeled data. The labeled data includes audio and video quality labels.

[0110] In this embodiment of the application, the computer device can acquire an audio and video training dataset, which includes video training data, audio training data, and labeled data, and the labeled data includes audio and video quality labels.

[0111] S802: Perform feature extraction processing on the video training data to obtain the target video feature vector, and perform feature extraction processing on the audio training data to obtain the target audio feature vector.

[0112] In this embodiment of the application, the computer device can perform feature extraction processing on video training data to obtain a target video feature vector, and perform feature extraction processing on audio training data to obtain a target audio feature vector.

[0113] S803: The target video feature vector and the target audio feature vector are fused to obtain the target feature vector.

[0114] In this embodiment of the application, the computer device can perform fusion processing on the target video feature vector and the target audio feature vector to obtain the target feature vector.

[0115] In one embodiment, after the computer device has learned and output the target video feature vector corresponding to the video content through the video content multi-task model and the target audio feature vector corresponding to the audio content through the audio multi-task model, it can input the two features, the target video feature vector and the target audio feature vector, into the multimodal transformer module of the first deep learning neural network model for fusion. After fusion, the audio content and video content can be used to jointly evaluate the audio and video quality level.

[0116] S804: Input the target feature vector, target audio feature vector, target video feature vector, and audio / video quality labels into the first deep learning neural network model for training to obtain the audio / video multi-task evaluation model.

[0117] In this embodiment of the application, the computer device can input the target feature vector, the target audio feature vector, the target video feature vector, and the audio and video quality labels into the first deep learning neural network model for training, thereby obtaining an audio and video multi-task evaluation model.

[0118] In one embodiment, the computer device can input the target feature vector, target audio feature vector, target video feature vector, and audio / video quality label into a first deep learning neural network model for training to obtain a target loss function value; adjust the model parameters based on the target loss function value, and input the target feature vector, target audio feature vector, target video feature vector, and audio / video quality label into the deep learning neural network model after adjusting the model parameters for training; when the target loss function value obtained after retraining is less than the function threshold, the audio / video multi-task evaluation model is determined to be obtained.

[0119] In one embodiment, when the computer device trains a first deep learning neural network model by inputting the target feature vector, target audio feature vector, target video feature vector, and audio / video quality labels to obtain the target loss function value, it can input the target audio feature vector, audio quality level label, and audio quality cause label into the first deep learning neural network model to obtain the first loss function value and the second loss function value; input the target video feature vector, video quality level label, and video quality cause label into the first deep learning neural network model to obtain the third loss function value and the fourth loss function value; and input the target feature vector and the comprehensive audio / video quality level label into the first deep learning neural network model to obtain the third task loss function value; further, the target loss function value is determined based on the first loss function value, the second loss function value, the third loss function value, the fourth loss function value, and the third task loss function value.

[0120] In one embodiment, when a computer device inputs a target audio feature vector, an audio quality level label, and an audio quality cause label into a first deep learning neural network model for training to obtain a first loss function value and a second loss function value, it can input the target audio feature vector, the audio quality level label, and the audio quality cause label into the first deep learning neural network model to evaluate and obtain audio quality level data and audio quality cause data; determine the first loss function value based on the audio quality level data and the audio quality level label; and determine the second loss function value based on the audio quality cause data and the audio quality cause label.

[0121] In one embodiment, when a computer device inputs a target video feature vector, a video quality level label, and a video quality cause label into a first deep learning neural network model for training to obtain a third loss function value and a fourth loss function value, it can input the target video feature vector, the video quality level label, and the video quality cause label into the first deep learning neural network model to evaluate and obtain video quality level data and video quality cause data; determine the third loss function value based on the video quality level data and the video quality level label; and determine the fourth loss function value based on the video quality cause data and the video quality cause label.

[0122] In one embodiment, when a computer device inputs the target feature vector and the comprehensive audio / video quality level label into a first deep learning neural network model for training to obtain a third task loss function value, it can input the target feature vector and the comprehensive audio / video quality level label into the first deep learning neural network model to evaluate the comprehensive audio / video quality level data, and determine the third task loss function value based on the comprehensive audio / video quality level data and the comprehensive audio / video quality level label.

[0123] In one embodiment, when determining the target loss function value based on the first loss function value, the second loss function value, the third loss function value, the fourth loss function value, and the third task loss function value, the computer device can perform a weighted summation of the first loss function value, the second loss function value, the third loss function value, the fourth loss function value, and the third task loss function value to obtain the total loss function. The specific formula for the weighted summation is shown in formula (1) below.

[0124]

[0125] Where L is the target loss function, L i Used to indicate the loss function, ω i denoted by weight, and i represents a coefficient, including 1, 2, 3, 4, and 5.

[0126] Furthermore, the computer device can adjust formula (1) according to the different learning stages, the difficulty of learning, and even the learning effect, to obtain the method of dynamically determining the loss function by introducing time, as shown in formula (2), to determine the target loss function.

[0127]

[0128] Where L is the target loss function, L i Used to indicate the loss function, ω i Let i be the weight, i be the coefficient (including 1, 2, 3, 4, 5), and t be the time.

[0129] Furthermore, in model learning, tasks typically involve two types of uncertainty: cognitive uncertainty and random uncertainty. Cognitive uncertainty can be mitigated by adding more data, while random uncertainty requires standardized data processing. Random uncertainty can be further categorized into two types: 1. Data-dependent uncertainty (heteroscedastic uncertainty), which depends on the input data, meaning the variance of the residuals in the model's predictions changes with the input data; 2. Task-dependent uncertainty (homoscedastic uncertainty), which is arbitrary uncertainty independent of the input data. It is unrelated to the model output and is a quantity that varies across different tasks while keeping all input data constant. Therefore, it can be described as task-related uncertainty. Homoscedastic uncertainty can be caused by task-related weights. Here, we assume the model conforms to the assumption of homoscedastic uncertainty.

[0130] Therefore, the uncertainty for regression tasks is defined as shown in the following formula (3):

[0131]

[0132] For classification tasks, the uncertainty is defined as shown in the following formula (4):

[0133] p(y|f W (x))=Softmax(f W (x)) (4)

[0134] Under the assumption of homoscedastic uncertainty, the minimization objective function L(W, σ1, σ2) of the multi-output model is obtained as shown in the following formula (5):

[0135]

[0136] Where y indicates the output data, x indicates the input data, w indicates the parameter matrix of the model, and σ indicates the variance. Used to indicate the first loss function, Used to indicate the second loss function.

[0137] σ is used to measure homoscedasticity uncertainty. Homoscedasticity uncertainty is task-related. The higher the homoscedasticity uncertainty of a task, the more noise there is in the output related to the task, and the more difficult the task is to learn. Therefore, during the training of a multi-task model, the corresponding σ will increase, weakening the weight of such tasks and making the training of the overall multi-task model smoother and more effective.

[0138] In the two multi-task evaluations of audio / video quality levels and causes, the primary task for improvement is the audio / video quality level label, whose homoscedastic uncertainty is lower than that of the audio / video quality cause label prediction task. Therefore, this method is well-suited for the audio / video multi-task evaluation scenario. Combining the two multi-tasks during training will further improve the prediction results of the audio / video quality level label. Furthermore, utilizing automated dynamic loss function weights eliminates the need for manual parameter tuning, saving significant time and model training resources spent searching for optimal parameters.

[0139] This application embodiment fuses the target video feature vector and the target audio feature vector to obtain the target feature vector. The target audio feature vector, audio quality level label, audio quality cause label, target video feature vector, video quality level label, video quality cause label, and comprehensive audio-video quality level label are input into a first deep learning neural network model for training to obtain an audio-video multi-task evaluation model. By combining the audio-video quality level, audio-video quality cause, and comprehensive audio-video quality level data into the training model, it helps to improve the performance and diversity of the model, and improves the efficiency and accuracy of audio-video multi-task evaluation.

[0140] Please see Figure 9 , Figure 9 This is a flowchart illustrating an audio / video multitasking evaluation method provided in this application embodiment. The audio / video multitasking evaluation method of this application embodiment can be executed by an audio / video multitasking evaluation device, which is located in a terminal or computer device, as explained above. Specifically, this application embodiment mainly describes the evaluation process of the audio / video multitasking evaluation model, specifically including the following steps.

[0141] S901: Obtain the audio and video data to be evaluated, which includes the audio data and video data to be evaluated.

[0142] In this embodiment of the application, the computer device can acquire audio and video data to be evaluated, which includes audio data to be evaluated and video data to be evaluated.

[0143] S902: Perform feature extraction processing on the video data to be evaluated to obtain the feature vector of the video to be evaluated, and perform feature extraction processing on the audio data to be evaluated to obtain the feature vector of the audio to be evaluated.

[0144] In this embodiment of the application, the computer device can perform feature extraction processing on the video data to be evaluated to obtain the video feature vector to be evaluated, and perform feature extraction processing on the audio data to be evaluated to obtain the audio feature vector to be evaluated.

[0145] S903: Concatenate the audio feature vector and the video feature vector to be evaluated into a sequence, and perform multimodal fusion processing on the sequence to obtain the feature vector to be evaluated.

[0146] In this embodiment of the application, the computer device can concatenate the audio feature vector to be evaluated and the video feature vector to be evaluated into a sequence, and perform multimodal fusion processing on the sequence to obtain the feature vector to be evaluated.

[0147] In one embodiment, the computer device can stitch together the audio data and video data to be evaluated to obtain multimodal data; and extract features from the multimodal data to obtain the feature vector to be evaluated.

[0148] S904: Input the feature vector to be evaluated into the pre-trained audio and video multi-task evaluation model to obtain the evaluation data of the audio and video data to be evaluated. The evaluation data includes comprehensive data of audio and video quality levels, video quality data and audio quality data.

[0149] In this embodiment of the application, the computer device can input the feature vector to be evaluated into a pre-trained audio and video multi-task evaluation model to obtain evaluation data of the audio and video data to be evaluated. The evaluation data includes comprehensive data of audio and video quality levels, video quality data and audio quality data.

[0150] In one embodiment, a computer device can input the feature vector to be evaluated into a pre-trained audio-visual multi-task evaluation model to obtain comprehensive data on audio-visual quality levels.

[0151] In one embodiment, a computer device can input the feature vector of the video to be evaluated into a pre-trained audio-visual multi-task evaluation model to obtain video quality data of the video data to be evaluated, which includes video quality level data and video quality cause data.

[0152] In one embodiment, a computer device can input the audio feature vector to be evaluated into a pre-trained audio-video multi-task evaluation model to obtain audio quality data of the audio data to be evaluated, which includes audio quality level data and audio quality cause data.

[0153] This application embodiment can perform feature extraction processing on the video data to be evaluated to obtain a feature vector of the video to be evaluated, perform feature extraction processing on the audio data to be evaluated to obtain a feature vector of the audio to be evaluated, perform feature extraction processing on the audio data to be evaluated to obtain a feature vector of the audio to be evaluated, concatenate the audio feature vector and the video feature vector of the audio to be evaluated into a sequence, perform multimodal fusion processing on the sequence to obtain a feature vector of the audio to be evaluated, and input the feature vector of the audio to be evaluated into a pre-trained audio and video multi-task evaluation model to obtain comprehensive audio and video quality level data, video quality data and audio quality data of the audio and video data to be evaluated. By using the audio and video multi-task evaluation model, multi-task evaluation of audio and video is completed, improving the efficiency and accuracy of audio and video multi-task evaluation.

[0154] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of an audio and video multi-task learning device provided in an embodiment of this application. Specifically, the audio and video multi-task learning device is disposed in a computer device, and the device includes: a first acquisition unit 1001, a first extraction unit 1002, a first training unit 1003, and a second training unit 1004;

[0155] The first acquisition unit 1001 is used to acquire an audio and video training dataset, which includes video training data, audio training data, and labeled data, and the labeled data includes audio and video quality labels.

[0156] The first extraction unit 1002 is used to perform feature extraction processing on the video training data to obtain a target video feature vector, and to perform feature extraction processing on the audio training data to obtain a target audio feature vector.

[0157] The first training unit 1003 is used to train the first deep learning neural network model using the target audio feature vector, the target video feature vector, and the audio and video quality labels to obtain the target loss function value. The target loss function value includes a homoscedasticity parameter, which is associated with the multi-task weights.

[0158] The second training unit 1004 is used to adjust the target loss function value according to the homoscedasticity parameter, adjust the model parameters according to the target loss function value, and retrain the first deep learning neural network model after adjusting the model parameters using the target audio feature vector, the target video feature vector, and the audio and video quality labels to obtain the audio and video multi-task evaluation model.

[0159] Further, the audio and video quality labels include audio quality labels, video quality labels, and a comprehensive audio and video quality level label; when the first training unit 1003 trains the first deep learning neural network model using the target audio feature vector, the target video feature vector, and the audio and video quality labels to obtain the target loss function value, it is specifically used for:

[0160] The target audio feature vector and the target video feature vector are fused to obtain the target feature vector;

[0161] The target audio feature vector and audio quality label, the target video feature vector and video quality label, and the target feature vector and audio / video quality level integrated label are input into the first deep learning neural network model for training to obtain the target loss function value.

[0162] Further, when the first training unit 1003 inputs the target audio feature vector and audio quality label, the target video feature vector and video quality label, and the target feature vector and comprehensive audio-video quality level label into the first deep learning neural network model for training, and obtains the target loss function value, it is specifically used for:

[0163] The first deep learning neural network model is trained using the target audio feature vector and audio quality label to obtain the first task loss function value;

[0164] The first deep learning neural network model is trained using the target video feature vector and video quality label to obtain the second task loss function value;

[0165] The first deep learning neural network model is trained using the target feature vector and the comprehensive audio / video quality level label to obtain the third task loss function value;

[0166] The target loss function value is determined based on the first task loss function value, the second task loss function value, and the third task loss function value.

[0167] Further, the audio quality label includes an audio quality level label and an audio quality cause label; when the first training unit 1003 trains the first deep learning neural network model using the target audio feature vector and the audio quality label to obtain the first task loss function value, it is specifically used for:

[0168] The target audio feature vector, audio quality level label, and audio quality reason label are input into the first deep learning neural network model for training to obtain the first loss function value and the second loss function value.

[0169] The first loss function value and the second loss function value are weighted and summed according to the first preset weight to obtain the first task loss function value.

[0170] Further, the video quality label includes a video quality level label and a video quality cause label; when the first training unit 1003 trains the first deep learning neural network model using the target video feature vector and the video quality label to obtain the second task loss function value, it is specifically used for:

[0171] The target video feature vector, video quality level label, and video quality reason label are input into the first deep learning neural network model for training to obtain the third loss function value and the fourth loss function value.

[0172] The third and fourth loss function values ​​are weighted and summed according to the second preset weights to obtain the second task loss function value.

[0173] Furthermore, when the second training unit 1004 determines the target loss function value based on the first task loss function value, the second task loss function value, and the third task loss function value, it is specifically used for:

[0174] The first task loss function value, the second task loss function value, and the third task loss function value are weighted and summed according to the third preset weight to obtain the target loss function value.

[0175] This application embodiment uses audio and video training data and labeled data to train an audio and video multi-task evaluation model, which increases the model's performance and diversity, reduces memory usage and resource consumption, and helps improve the efficiency and accuracy of audio and video multi-task evaluation.

[0176] Please see Figure 11 , Figure 11 This is a schematic diagram of the structure of an audio-visual multi-task evaluation device provided in an embodiment of this application. Specifically, the audio-visual multi-task evaluation device is disposed in a computer device, and the device includes: a second acquisition unit 1101, a second extraction unit 1102, a fusion unit 1103, and an evaluation unit 1104;

[0177] The second acquisition unit 1101 is used to acquire audio and video data to be evaluated, wherein the audio and video data to be evaluated includes audio data to be evaluated and video data to be evaluated.

[0178] The second extraction unit 1102 is used to perform feature extraction processing on the video data to be evaluated to obtain the video feature vector to be evaluated, and to perform feature extraction processing on the audio data to be evaluated to obtain the audio feature vector to be evaluated.

[0179] The fusion unit 1103 is used to concatenate the audio feature vector to be evaluated and the video feature vector to be evaluated into a sequence, and perform multimodal fusion processing on the sequence to obtain the feature vector to be evaluated;

[0180] The evaluation unit 1104 is used to input the feature vector to be evaluated, the video feature vector to be evaluated, and the audio feature vector to be evaluated into a pre-trained audio-visual multi-task evaluation model to obtain the evaluation data of the audio-visual data to be evaluated. The evaluation data includes comprehensive audio-visual quality level data, video quality data, and audio quality data.

[0181] This application embodiment improves the efficiency and accuracy of audio and video multi-task evaluation by performing multimodal fusion processing on the audio data and video data to be evaluated, obtaining the feature vector to be evaluated, and using the audio and video multi-task evaluation model and the feature vector to be evaluated.

[0182] Please see Figure 12 , Figure 12 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Specifically, the computer device includes: a memory 1201 and a processor 1202.

[0183] In one embodiment, the computer device further includes a data interface 1203 for transmitting data information between the computer device and other devices.

[0184] The memory 1201 may include volatile memory; the memory 1201 may also include non-volatile memory; the memory 1201 may also include a combination of the above types of memory. The processor 1202 may be a central processing unit (CPU). The processor 1202 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), or any combination thereof.

[0185] The memory 1201 is used to store programs, and the processor 1202 can call the programs stored in the memory 1201 to perform the following steps:

[0186] Obtain an audio and video training dataset, which includes video training data, audio training data, and labeled data, wherein the labeled data includes audio and video quality labels;

[0187] The video training data is subjected to feature extraction processing to obtain a target video feature vector, and the audio training data is subjected to feature extraction processing to obtain a target audio feature vector;

[0188] The first deep learning neural network model is trained using the target audio feature vector, the target video feature vector, and the audio and video quality labels to obtain the target loss function value. The target loss function value includes a homoscedasticity parameter, which is associated with the multi-task weights.

[0189] The target loss function value is adjusted according to the homoscedasticity parameter, the model parameters are adjusted according to the target loss function value, and the first deep learning neural network model after adjusting the model parameters is retrained using the target audio feature vector, the target video feature vector, and the audio and video quality labels to obtain the audio and video multi-task evaluation model.

[0190] Further, the audio and video quality labels include audio quality labels, video quality labels, and a comprehensive audio and video quality level label; when the processor 1002 trains the first deep learning neural network model using the target audio feature vector, the target video feature vector, and the audio and video quality labels to obtain the target loss function value, it is specifically used for:

[0191] The target audio feature vector and the target video feature vector are fused to obtain the target feature vector;

[0192] The target audio feature vector and audio quality label, the target video feature vector and video quality label, and the target feature vector and audio / video quality level integrated label are input into the first deep learning neural network model for training to obtain the target loss function value.

[0193] Further, when the processor 1002 inputs the target audio feature vector and audio quality label, the target video feature vector and video quality label, and the target feature vector and comprehensive audio-visual quality level label into the first deep learning neural network model for training, and obtains the target loss function value, it is specifically used for:

[0194] The first deep learning neural network model is trained using the target audio feature vector and audio quality label to obtain the first task loss function value;

[0195] The first deep learning neural network model is trained using the target video feature vector and video quality label to obtain the second task loss function value;

[0196] The first deep learning neural network model is trained using the target feature vector and the comprehensive audio / video quality level label to obtain the third task loss function value;

[0197] The target loss function value is determined based on the first task loss function value, the second task loss function value, and the third task loss function value.

[0198] Further, the audio quality label includes an audio quality level label and an audio quality cause label; when the processor 1002 trains the first deep learning neural network model using the target audio feature vector and the audio quality label to obtain the first task loss function value, it is specifically used for:

[0199] The target audio feature vector, audio quality level label, and audio quality reason label are input into the first deep learning neural network model for training to obtain the first loss function value and the second loss function value.

[0200] The first loss function value and the second loss function value are weighted and summed according to the first preset weight to obtain the first task loss function value.

[0201] Further, the video quality label includes a video quality level label and a video quality cause label; when the processor 1002 trains the first deep learning neural network model using the target video feature vector and the video quality label to obtain the second task loss function value, it is specifically used for:

[0202] The target video feature vector, video quality level label, and video quality reason label are input into the first deep learning neural network model for training to obtain the third loss function value and the fourth loss function value.

[0203] The third and fourth loss function values ​​are weighted and summed according to the second preset weights to obtain the second task loss function value.

[0204] Further, when the processor 1002 determines the target loss function value based on the first task loss function value, the second task loss function value, and the third task loss function value, it is specifically used for:

[0205] The first task loss function value, the second task loss function value, and the third task loss function value are weighted and summed according to the third preset weight to obtain the target loss function value.

[0206] This application embodiment uses audio and video training data and labeled data to train an audio and video multi-task evaluation model, which increases the model's performance and diversity, reduces memory usage and resource consumption, and helps improve the efficiency and accuracy of audio and video multi-task evaluation.

[0207] Please see Figure 13 , Figure 13 This is a schematic diagram of another computer device provided in an embodiment of this application. Specifically, the computer device includes: a memory 1301 and a processor 1302.

[0208] In one embodiment, the computer device further includes a data interface 1303 for transmitting data information between the computer device and other devices.

[0209] The memory 1301 may include volatile memory; the memory 1301 may also include non-volatile memory; the memory 1301 may also include a combination of the above types of memory. The processor 1302 may be a central processing unit (CPU). The processor 1302 may further include hardware chips. The hardware chips may be application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof. The PLDs may be complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or any combination thereof.

[0210] The memory 1301 is used to store programs, and the processor 1302 can call the programs stored in the memory 1301 to perform the following steps:

[0211] Acquire audio and video data to be evaluated, which includes audio data and video data to be evaluated;

[0212] The video data to be evaluated is subjected to feature extraction processing to obtain the video feature vector to be evaluated, and the audio data to be evaluated is subjected to feature extraction processing to obtain the audio feature vector to be evaluated.

[0213] The audio feature vector and the video feature vector to be evaluated are concatenated into a sequence, and the sequence is subjected to multimodal fusion processing to obtain the feature vector to be evaluated.

[0214] The feature vector to be evaluated is input into a pre-trained audio-visual multi-task evaluation model to obtain the evaluation data of the audio-visual data to be evaluated. The evaluation data includes comprehensive audio-visual quality level data, video quality data, and audio quality data.

[0215] This application embodiment performs multimodal fusion processing on the audio data and video data to be evaluated to obtain the feature vector to be evaluated. The audio and video multi-task evaluation model, the feature vector to be evaluated, the video feature vector to be evaluated, and the audio feature vector to be evaluated are used to evaluate the audio and video data to be evaluated, thereby improving the efficiency and accuracy of audio and video multi-task evaluation.

[0216] Embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements this application. Figure 1 , Figure 3 or Figure 8 The method described in the corresponding embodiment can also be implemented. Figure 9 The device described in the embodiments corresponding to this application will not be further elaborated here.

[0217] The computer-readable storage medium can be an internal storage unit of the device described in any of the foregoing embodiments, such as the device's hard drive or memory. The computer-readable storage medium can also be an external storage device of the device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the device. Further, the computer-readable storage medium can include both internal and external storage units of the device. The computer-readable storage medium is used to store the computer program and other programs and data required by the terminal. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0218] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0219] The above-disclosed embodiments are merely some of the embodiments of this application, and should not be construed as limiting the scope of this application. Those skilled in the art can understand that implementing all or part of the above embodiments and making equivalent changes in accordance with the claims of this application still fall within the scope of the invention.

Claims

1. A multi-task audio and video learning method, characterized in that, include: Obtain an audio and video training dataset, which includes video training data, audio training data, and labeled data, wherein the labeled data includes audio and video quality labels; The video training data is subjected to feature extraction processing to obtain a target video feature vector, and the audio training data is subjected to feature extraction processing to obtain a target audio feature vector; The first deep learning neural network model is trained using the target audio feature vector, the target video feature vector, and the audio and video quality labels to obtain the target loss function value. The target loss function value includes a homoscedasticity parameter, which is associated with the multi-task weights. The target loss function value is adjusted according to the homoscedasticity parameter, the model parameters are adjusted according to the target loss function value, and the first deep learning neural network model after adjusting the model parameters is retrained using the target audio feature vector, the target video feature vector, and the audio and video quality labels to obtain the audio and video multi-task evaluation model. The audio and video quality labels include audio quality labels, video quality labels, and a comprehensive audio and video quality level label; the step of training the first deep learning neural network model using the target audio feature vector, target video feature vector, and audio and video quality labels to obtain the target loss function value includes: The target audio feature vector and the target video feature vector are fused to obtain the target feature vector; The target audio feature vector and audio quality label, the target video feature vector and video quality label, and the target feature vector and comprehensive audio-video quality level label are input into the first deep learning neural network model for training to obtain the target loss function value; The step of inputting the target audio feature vector and audio quality label, the target video feature vector and video quality label, and the target feature vector and comprehensive audio-video quality level label into the first deep learning neural network model for training to obtain the target loss function value includes: The first deep learning neural network model is trained using the target audio feature vector and audio quality label to obtain the first task loss function value; The first deep learning neural network model is trained using the target video feature vector and video quality label to obtain the second task loss function value; The first deep learning neural network model is trained using the target feature vector and the comprehensive audio and video quality level label to obtain the third task loss function value; The target loss function value is determined based on the first task loss function value, the second task loss function value, and the third task loss function value.

2. The method according to claim 1, characterized in that, The audio quality label includes an audio quality level label and an audio quality cause label; the step of training the first deep learning neural network model using the target audio feature vector and the audio quality label to obtain the first task loss function value includes: The target audio feature vector, audio quality level label, and audio quality reason label are input into the first deep learning neural network model for training to obtain the first loss function value and the second loss function value. The first loss function value and the second loss function value are weighted and summed according to the first preset weight to obtain the first task loss function value.

3. The method according to claim 1, characterized in that, The video quality label includes a video quality level label and a video quality reason label; the step of training the first deep learning neural network model using the target video feature vector and the video quality label to obtain the second task loss function value includes: The target video feature vector, video quality level label, and video quality reason label are input into the first deep learning neural network model for training to obtain the third loss function value and the fourth loss function value. The third and fourth loss function values ​​are weighted and summed according to the second preset weights to obtain the second task loss function value.

4. The method according to claim 1, characterized in that, Determining the target loss function value based on the first task loss function value, the second task loss function value, and the third task loss function value includes: The first task loss function value, the second task loss function value, and the third task loss function value are weighted and summed according to the third preset weight to obtain the target loss function value.

5. A method for evaluating multi-task audio and video, characterized in that, include: Acquire audio and video data to be evaluated, which includes audio data and video data to be evaluated; The video data to be evaluated is subjected to feature extraction processing to obtain the video feature vector to be evaluated, and the audio data to be evaluated is subjected to feature extraction processing to obtain the audio feature vector to be evaluated. The audio feature vector and the video feature vector to be evaluated are concatenated into a sequence, and the sequence is subjected to multimodal fusion processing to obtain the feature vector to be evaluated. The feature vector to be evaluated is input into a pre-trained audio-visual multi-task evaluation model to obtain evaluation data of the audio-visual data to be evaluated. The evaluation data includes comprehensive audio-visual quality level data, video quality data and audio quality data. The audio-visual multi-task evaluation model is trained by an audio-visual multi-task learning method according to any one of claims 1-4. The step of inputting the feature vector to be evaluated into a pre-trained audio-visual multi-task evaluation model to obtain evaluation data for the audio-visual data to be evaluated includes: The feature vector to be evaluated is input into a pre-trained audio-visual multi-task evaluation model to obtain comprehensive audio-visual quality level data; and, The feature vector of the video to be evaluated is input into a pre-trained audio-visual multi-task evaluation model to obtain video quality data of the video data to be evaluated. The video quality data includes video quality level data and video quality cause data; and... The audio feature vector to be evaluated is input into a pre-trained audio-video multi-task evaluation model to obtain audio quality data of the audio data to be evaluated. The audio quality data includes audio quality level data and audio quality cause data.

6. A computer device, characterized in that, The device includes a processor, an input device, an output device, and a memory, which are interconnected. The memory is used to store a computer program, which includes a program, and the processor is configured to call the program to perform the method as described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that, when executed, are used to implement the method as described in any one of claims 1-5.

Images