Method and device for joint video and audio quality assessment based on neural network

Abstract

The invention provides a neural network-based video and audio joint quality evaluation method, comprising: from the video frame, the video image block adapted to the input of the neural network is intercepted, and the one-dimensional audio signal of the audio segment is converted by short-time Fourier transform For the two-dimensional spectrogram representation, the neural network is used to extract the perceptual quality features from the video image block and the two-dimensional spectrogram respectively, and the extracted audio and video deep neural network perceptual quality features are post-processed to obtain two modes based on The quality features of the deep neural network, the quality features of the two modalities are fused to obtain the joint perceptual quality of video frames and audio clips, and the joint perceptual quality of video frames and audio clips is pooled in the time domain to obtain the joint perceptual quality of the overall audio and video. At the same time, a combined quality evaluation device is provided. The neural network-based video and audio joint quality evaluation method provided by the present invention can effectively evaluate the overall experience quality of audio and video.

Description

technical field [0001] The present invention relates to the technical field of multimedia quality evaluation, in particular to a neural network-based video and audio joint quality evaluation method and device. Background technique [0002] With the progress of society and the development of science and technology, the way people convey information is constantly changing. In particular, the rapid development of information technology has made multimedia represented by video and audio gradually become an indispensable way for people to convey information and communicate. Statistics show that people around the world take more than one trillion photos every year, and other types of multimedia information such as audio and video are also experiencing explosive growth. In this context, related multimedia signal processing technology has also become a research hotspot. Multimedia information may go through various stages such as collection, compression, transmission, processing, ...

AI Technical Summary

Problems solved by technology

Generally speaking, in order to study the interaction between audiovisual signals and other factors that affect the evaluation of audiovisual quality, it is usually necessary to conduct basic research on multimodal perception, and these studies are usually achieved through some audiovisual experiments, while the current field Intrinsic and objective audio-video joint quality evaluation models are extremely scarce

[0010] At present, there is no description or report of the similar technology of the present invention, and no similar data at home and abroad have been collected yet.

Images