Video quality evaluation method and device, equipment and storage medium

Abstract

The invention discloses a video quality evaluation method and device, equipment and a storage medium, and belongs to the technical field of video processing. The method comprises the following steps: acquiring to-be-evaluated video data; extracting saliency semantic information corresponding to the visual attention area in the to-be-evaluated video data, and performing space-time modeling and context analysis on the saliency semantic information to obtain saliency features of the to-be-evaluated video data; extracting multi-scale image features of the to-be-evaluated video data; extracting coding domain features of the to-be-evaluated video data; fusing the saliency features, the multi-scale image features and the coding domain features to obtain fused feature representation; and predicting the fusion feature representation by using a pre-trained video quality evaluation model to obtain a quality evaluation result of the to-be-evaluated video data. According to the method, the video quality can be evaluated by integrating the multi-dimensional features, the problems that various distortion types are difficult to consider and subjective perception of human eyes is difficult to reflect in the prior art are effectively solved, and the video quality evaluation accuracy is improved.

Description

Technical Field

[0001] This application belongs to the field of video processing technology, and in particular relates to a video quality assessment method, apparatus, device and storage medium. Background Technology

[0002] With the increasing number of videos posted by users on social media and digital platforms, the ability to efficiently and accurately assess video quality has become a crucial need.

[0003] However, existing video quality assessment methods generally have limitations. Most technologies can only detect a certain type of video defect and are difficult to accurately assess the overall quality of the video. This results in a gap between the assessment results and human perception, affecting the reliability and practicality of the assessment. Summary of the Invention

[0004] This application provides a video quality assessment method, apparatus, device, and storage medium that can efficiently and accurately assess video quality.

[0005] In a first aspect, embodiments of this application provide a video quality assessment method, the method comprising: acquiring video data to be assessed; extracting salient semantic information corresponding to visually relevant regions in the video data to be assessed, performing spatiotemporal modeling and contextual analysis on the salient semantic information to obtain salient features of the video data to be assessed; extracting multi-scale image features of the video data to be assessed; extracting coding domain features of the video data to be assessed; fusing the salient features, multi-scale image features, and coding domain features to obtain a fused feature representation; and using a pre-trained video quality assessment model to predict the fused feature representation to obtain a quality assessment result of the video data to be assessed.

[0006] In one possible implementation of the first aspect, the salient semantic information corresponding to the visually concerned region in the video data to be evaluated is extracted. The method includes: using a saliency detection network to identify the visually concerned region in the video data to be evaluated and generating the corresponding salient semantic information. The saliency detection network is built based on the U2-Net architecture.

[0007] In one possible implementation of the first aspect, spatiotemporal modeling and contextual analysis are performed on salient semantic information to obtain salient features of the video data to be evaluated. The method includes: inputting salient semantic information as prior information into an image perception network; and using the image perception network, based on an attention mechanism, modeling the dependency relationship between the visual attention area and the overall picture in a global scope to generate salient features.

[0008] In one possible implementation of the first aspect, extracting multi-scale image features from the video data to be evaluated includes: cropping multiple small blocks from each frame of the video data to be evaluated and merging the multiple small blocks into a local image block, the local image block being used to represent the local feature information of each frame; adjusting the size of each frame to be consistent with the size of the corresponding local image block to obtain a global image block, the global image block being used to represent the global feature information of each frame; and inputting the local image block and the global image block into a self-attention model for feature extraction to obtain the multi-scale image features of each frame.

[0009] In one possible implementation of the first aspect, the coding domain features include at least one of the following: average quantization parameter, motion amplitude parameter, frame rate, and bit rate.

[0010] In one possible implementation of the first aspect, saliency features, local features, global features, and coding domain features are fused to obtain a fused feature representation. The method includes: performing linear weighted fusion of saliency features, local features, global features, and coding domain features based on preset weights to obtain a fused feature representation.

[0011] In one possible implementation of the first aspect, before acquiring the video data to be evaluated, the method further includes: acquiring video data samples and quality labels for the video data samples; extracting salient features, local features, global features, and coding domain features from the video data samples respectively, and fusing them to obtain sample fusion features; analyzing the sample fusion features using a video quality assessment model to obtain quality prediction results; determining the model prediction loss function based on the difference between the quality prediction results and the quality labels; minimizing the model prediction loss using an optimization algorithm and updating the model parameters; and outputting the video quality assessment model when the training stopping condition is met.

[0012] In one implementation of the first aspect, acquiring video data samples includes: acquiring raw video data; performing quality transformation processing on the raw video data to generate a training dataset, the training dataset containing multiple video data samples; the quality transformation includes at least one of blur transformation, noise transformation, color transformation, and encoding transformation.

[0013] Secondly, embodiments of this application provide a video quality assessment device, comprising: an acquisition module for acquiring video data to be assessed; a first feature extraction module for extracting salient semantic information corresponding to visually relevant regions in the video data to be assessed, performing spatiotemporal modeling and contextual analysis on the salient semantic information to obtain salient features of the video data to be assessed; a second feature extraction module for extracting multi-scale image features of the video data to be assessed; a third feature extraction module for extracting coding domain features of the video data to be assessed from the video coding domain; a feature fusion module for fusing salient features, multi-scale image features, and coding domain features to obtain a fused feature representation; and a quality assessment module for using a pre-trained video quality assessment model to predict the fused feature representation to obtain a quality assessment result for the video data to be assessed.

[0014] Thirdly, embodiments of this application provide a computer device, the device including: a processor, and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the method of any one of the first aspects above.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the method of any one of the first aspects described above.

[0016] Fifthly, embodiments of this application provide a computer program product comprising a computer program that, when executed by a processor, implements the method of any one of the first aspects described above.

[0017] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects: The video quality assessment method provided in this application extracts salient semantic features, multi-scale image features, and coding domain features from the video data to be assessed, and fuses them into a comprehensive feature representation covering multi-dimensional quality information. This feature representation is then analyzed and predicted using a pre-trained video quality assessment model to obtain the video quality assessment result. This method can comprehensively capture all aspects of video quality information, including salient regions, local textures, global structure, and coding information. It effectively solves the problems of existing technologies that struggle to accommodate multiple distortion types and reflect subjective human perception. Therefore, without the need for a reference video, it can more accurately and robustly simulate human visual perception and accurately assess video quality. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below.

[0019] Figure 1 This is a flowchart illustrating a video quality assessment method provided in an embodiment of this application; Figure 2 This is a flowchart illustrating a video quality assessment model training method provided in an embodiment of this application; Figure 3 This is a flowchart illustrating a video quality assessment method provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a video quality assessment device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solutions of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0021] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples consistent with some aspects of this application as detailed in the appended claims.

[0022] In the digital age, video has become one of the main ways of information transmission and visual presentation. Video quality directly affects users' acquisition and perception of information, and is also related to the platform's attractiveness, user experience, and content dissemination effectiveness. However, due to limitations in shooting equipment performance, compression and packet loss during network transmission, environmental noise, and other factors, video quality is often affected to varying degrees.

[0023] As described in the background section, the sheer volume of user-generated videos on social media and digital platforms makes manual review insufficient for meeting the demands of efficiency and scalability. Therefore, there is an urgent need for automated Video Quality Assessment (VQA) technology. VQA enables computers to perceive video quality like humans and obtain assessment results consistent with subjective quality perception through quantitative indicators.

[0024] Currently, based on whether reference videos are provided, VQA methods can be divided into three categories: Full-reference VQA requires both the distorted video and the reference video. When the dataset is insufficient, this method is less robust and struggles to handle complex and degraded video scenarios. Semi-referenced VQA only requires providing partial information from the reference video (such as histograms, structural features, etc.), but this is often difficult to obtain in practical applications; No-reference VQA, which relies solely on distorted video for evaluation, is more in line with practical application needs, but it is also the most challenging method.

[0025] Despite continuous development, existing methods still have significant limitations. Most techniques can only detect single types of video defects (such as blurring or compression artifacts), making it difficult to comprehensively assess the overall quality of the video. This leads to discrepancies with human subjective perception, affecting the reliability and practicality of the assessment.

[0026] To address the aforementioned technical problems, embodiments of this application provide a video quality assessment method, apparatus, device, and storage medium. Specifically, it presents a reference-free video quality assessment scheme that improves the accuracy and practicality of video quality assessment by analyzing and fusing multi-dimensional features and simulating human eye perception.

[0027] The video quality assessment method provided in this application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] Figure 1 A flowchart illustrating a video quality assessment method provided in an embodiment of this application is shown.

[0029] like Figure 1 As shown, the method may include the following steps.

[0030] S110. Obtain the video data to be evaluated.

[0031] S120. Extract the salient semantic information corresponding to the visual attention region in the video data to be evaluated, perform spatiotemporal modeling and context analysis on the salient semantic information, and obtain the salient features of the video data to be evaluated.

[0032] The visual attention area refers to the key area in a video that is most attractive to the human eye and most likely to attract attention.

[0033] Understandably, the video focus area carries key semantic information and perceptual value of the video data to be evaluated. For example, it could be the face of a person in the video, the main human figure, or an object in motion, or it could be an area with prominent lighting or color contrast.

[0034] In one embodiment, the method for extracting salient semantic information corresponding to visually relevant regions in the video data to be evaluated includes: S121. Using a saliency detection network, visual attention regions in the video data to be evaluated are identified, and corresponding saliency semantic information is generated.

[0035] A saliency detection network is used to simulate the visual attention mechanism of the human eye and automatically identify the visual attention region from video frames.

[0036] For example, the saliency detection network can be a nested U-shaped saliency detection network, such as one built on the U2-Net architecture. Compared to the traditional U-Net architecture, the saliency detection network based on the U2-Net architecture has advantages in multi-scale feature extraction and detail preservation, and can highlight key local regions in the video while ensuring global perception.

[0037] Saliency semantic information refers to the semantic feature representation generated by the saliency detection network for the identified visually relevant regions.

[0038] For example, salient semantic information is used as a semantic feature mask for video frames to segment visual attention regions such as faces, human subjects, and moving objects from the video frames.

[0039] This embodiment utilizes a saliency detection network based on the U2-Net architecture to identify visually relevant regions in a video and generate saliency semantic information. This highlights the most important regional features in the video, providing accurate prior information for subsequent spatiotemporal modeling and quality assessment.

[0040] In one embodiment, spatiotemporal modeling and contextual analysis are performed on salient semantic information to obtain salient features of the video data to be evaluated. The method includes: S122. Input salient semantic information as prior information into the image quality perception network; S123. Using an image quality perception network, based on an attention mechanism, model the dependency relationship between the visual attention area and the overall picture in the global scope, and generate salient features.

[0041] Image quality perceptual networks are used to extract multi-level semantic features from video frames that are consistent with human visual perception and to model the interaction between local and global features.

[0042] For example, an image quality-aware network can be built using a Shifted Window Transformer (Swin Transformer) architecture. It should be understood that, compared to traditional Convolutional Neural Networks (CNNs), which can only extract local features within a fixed receptive field, or ordinary Transformers, which suffer from excessive computational overhead when processing high-resolution videos, the Swin Transformer possesses stronger multi-scale feature modeling capabilities and higher computational efficiency, making it more suitable for saliency modeling in complex video scenes.

[0043] Saliency features refer to the deep feature representations extracted by the image quality perception network after introducing saliency semantic information. They are used to characterize the semantic and structural relationship between the visual attention area and the overall picture in the video data to be evaluated.

[0044] In one implementation of this embodiment, salient semantic information is first input as a priori tokens into an image quality perception network based on the Swin Transformer. Then, the image quality perception network dynamically models the dependency between salient regions (i.e., visually relevant regions) and non-salient regions (i.e., non-visually relevant regions) globally using a self-attention mechanism. Attention weights are adaptively assigned to salient and non-salient regions, making the model more focused on salient regions without losing contextual information about the overall scene. Finally, salient features are output.

[0045] The method in this embodiment uses salient semantic information as prior input to the self-attention model and models the dependency relationship between the visual attention region and the overall picture in a global scope. It can fully integrate local salient regions and global contextual information, thereby generating more accurate and robust salient features.

[0046] Assume that the saliency detection network is represented as The image quality-aware network based on the Swing Transformer is represented as follows: The video frame I is input into the saliency detection network. Based on the processing in step S120, the saliency features output by the image quality perception network can be represented as follows: (1) S130. Extract multi-scale image features from the video data to be evaluated.

[0047] For example, multi-scale image features can include local features and global features.

[0048] Local features are feature information used to characterize local regions within video frames of video data to be evaluated.

[0049] Global features are feature information used to characterize the overall structure of video frames in the video data to be evaluated.

[0050] In one implementation, each frame of the video data to be evaluated can be processed based on a multi-scale convolutional neural network. Local features are extracted using shallow convolutional neural networks in the multi-scale convolutional neural network, and global features are extracted using deep convolutional neural networks in the multi-scale convolutional neural network.

[0051] In one implementation, each frame of the video data to be evaluated can be modeled from two levels: local region and global image. Local features are extracted based on the local level model, and global features are extracted based on the global level model.

[0052] In one embodiment, each frame of the video data to be evaluated is modeled from two levels: local region and global image. Local features are extracted based on the local level model, and global features are extracted based on the global level model. This may include the following steps S131-S133.

[0053] S131. Cut out multiple small blocks from each frame of the video data to be evaluated, and merge the multiple small blocks into a local image block.

[0054] For example, 49 32x32 patch blocks are cropped from each frame of the video data to be evaluated, each patch block being a small block. The 49 patches are then merged into a single 224x224 patch block, which is the local image block.

[0055] Local image patches are used to represent local feature information of each frame of an image.

[0056] S132. Adjust the size of each frame image to match the size of the corresponding local image block to obtain the global image block.

[0057] Continuing from the previous example, the size of each frame image is scaled to 224x224 to obtain a resized image, which is the global image block.

[0058] Global image patches are used to represent global feature information of each frame of image.

[0059] S133. Input the local image patch and the global image patch into the self-attention model for feature extraction to obtain the multi-scale image features of each frame.

[0060] Self-attention models are used to capture local details and global semantic information from an input image, model the dependencies between different regions of the image, and generate multi-scale feature representations. For example, a self-attention model could be a SwinTransformer network.

[0061] Continuing from the previous example, the local image patch 224x224 and the global image patch resize 224x224 are input into the Swin Transformer network. The relationship between regions is modeled globally through the self-attention mechanism, local and global features of each frame are extracted and fused into multi-scale image features.

[0062] It should be understood that multi-scale image features are low-level features, represented as... A local image patch can be represented as The global image patch can be represented as After using the Swing Transformer network to extract features based on local and global image patches, the corresponding multi-scale image features can be represented as follows: (2) This embodiment proposes a method for extracting local and global features. Local features are extracted by cropping and recombining small image patches, while global features are obtained by scaling the entire frame image. The method also uses a self-attention model to fuse local and global information, which can comprehensively represent the fine-grained details and overall semantic structure of each frame image.

[0063] S140. Extract the coding domain features of the video data to be evaluated.

[0064] Coding domain features refer to features extracted from video coding information, which are used to reflect the distortion of the video data to be evaluated during compression and transmission.

[0065] Given the shortcomings of traditional evaluation models in handling coding distortion and transmission effects, this application introduces coding domain information to supplement visual feature-based evaluation, thereby providing a more comprehensive assessment of the quality of coded video.

[0066] For example, the coding domain features include at least one of the following: average quantization parameter, motion amplitude parameter, frame rate, and bit rate.

[0067] For the video data V to be evaluated, the extracted coding domain features can be represented as follows: (3) in, The average quantization parameter represents the average value of the quantization parameters used during video encoding. The larger the quantization parameter, the stronger the video compression, which may lead to loss of detail. The standard deviation of the quantization parameter reflects the magnitude of the change of the quantization parameter between video frames. The greater the fluctuation of the quantization parameter, the more unstable the video quality may be. The mean motion vector represents the average value of motion vectors between video frames, reflecting the dynamics of the video content. The stronger or faster the motion, the more likely encoding distortion may occur. The mean of motion vector differences is used to measure the change in motion vectors between adjacent frames, reflecting the smoothness of motion. Large changes may lead to motion blur or instability. The standard deviation of motion amplitude represents the degree of fluctuation of the motion vector and characterizes the complexity of the motion in the video content.

[0068] The gray-level co-occurrence matrix features are used to statistically analyze the texture information of video frames.

[0069] Frame rate is used to reflect the smoothness of video playback.

[0070] Bitrate is used to reflect compression strength and video quality.

[0071] S150, by fusing saliency features, multi-scale image features and coding domain features, a fused feature representation is obtained.

[0072] In one implementation, based on preset weights, saliency features, multi-scale image features, and coding domain features are linearly weighted and fused to obtain a fused feature representation.

[0073] Fusion feature representation The calculation formula is as follows: (4) in, To indicate the first to be merged Class features include saliency features, multi-scale image features, and coding domain features; Preset weights for the corresponding features.

[0074] The preset weights for each type of feature can be allocated based on feature importance or experience, or can be adaptively adjusted through learning. This application does not impose any restrictions on this.

[0075] S160. Use a pre-trained video quality assessment model to predict the fusion feature representation and obtain the quality assessment results of the video data to be evaluated.

[0076] Video quality assessment models can be built based on deep neural networks or self-attention architectures.

[0077] For example, the quality assessment result can be represented as a numerical quality score.

[0078] The video quality assessment method provided in this application extracts salient semantic features, multi-scale image features, and coding domain features from the video data to be assessed, and fuses them into a comprehensive feature representation covering multi-dimensional quality information. This feature representation is then analyzed and predicted using a pre-trained video quality assessment model to obtain the video quality assessment result. This method can comprehensively capture all aspects of video quality information, including salient regions, local textures, global structure, and coding information. It effectively solves the problems of existing technologies that struggle to accommodate multiple distortion types and reflect subjective human perception. Therefore, without the need for a reference video, it can more accurately and robustly simulate human visual perception and accurately assess video quality.

[0079] This application also provides a video quality assessment model and its training method.

[0080] Figure 2 A flowchart illustrating a training method for a video quality assessment model provided in an embodiment of this application is shown below. Figure 2 The training process of this model is described.

[0081] S210. Obtain video data samples and quality labels for the video data samples.

[0082] In one implementation, step S210, acquiring video data samples, may include: S211. Obtain the raw video data.

[0083] For example, unlabeled video data from various scenarios can be collected from business operations or open-source video resources as raw video data.

[0084] S212. Perform quality transformation processing on the original video data to generate a training dataset, which contains multiple video data samples; the quality transformation includes at least one of blur transformation, noise transformation, color transformation and encoding transformation.

[0085] This implementation expands the sample set through quality transformation, significantly increasing the sample size and diversity, and improving the model's generalization ability and robustness under different quality scenarios.

[0086] In one implementation, the quality label of a video data sample can be obtained through subjective ratings by volunteers to reflect the human eye's perception of the overall visual quality of the video.

[0087] S220. Extract the salient features, multi-scale image features and coding domain features of the video data samples respectively, and fuse them to obtain the sample fusion features.

[0088] It should be understood that the implementation of step S220 can be referred to steps S120-S140, and will not be repeated here.

[0089] S230. Analyze the sample fusion features using a video quality assessment model to obtain quality prediction results.

[0090] The video quality assessment model internally includes a fully connected layer. It fuses features from samples. The input video quality assessment model is further processed by this fully connected layer, and the output is the predicted video quality value Q. The regression task can be represented as: (5) in, This represents the video quality assessment model and its subsequent mapping layer processing functions.

[0091] S240. Determine the model prediction loss based on the difference between the quality prediction results and the quality labels.

[0092] Based on the difference between the quality prediction results and the quality labels, a loss function is constructed for training and optimization. For example, the loss function is defined as follows: (6) in, This represents the absolute error between the predicted value and the true label. The loss represents the Pearson linear correlation coefficient between the predicted value and the true label; Indicates the sorting loss; and These are the weighting coefficients for each loss term.

[0093] S250. Minimize the model prediction loss using an optimization algorithm and update the model parameters.

[0094] An optimization algorithm could be, for example, the gradient descent algorithm.

[0095] S260. If the training stopping condition is met, output the video quality evaluation model.

[0096] Training can be stopped for reasons such as loss convergence or reaching a preset number of training iterations.

[0097] In this embodiment, salient features, multi-scale image features, and coding domain features are systematically extracted using video data samples and their subjective quality labels, and then fused into a comprehensive feature representation for model training. By constructing a multi-dimensional loss function and using optimization algorithms to iteratively update the model parameters, the trained video quality assessment model can effectively learn the mapping relationship between fused features and human subjective perception. This enables accurate and robust quality assessment of videos with various distortion types during prediction, significantly improving the consistency between model prediction results and human perception, and enhancing adaptability to real-world video scenes.

[0098] Figure 3 A flowchart illustrating a video quality assessment method provided in an embodiment of this application is shown. It should be understood that... Figure 3 The illustrated embodiment can be considered as Figure 1 A specific example of the embodiment is described below in conjunction with... Figure 3 The methodology of this application will be explained again.

[0099] S1. Obtain the video data to be evaluated.

[0100] S2. Extract the saliency detection results corresponding to the visual attention regions in the video data to be evaluated based on the U2-Net saliency detection network.

[0101] It should be understood that step S2 can be seen as an example of step S120, and the saliency detection result can be seen as a type of saliency semantic information.

[0102] S3. Input the saliency detection results into the Transformer image perception network to extract saliency features.

[0103] S4. Patch and resize the image frames of the video data to be evaluated.

[0104] S5. Input the sampled image information into the Transformer model to extract multi-scale image features.

[0105] It should be understood that steps S4 and S5 can be seen as an example of step S130.

[0106] S6. Extract the coding domain features of the video data to be evaluated.

[0107] It should be understood that step S6 can be seen as an example of step S140.

[0108] S7. Fuse the salient features, multi-scale image features and coding domain features extracted in the above steps.

[0109] It should be understood that step S6 can be seen as an example of step S150.

[0110] S8. Input the fused feature representation into the quality regression module for analysis and prediction to obtain the quality score of the video to be evaluated.

[0111] It should be understood that the quality regression module is an implementation example of the video quality assessment model, and step S7 can be regarded as an example of step S160.

[0112] This embodiment goes beyond global features; it integrates saliency features, multi-size image features, and coding domain features to construct a more comprehensive feature set. This enables the model to deeply understand video quality from multiple perspectives, including regions of interest, detail perception, structural analysis, and coding quality, thus more closely resembling the multi-dimensional perception of the human eye and improving the comprehensiveness and accuracy of the evaluation.

[0113] The foregoing mainly describes a video quality assessment method according to an embodiment of this application with reference to the accompanying drawings. It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially, these steps are not necessarily executed in the order shown in the figures. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of steps or stages in other steps. The following describes a video quality assessment apparatus according to an embodiment of this application with reference to the accompanying drawings. For brevity, appropriate omissions will be made in the following description of the apparatus; relevant content can be referred to in the relevant descriptions of the method above, and will not be repeated.

[0114] Corresponding to the method described in the above embodiments, Figure 4 A structural block diagram of a video quality assessment device 1000 provided in an embodiment of this application is shown. For ease of explanation, only the parts related to the embodiment of this application are shown.

[0115] like Figure 4 As shown, the device 1000 may include: Module 1001 is used to acquire video data to be evaluated; The first feature extraction module 1002 is used to extract salient semantic information corresponding to the visual attention region in the video data to be evaluated, perform spatiotemporal modeling and context analysis on the salient semantic information, and obtain salient features of the video data to be evaluated. The second feature extraction module 1003 is used to extract local and global features of the video data to be evaluated. The third feature extraction module 1004 is used to extract coding domain features of the video data to be evaluated from the video coding domain. The feature fusion module 1005 is used to fuse salient features, local features, global features, and coding domain features to obtain a fused feature representation; The quality assessment module 1006 is used to predict the fused feature representation using a pre-trained video quality assessment model to obtain the quality assessment result of the video data to be assessed.

[0116] Figure 5 A schematic diagram of the hardware structure of a computer device provided in an embodiment of this application is shown.

[0117] The computer device may include a processor 7001 and a memory 7002 storing computer program instructions.

[0118] Specifically, the processor 7001 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0119] Memory 7002 may include mass storage for data or instructions. For example, and not limitingly, memory 7002 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. In one instance, memory 7002 may include removable or non-removable (or fixed) media, or memory 7002 may be non-volatile solid-state memory. Memory 7002 may be internal or external to the integrated gateway disaster recovery device.

[0120] In one instance, the memory 7002 may be a read-only memory (ROM). In one instance, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0121] The memory 7002 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, generally, a memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this disclosure.

[0122] The processor 7001 reads and executes computer program instructions stored in memory 7002 to achieve... Figure 1 The video quality assessment method in the illustrated embodiment.

[0123] In one example, the computer device may also include a communication interface 7003 and a bus 7004. Wherein, as... Figure 5 As shown, the processor 7001, memory 7002, and communication interface 7003 are connected through bus 7004 and complete communication with each other.

[0124] The communication interface 7003 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0125] Bus 7004 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 7004 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0126] Furthermore, in conjunction with the video quality assessment methods described in the above embodiments, this application embodiment can provide a computer storage medium for implementation. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the video quality assessment methods described in the above embodiments.

[0127] This application also provides a computer program product, including a computer program, which, when executed, implements any of the video quality assessment methods described in the above embodiments.

[0128] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0129] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memory (ROM), flash memory, erasable read-only memory (EROM), floppy disks, compact disc read-only memory (CD-ROM), optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0130] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0131] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A video quality assessment method, characterized in that, include: Obtain the video data to be evaluated; Extract salient semantic information corresponding to the visual attention region in the video data to be evaluated, perform spatiotemporal modeling and context analysis on the salient semantic information, and obtain the salient features of the video data to be evaluated; Extract multi-scale image features from the video data to be evaluated; Extract the coding domain features of the video data to be evaluated; By fusing the saliency features, the multi-scale image features, and the coding domain features, a fused feature representation is obtained; The quality assessment result of the video data to be evaluated is obtained by using a pre-trained video quality assessment model to predict the fused feature representation.

2. The method according to claim 1, characterized in that, The extraction of salient semantic information corresponding to the visually attention-grabbing regions in the video data to be evaluated includes: The visual attention region in the video data to be evaluated is identified using a saliency detection network, and the corresponding saliency semantic information is generated. The saliency detection network is built based on the U2-Net architecture.

3. The method according to claim 1, characterized in that, The process of performing spatiotemporal modeling and contextual analysis on the salient semantic information to obtain the salient features of the video data to be evaluated includes: The salient semantic information is input as prior information into the image quality perception network; Using the image quality perception network, based on the attention mechanism, the dependency relationship between the visual attention area and the overall image is modeled globally, and the salient features are generated.

4. The method according to any one of claims 1 to 3, characterized in that, The extraction of multi-scale image features from the video data to be evaluated includes: Multiple small blocks are cropped from each frame of the video data to be evaluated, and the multiple small blocks are merged into a local image block, which is used to characterize the local feature information of each frame of the image; The size of each frame image is adjusted to match the size of the corresponding local image block to obtain a global image block, which is used to represent the global feature information of each frame image; The local image patch and the global image patch are input into the self-attention model for feature extraction to obtain the multi-scale image features of each frame.

5. The method according to any one of claims 1 to 3, characterized in that, The coding domain features include at least one of the following: average quantization parameter, motion amplitude parameter, frame rate, and bit rate.

6. The method according to any one of claims 1 to 3, characterized in that, The fusion of the saliency features, the multi-scale image features, and the coding domain features yields a fused feature representation, including: Based on preset weights, the saliency features, the multi-scale image features, and the coding domain features are linearly weighted and fused to obtain the fused feature representation.

7. The method according to any one of claims 1 to 3, characterized in that, Before acquiring the video data to be evaluated, the method further includes: Obtain video data samples and their quality labels; The saliency features, multi-scale image features, and coding domain features of the video data samples are extracted respectively, and then fused to obtain the sample fusion features; The video quality assessment model is used to analyze the sample fusion features to obtain quality prediction results; The model prediction loss is determined based on the difference between the quality prediction result and the quality label; The model parameters are updated by minimizing the model's prediction loss using an optimization algorithm; If the training stopping condition is met, the video quality assessment model is output.

8. The method according to claim 7, characterized in that, The acquisition of video data samples includes: Obtain raw video data; The original video data is subjected to quality transformation processing to generate a training dataset, which contains multiple video data samples; the quality transformation includes at least one of blur transformation, noise transformation, color transformation and encoding transformation.

9. A video quality assessment device, characterized in that, include: The acquisition module is used to acquire the video data to be evaluated. The first feature extraction module is used to extract salient semantic information corresponding to the visual attention region in the video data to be evaluated, perform spatiotemporal modeling and context analysis on the salient semantic information, and obtain the salient features of the video data to be evaluated. The second feature extraction module is used to extract multi-scale image features from the video data to be evaluated. The third feature extraction module is used to extract the coding domain features of the video data to be evaluated from the video coding domain. The feature fusion module is used to fuse the saliency features, the multi-scale image features, and the coding domain features to obtain a fused feature representation; The quality assessment module is used to predict the fused feature representation using a pre-trained video quality assessment model to obtain the quality assessment result of the video data to be assessed.

10. A computer device, characterized in that, The device includes: a processor and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the method as claimed in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, implement the method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of a computer device, the computer device causes the computer device to perform the method as described in any one of claims 1 to 8.

Images