Screen content video quality evaluation method and device based on layered spatiotemporal perception
Through a dual branch quality evaluation model based on hierarchical space-time perception, combined with airspace and time-domain evaluation, the problem of screen content video quality evaluation is solved, and the accurate evaluation and optimization of screen content video quality is achieved, thereby improving user experience.
Patent Information
- Application Number
- CN202411319739.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-23
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-09-23
AI Technical Summary
The prior art is difficult to effectively evaluate and optimize the visual quality of screen content videos, especially when noise is disturbed by different processing stages, affecting the user experience.
Using the screen content video quality evaluation method based on hierarchical space-time perception, a dual-branch quality evaluation model is constructed, combined with spatial and time-domain perception evaluation, using significance calculation, dual-channel convolutional neural network, spatial gate feature enhancement and self-direction Adapt weighting strategies, evaluate the quality of the video block and calculate the overall quality score of the screen content video.
This method can effectively simulate the perception of the video quality of screen content by the human eye visual system, reduce the impact of visual redundancy on the perceived quality, and improve the accuracy and reliability of video quality evaluation.
Smart Images

Figure CN118865075B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of video evaluation, and in particular to a method and device for evaluating screen content video quality based on layered spatiotemporal perception. Background Art
[0002] With the rapid development of artificial intelligence and digital multimedia technology and the popularity of various portable devices, screen content videos are increasingly used in digital multimedia application fields (such as online education, video conferencing, webcasting, game videos, etc.), providing users with a more flexible and free multimedia experience. Unlike natural videos, screen content videos are a mixture of images and computer-generated text / image areas.
[0003] In different processing stages of screen content videos (such as acquisition, transmission, display, etc.), they will inevitably be disturbed by various noises, resulting in varying degrees of degradation of perceived quality, which seriously affects the user experience. Therefore, it is urgent to develop an effective video quality assessment method to optimize the visual quality of screen content videos and improve the performance of related tasks. Summary of the invention
[0004] The purpose of this application is to propose a method and device for evaluating screen content video quality based on layered spatiotemporal perception in response to the above-mentioned technical problems.
[0005] In a first aspect, the present invention provides a method for evaluating screen content video quality based on layered spatiotemporal perception, characterized in that it comprises the following steps:
[0006] Acquire a screen content video, and extract a plurality of fragmented videos and a plurality of key frames from the screen content video;
[0007] A screen content video quality assessment model based on two branches is constructed and trained to obtain a trained screen content video quality assessment model, wherein the screen content video quality assessment model includes a spatial perception quality assessment branch and a temporal perception quality assessment branch, wherein the spatial perception quality assessment branch includes a saliency calculation module, a dual-channel convolutional neural network, a block-level quality assessment module and an adaptive weighting module, and the block-level quality assessment module includes a spatial gate feature enhancement module and a block-level quality aggregation module connected in sequence;
[0008] Several key frames and several fragmented videos are input into the trained screen content video quality evaluation model, each key frame is input into the saliency calculation module to calculate the average saliency value of each video block in the key frame, several salient video blocks are screened out according to the average saliency value, each salient video block is input into the dual-channel convolutional neural network to obtain the multi-layer features of each stage, the multi-layer features of each stage are input into the block-level quality evaluation module, after the spatial gate feature enhancement module, the enhanced features of each stage are obtained, the enhanced features of each stage are input into the block-level quality aggregation module to obtain the block-level quality score of each video block; the average saliency value and block-level quality score of each video block are input into the adaptive weighting module, the frame-level quality score corresponding to each key frame is calculated by using the adaptive weighting strategy, and the spatial domain perception quality score is calculated according to the frame-level quality scores corresponding to all key frames; several fragmented videos are input into the temporal domain perception quality evaluation branch to obtain the temporal domain perception quality score; the quality score of the screen content video is calculated according to the spatial domain perception quality score and the temporal domain perception quality score.
[0009] Preferably, each key frame is input into the saliency calculation module to calculate the average saliency value of each video block in the key frame, specifically including:
[0010] The motion blur convolution kernel is used to convolve each key frame to obtain the corresponding blurred key frame, as shown in the following formula:
[0011] ;
[0012] Among them, I represents the key frame, I m represents a blurred keyframe, represents the convolution operation, K m Represents the motion blur convolution kernel; the motion blur convolution kernel is expressed as follows:
[0013] ;
[0014] in, For exercise volume, is the direction of motion and its angle with the horizontal axis, represents the transpose operation, and Represents the motion blur kernel coordinates The corresponding value;
[0015] Based on the Sobel filter, the gradient value corresponding to the key frame of the coordinates of the pixel points located in the same video block and the gradient value corresponding to the blurred key frame are calculated, and the similarity between the gradient value corresponding to the key frame of the coordinates of the pixel points located in the same video block and the gradient value corresponding to the blurred key frame is calculated to obtain the first similarity, as shown in the following formula:
[0016] ;
[0017] ;
[0018] ;
[0019] in, represents the first similarity, Represents the coordinates of the pixel points in the video block, The coordinates of the pixel points in the key frame located in the video block are The gradient value of The coordinates of the pixel points in the video block in the blurred keyframe are The gradient value of and represents the horizontal Sobel filter and the vertical Sobel filter, is a positive constant;
[0020] The Gaussian convolution kernel is used to convolve each key frame to obtain the corresponding focused key frame, as shown in the following formula:
[0021] ;
[0022] in, represents a blurred keyframe, Represents the Gaussian convolution kernel, which is expressed as follows:
[0023] ;
[0024] in, and represents the Gaussian kernel coordinates The corresponding value, represents the standard deviation of the Gaussian kernel function;
[0025] The gradient value corresponding to the focused key frame is calculated based on the Sobel filter, and the similarity between the gradient value corresponding to the key frame of the coordinates of the pixel point located in the same video block and the gradient value corresponding to the blurred key frame is calculated to obtain the second similarity, as shown in the following formula:
[0026] ;
[0027] ;
[0028] in, The coordinates of the pixel points in the video block in the blurred keyframe are The gradient value of represents the second similarity, represents a positive constant;
[0029] The comprehensive significance value is calculated according to the first similarity and the second similarity, as shown in the following formula:
[0030] ;
[0031] in, A hyperparameter representing the ratio of fusion weights, The coordinates of the pixel points in the key frame located in the video block are The comprehensive significance value of
[0032] The average saliency value of each video block is calculated based on the comprehensive saliency values of all pixels in each video block of the key frame, as shown in the following formula:
[0033] ;
[0034] in, represents the average saliency value of the video block, and M is the total number of pixels in the video block.
[0035] Preferably, a number of significant video blocks are screened out according to the average significance value, and each significant video block is input into a dual-channel convolutional neural network to obtain multi-layer features at each stage, specifically including:
[0036] Sort the average saliency values of all video blocks in each key frame from large to small, and select the top N video blocks as salient video blocks;
[0037] The dual-channel convolutional neural network includes a first convolutional neural network and a second convolutional neural network. According to different block attributes, the salient video blocks are input into the first convolutional neural network and the second convolutional neural network respectively to obtain the nth salient video block. The multi-layer features of the stages are shown in the following formula:
[0038] ;
[0039] in, represents the nth salient video block, n=1,2,…,N, Represents the function corresponding to the first convolutional neural network or the second convolutional neural network. If The block attribute of is a text block, then is the function corresponding to the first convolutional neural network, if The block attribute of is an image block, then is the function corresponding to the second convolutional neural network, represents the multi-layer features of the kth stage of the nth salient video block, .
[0040] Preferably, the multi-layer features of each stage are input into the block-level quality assessment module, and the enhanced features of each stage are obtained through the spatial gate feature enhancement module. The enhanced features of each stage are input into the block-level quality aggregation module to obtain the block-level quality score of each video block, which specifically includes:
[0041] The multi-layer features of each stage are input into the spatial gate feature enhancement module to obtain the enhanced features of each stage. The spatial gate feature enhancement module includes an upsampling enhancement module and a downsampling enhancement module, as shown in the following formula:
[0042] ;
[0043] in, represents the upsampling enhancement module, Represents the downsampling enhancement module, and the upsampling enhancement module includes The first spatial gate sampling forward network, the downsampling enhancement module includes The second spatial gate sampling forward network, represents the r first spatial gate sampling forward network in the upsampling enhancement module, represents the r second spatial gate sampling forward network in the downsampling enhancement module, , represents the enhanced features of the kth stage of the nth salient video block;
[0044] The global average values of the enhanced features of all stages of the nth salient video block are calculated respectively, and input into the block-level quality aggregation module for serial aggregation calculation to obtain the block-level quality score, as shown in the following formula:
[0045] ;
[0046] in, represents a block-level quality aggregation module, which includes a first fully connected layer and a second fully connected layer connected in sequence. represents the first fully connected layer, represents the second fully connected layer, represents the enhanced features of the first stage of the nth salient video block, represents the enhanced features of the second stage of the nth salient video block, The nth salient video block The enhanced features of each stage, represents the splicing operation along the channel, represents the global average pooling operation, represents the block-level quality score of the nth salient video block, n=1,2,…,N.
[0047] Preferably, the first spatial gate sampling forward network includes a first depth-separable convolutional layer and a first convolutional layer, the step size of the first convolutional layer is 1, the convolution kernel size is 1×1, and the calculation process of each first spatial gate sampling forward network is shown in the following formula:
[0048] ;
[0049] ;
[0050] in, represents the function corresponding to the first separable convolutional layer, represents the function corresponding to the first convolutional layer, represents the channel split operation, represents the spatial sampling operation, represents element-wise multiplication, It indicates that the first input feature of the forward network is sampled by the first spatial gate. The first input feature of the forward network sampled by the rth first spatial gate is , It indicates that the second input feature of the first spatial gate sampling forward network that needs spatial sampling, the output feature of the previous first spatial gate sampling forward network is used as the second input feature of the next first spatial gate sampling forward network that needs spatial sampling, and the second input feature of the first first spatial gate sampling forward network that needs spatial sampling is , and express The first and second features obtained by cross-channel segmentation, represents the function corresponding to the first spatial gate sampling forward network, Represents the output features of the first spatial gate sampling forward network;
[0051] The second spatial gate sampling forward network includes a second depth-separable convolutional layer and a second convolutional layer. The step size of the second convolutional layer is 2, and the convolution kernel size is 1×1. The calculation process of each second spatial gate sampling forward network is shown in the following formula:
[0052] ;
[0053] ;
[0054] in, represents the function corresponding to the second separable convolutional layer, represents the function corresponding to the second convolutional layer, Indicates the third input feature of the second spatial gate sampling forward network. The third input feature of the rth second spatial gate sampling forward network is The first spatial gate samples the second input feature of the forward network that needs spatial sampling, Indicates the fourth input feature of the second spatial gate sampling forward network that needs spatial sampling, the output feature of the previous second spatial gate sampling forward network is used as the fourth input feature of the next second spatial gate sampling forward network that needs spatial sampling, and the fourth input feature of the first second spatial gate sampling forward network that needs spatial sampling is the fourth input feature of the first second spatial gate sampling forward network. The first spatial gate samples the output features of the forward network, and express The third and fourth features obtained by cross-channel segmentation, represents the function corresponding to the second spatial gate sampling forward network, represents the output feature of the second spatial gate sampling forward network. The output feature of the rth second spatial gate sampling forward network is , No. The output characteristics of the first spatial gate sampling forward network are .
[0055] Preferably, the average significance value and block-level quality score of each video block are input into an adaptive weighting module, and the frame-level quality score corresponding to each key frame is calculated using an adaptive weighting strategy. The spatial perceptual quality score is calculated based on the frame-level quality scores corresponding to all key frames; a plurality of fragmented videos are input into a temporal perceptual quality assessment branch to obtain a temporal perceptual quality score; the quality score of the screen content video is calculated based on the spatial perceptual quality score and the temporal perceptual quality score, specifically including:
[0056] The frame-level quality score corresponding to the t-th video frame is calculated using the following formula:
[0057] ;
[0058] in, represents the frame-level quality score corresponding to the t-th video frame, represents the average saliency value of the nth video block in the tth video frame, represents the block-level quality score of the nth video block in the tth video frame, t=1,2,…,L;
[0059] The spatial perception quality score is calculated using the following formula:
[0060] ;
[0061] in, represents the spatial perception quality score;
[0062] The temporal perceptual quality assessment branch uses a video swim transformer. Several fragmented videos are input into the video swim transformer to obtain a temporal perceptual quality score. ;
[0063] The quality score of the screen content video is calculated using the following formula:
[0064] ;
[0065] in, Indicates the quality score of the screen content video. Indicates the weight value.
[0066] In a second aspect, the present invention provides a device for evaluating screen content video quality based on layered spatiotemporal perception, comprising:
[0067] A data extraction module is configured to obtain a screen content video and extract a plurality of fragmented videos and a plurality of key frames from the screen content video;
[0068] A model building module is configured to build and train a screen content video quality assessment model based on two branches to obtain a trained screen content video quality assessment model, wherein the screen content video quality assessment model includes a spatial perception quality assessment branch and a temporal perception quality assessment branch, wherein the spatial perception quality assessment branch includes a saliency calculation module, a dual-channel convolutional neural network, a block-level quality assessment module and an adaptive weighting module, and the block-level quality assessment module includes a spatial gate feature enhancement module and a block-level quality aggregation module connected in sequence;
[0069] The prediction module is configured to input several key frames and several fragmented videos into the trained screen content video quality evaluation model, each key frame is input into the significance calculation module to calculate the average significance value of each video block in the key frame, several significant video blocks are screened out according to the average significance value, each significant video block is input into the dual-channel convolutional neural network to obtain the multi-layer features of each stage, the multi-layer features of each stage are input into the block-level quality evaluation module, and the enhanced features of each stage are obtained through the spatial gate feature enhancement module, and the enhanced features of each stage are input into the block-level quality aggregation module to obtain the block-level quality score of each video block; the average significance value and block-level quality score of each video block are input into the adaptive weighting module, and the frame-level quality score corresponding to each key frame is calculated by using the adaptive weighting strategy, and the spatial domain perception quality score is calculated according to the frame-level quality scores corresponding to all key frames; several fragmented videos are input into the temporal domain perception quality evaluation branch to obtain the temporal domain perception quality score; the quality score of the screen content video is calculated according to the spatial domain perception quality score and the temporal domain perception quality score.
[0070] In a third aspect, the present invention provides an electronic device comprising one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.
[0071] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.
[0072] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.
[0073] Compared with the prior art, the present invention has the following beneficial effects:
[0074] (1) The screen content video quality assessment method based on layered spatiotemporal perception proposed in the present invention adopts a weighting strategy based on an adaptive fuzzy focusing visual mechanism. This strategy promotes quality aggregation from block level to frame level, models the perceptual differences of the human visual system in different attribute areas, and reduces the impact of visual redundancy on perceptual quality assessment.
[0075] (2) The spatial gate feature enhancement module in the screen content video quality assessment method based on layered spatiotemporal perception proposed in the present invention performs upsampling enhancement and downsampling enhancement through an upsampling enhancement module and a downsampling enhancement module respectively, so as to conveniently express the respective features of different attribute blocks while reducing redundancy.
[0076] (3) The screen content video quality assessment method based on layered spatiotemporal perception proposed in the present invention makes full use of the quality-related features in multiple video levels to perform multi-level video quality perception, and realizes quality assessment of distorted screen content videos through a reliable perception fusion method, which can effectively simulate the perception of the screen content video quality by the human visual system. BRIEF DESCRIPTION OF THE DRAWINGS
[0077] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0078] Figure 1 A schematic diagram of a flow chart of a method for evaluating screen content video quality based on layered spatiotemporal perception according to an embodiment of the present application;
[0079] Figure 2A schematic diagram of the structure of a screen content video quality assessment model of a screen content video quality assessment method based on layered spatiotemporal perception according to an embodiment of the present application;
[0080] Figure 3 A schematic diagram of the structure of a spatial gate feature enhancement module of a screen content video quality assessment method based on layered spatiotemporal perception according to an embodiment of the present application;
[0081] Figure 4 This is a structural schematic diagram of an upsampling enhancement module of a screen content video quality assessment method based on layered spatiotemporal perception according to an embodiment of the present application;
[0082] Figure 5 This is a structural schematic diagram of a downsampling enhancement module of a screen content video quality assessment method based on layered spatiotemporal perception according to an embodiment of the present application;
[0083] Figure 6 A schematic diagram of a screen content video quality assessment device based on layered spatiotemporal perception according to an embodiment of the present application;
[0084] Figure 7 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0085] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0086] Figure 1 A method for evaluating screen content video quality based on layered spatiotemporal perception provided by an embodiment of the present application is shown, comprising the following steps:
[0087] S1, obtaining a screen content video, and extracting a plurality of fragmented videos and a plurality of key frames from the screen content video.
[0088] Specifically, each video frame in the screen content video is divided into several grids, several video blocks are selected from each grid, and they are spliced into fragmented video frames according to the original positions, and each fragmented video frame is spliced into a fragmented video. In one embodiment, a video frame in the screen content video is divided into 7×7 grids, and a video block with a pixel size of 32×32 is selected from each grid, and finally spliced into a video frame with a pixel size of 224×224 according to the original position. In other embodiments, the number of divided grids and the size of the selected video blocks can be selected according to requirements. All key frames in the video frames in the screen content video are extracted.
[0089] S2, construct and train a screen content video quality evaluation model based on two branches to obtain a trained screen content video quality evaluation model, wherein the screen content video quality evaluation model includes a spatial perception quality evaluation branch and a temporal perception quality evaluation branch, wherein the spatial perception quality evaluation branch includes a saliency calculation module, a dual-channel convolutional neural network, a block-level quality evaluation module and an adaptive weighting module, and the block-level quality evaluation module includes a spatial gate feature enhancement module and a block-level quality aggregation module connected in sequence.
[0090] Specifically, refer to Figure 2 The screen content video quality evaluation model proposed in the embodiment of the present application has two branches, namely, a spatial perception quality evaluation branch and a temporal perception quality evaluation branch, which are respectively used to calculate the spatial perception quality score and the temporal perception quality score. First, more prominent salient video blocks are selected from key frames using visual saliency, and the salient video blocks are classified according to different block attributes to obtain text blocks and image blocks. Corresponding convolutional neural networks are respectively constructed for the text blocks and image blocks to obtain a dual-channel convolutional neural network. A block-level quality evaluation module including an integrated spatial gate feature enhancement module is designed to evaluate the quality of video blocks with different block attributes at the block level. Combined with an adaptive weighting strategy based on a visual mechanism, the block-level quality scores are aggregated into frame-level quality scores, and pooled to obtain the spatial perception quality score, which is directly combined with the temporal perception quality score predicted by the temporal perception quality evaluation branch, thereby effectively realizing the quality evaluation of the screen content video.
[0091] S3, input several key frames and several fragmented videos into the trained screen content video quality evaluation model, input each key frame into the saliency calculation module to calculate the average saliency value of each video block in the key frame, select several salient video blocks according to the average saliency value, input each salient video block into the dual-channel convolutional neural network to obtain the multi-layer features of each stage, input the multi-layer features of each stage into the block-level quality evaluation module, pass through the spatial gate feature enhancement module to obtain the enhanced features of each stage, input the enhanced features of each stage into the block-level quality aggregation module to obtain the block-level quality score of each video block; input the average saliency value and block-level quality score of each video block into the adaptive weighting module, adopt the adaptive weighting strategy to calculate the frame-level quality score corresponding to each key frame, calculate the spatial domain perception quality score according to the frame-level quality scores corresponding to all key frames; input several fragmented videos into the temporal domain perception quality evaluation branch to obtain the temporal domain perception quality score; calculate the quality score of the screen content video according to the spatial domain perception quality score and the temporal domain perception quality score.
[0092] In a specific embodiment, each key frame is input into the saliency calculation module to calculate the average saliency value of each video block in the key frame, specifically including:
[0093] The motion blur convolution kernel is used to convolve each key frame to obtain the corresponding blurred key frame, as shown in the following formula:
[0094] ;
[0095] Among them, I represents the key frame, I m represents a blurred keyframe, represents the convolution operation, K m Represents the motion blur convolution kernel; the motion blur convolution kernel is expressed as follows:
[0096] ;
[0097] in, For exercise volume, is the direction of motion and its angle with the horizontal axis, represents the transpose operation, and Represents the motion blur kernel coordinates The corresponding value;
[0098] Based on the Sobel filter, the gradient value corresponding to the key frame of the coordinates of the pixel points located in the same video block and the gradient value corresponding to the blurred key frame are calculated, and the similarity between the gradient value corresponding to the key frame of the coordinates of the pixel points located in the same video block and the gradient value corresponding to the blurred key frame is calculated to obtain the first similarity, as shown in the following formula:
[0099] ;
[0100] ;
[0101] ;
[0102] in, represents the first similarity, Represents the coordinates of the pixel points in the video block, The coordinates of the pixel points in the key frame located in the video block are The gradient value of The coordinates of the pixel points in the video block in the blurred keyframe are The gradient value of and represents the horizontal Sobel filter and the vertical Sobel filter, is a positive constant;
[0103] The Gaussian convolution kernel is used to convolve each key frame to obtain the corresponding focused key frame, as shown in the following formula:
[0104] ;
[0105] in, represents a blurred keyframe, Represents the Gaussian convolution kernel, which is expressed as follows:
[0106] ;
[0107] in, and represents the Gaussian kernel coordinates The corresponding value, represents the standard deviation of the Gaussian kernel function;
[0108] The gradient value corresponding to the focused key frame is calculated based on the Sobel filter, and the similarity between the gradient value corresponding to the key frame of the coordinates of the pixel point located in the same video block and the gradient value corresponding to the blurred key frame is calculated to obtain the second similarity, as shown in the following formula:
[0109] ;
[0110] ;
[0111] in, The coordinates of the pixel points in the video block in the blurred keyframe are The gradient value of represents the second similarity, represents a positive constant;
[0112] The comprehensive significance value is calculated according to the first similarity and the second similarity, as shown in the following formula:
[0113] ;
[0114] in, A hyperparameter representing the ratio of fusion weights, The coordinates of the pixel points in the key frame located in the video block are The comprehensive significance value of
[0115] The average saliency value of each video block is calculated based on the comprehensive saliency values of all pixels in each video block of the key frame, as shown in the following formula:
[0116] ;
[0117] in, represents the average saliency value of the video block, and M is the total number of pixels in the video block.
[0118] In a specific embodiment, a number of significant video blocks are screened out according to the average significance value, and each significant video block is input into a dual-channel convolutional neural network to obtain multi-layer features at each stage, specifically including:
[0119] Sort the average saliency values of all video blocks in each key frame from large to small, and select the top N video blocks as salient video blocks;
[0120] The dual-channel convolutional neural network includes a first convolutional neural network and a second convolutional neural network. According to different block attributes, the salient video blocks are input into the first convolutional neural network and the second convolutional neural network respectively to obtain the nth salient video block. The multi-layer features of the stages are shown in the following formula:
[0121] ;
[0122] in, represents the nth salient video block, n=1,2,…,N, Represents the function corresponding to the first convolutional neural network or the second convolutional neural network. If The block attribute of is a text block, then is the function corresponding to the first convolutional neural network, if The block attribute of is an image block, then is the function corresponding to the second convolutional neural network, represents the multi-layer features of the kth stage of the nth salient video block, .
[0123] In a specific embodiment, the multi-layer features of each stage are input into the block-level quality assessment module, and the enhanced features of each stage are obtained through the spatial gate feature enhancement module. The enhanced features of each stage are input into the block-level quality aggregation module to obtain the block-level quality score of each video block, which specifically includes:
[0124] The multi-layer features of each stage are input into the spatial gate feature enhancement module to obtain the enhanced features of each stage. The spatial gate feature enhancement module includes an upsampling enhancement module and a downsampling enhancement module, as shown in the following formula:
[0125] ;
[0126] in, represents the upsampling enhancement module, Represents the downsampling enhancement module, and the upsampling enhancement module includes The first spatial gate sampling forward network, the downsampling enhancement module includes The second spatial gate sampling forward network, represents the r first spatial gate sampling forward network in the upsampling enhancement module, represents the r second spatial gate sampling forward network in the downsampling enhancement module, , represents the enhanced features of the kth stage of the nth salient video block;
[0127] The global average values of the enhanced features of all stages of the nth salient video block are calculated respectively, and input into the block-level quality aggregation module for serial aggregation calculation to obtain the block-level quality score, as shown in the following formula:
[0128] ;
[0129] in, represents a block-level quality aggregation module, which includes a first fully connected layer and a second fully connected layer connected in sequence. represents the first fully connected layer, represents the second fully connected layer, represents the enhanced features of the first stage of the nth salient video block, represents the enhanced features of the second stage of the nth salient video block, The nth salient video block The enhanced features of each stage, represents the splicing operation along the channel, represents the global average pooling operation, represents the block-level quality score of the nth salient video block, n=1,2,…,N.
[0130] In a specific embodiment, the first spatial gate sampling forward network includes a first depth-separable convolutional layer and a first convolutional layer, the step size of the first convolutional layer is 1, the convolution kernel size is 1×1, and the calculation process of each first spatial gate sampling forward network is shown in the following formula:
[0131] ;
[0132] ;
[0133] in, represents the function corresponding to the first separable convolutional layer, represents the function corresponding to the first convolutional layer, represents the channel split operation, represents the spatial sampling operation, represents element-wise multiplication, It indicates that the first input feature of the forward network is sampled by the first spatial gate. The first input feature of the forward network sampled by the rth first spatial gate is , It indicates that the second input feature of the first spatial gate sampling forward network that needs spatial sampling, the output feature of the previous first spatial gate sampling forward network is used as the second input feature of the next first spatial gate sampling forward network that needs spatial sampling, and the second input feature of the first first spatial gate sampling forward network that needs spatial sampling is , and express The first and second features obtained by cross-channel segmentation, represents the function corresponding to the first spatial gate sampling forward network, Represents the output features of the first spatial gate sampling forward network;
[0134] The second spatial gate sampling forward network includes a second depth-separable convolutional layer and a second convolutional layer. The step size of the second convolutional layer is 2, and the convolution kernel size is 1×1. The calculation process of each second spatial gate sampling forward network is shown in the following formula:
[0135] ;
[0136] ;
[0137] in, represents the function corresponding to the second separable convolutional layer, represents the function corresponding to the second convolutional layer, Indicates the third input feature of the second spatial gate sampling forward network. The third input feature of the rth second spatial gate sampling forward network is The first spatial gate samples the second input feature of the forward network that needs spatial sampling, Indicates the fourth input feature of the second spatial gate sampling forward network that needs spatial sampling, the output feature of the previous second spatial gate sampling forward network is used as the fourth input feature of the next second spatial gate sampling forward network that needs spatial sampling, and the fourth input feature of the first second spatial gate sampling forward network that needs spatial sampling is the fourth input feature of the first second spatial gate sampling forward network. The first spatial gate samples the output features of the forward network, and express The third and fourth features obtained by cross-channel segmentation, represents the function corresponding to the second spatial gate sampling forward network, represents the output feature of the second spatial gate sampling forward network. The output feature of the rth second spatial gate sampling forward network is , No. The output characteristics of the first spatial gate sampling forward network are .
[0138] In a specific embodiment, the average significance value and block-level quality score of each video block are input into an adaptive weighting module, and the frame-level quality score corresponding to each key frame is calculated using an adaptive weighting strategy. The spatial perceptual quality score is calculated based on the frame-level quality scores corresponding to all key frames; a plurality of fragmented videos are input into a temporal perceptual quality evaluation branch to obtain a temporal perceptual quality score; the quality score of the screen content video is calculated based on the spatial perceptual quality score and the temporal perceptual quality score, specifically including:
[0139] The frame-level quality score corresponding to the t-th video frame is calculated using the following formula:
[0140] ;
[0141] in, represents the frame-level quality score corresponding to the t-th video frame, represents the average saliency value of the nth video block in the tth video frame, represents the block-level quality score of the nth video block in the tth video frame, t=1,2,…,L;
[0142] The spatial perception quality score is calculated using the following formula:
[0143] ;
[0144] in, represents the spatial perception quality score;
[0145] The temporal perceptual quality assessment branch uses a video swim transformer. Several fragmented videos are input into the video swim transformer to obtain a temporal perceptual quality score. ;
[0146] The quality score of the screen content video is calculated using the following formula:
[0147] ;
[0148] in, Indicates the quality score of the screen content video. Indicates the weight value.
[0149] Specifically, the average significance value is calculated in the video block corresponding to the 32×32 size area of the key frame. The significance is calculated based on the gradient similarity between the key frame and the blurred key frame simulating the blur effect, that is, the significance of the "blurred" effect is characterized by the first similarity, and the significance is calculated based on the gradient similarity between the key frame and the focused key frame simulating the focus effect, that is, the significance of the "focus" effect is characterized by the second similarity. Finally, a comprehensive significance value is calculated based on the first similarity and the second similarity, and the average significance value of the video block is calculated based on the comprehensive significance value. In one embodiment, the average significance values of the video blocks are sorted from large to small, and the video blocks ranked in the top 60% are selected to form a significant video block set. The significant video blocks in the significant video block set are , where N is equal to 60% of the total number of blocks, According to the block attributes (text block and image block), feed it to the corresponding convolutional neural network In, get Multi-layer features at each stage . In other embodiments, the total number of significant video blocks can be selected according to specific circumstances. The multi-layer features of each stage are input into the spatial gate feature enhancement module, which includes a spatial gate sampling forward network (SSFN). The SSFN is mainly composed of a deep separable convolution layer (DWConv) and a convolution layer with a convolution kernel size of 1×1. The enhanced features of each stage are obtained through the spatial gate sampling forward network (SSFN). The global average of the enhanced features of all stages is calculated based on the enhanced features of each stage, and the enhanced features are input into the block-level quality aggregation module for serial aggregation calculation to obtain the block-level quality score. The block-level quality score of each significant video block in each significant key frame of the screen content video is calculated respectively, and the average significance value of each significant video block is used as the weight. The frame-level quality score of each key frame is calculated using an adaptive weighting strategy, and the average frame-level quality score of L key frames is further calculated as the spatial domain perception quality score. The block-level quality aggregation module consists of two fully connected layers.
[0150] For further reference, Figure 3-5 The spatial gate feature enhancement module includes an upsampling enhancement module and a downsampling enhancement module. The upsampling enhancement module and the downsampling enhancement module respectively contain corresponding spatial gate sampling forward networks (SSFNs), namely the first spatial gate sampling forward network and the second spatial gate sampling forward network. The first spatial gate sampling forward network and the second spatial gate sampling forward network serve as basic units in the upsampling enhancement module and the downsampling enhancement module, and perform upsampling enhancement and downsampling enhancement on the multi-layer features of each stage respectively.
[0151] In addition, the temporal perceptual quality assessment branch uses a video swim transformer, which is an existing network structure and its specific details are not repeated here. The fragmented video is input into the video swim transformer to obtain the temporal perceptual quality score, and the spatial perceptual quality score and the temporal perceptual quality score are weighted to obtain the quality score of the screen content video.
[0152] Further references Figure 6 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a screen content video quality evaluation device based on layered spatiotemporal perception, and the device embodiment is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0153] The embodiment of the present application provides a screen content video quality assessment device based on layered spatiotemporal perception, comprising:
[0154] The data extraction module 1 is configured to obtain a screen content video and extract a plurality of fragmented videos and a plurality of key frames from the screen content video;
[0155] Model construction module 2 is configured to construct and train a screen content video quality assessment model based on two branches to obtain a trained screen content video quality assessment model, wherein the screen content video quality assessment model includes a spatial perception quality assessment branch and a temporal perception quality assessment branch, wherein the spatial perception quality assessment branch includes a saliency calculation module, a dual-channel convolutional neural network, a block-level quality assessment module and an adaptive weighting module, and the block-level quality assessment module includes a spatial gate feature enhancement module and a block-level quality aggregation module connected in sequence;
[0156] The prediction module 3 is configured to input several key frames and several fragmented videos into the trained screen content video quality evaluation model, each key frame is input into the significance calculation module to calculate the average significance value of each video block in the key frame, and several significant video blocks are screened out according to the average significance value, and each significant video block is input into the dual-channel convolutional neural network to obtain the multi-layer features of each stage, and the multi-layer features of each stage are input into the block-level quality evaluation module, and the enhanced features of each stage are obtained through the spatial gate feature enhancement module, and the enhanced features of each stage are input into the block-level quality aggregation module to obtain the block-level quality score of each video block; the average significance value and block-level quality score of each video block are input into the adaptive weighting module, and the frame-level quality score corresponding to each key frame is calculated by using the adaptive weighting strategy, and the spatial domain perception quality score is calculated according to the frame-level quality scores corresponding to all key frames; several fragmented videos are input into the temporal domain perception quality evaluation branch to obtain the temporal domain perception quality score; the quality score of the screen content video is calculated according to the spatial domain perception quality score and the temporal domain perception quality score.
[0157] Figure 7 Schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention. Figure 7 As shown, the electronic device of this embodiment includes: a processor 701 and a memory 702; wherein the memory 702 is used to store computer-executable instructions; the processor 701 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant description in the above method embodiment.
[0158] Optionally, the memory 702 may be independent or integrated with the processor 701 .
[0159] When the memory 702 is independently provided, the electronic device further includes a bus 703 for connecting the memory 702 and the processor 701 .
[0160] The embodiment of the present invention further provides a computer storage medium, in which computer execution instructions are stored. When the processor 701 executes the computer execution instructions, the above method is implemented.
[0161] The embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by the processor 701, the above method is implemented.
[0162] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.
[0163] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.
[0164] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each module may exist physically separately, or two or more modules may be integrated into one unit. The unit formed by the above modules may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0165] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor 701 to perform some steps of the methods of various embodiments of the present application.
[0166] It should be understood that the processor 701 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or the processor 701 may be any conventional processor 701, etc. The steps of the method disclosed in the invention may be directly embodied in the execution of the hardware processor 701, or may be executed by a combination of hardware and software modules in the processor 701.
[0167] The memory 702 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disk.
[0168] The bus 703 may be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 703 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus 703 in the drawings of the present application is not limited to only one bus 703 or one type of bus 703.
[0169] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.
[0170] An exemplary storage medium is coupled to the processor 701, so that the processor 701 can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor 701. The processor 701 and the storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as ASIC). Of course, the processor 701 and the storage medium can also exist as discrete components in an electronic device or a main control device.
[0171] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.
[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for evaluating screen content video quality based on layered spatiotemporal perception, characterized in that: The following steps are involved: Acquire a screen content video, and extract a plurality of fragmented videos and a plurality of key frames from the screen content video; A screen content video quality assessment model based on two branches is constructed and trained to obtain a trained screen content video quality assessment model, wherein the screen content video quality assessment model includes a spatial perception quality assessment branch and a temporal perception quality assessment branch, wherein the spatial perception quality assessment branch includes a saliency calculation module, a dual-channel convolutional neural network, a block-level quality assessment module and an adaptive weighting module, and wherein the block-level quality assessment module includes a spatial gate feature enhancement module and a block-level quality aggregation module connected in sequence; Input the several key frames and several fragmented videos into the trained screen content video quality evaluation model, input each key frame into the saliency calculation module to calculate the average saliency value of each video block in the key frame, select several saliency video blocks according to the average saliency value, input each saliency video block into the dual-channel convolutional neural network to obtain multi-layer features of each stage, input the multi-layer features of each stage into the block-level quality evaluation module, pass through the spatial gate feature enhancement module to obtain enhanced features of each stage, input the enhanced features of each stage into the block-level quality aggregation module to obtain the block-level quality score of each video block, specifically including: The multi-layer features of each stage are input into the spatial gate feature enhancement module to obtain enhanced features of each stage. The spatial gate feature enhancement module includes an upsampling enhancement module and a downsampling enhancement module, as shown in the following formula: Wherein, USE(·) represents an upsampling enhancement module, DSE(·) represents a downsampling enhancement module, and the upsampling enhancement module includes N s -1 first spatial gate sampling forward network, the downsampling enhancement module includes N s -1 second spatial gate sampling forward network, represents the r first spatial gate sampling forward network in the upsampling enhancement module, represents the r second spatial gate sampling forward network in the downsampling enhancement module, r = 1, 2, ..., N s -1, represents the enhanced features of the kth stage of the nth salient video block, represents the multi-layer features of the kth stage of the nth salient video block, k = 1, ..., N s ; The global average values of the enhanced features of all stages of the nth salient video block are calculated respectively, and input into the block-level quality aggregation module for serial aggregation calculation to obtain the block-level quality score, as shown in the following formula: Wherein, PQAM(·) represents a block-level quality aggregation module, and the block-level quality aggregation module includes a first fully connected layer and a second fully connected layer connected in sequence, FC1(·) represents the first fully connected layer, FC2(·) represents the second fully connected layer, represents the enhanced features of the first stage of the nth salient video block, represents the enhanced features of the second stage of the nth salient video block, The Nth salient video block s The enhanced features of each stage, represents the concatenation operation along the channel, ρ(·) represents the global average pooling operation, Q n represents the block-level quality score of the nth significant video block, n=1,2,...,N; inputting the average significance value and block-level quality score of each video block into the adaptive weighting module, using the adaptive weighting strategy to calculate the frame-level quality score corresponding to each key frame, and calculating the spatial domain perceptual quality score according to the frame-level quality scores corresponding to all key frames; the plurality of fragmented videos are input into the temporal domain perceptual quality assessment branch to obtain the temporal domain perceptual quality score; calculating the quality score of the screen content video according to the spatial domain perceptual quality score and the temporal domain perceptual quality score, specifically including: The frame-level quality score corresponding to the t-th video frame is calculated using the following formula: in, represents the frame-level quality score corresponding to the t-th video frame, represents the average saliency value of the nth salient video block in the tth video frame, represents the block-level quality score of the n-th salient video block in the t-th video frame, t=1, 2, ..., L; The spatial perception quality score is calculated using the following formula: Among them, Q s represents the spatial perception quality score; The temporal perceptual quality assessment branch uses a video swim transformer, and the plurality of fragmented videos are input into the video swim transformer to obtain a temporal perceptual quality score Q d ; The quality score of the screen content video is calculated using the following formula: Q sc =λ·Q s +(1-λ)·Q d ; Among them, Q sc represents the quality score of the screen content video, and λ represents the weight value.
2. The screen content video quality assessment method based on layered spatiotemporal perception according to claim 1 is characterized in that: Each key frame is input into the saliency calculation module to calculate the average saliency value of each video block in the key frame, specifically including: The motion blur convolution kernel is used to convolve each key frame to obtain the corresponding blurred key frame, as shown in the following formula: Among them, I represents the key frame, I m represents a blurred keyframe, represents the convolution operation, K m Represents the motion blur convolution kernel; the motion blur convolution kernel is expressed as follows: Among them, φ is the amount of motion, θ is the direction of motion and its angle with the horizontal axis, T represents the transposition operation, and x1 and y1 represent the values corresponding to the motion blur kernel coordinates (x1, y1); The gradient value corresponding to the key frame and the gradient value corresponding to the blurred key frame of the coordinates of the pixel points located in the same video block are calculated based on the Sobel filter, and the similarity between the gradient value corresponding to the key frame and the gradient value corresponding to the blurred key frame of the coordinates of the pixel points located in the same video block is calculated to obtain a first similarity, as shown in the following formula: Among them, Sim M (x, y) represents the first similarity, (x, y) represents the coordinates of the pixel point in the video block, G I (x, y) represents the gradient value of the pixel point with coordinates (x, y) in the video block in the key frame, represents the gradient value of the pixel point in the video block with coordinates (x, y) in the blurred key frame, S x and S y represents horizontal Sobel filter and vertical Sobel filter, c1 is a positive constant; The Gaussian convolution kernel is used to convolve each key frame to obtain the corresponding focused key frame, as shown in the following formula: Among them, I g Indicates the focus keyframe, K g Represents the Gaussian convolution kernel, which is expressed as follows: Among them, x2 and y2 represent the values corresponding to the Gaussian kernel coordinates (x2, y2), and σ represents the standard deviation of the Gaussian kernel function; The gradient value corresponding to the focused key frame is calculated based on the Sobel filter, and the similarity between the gradient value corresponding to the key frame and the gradient value corresponding to the focused key frame of the coordinates of the pixel point located in the same video block is calculated to obtain a second similarity, as shown in the following formula: in, represents the gradient value of the pixel point in the video block with coordinates (x, y) in the focus key frame, Sim G (x, y) represents the second similarity, and c2 represents a positive constant; The comprehensive significance value is calculated according to the first similarity and the second similarity, as shown in the following formula: S(x,y)=α·(1-Sim M (x,y))+(1-α)·(1-Sim G (x,y)); Wherein, α represents a hyperparameter of the fusion weight ratio, and S(x, y) represents the comprehensive significance value of the pixel point with coordinates (x, y) in the video block in the key frame; The average saliency value of each video block is calculated according to the comprehensive saliency values of all pixels in each video block of the key frame, as shown in the following formula: in, represents the average saliency value of the video block, and M is the total number of pixels in the video block.
3. The screen content video quality assessment method based on layered spatiotemporal perception according to claim 1 is characterized in that: According to the average significance value, a number of significant video blocks are screened out, and each significant video block is input into the dual-channel convolutional neural network to obtain multi-layer features of each stage, specifically including: Sort the average saliency values of all video blocks in each key frame from large to small, and take the top N video blocks as salient video blocks; The dual-channel convolutional neural network includes a first convolutional neural network and a second convolutional neural network. The salient video blocks are respectively input into the first convolutional neural network and the second convolutional neural network according to different block attributes to obtain N of the nth salient video blocks. s The multi-layer features of the stages are shown in the following formula: Among them, P n represents the nth salient video block, n = 1, 2, ..., N, if P n The block attribute of is a text block, then is the function corresponding to the first convolutional neural network, if P n The block attribute of is an image block, then is the function corresponding to the second convolutional neural network.
4. The screen content video quality assessment method based on layered spatiotemporal perception according to claim 1, characterized in that: The first spatial gate sampling forward network includes a first depth-separable convolutional layer and a first convolutional layer. The step size of the first convolutional layer is 1, and the convolution kernel size is 1×1. The calculation process of each first spatial gate sampling forward network is shown in the following formula: in, represents the function corresponding to the first separable convolutional layer, represents the function corresponding to the first convolutional layer, represents the channel splitting operation, ψ(·) represents the spatial sampling operation, ⊙ represents the element multiplication, and F iu It indicates that the first input feature of the forward network is sampled by the first spatial gate. The first input feature of the forward network sampled by the rth first spatial gate is It indicates that the second input feature of the first spatial gate sampling forward network that needs spatial sampling, the output feature of the previous first spatial gate sampling forward network is used as the second input feature of the next first spatial gate sampling forward network that needs spatial sampling, and the second input feature of the first first spatial gate sampling forward network that needs spatial sampling is and express The first and second features obtained by cross-channel segmentation, SSFN up (·) represents the function corresponding to the first spatial gate sampling forward network, F ou Represents the output features of the first spatial gate sampling forward network; The second spatial gate sampling forward network includes a second depth-separable convolutional layer and a second convolutional layer. The step size of the second convolutional layer is 2, and the convolution kernel size is 1×1. The calculation process of each second spatial gate sampling forward network is shown in the following formula: in, represents the function corresponding to the second separable convolutional layer, Represents the function corresponding to the second convolutional layer, F id Indicates the third input feature of the second spatial gate sampling forward network. The third input feature of the rth second spatial gate sampling forward network is the Nth s -r first spatial gates sample the second input features of the forward network that need spatial sampling, Indicates that the fourth input feature of the second spatial gate sampling forward network that needs spatial sampling, the output feature of the previous second spatial gate sampling forward network is used as the fourth input feature of the next second spatial gate sampling forward network that needs spatial sampling, and the fourth input feature of the first second spatial gate sampling forward network that needs spatial sampling is the Nth s -1 first spatial gate samples the output features of the forward network, and express The third and fourth features obtained by cross-channel segmentation, SSFN down (·) represents the function corresponding to the second spatial gate sampling forward network, F od represents the output feature of the second spatial gate sampling forward network. The output feature of the rth second spatial gate sampling forward network is Nth s -1 The output characteristics of the first spatial gate sampling forward network are 5. A screen content video quality assessment device based on layered spatiotemporal perception, characterized in that: include: A data extraction module is configured to obtain a screen content video and extract a plurality of fragmented videos and a plurality of key frames from the screen content video; A model building module is configured to build and train a screen content video quality assessment model based on two branches to obtain a trained screen content video quality assessment model, wherein the screen content video quality assessment model includes a spatial perception quality assessment branch and a temporal perception quality assessment branch, wherein the spatial perception quality assessment branch includes a saliency calculation module, a dual-channel convolutional neural network, a block-level quality assessment module and an adaptive weighting module, and wherein the block-level quality assessment module includes a spatial gate feature enhancement module and a block-level quality aggregation module connected in sequence; The prediction module is configured to input the several key frames and the several fragmented videos into the trained screen content video quality evaluation model, each key frame is input into the saliency calculation module to calculate the average saliency value of each video block in the key frame, and several saliency video blocks are screened out according to the average saliency value, each saliency video block is input into the dual-channel convolutional neural network to obtain multi-layer features of each stage, the multi-layer features of each stage are input into the block-level quality assessment module, and the enhanced features of each stage are obtained through the spatial gate feature enhancement module, and the enhanced features of each stage are input into the block-level quality aggregation module to obtain the block-level quality score of each video block, specifically including: The multi-layer features of each stage are input into the spatial gate feature enhancement module to obtain enhanced features of each stage. The spatial gate feature enhancement module includes an upsampling enhancement module and a downsampling enhancement module, as shown in the following formula: Wherein, USE(·) represents an upsampling enhancement module, DSE(·) represents a downsampling enhancement module, and the upsampling enhancement module includes N s -1 first spatial gate sampling forward network, the downsampling enhancement module includes N s -1 second spatial gate sampling forward network, represents the r first spatial gate sampling forward network in the upsampling enhancement module, represents the r second spatial gate sampling forward network in the downsampling enhancement module, r = 1, 2, ..., N s -1, represents the enhanced features of the kth stage of the nth salient video block, represents the multi-layer features of the kth stage of the nth salient video block, k = 1, ..., N s ; The global average values of the enhanced features of all stages of the nth salient video block are calculated respectively, and input into the block-level quality aggregation module for serial aggregation calculation to obtain the block-level quality score, as shown in the following formula: Wherein, PQAM(·) represents a block-level quality aggregation module, and the block-level quality aggregation module includes a first fully connected layer and a second fully connected layer connected in sequence, FC1(·) represents the first fully connected layer, FC2(·) represents the second fully connected layer, represents the enhanced features of the first stage of the nth salient video block, represents the enhanced features of the second stage of the nth salient video block, The Nth salient video block s The enhanced features of each stage, represents the concatenation operation along the channel, ρ(·) represents the global average pooling operation, Q n represents the block-level quality score of the nth significant video block, n=1,2,...,N; inputting the average significance value and block-level quality score of each video block into the adaptive weighting module, using the adaptive weighting strategy to calculate the frame-level quality score corresponding to each key frame, and calculating the spatial domain perceptual quality score according to the frame-level quality scores corresponding to all key frames; the plurality of fragmented videos are input into the temporal domain perceptual quality assessment branch to obtain the temporal domain perceptual quality score; calculating the quality score of the screen content video according to the spatial domain perceptual quality score and the temporal domain perceptual quality score, specifically including: The frame-level quality score corresponding to the t-th video frame is calculated using the following formula: in, represents the frame-level quality score corresponding to the t-th video frame, represents the average saliency value of the nth salient video block in the tth video frame, represents the block-level quality score of the nth salient video block in the tth video frame, t=1,2,...,L; The spatial perception quality score is calculated using the following formula: Among them, Q s represents the spatial perception quality score; The temporal perceptual quality assessment branch uses a video swim transformer, and the plurality of fragmented videos are input into the video swim transformer to obtain a temporal perceptual quality score Q d ; The quality score of the screen content video is calculated using the following formula: Q sc =λ·Q s +(1-λ)·Q d ; Among them, Q sc represents the quality score of the screen content video, and λ represents the weight value.
6. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 4.
7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.
8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.
Citation Information
Patent Citations
Urban tree detection method based on improved bidirectional weighted feature pyramid
CN116259047A
Underground mine abnormal action recognition method based on video understanding
CN116798117A
