Screen content video quality evaluation method and device based on attention guidance

Abstract

The present invention discloses a method and device for evaluating the quality of screen content video based on attention guidance, which relates to the field of video quality evaluation, including: constructing and training a time-space fused screen content video quality evaluation model to obtain a trained screen content video quality evaluation model, wherein the screen content video quality evaluation model includes a spatial quality evaluation branch and a temporal quality evaluation branch; obtaining a screen content video and extracting a plurality of fragmented videos and a plurality of key frames; inputting each fragmented video and key frame into the trained screen content video quality evaluation model, respectively passing through the spatial quality evaluation branch and the temporal quality evaluation branch to obtain a spatial quality score and a temporal quality score, and combining the two to obtain a quality score of the screen content video. The present invention solves the problems of poor sensitivity and low accuracy of visual quality evaluation in existing video quality evaluation methods.

Description

Technical Field

[0001] The present invention relates to the field of video quality evaluation, and in particular to a method and device for evaluating screen content video quality based on attention guidance. Background Art

[0002] With the rapid development of artificial intelligence and digital multimedia technology, as well as the popularity of various portable devices, screen content videos are increasingly used in digital multimedia application fields (such as online education, video conferencing, webcasting, game videos, etc.), providing users with a more flexible and free multimedia experience. Unlike natural videos, screen content videos are a mixture of images and computer-generated text / image areas.

[0003] In different processing stages of screen content videos (such as acquisition, transmission, display, etc.), they will inevitably be disturbed by various noises, resulting in varying degrees of degradation of perceived quality, which seriously affects the user experience. Therefore, it is urgent to develop an effective video quality assessment method to evaluate the visual quality of screen content videos and improve the algorithm performance of related tasks. Summary of the invention

[0004] The purpose of this application is to propose a method and device for evaluating screen content video quality based on attention guidance in response to the above-mentioned technical problems.

[0005] In a first aspect, the present invention provides a method for evaluating screen content video quality based on attention guidance, comprising the following steps:

[0006] Constructing and training a spatiotemporal fusion screen content video quality evaluation model to obtain a trained screen content video quality evaluation model, wherein the screen content video quality evaluation model includes a spatial quality evaluation branch and a temporal quality evaluation branch, wherein the spatial quality evaluation branch includes a texture feature extraction unit, a semantic feature extraction unit, a first fusion module, and a spatial quality aggregation module, wherein the texture feature extraction unit includes a saliency calculation module, an initial multi-scale texture representation calculation module, and a first pre-trained network; and the semantic feature extraction unit includes a second pre-trained network and a second fusion module;

[0007] A screen content video is obtained and a plurality of fragmented videos and a plurality of key frames are extracted; each fragmented video and key frame is input into a trained screen content video quality evaluation model, each key frame is firstly subjected to channel segmentation and extracted by an initial multi-scale texture representation calculation module of a texture feature extraction unit to obtain an initial multi-scale texture representation of each key frame, each key frame is subjected to a saliency calculation module to obtain a saliency map of each key frame, a multi-scale texture representation of each key frame is obtained according to the initial multi-scale texture representation and the saliency map of each key frame, the multi-scale texture representation of each key frame is input into a first pre-trained network to extract the texture features of each key frame; each key frame is input into a semantic feature extraction unit The image block is first divided and cropped to obtain several image blocks, and reconstructed to obtain reconstructed key frames. Each image block and the reconstructed key frame are respectively input into the second pre-trained network to obtain the semantic features of each image block and the semantic features of the reconstructed key frame. The semantic features of all image blocks and the semantic features of the reconstructed key frame are input into the second fusion module to obtain the semantic features of each key frame; the texture features and semantic features of each key frame are input into the first fusion module to obtain the spatial features, and the spatial features are passed through the spatial quality aggregation module to obtain the spatial quality score; each fragmented video is input into the time domain quality evaluation branch to obtain the time domain quality score; the quality score of the screen content video is calculated according to the spatial quality score and the time domain quality score.

[0008] Preferably, each key frame is input into the initial multi-scale texture representation calculation module, and the channel segmentation is first performed and the initial multi-scale texture representation of each key frame is extracted. Each key frame is calculated by the saliency calculation module to obtain the saliency map of each key frame, which specifically includes:

[0009] The keyframe Pixel at position According to the three channels of RGB, the channels in the key frames are divided into Pixels at position The three channels represent , , , and use multi-scale Gaussian Laplacian filter to process each channel representation separately to obtain Multi-scale texture representation of the pixel at each channel , , , as shown below:

[0010] ;

[0011] ;

[0012] ;

[0013] ;

[0014] in, Represents the coordinates of the pixel points in the key frame, represents a multi-scale Laplacian of Gaussian filter, represents the Gaussian kernel scale, represents the standard deviation of the multi-scale Laplacian of Gaussian filter, Represents the convolution operation;

[0015] The keyframe The multi-scale texture representations of all channels of the pixel at position are added to obtain the key frame an initial multi-scale texture representation at a location;

[0016] ;

[0017] in, Indicates the key frame The initial multi-scale texture representation of the pixel point at the position, the initial multi-scale texture representation of all the pixel points of each key frame constitutes the initial multi-scale texture representation of each key frame;

[0018] The motion blur convolution kernel is used to convolve each key frame to obtain the corresponding blurred key frame, as shown in the following formula:

[0019] ;

[0020] ;

[0021] in, represents a key frame, represents a blurred keyframe, is the motion blur convolution kernel, For exercise volume, is the direction of motion and its angle with the horizontal axis, represents the transpose operation, and Represents the coordinates of the motion blur convolution kernel The corresponding value, Represents the convolution operation;

[0022] Use Sobel filter to calculate the keyframe Gradient mapping corresponding to the pixel at position and blur keyframes Gradient mapping corresponding to the pixel at position , and calculate the keyframe Gradient mapping corresponding to the pixel at position and blur keyframes Gradient mapping corresponding to the pixel at position The similarity between , as shown below:

[0023] ;

[0024] ;

[0025] ;

[0026] in, and represents horizontal Sobel filtering and vertical Sobel filtering, represents a positive constant;

[0027] The Gaussian convolution kernel is used to convolve each key frame to obtain the corresponding focused key frame, as shown in the following formula:

[0028] ;

[0029] ;

[0030] in, Indicates the focus keyframe. represents the Gaussian convolution kernel, represents the convolution operation, and Represents the coordinates of the Gaussian convolution kernel The corresponding value, represents the standard deviation of the Gaussian kernel function;

[0031] Use the Sobel filter to calculate the gradient map corresponding to the focused keyframe , and calculate the keyframe Gradient mapping corresponding to the pixel at position and in the focus keyframe Gradient mapping corresponding to the pixel at position The similarity between , as shown below:

[0032] ;

[0033] in, is a positive constant;

[0034] according to and Calculate the keyframe The saliency value of the pixel at the position is as follows:

[0035] ;

[0036] in, A hyperparameter representing the significant fusion ratio, Indicates the key frame The saliency value of the pixel at the position, the saliency values ​​of all pixels in each key frame constitute the saliency map of each key frame.

[0037] Preferably, the first pre-trained network and the second pre-trained network both include multiple convolutional layers with different convolution kernel sizes arranged in parallel and a global average pooling layer and a global standard deviation pooling layer respectively connected to each convolutional layer. During the training process of the screen content video quality assessment model, the parameters of the first pre-trained network and the second pre-trained network are frozen.

[0038] Preferably, a multi-scale texture representation of each key frame is obtained according to the initial multi-scale texture representation and the saliency map of each key frame, and the multi-scale texture representation of each key frame is input into the first pre-trained network to extract the texture features of each key frame, specifically including:

[0039] The initial multi-scale texture representation of each pixel in each key frame is multiplied by the corresponding saliency value to obtain the multi-scale texture representation of each pixel in each key frame, as shown in the following formula:

[0040] ;

[0041] in, Indicates the key frame The initial multi-scale texture representation of the pixel at position, Indicates the key frame The saliency value of the pixel at position, Indicates the key frame Multi-scale texture representation of pixel points at position;

[0042] The keyframe The multi-scale texture representation of the pixel at position is input into the first pre-trained network to obtain the key frame Texture features of the pixel at the position;

[0043] ;

[0044] ;

[0045] in, Represents the first pre-trained network The functions corresponding to all convolutional layers in the kth stage, , represents the texture feature of the kth stage, Represents the function corresponding to the global average pooling layer, Represents the function corresponding to the global standard error pooling layer, Represents a splicing operation, Represents the texture features of the keyframe.

[0046] Preferably, each key frame is input into a semantic feature extraction unit, and is first divided and cropped to obtain a number of image blocks, and reconstructed to obtain a reconstructed key frame, each image block and the reconstructed key frame are respectively input into a second pre-trained network to obtain the semantic features of each image block and the semantic features of the reconstructed key frame, and the semantic features of all image blocks and the semantic features of the reconstructed key frame are input into a second fusion module to obtain the semantic features of each key frame, specifically including:

[0047] The key frame is divided into several equal parts to obtain several grids, and a fixed-size image block is randomly divided and cropped in each grid to obtain several image blocks, and the image blocks are reconstructed according to the grid positions to obtain a reconstructed key frame;

[0048] The nth image block and the reconstructed key frame are respectively input into the second pre-trained network to obtain the semantic features of the nth image block and the semantic features of the reconstructed key frame, as shown in the following formula:

[0049] ;

[0050] ;

[0051] ;

[0052] ;

[0053] in, represents the nth image block, , N represents the total number of image blocks, represents the reconstructed keyframe, Represents the second pre-trained network The functions corresponding to all convolutional layers in the kth stage, , Represents the function corresponding to the global average pooling layer, Represents the function corresponding to the global standard error pooling layer, represents the semantic features of the kth stage corresponding to the nth image block, represents the semantic features of the kth stage corresponding to the reconstructed key frame, Represents a splicing operation, represents the semantic feature corresponding to the nth image block, Represents the semantic features corresponding to the reconstructed keyframe;

[0054] The second fusion module includes a first cross attention module. The semantic features corresponding to the nth image block and the semantic features corresponding to the reconstructed key frame are passed through the first cross attention module. The semantic features corresponding to the reconstructed key frame are used as the query vector, and the semantic features corresponding to the nth image block are used as the key vector and the value vector to obtain the semantic features of the nth image block. , the semantic features of all image blocks are spliced ​​to obtain the semantic features of the key frame, as shown in the following formula:

[0055] ;

[0056] ;

[0057] in, express function, represents the transposed matrix, represents a constant, Represents a splicing operation, Represents the semantic features of key frames.

[0058] Preferably, the texture features and semantic features of each key frame are input into the first fusion module to obtain spatial features, and the spatial features are passed through the spatial quality aggregation module to obtain a spatial quality score; each fragmented video is input into the temporal quality evaluation branch to obtain a temporal quality score; and the quality score of the screen content video is calculated according to the spatial quality score and the temporal quality score, specifically including:

[0059] The first fusion module includes a second cross attention module and a third cross attention module; the texture features of the key frame and the semantic features of the corresponding key frame are respectively passed through the second cross attention module and the third cross attention module, and the output features of the second cross attention module and the output features of the third cross attention module are added to obtain the spatial features of the key frame, as shown in the following formula:

[0060] ;

[0061] in, express function, represents the transposed matrix, represents the spatial features of the keyframes, represents the texture features of the keyframe, Represents the semantic features of key frames, represents a constant;

[0062] The spatial quality aggregation module consists of two fully connected layers connected in sequence to obtain the spatial quality score of the keyframe, as shown in the following formula:

[0063] ;

[0064] in, Represents the function corresponding to the spatial quality aggregation module, represents the airspace quality fraction;

[0065] The temporal quality evaluation branch uses the video Swim Transformer network, and the fragmented video is input into the video Swim Transformer network to obtain the temporal quality score. ;

[0066] The quality score of the screen content video is calculated using the following formula:

[0067] ;

[0068] in, Indicates the weight value.

[0069] In a second aspect, the present invention provides a device for evaluating screen content video quality based on attention guidance, comprising:

[0070] A model building module is configured to build and train a spatiotemporal fusion screen content video quality evaluation model to obtain a trained screen content video quality evaluation model, wherein the screen content video quality evaluation model includes a spatial quality evaluation branch and a temporal quality evaluation branch, wherein the spatial quality evaluation branch includes a texture feature extraction unit, a semantic feature extraction unit, a first fusion module, and a spatial quality aggregation module, wherein the texture feature extraction unit includes a saliency calculation module, an initial multi-scale texture representation calculation module, and a first pre-trained network; and the semantic feature extraction unit includes a second pre-trained network and a second fusion module;

[0071] The quality score calculation module is configured to obtain a screen content video and extract a plurality of fragmented videos and a plurality of key frames; each fragmented video and key frame is input into a trained screen content video quality evaluation model; each key frame is firstly subjected to channel segmentation and extracted by an initial multi-scale texture representation calculation module of a texture feature extraction unit to obtain an initial multi-scale texture representation of each key frame; each key frame is subjected to a saliency calculation module to obtain a saliency map of each key frame; a multi-scale texture representation of each key frame is obtained according to the initial multi-scale texture representation and the saliency map of each key frame; the multi-scale texture representation of each key frame is input into a first pre-trained network to extract the texture features of each key frame; each key frame is input The semantic feature extraction unit first divides and crops to obtain several image blocks, and reconstructs them to obtain reconstructed key frames. Each image block and the reconstructed key frame are respectively input into the second pre-trained network to obtain the semantic features of each image block and the semantic features of the reconstructed key frame. The semantic features of all image blocks and the semantic features of the reconstructed key frame are input into the second fusion module to obtain the semantic features of each key frame; the texture features and semantic features of each key frame are input into the first fusion module to obtain the spatial features, and the spatial features are passed through the spatial quality aggregation module to obtain the spatial quality score; each fragmented video is input into the time domain quality evaluation branch to obtain the time domain quality score; the quality score of the screen content video is calculated based on the spatial quality score and the time domain quality score.

[0072] In a third aspect, the present invention provides an electronic device comprising one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0073] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0074] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0075] Compared with the prior art, the present invention has the following beneficial effects:

[0076] (1) The attention-guided screen content video quality assessment method proposed in this paper makes full use of the rich texture information and semantic information in the video that are closely related to the quality, and realizes the quality assessment of distorted screen content videos through a reliable cross-attention fusion mechanism.

[0077] (2) The screen content video quality assessment method based on attention guidance proposed in the present invention adopts a multi-scale texture representation method of key frames in the first pre-trained network and a multi-scale semantic representation method of key frames in the second pre-trained network, simulating the sensitivity of the human eye to edge information.

[0078] (3) The screen content video quality assessment method based on attention guidance proposed in the present invention designs a multi-stage cross-attention mechanism, which not only effectively integrates the semantic features of multi-region image blocks, but also combines texture features with semantic features, effectively improving the accuracy of screen content video quality assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0080] Figure 1 A schematic diagram of a flow chart of a method for evaluating screen content video quality based on attention guidance according to an embodiment of the present application;

[0081] Figure 2 A schematic diagram of a screen content video quality assessment model of a screen content video quality assessment method based on attention guidance according to an embodiment of the present application;

[0082] Figure 3 A schematic diagram of a device for evaluating screen content video quality based on attention guidance according to an embodiment of the present application;

[0083] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0084] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0085] Figure 1 A method for evaluating the quality of screen content video based on attention guidance provided by an embodiment of the present application is shown, comprising the following steps:

[0086] S1, construct and train a spatiotemporal fusion screen content video quality evaluation model to obtain a trained screen content video quality evaluation model, wherein the screen content video quality evaluation model includes a spatial quality evaluation branch and a temporal quality evaluation branch, wherein the spatial quality evaluation branch includes a texture feature extraction unit, a semantic feature extraction unit, a first fusion module and a spatial quality aggregation module, wherein the texture feature extraction unit includes a saliency calculation module, an initial multi-scale texture representation calculation module and a first pre-trained network; and the semantic feature extraction unit includes a second pre-trained network and a second fusion module.

[0087] In a specific embodiment, the first pre-trained network and the second pre-trained network both include multiple convolutional layers with different convolution kernel sizes set in parallel and a global average pooling layer and a global standard deviation pooling layer respectively connected to each convolutional layer. During the training process of the screen content video quality assessment model, the parameters of the first pre-trained network and the second pre-trained network are frozen.

[0088] Specifically, refer to Figure 2 The screen content video quality evaluation model consists of a spatial quality evaluation branch and a temporal quality evaluation branch. The spatial quality evaluation branch and the temporal quality evaluation branch are used to calculate the spatial quality score and the temporal quality score respectively. Finally, the spatial quality score and the temporal quality score are combined to achieve the quality evaluation of the screen content video. In the spatial quality evaluation branch, there are first two parallel texture feature extraction units and semantic feature extraction units, which are used to extract the texture features and semantic features of the key frames respectively. The texture features and semantic features of the key frames are sequentially passed through the first fusion module and the spatial quality aggregation module to obtain the spatial quality score.

[0089] Furthermore, a first pre-trained network and a second pre-trained network are respectively provided in the texture feature extraction unit and the semantic feature extraction unit. The network structures of the first pre-trained network and the second pre-trained network are the same. During the training process of the screen content video quality assessment model, the parameters of the first pre-trained network and the second pre-trained network are frozen and do not participate in the training, thereby reducing the amount of data and complexity of the training.

[0090] S2, obtaining a screen content video and extracting a number of fragmented videos and a number of key frames; inputting each fragmented video and key frame into a trained screen content video quality evaluation model, each key frame is passed through an initial multi-scale texture representation calculation module of a texture feature extraction unit, first performing channel segmentation and extracting an initial multi-scale texture representation of each key frame, each key frame is passed through a saliency calculation module to calculate a saliency map of each key frame, a multi-scale texture representation of each key frame is obtained based on the initial multi-scale texture representation and saliency map of each key frame, the multi-scale texture representation of each key frame is input into a first pre-trained network, and the texture feature of each key frame is extracted; each key frame is input into a semantic feature extraction unit, and then the texture feature of each key frame is extracted. Take a unit, divide and crop it first to obtain several image blocks, and reconstruct them to obtain reconstructed key frames. Input each image block and reconstructed key frame into the second pre-trained network respectively to obtain the semantic features of each image block and the semantic features of the reconstructed key frame. Input the semantic features of all image blocks and the semantic features of the reconstructed key frame into the second fusion module to obtain the semantic features of each key frame. Input the texture features and semantic features of each key frame into the first fusion module to obtain the spatial features. The spatial features are passed through the spatial quality aggregation module to obtain the spatial quality score. Each fragmented video is input into the time domain quality evaluation branch to obtain the time domain quality score. The quality score of the screen content video is calculated according to the spatial quality score and the time domain quality score.

[0091] In a specific embodiment, each key frame is input into an initial multi-scale texture representation calculation module, and channel segmentation is first performed and extracted to obtain an initial multi-scale texture representation of each key frame. Each key frame is calculated by a saliency calculation module to obtain a saliency map of each key frame, which specifically includes:

[0092] The keyframe Pixels at position According to the three channels of RGB, the channels in the key frames are divided into Pixels at position The three channels represent , , , and use multi-scale Gaussian Laplacian filter to process each channel representation separately to obtain Multi-scale texture representation of the pixel at each channel , , , as shown below:

[0093] ;

[0094] ;

[0095] ;

[0096] ;

[0097] in, Represents the coordinates of the pixel points in the key frame, represents a multi-scale Laplacian of Gaussian filter, represents the Gaussian kernel scale, represents the standard deviation of the multi-scale Laplacian of Gaussian filter, Represents the convolution operation;

[0098] The keyframe The multi-scale texture representations of all channels of the pixel at position are added to obtain the key frame an initial multi-scale texture representation at a location;

[0099] ;

[0100] in, Indicates the key frame The initial multi-scale texture representation of the pixel point at the position, the initial multi-scale texture representation of all the pixel points of each key frame constitutes the initial multi-scale texture representation of each key frame;

[0101] The motion blur convolution kernel is used to convolve each key frame to obtain the corresponding blurred key frame, as shown in the following formula:

[0102] ;

[0103] ;

[0104] in, represents a key frame, represents a blurred keyframe, is the motion blur convolution kernel, For exercise volume, is the direction of motion and its angle with the horizontal axis, represents the transpose operation, and Represents the coordinates of the motion blur convolution kernel The corresponding value, Represents the convolution operation;

[0105] Use Sobel filter to calculate the keyframe Gradient mapping corresponding to the pixel at position and blur keyframes Gradient mapping corresponding to the pixel at position , and calculate the keyframe Gradient mapping corresponding to the pixel at position and blur keyframes Gradient mapping corresponding to the pixel at position The similarity between , as shown below:

[0106] ;

[0107] ;

[0108] ;

[0109] in, and represents horizontal Sobel filtering and vertical Sobel filtering, represents a positive constant;

[0110] The Gaussian convolution kernel is used to convolve each key frame to obtain the corresponding focused key frame, as shown in the following formula:

[0111] ;

[0112] ;

[0113] in, Indicates the focus keyframe. represents the Gaussian convolution kernel, represents the convolution operation, and Represents the coordinates of the Gaussian convolution kernel The corresponding value, represents the standard deviation of the Gaussian kernel function;

[0114] Use the Sobel filter to calculate the gradient map corresponding to the focused keyframe , and calculate the keyframe Gradient mapping corresponding to the pixel at position and in the focus keyframe Gradient mapping corresponding to the pixel at position The similarity between , as shown below:

[0115] ;

[0116] in, is a positive constant;

[0117] according to and Calculate the keyframe The saliency value of the pixel at the position is as follows:

[0118] ;

[0119] in, A hyperparameter representing the significant fusion ratio, Indicates the key frame The saliency value of the pixel at the position, the saliency values ​​of all pixels in each key frame constitute the saliency map of each key frame.

[0120] In a specific embodiment, a multi-scale texture representation of each key frame is obtained according to the initial multi-scale texture representation and the saliency map of each key frame, and the multi-scale texture representation of each key frame is input into the first pre-trained network to extract the texture features of each key frame, which specifically includes:

[0121] The initial multi-scale texture representation of each pixel in each key frame is multiplied by the corresponding saliency value to obtain the multi-scale texture representation of each pixel in each key frame, as shown in the following formula:

[0122] ;

[0123] in, Indicates the key frame The initial multi-scale texture representation of the pixel at position, Indicates the key frame The saliency value of the pixel at position, Indicates the key frame Multi-scale texture representation of pixel points at position;

[0124] The keyframe The multi-scale texture representation of the pixel at position is input into the first pre-trained network to obtain the key frame Texture features of the pixel at the position;

[0125] ;

[0126] ;

[0127] in, Represents the first pre-trained network The functions corresponding to all convolutional layers in the kth stage, , represents the texture feature of the kth stage, Represents the function corresponding to the global average pooling layer, Represents the function corresponding to the global standard error pooling layer, Represents a splicing operation, Represents the texture features of the keyframe.

[0128] Specifically, the key frame is divided into three channels of RGB, and each channel is processed by a multi-scale Gaussian Laplacian filter to further extract the texture features of the key frame. The standard deviation of the multi-scale Gaussian Laplacian filter is =1.487.

[0129] In a specific embodiment, each key frame is input into a semantic feature extraction unit, which is first divided and cropped to obtain a plurality of image blocks, and reconstructed to obtain a reconstructed key frame. Each image block and the reconstructed key frame are respectively input into a second pre-trained network to obtain the semantic features of each image block and the semantic features of the reconstructed key frame. The semantic features of all image blocks and the semantic features of the reconstructed key frame are input into a second fusion module to obtain the semantic features of each key frame, which specifically includes:

[0130] The key frame is divided into several equal parts to obtain several grids, and a fixed-size image block is randomly divided and cropped in each grid to obtain several image blocks, and the image blocks are reconstructed according to the grid positions to obtain a reconstructed key frame;

[0131] The nth image block and the reconstructed key frame are respectively input into the second pre-trained network to obtain the semantic features of the nth image block and the semantic features of the reconstructed key frame, as shown in the following formula:

[0132] ;

[0133] ;

[0134] ;

[0135] ;

[0136] in, represents the nth image block, , N represents the total number of image blocks, represents the reconstructed keyframe, Represents the second pre-trained network The functions corresponding to all convolutional layers in the kth stage, , Represents the function corresponding to the global average pooling layer, Represents the function corresponding to the global standard error pooling layer, represents the semantic features of the kth stage corresponding to the nth image block, represents the semantic features of the kth stage corresponding to the reconstructed key frame, Represents a splicing operation, represents the semantic feature corresponding to the nth image block, Represents the semantic features corresponding to the reconstructed keyframe;

[0137] The second fusion module includes a first cross attention module. The semantic features corresponding to the nth image block and the semantic features corresponding to the reconstructed key frame are passed through the first cross attention module. The semantic features corresponding to the reconstructed key frame are used as the query vector, and the semantic features corresponding to the nth image block are used as the key vector and the value vector to obtain the semantic features of the nth image block. , the semantic features of all image blocks are spliced ​​to obtain the semantic features of the key frame, as shown in the following formula:

[0138] ;

[0139] ;

[0140] in, express function, represents the transposed matrix, represents a constant, Represents a splicing operation, Represents the semantic features of key frames.

[0141] Specifically, the key frame is divided into multiple regions, and the semantic features that conform to the visual characteristics of the human eye are extracted using the second pre-trained network and the cross attention mechanism. In one embodiment, the key frame is divided into ten equal parts to obtain 10 grids, and a random number is cut out in each grid. The image blocks of different sizes are reconstructed according to the original grid positions to obtain the reconstructed key frames. All image blocks and reconstructed key frames are passed through the second pre-trained network and the second fusion module to calculate the semantic features of the key frames. The second fusion module uses the first cross attention module to assign different importance to the semantic features corresponding to the multi-region image blocks to obtain the semantic features of each image block.

[0142] In a specific embodiment, the texture features and semantic features of each key frame are input into the first fusion module to obtain spatial features, and the spatial features are passed through the spatial quality aggregation module to obtain a spatial quality score; each fragmented video is input into the temporal quality evaluation branch to obtain a temporal quality score; the quality score of the screen content video is calculated based on the spatial quality score and the temporal quality score, specifically including:

[0143] The first fusion module includes a second cross attention module and a third cross attention module; the texture features of the key frame and the semantic features of the corresponding key frame are respectively passed through the second cross attention module and the third cross attention module, and the output features of the second cross attention module and the output features of the third cross attention module are added to obtain the spatial features of the key frame, as shown in the following formula:

[0144] ;

[0145] in, express function, represents the transposed matrix, represents the spatial features of the keyframes, represents the texture features of the keyframe, Represents the semantic features of key frames, represents a constant;

[0146] The spatial quality aggregation module consists of two fully connected layers connected in sequence to obtain the spatial quality score of the keyframe, as shown in the following formula:

[0147] ;

[0148] in, Represents the function corresponding to the spatial quality aggregation module, represents the airspace quality fraction;

[0149] The temporal quality evaluation branch uses the video Swim Transformer network, and the fragmented video is input into the video Swim Transformer network to obtain the temporal quality score. ;

[0150] The quality score of the screen content video is calculated using the following formula:

[0151] ;

[0152] in, Indicates the weight value.

[0153] Specifically, multi-layer cross-attention is used to fuse the texture features and semantic features of key frames to predict the spatial quality score, which is then combined with the temporal quality score representing the temporal perception of the video predicted by the temporal quality evaluation branch, and the quality score of the screen content video is obtained by weighted calculation.

[0154] Further references Figure 3 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a screen content video quality evaluation device based on attention guidance, and the device embodiment is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0155] The embodiment of the present application provides a device for evaluating screen content video quality based on attention guidance, comprising:

[0156] The model building module 1 is configured to build and train a spatiotemporal fusion screen content video quality evaluation model to obtain a trained screen content video quality evaluation model, wherein the screen content video quality evaluation model includes a spatial quality evaluation branch and a temporal quality evaluation branch, wherein the spatial quality evaluation branch includes a texture feature extraction unit, a semantic feature extraction unit, a first fusion module, and a spatial quality aggregation module, wherein the texture feature extraction unit includes a saliency calculation module, an initial multi-scale texture representation calculation module, and a first pre-trained network; and the semantic feature extraction unit includes a second pre-trained network and a second fusion module;

[0157] The quality score calculation module 2 is configured to obtain the screen content video and extract a plurality of fragmented videos and a plurality of key frames; each fragmented video and key frame is input into the trained screen content video quality evaluation model; each key frame is firstly subjected to the initial multi-scale texture representation calculation module of the texture feature extraction unit, and the channel segmentation is performed and the initial multi-scale texture representation of each key frame is extracted; each key frame is subjected to the saliency calculation module to calculate the saliency map of each key frame; the multi-scale texture representation of each key frame is obtained according to the initial multi-scale texture representation and the saliency map of each key frame; the multi-scale texture representation of each key frame is input into the first pre-trained network to extract the texture feature of each key frame; each key frame is input The semantic feature extraction unit first divides and crops to obtain several image blocks, and reconstructs them to obtain reconstructed key frames. Each image block and the reconstructed key frame are respectively input into the second pre-trained network to obtain the semantic features of each image block and the semantic features of the reconstructed key frame. The semantic features of all image blocks and the semantic features of the reconstructed key frame are input into the second fusion module to obtain the semantic features of each key frame; the texture features and semantic features of each key frame are input into the first fusion module to obtain the spatial features, and the spatial features are passed through the spatial quality aggregation module to obtain the spatial quality score; each fragmented video is input into the time domain quality evaluation branch to obtain the time domain quality score; the quality score of the screen content video is calculated based on the spatial quality score and the time domain quality score.

[0158] Figure 4 Schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention. Figure 4 As shown, the electronic device of this embodiment includes: a processor 401 and a memory 402; wherein the memory 402 is used to store computer-executable instructions; the processor 401 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0159] Optionally, the memory 402 may be independent or integrated with the processor 401 .

[0160] When the memory 402 is independently provided, the electronic device further includes a bus 403 for connecting the memory 402 and the processor 401 .

[0161] The embodiment of the present invention further provides a computer storage medium, in which computer execution instructions are stored. When the processor 401 executes the computer execution instructions, the above method is implemented.

[0162] The embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by the processor 401, the above method is implemented.

[0163] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0164] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0165] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each module may exist physically separately, or two or more modules may be integrated into one unit. The unit formed by the above modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0166] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor 401 to perform some steps of the methods of various embodiments of the present application.

[0167] It should be understood that the processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or the processor 401 may be any conventional processor 401, etc. The steps of the method disclosed in the invention may be directly embodied in the hardware processor 401 for execution, or may be executed by a combination of hardware and software modules in the processor 401.

[0168] The memory 402 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disk.

[0169] The bus 403 may be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 403 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus 403 in the drawings of the present application is not limited to only one bus 403 or one type of bus 403.

[0170] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0171] An exemplary storage medium is coupled to the processor 401, so that the processor 401 can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor 401. The processor 401 and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor 401 and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0172] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for evaluating screen content video quality based on attention guidance, characterized in that: The following steps are involved: Constructing and training a spatiotemporal fusion screen content video quality assessment model to obtain a trained screen content video quality assessment model, wherein the screen content video quality assessment model includes a spatial quality assessment branch and a temporal quality assessment branch, wherein the spatial quality assessment branch includes a texture feature extraction unit, a semantic feature extraction unit, a first fusion module, and a spatial quality aggregation module, wherein the texture feature extraction unit includes a saliency calculation module, an initial multi-scale texture representation calculation module, and a first pre-trained network; and the semantic feature extraction unit includes a second pre-trained network and a second fusion module; The screen content video is obtained and a plurality of fragmented videos and a plurality of key frames are extracted; each fragmented video and key frame is input into the trained screen content video quality evaluation model, each key frame is firstly subjected to channel segmentation and extracted to obtain the initial multi-scale texture representation of each key frame by the initial multi-scale texture representation calculation module of the texture feature extraction unit, each key frame is subjected to the saliency calculation module to calculate the saliency map of each key frame, a multi-scale texture representation of each key frame is obtained according to the initial multi-scale texture representation and the saliency map of each key frame, the multi-scale texture representation of each key frame is input into the first pre-trained network, and the texture feature of each key frame is extracted, specifically including: The initial multi-scale texture representation of each pixel in each key frame is multiplied by the corresponding saliency value to obtain the multi-scale texture representation of each pixel in each key frame, as shown in the following formula: in, Indicates the key frame The initial multi-scale texture representation of the pixel at position, Indicates the key frame The saliency value of the pixel at position, Indicates the key frame Multi-scale texture representation of pixels at positions; The key frame The multi-scale texture representation of the pixel point at the position is input into the first pre-trained network to obtain the key frame Texture features of the pixel at the position; in, Represents the first pre-trained network The first of the The functions corresponding to all convolutional layers in the stage are: Indicates The texture features of each stage, Represents the function corresponding to the global average pooling layer, Represents the function corresponding to the global standard error pooling layer, Represents a splicing operation, Represents the texture features of the key frame; each key frame is input into the semantic feature extraction unit, first divided and cropped to obtain a number of image blocks, and reconstructed to obtain a reconstructed key frame, each image block and the reconstructed key frame are respectively input into the second pre-trained network to obtain the semantic features of each image block and the semantic features of the reconstructed key frame, the semantic features of all image blocks and the semantic features of the reconstructed key frame are input into the second fusion module to obtain the semantic features of each key frame, specifically including: Divide the key frame into several equal parts to obtain several grids, randomly divide and crop image blocks of a fixed size in each grid to obtain several image blocks, and reconstruct them according to grid positions to obtain reconstructed key frames; Separately The image blocks and reconstructed key frames are input into the second pre-trained network to obtain the The semantic features of the image blocks and the semantic features of the reconstructed key frames are shown in the following formula: in, Indicates image blocks, represents the total number of image blocks, represents the reconstructed keyframe, Represents the second pre-trained network The first of the The functions corresponding to all convolutional layers in the stage are: Represents the function corresponding to the global average pooling layer, Represents the function corresponding to the global standard error pooling layer, Indicates The image block corresponds to The semantic features of each stage, Represents the first key frame corresponding to the reconstruction The semantic features of each stage, Represents a splicing operation, Indicates The semantic features corresponding to the image blocks are Represents the semantic features corresponding to the reconstructed keyframe; The second fusion module includes a first cross attention module, The semantic features corresponding to the image blocks and the semantic features corresponding to the reconstructed key frames are passed through the first cross attention module, and the semantic features corresponding to the reconstructed key frames are used as query vectors. The semantic features corresponding to the image blocks are used as key vectors and value vectors to obtain the Semantic features of image patches The semantic features of all image blocks are concatenated to obtain the semantic features of the key frame, as shown in the following formula: in, express function, represents the transposed matrix, represents a constant, Represents a splicing operation, Representing semantic features of key frames; inputting texture features and semantic features of each key frame into the first fusion module to obtain spatial features, and the spatial features are passed through the spatial quality aggregation module to obtain a spatial quality score; each fragmented video is input into the temporal quality evaluation branch to obtain a temporal quality score; and the quality score of the screen content video is calculated based on the spatial quality score and the temporal quality score.

2. The screen content video quality assessment method based on attention guidance according to claim 1 is characterized in that: Each key frame is input into the initial multi-scale texture representation calculation module, and the channel segmentation is first performed and the initial multi-scale texture representation of each key frame is extracted. Each key frame is calculated by the saliency calculation module to obtain the saliency map of each key frame, which specifically includes: The keyframe Pixel at position According to the three channels of RGB, the channels in the key frames are divided into Pixel at position The three channels represent And use multi-scale Gaussian Laplacian filter to process each channel representation separately to obtain Multi-scale texture representation of the pixel at each channel As shown below: in, Represents the coordinates of the pixel points in the key frame, represents a multi-scale Laplacian of Gaussian filter, represents the Gaussian kernel scale, represents the standard deviation of the multi-scale Laplacian of Gaussian filter, Represents the convolution operation; The keyframe The multi-scale texture representations of all channels of the pixel at position are added to obtain the key frame an initial multi-scale texture representation at a location; in, Indicates the key frame The initial multi-scale texture representation of the pixel point at the position, the initial multi-scale texture representation of all the pixel points of each key frame constitutes the initial multi-scale texture representation of each key frame; The motion blur convolution kernel is used to convolve each key frame to obtain the corresponding blurred key frame, as shown in the following formula: in, represents a key frame, represents a blurred keyframe, is the motion blur convolution kernel, For exercise volume, is the direction of motion and its angle with the horizontal axis, represents the transpose operation, and Represents the coordinates of the motion blur convolution kernel The corresponding value, Represents the convolution operation; Use Sobel filter to calculate the keyframe Gradient mapping corresponding to the pixel at position and blur keyframes Gradient mapping corresponding to the pixel at position And calculate the key frame Gradient mapping corresponding to the pixel at position and blur keyframes Gradient mapping corresponding to the pixel at position Similarity between As shown below: in, and represents horizontal Sobel filtering and vertical Sobel filtering, represents a positive constant; The Gaussian convolution kernel is used to convolve each key frame to obtain the corresponding focused key frame, as shown in the following formula: in, Indicates the focus keyframe. represents the Gaussian convolution kernel, represents the convolution operation, and Represents the coordinates of the Gaussian convolution kernel The corresponding value, represents the standard deviation of the Gaussian kernel function; Use the Sobel filter to calculate the gradient map corresponding to the focused keyframe And calculate the key frame Gradient mapping corresponding to the pixel at position and the focus keyframe Gradient mapping corresponding to the pixel at position Similarity between As shown below: in, is a positive constant; according to and Calculate the keyframe The saliency value of the pixel at the position is as follows: in, The hyperparameter representing the significant fusion ratio, Indicates the key frame The saliency value of the pixel at the position, the saliency values ​​of all pixels in each key frame constitute the saliency map of each key frame.

3. The screen content video quality assessment method based on attention guidance according to claim 1 is characterized in that: The first pre-trained network and the second pre-trained network both include multiple convolutional layers with different convolution kernel sizes set in parallel and a global average pooling layer and a global standard deviation pooling layer respectively connected to each convolutional layer. During the training process of the screen content video quality assessment model, the parameters of the first pre-trained network and the second pre-trained network are frozen.

4. The screen content video quality assessment method based on attention guidance according to claim 1 is characterized in that: Inputting the texture features and semantic features of each key frame into the first fusion module to obtain spatial features, and passing the spatial features through the spatial quality aggregation module to obtain a spatial quality score; inputting each fragmented video into the temporal quality evaluation branch to obtain a temporal quality score; and calculating the quality score of the screen content video according to the spatial quality score and the temporal quality score, specifically including: The first fusion module includes a second cross attention module and a third cross attention module; the texture features of the key frame and the semantic features of the corresponding key frame are respectively passed through the second cross attention module and the third cross attention module, and the output features of the second cross attention module and the output features of the third cross attention module are added to obtain the spatial features of the key frame, as shown in the following formula: in, express function, represents the transposed matrix, represents the spatial features of the keyframes, represents the texture features of the keyframe, represents the semantic features of key frames, represents a constant; The spatial quality aggregation module includes two fully connected layers connected in sequence to obtain the spatial quality score of the key frame, as shown in the following formula: in, Represents the function corresponding to the spatial quality aggregation module, represents the airspace quality fraction; The temporal quality evaluation branch uses a video Swim Transformer network, and the fragmented video is input into the video Swim Transformer network to obtain a temporal quality score. The quality score of the screen content video is calculated using the following formula: in, Indicates the weight value.

5. A screen content video quality assessment device based on attention guidance, characterized in that: include: A model building module is configured to build and train a spatiotemporal fusion screen content video quality evaluation model to obtain a trained screen content video quality evaluation model, wherein the screen content video quality evaluation model includes a spatial quality evaluation branch and a temporal quality evaluation branch, wherein the spatial quality evaluation branch includes a texture feature extraction unit, a semantic feature extraction unit, a first fusion module, and a spatial quality aggregation module, wherein the texture feature extraction unit includes a saliency calculation module, an initial multi-scale texture representation calculation module, and a first pre-trained network; and the semantic feature extraction unit includes a second pre-trained network and a second fusion module; The quality score calculation module is configured to obtain a screen content video and extract a plurality of fragmented videos and a plurality of key frames; each fragmented video and key frame is input into the trained screen content video quality evaluation model; each key frame is firstly subjected to channel segmentation and extracted by the initial multi-scale texture representation calculation module of the texture feature extraction unit to obtain the initial multi-scale texture representation of each key frame; each key frame is subjected to the saliency calculation module to calculate the saliency map of each key frame; a multi-scale texture representation of each key frame is obtained according to the initial multi-scale texture representation and the saliency map of each key frame; the multi-scale texture representation of each key frame is input into the first pre-trained network to extract the texture feature of each key frame, specifically including: The initial multi-scale texture representation of each pixel in each key frame is multiplied by the corresponding saliency value to obtain the multi-scale texture representation of each pixel in each key frame, as shown in the following formula: in, Indicates the key frame The initial multi-scale texture representation of the pixel at position, Indicates the key frame The saliency value of the pixel at position, Indicates the key frame Multi-scale texture representation of pixels at positions; The key frame The multi-scale texture representation of the pixel point at the position is input into the first pre-trained network to obtain the key frame Texture features of the pixel at the position; in, Represents the first pre-trained network The first of the The functions corresponding to all convolutional layers in the stage are: Indicates The texture features of each stage, Represents the function corresponding to the global average pooling layer, Represents the function corresponding to the global standard error pooling layer, Represents a splicing operation, Represents the texture features of the key frame; each key frame is input into the semantic feature extraction unit, first divided and cropped to obtain a number of image blocks, and reconstructed to obtain a reconstructed key frame, each image block and the reconstructed key frame are respectively input into the second pre-trained network to obtain the semantic features of each image block and the semantic features of the reconstructed key frame, the semantic features of all image blocks and the semantic features of the reconstructed key frame are input into the second fusion module to obtain the semantic features of each key frame, specifically including: Divide the key frame into several equal parts to obtain several grids, randomly divide and crop image blocks of a fixed size in each grid to obtain several image blocks, and reconstruct them according to grid positions to obtain reconstructed key frames; Separately The image blocks and reconstructed key frames are input into the second pre-trained network to obtain the The semantic features of the image blocks and the semantic features of the reconstructed key frames are shown in the following formula: in, Indicates image blocks, represents the total number of image blocks, represents the reconstructed keyframe, Represents the second pre-trained network The first of the The functions corresponding to all convolutional layers in the stage are: Represents the function corresponding to the global average pooling layer, Represents the function corresponding to the global standard error pooling layer, Indicates The image block corresponds to The semantic features of each stage, Represents the first key frame corresponding to the reconstruction The semantic features of each stage, Represents a splicing operation, Indicates The semantic features corresponding to the image blocks are Represents the semantic features corresponding to the reconstructed keyframe; The second fusion module includes a first cross attention module, The semantic features corresponding to the image blocks and the semantic features corresponding to the reconstructed key frames are passed through the first cross attention module, and the semantic features corresponding to the reconstructed key frames are used as query vectors. The semantic features corresponding to the image blocks are used as key vectors and value vectors to obtain the Semantic features of image patches The semantic features of all image blocks are concatenated to obtain the semantic features of the key frame, as shown in the following formula: in, express function, represents the transposed matrix, represents a constant, Represents a splicing operation, Representing semantic features of key frames; inputting texture features and semantic features of each key frame into the first fusion module to obtain spatial features, and the spatial features are passed through the spatial quality aggregation module to obtain a spatial quality score; each fragmented video is input into the temporal quality evaluation branch to obtain a temporal quality score; and the quality score of the screen content video is calculated based on the spatial quality score and the temporal quality score.

6. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images