Compressed video quality evaluation method and device and electronic equipment

Abstract

The embodiment of the invention discloses a compressed video quality evaluation method and device and electronic equipment, and the method comprises the steps: inputting an image included in a compressed video into a quality evaluation model which comprises a feature extraction unit, a first predictor and a second predictor; the feature extraction unit performs feature extraction on an input image to obtain local features and global features respectively; the first predictor performs quality evaluation on a plurality of regions in the image according to the local features to obtain quality scores of the plurality of regions, and generates a quality score map according to the quality scores of the plurality of regions; the second predictor evaluates the quality of the image according to the global features to obtain a global quality score; and the quality evaluation model outputs a quality score graph and a global quality score as a quality evaluation result of the compressed video.

Description

Technical Field

[0001] The present invention relates to the technical field of video encoding and decoding, and in particular to a compressed video quality assessment method, device and electronic equipment. Background Art

[0002] Video compression can significantly improve transmission efficiency, save storage space and transmission bandwidth, but it also brings problems such as data loss and quality loss. Compressed video quality enhancement refers to improving the quality of compressed videos through specific technical means, reducing the image quality loss caused by compression, making the video clearer and smoother, thereby improving the visual effect and viewing experience of the video. Compressed video quality enhancement can usually select different technical means for enhancement processing based on the quality evaluation results of the compressed video, such as high-quality video or low-quality video. Therefore, the quality evaluation results of compressed videos are particularly important in compressed video quality enhancement.

[0003] At present, the training of DNN (Deep Neural Networks) models is commonly used to achieve quality assessment of compressed videos. In model training, the input compressed video is usually encoded using a constant quantization parameter. When training compressed videos encoded using multiple different quantization parameters, it is necessary to train each quantization parameter to obtain a corresponding model. In the inference stage, multiple trained models are required to complete quality assessment, which is highly complex and lacks flexibility. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a compressed video quality assessment method, device and electronic device to solve the problems in the prior art.

[0005] To solve the above technical problems, the embodiments of the present invention are implemented as follows: In a first aspect, an embodiment of the present invention provides a compressed video quality assessment method, comprising: Inputting an image included in the compressed video into a quality assessment model, the quality assessment model including a feature extraction unit, a first predictor, and a second predictor; The feature extraction unit performs feature extraction on the input image to obtain local features and global features respectively; The first predictor performs quality assessment on multiple regions in the image according to the local features to obtain quality scores of the multiple regions, and generates a quality score map according to the quality scores of the multiple regions; The second predictor performs quality assessment on the image according to the global feature to obtain a global quality score; The quality assessment model outputs the quality score map and the global quality score as a quality assessment result of the compressed video.

[0006] In a second aspect, an embodiment of the present invention provides a compressed video quality assessment device, the device comprising a quality assessment model, the quality assessment model comprising a feature extraction unit, a first predictor, and a second predictor; The feature extraction unit is used to extract features from images included in the input compressed video to obtain local features and global features respectively; The first predictor is used to respectively perform quality assessment on multiple regions in the image according to the local features to obtain quality scores of the multiple regions, and generate a quality score map according to the quality scores of the multiple regions; The second predictor is used to perform quality assessment on the image according to the global feature to obtain a global quality score; The input of the quality assessment model is the image included in the compressed video, and the output of the quality assessment model is the quality score map and the global quality score as the quality assessment result of the compressed video.

[0007] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the above-mentioned compressed video quality assessment method when executed by the processor.

[0008] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned compressed video quality assessment method are implemented.

[0009] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program, which implements the steps of the above-mentioned compressed video quality assessment method when executed by a processor.

[0010] It can be seen from the technical solution provided by the above embodiment of the present invention that the embodiment of the present invention inputs the image included in the compressed video into the quality assessment model, and the quality assessment model includes a feature extraction unit, a first predictor and a second predictor. The feature extraction unit extracts features from the input image to obtain local features and global features respectively. The first predictor performs quality assessment on multiple regions in the image according to the local features to obtain quality scores of the multiple regions, generates a quality score map according to the quality scores of the multiple regions, and the second predictor performs quality assessment on the image according to the global features to obtain a global quality score. Finally, the quality assessment model outputs the quality score map and the global quality score as the quality assessment result of the compressed video. Therefore, the quality assessment of the compressed video can be achieved based on a quality assessment model, which greatly simplifies the complexity of the quality assessment and improves the processing efficiency. Moreover, the results output by the model include local scores and global scores, which improve the perception ability of the model and the accuracy of the quality assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0012] Figure 1 A schematic diagram of a flow chart of a compressed video quality assessment method provided by an embodiment of the present invention; Figure 2 A schematic diagram of a flow chart of another compressed video quality assessment method provided by an embodiment of the present invention; Figure 3 A schematic diagram of the structure of a quality assessment model provided by an embodiment of the present invention; Figure 4 A schematic diagram of the structure of a compressed video quality assessment device provided by an embodiment of the present invention; Figure 5 A schematic diagram of the structure of another compressed video quality assessment device provided by an embodiment of the present invention; Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0013] The embodiments of the present invention provide a compressed video quality assessment method, device and electronic equipment.

[0014] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0015] The compressed video quality assessment method and device provided in the embodiment of the present invention can be applied to an electronic device, which can be a terminal device or a server. The server can be an independent server, or a server cluster composed of multiple servers, or a cloud server capable of cloud computing, which is not specifically limited. The terminal device can be a device such as a personal computer, or a mobile terminal device such as a mobile phone or a tablet computer, or a smart wearable device, a vehicle-mounted terminal, etc., which is not specifically limited.

[0016] The application scenarios of the above method and device include, but are not limited to: various scenarios that require high-quality video transmission and storage, such as online video streaming, video conferencing or video editing, and other scenarios that require video enhancement, and quality assessment of compressed videos. The above method and device can achieve quality assessment of compressed videos based on a quality assessment model, which greatly simplifies the complexity of quality assessment and improves processing efficiency. Moreover, the results output by the model include local scores and global scores, which improve the perception ability of the model and the accuracy of quality assessment.

[0017] like Figure 1 As shown, an embodiment of the present invention provides a compressed video quality assessment method, which may include the following steps: S102: Inputting an image included in the compressed video into a quality assessment model, where the quality assessment model includes a feature extraction unit, a first predictor, and a second predictor.

[0018] In an embodiment of the present invention, the quality assessment model can be trained for compressed videos that are pre-encoded using multiple different quantization parameters. Different compressed videos can be obtained by encoding using multiple different quantization parameters. The obtained compressed videos are used as training sets to train the quality assessment model, so that the quality assessment model can learn the ability to perform quality assessment on different compressed videos. There is no need to train different models for different quantization parameters. Quality assessment can be achieved using one model, which greatly simplifies the training process. Moreover, the result of the training is one model instead of multiple models, which reduces the complexity of the model and improves the perception ability of the model.

[0019] In an embodiment of the present invention, the feature extraction unit may include a multi-layer network structure, each layer of the network structure includes a first convolutional layer and a second convolutional layer, the scale of the second convolutional layer in each layer of the network structure is smaller than the scale of the first convolutional layer, and the number of output channels of the multi-layer network structure increases successively. Among them, the number of layers of the network structure can be set as needed, and the specific value is not limited. The number of output channels of each layer of the network structure can also be set as needed, and the specific value is not limited.

[0020] S104: The feature extraction unit extracts features from the input image to obtain local features and global features.

[0021] S106: The first predictor performs quality assessment on multiple regions in the image according to the local features, obtains quality scores of the multiple regions, and generates a quality score map according to the quality scores of the multiple regions.

[0022] The number of regions in the image can be set as needed, and the specific value is not limited. The quality score map includes the quality score of each region of the image, which can fully reflect the local quality of the image, making the image quality assessment result more detailed and accurate.

[0023] S108: The second predictor performs quality assessment on the image according to the global features to obtain a global quality score.

[0024] In the embodiment of the present invention, the first predictor may include a convolutional layer, and the second predictor may include an adaptive average pooling layer and a plurality of fully connected layers, wherein the number of fully connected layers may be set as required, and the specific value is not limited.

[0025] S110: The quality assessment model outputs a quality score map and a global quality score as a quality assessment result of the compressed video.

[0026] In one implementation manner, the quality assessment model may further include a first correction unit. Before the quality assessment model outputs the quality score map, the model may further include: The first correction unit generates pixel-by-pixel affine parameter pairs using convolution, and uses the affine parameter pairs to apply affine transformation spatially to correct the quality score map.

[0027] Among them, using affine parameters to apply affine transformation in space to correct the quality score map can specifically include: Use the following formula to correct the quality score map: ; in, is the quality score map, is the corrected quality score map, is an affine parameter pair.

[0028] In another implementation manner, the quality assessment model may further include a second correction unit. Before the quality assessment model outputs the global quality score, the following may also be included: The second correction unit uses channel attention to correct the global quality score.

[0029] The second correction unit corrects the global quality score by using channel attention, which may specifically include: The global quality score is input into a convolutional layer to generate a vector containing the number of output channels, and the vector is used to correct the global quality score along the channel direction.

[0030] The above method provided by the embodiment of the present invention inputs the image included in the compressed video into a quality assessment model, and the quality assessment model includes a feature extraction unit, a first predictor and a second predictor. The feature extraction unit extracts features from the input image to obtain local features and global features respectively. The first predictor performs quality assessment on multiple regions in the image according to the local features to obtain quality scores of the multiple regions, generates a quality score map according to the quality scores of the multiple regions, and the second predictor performs quality assessment on the image according to the global features to obtain a global quality score. Finally, the quality assessment model outputs the quality score map and the global quality score as the quality assessment result of the compressed video. Therefore, the quality assessment of the compressed video can be achieved based on a quality assessment model, which greatly simplifies the complexity of the quality assessment and improves the processing efficiency. Moreover, the results output by the model include local scores and global scores, which improve the perception ability of the model and the accuracy of the quality assessment.

[0031] like Figure 2 As shown, the embodiment of the present invention provides another compressed video quality assessment method, which may include the following steps: S202: Inputting an image included in the compressed video into a quality assessment model, where the quality assessment model includes a feature extraction unit, a first predictor, a second predictor, a first correction unit, and a second correction unit.

[0032] In the embodiment of the present invention, the quality assessment model can be trained for compressed videos that are pre-encoded using multiple different quantization parameters. Different compressed videos can be obtained by encoding using multiple different quantization parameters. The obtained compressed videos are used as training sets to train the quality assessment model, so that the quality assessment model can learn the ability to perform quality assessment on different compressed videos.

[0033] During the training phase, multiple quality evaluation criteria such as PSNR or SSIM can be used to add labels to the images in the training set. When training based on labels, multiple loss functions can be used to estimate the global quality score and the local quality score map. Among them, there can be multiple loss functions, such as using L1, etc., which are not specifically limited.

[0034] In the above training process, there is no need to train different models for different quantization parameters. Quality assessment can be achieved by training one model, which greatly simplifies the training process. In addition, the result of the training is one model instead of multiple models, which reduces the complexity of the model and improves the perception ability of the model.

[0035] In an embodiment of the present invention, the feature extraction unit may include a multi-layer network structure, each layer of the network structure includes a first convolutional layer and a second convolutional layer, the scale of the second convolutional layer in each layer of the network structure is smaller than the scale of the first convolutional layer, and the number of output channels of the multi-layer network structure increases successively.

[0036] Among them, the number of layers of the network structure in the feature extraction unit can be 2, 4, 6 or 8, etc., which is not specifically limited. The scale of the first convolution layer and the second convolution layer can also be set as needed, that is, the size of the convolution kernel is also not limited. The number of output channels of each layer in the multi-layer network structure can also be set as needed, and the specific value is not limited.

[0037] Figure 3 A schematic diagram of the structure of the quality assessment model provided by an embodiment of the present invention. Figure 3 , the feature extraction unit consists of a 4-layer network structure from left to right. Each layer of the network structure includes two convolutional layers: the first convolutional layer on the left and the second convolutional layer on the right. In each layer of the network structure: the first convolutional layer and the second convolutional layer are composed of a convolution, a batch normalization (BN) and a ReLU activation function; the convolution kernel size of the first convolutional layer is 3×3, and the convolution kernel size of the second convolutional layer is 2×2 to perform scale reduction operations. From left to right, the number of output channels of each layer of the 4-layer network structure is set to 32, 64, 128 and 256, respectively, that is, increasing in sequence.

[0038] S204: The feature extraction unit performs feature extraction on the input image to obtain local features and global features, and then executes steps S206 and S210 respectively.

[0039] S206: The first predictor performs quality assessment on multiple regions in the image according to the local features, obtains quality scores of the multiple regions, and generates a quality score map according to the quality scores of the multiple regions.

[0040] In the embodiment of the present invention, the first predictor may include a convolution layer. For example, the image width and height of the input compressed video are H and W respectively. The first predictor uses a convolution layer to generate a size of Quality score map .

[0041] S208: The first correction unit generates pixel-by-pixel affine parameter pairs using convolution, applies affine transformation spatially to correct the quality score map using the affine parameter pairs, and then executes step S214.

[0042] The first correction unit may use two convolutions to generate pixel-by-pixel affine parameter pairs. Of course, other methods may also be used, which will not be described in detail here.

[0043] The above steps S206 and S208 are steps of using local features for prediction and correction, thereby obtaining a more accurate quality score map.

[0044] In one implementation manner, the above step S208 may specifically include: The first correction unit uses convolution to generate pixel-by-pixel affine parameter pairs, and corrects the quality score map based on the affine parameter pairs using the following formula: ; in, is the quality score map, is the corrected quality score map, is an affine parameter pair, is the first affine parameter, is the second affine parameter.

[0045] The above-mentioned pixel-by-pixel method refers to generating affine parameter pairs based on pixel points in the image, and then correcting the quality score map by spatial feature transformation based on pixel points. This method of adaptively adjusting the quality score map can improve the accuracy of model quality assessment.

[0046] S210: The second predictor performs quality assessment on the image according to the global features to obtain a global quality score.

[0047] In the embodiment of the present invention, the second predictor includes an adaptive average pooling layer and a plurality of fully connected layers, wherein the number of fully connected layers may be 2, 3, or 5, etc., and the specific value is not limited.

[0048] For example, see Figure 3 The second predictor includes an adaptive average pooling layer and three fully connected layers. The three fully connected layers are shown in the dashed box from left to right. The global quality score can be predicted by the second predictor. .

[0049] S212: The second correction unit corrects the global quality score by using a channel attention method, and then executes step S214.

[0050] The above steps S210 and S212 are steps of using global features to perform prediction and correction, thereby obtaining a more accurate global quality score.

[0051] In one implementation manner, the above step S212 may specifically include: The global quality score is input into a convolutional layer to generate a vector containing the number of output channels, and the vector is used to correct the global quality score along the channel direction.

[0052] The convolution layer may be a convolution layer with a convolution kernel size of 1x1, through which a vector may be obtained. Since the vector includes the number of output channels, the global quality score may be corrected along the channel direction through the vector.

[0053] S214: The quality assessment model outputs a quality score map and a global quality score as a quality assessment result of the compressed video, and the process ends.

[0054] The above method provided by the embodiment of the present invention inputs the image included in the compressed video into the quality assessment model, the quality assessment model includes a feature extraction unit, a first predictor and a second predictor, the feature extraction unit extracts features from the input image to obtain local features and global features respectively, the first predictor performs quality assessment on multiple regions in the image according to the local features to obtain quality scores of the multiple regions, generates a quality score map according to the quality scores of the multiple regions, and the second predictor performs quality assessment on the image according to the global features to obtain a global quality score, and finally the quality assessment model outputs the quality score map and the global quality score as the quality assessment result of the compressed video, therefore, the quality assessment of the compressed video can be realized based on a quality assessment model, which greatly simplifies the complexity of the quality assessment and improves the processing efficiency. Moreover, the results output by the model include local scores and global scores, which can effectively perceive the quality fluctuations of the compressed video between the time domain and the spatial domain, realize hierarchical perception, improve the perception ability of the model, and improve the accuracy of the quality assessment, thereby facilitating the quality enhancement processing of the compressed video, with better enhancement effect, and can also improve the recovery ability of the compressed video.

[0055] The above method is a process of reasoning using a quality assessment model. The quality assessment model estimates the quality based on the input image and corrects the obtained global quality score and local quality score, which can further improve the model's perception ability and enhance the model's representation ability.

[0056] The above is a compressed video quality assessment method provided by an embodiment of the present invention. Based on the same idea, an embodiment of the present invention also provides a compressed video quality assessment device, such as Figure 4 As shown, the device includes a quality assessment model 400 , which includes a feature extraction unit 401 , a first predictor 402 , and a second predictor 403 . A feature extraction unit 401 is used to extract features from images included in the input compressed video to obtain local features and global features respectively; A first predictor 402 is used to perform quality assessment on multiple regions in the image according to local features, obtain quality scores of the multiple regions, and generate a quality score map according to the quality scores of the multiple regions; The second predictor 403 is used to evaluate the quality of the image according to the global features to obtain a global quality score; The input of the quality assessment model 400 is the image included in the compressed video, and the output of the quality assessment model 400 is the quality score map and the global quality score as the quality assessment result of the compressed video.

[0057] In an embodiment of the present invention, the feature extraction unit includes a multi-layer network structure, each layer of the network structure includes a first convolutional layer and a second convolutional layer, the scale of the second convolutional layer in each layer of the network structure is smaller than the scale of the first convolutional layer, and the number of output channels of the multi-layer network structure increases successively.

[0058] In the embodiment of the present invention, the first predictor includes a convolutional layer, and the second predictor includes an adaptive average pooling layer and a plurality of fully connected layers.

[0059] See also Figure 5 The quality assessment model may further include a first correction unit 404. Wherein: The first correction unit 404 is used to generate pixel-by-pixel affine parameter pairs using convolution before the quality assessment model outputs the quality score map, and to correct the quality score map by applying affine transformation in space using the affine parameter pairs.

[0060] In one implementation manner, the first correction unit 404 uses the affine parameter to apply an affine transformation in space to correct the quality score map, which may specifically include: Use the following formula to correct the quality score map: ; in, is the quality score map, is the corrected quality score map, is an affine parameter pair.

[0061] See also Figure 5The quality assessment model may further include a second correction unit 405. Wherein: The second correction unit 405 is used to correct the global quality score by using a channel attention method before the quality assessment model outputs the global quality score.

[0062] In one implementation, the second correction unit 405 corrects the global quality score by using a channel attention method, which may specifically include: The global quality score is input into a convolutional layer to generate a vector containing the number of output channels, and the vector is used to correct the global quality score along the channel direction.

[0063] In the embodiment of the present invention, the quality assessment model 400 may be trained for compressed videos that are pre-encoded using a plurality of different quantization parameters.

[0064] The above-mentioned device provided by the embodiment of the present invention can execute the method in any of the above-mentioned method embodiments. For detailed processes, please refer to the description of each method embodiment, which will not be repeated here.

[0065] The above-mentioned device provided by the embodiment of the present invention inputs the image included in the compressed video into the quality assessment model, and the quality assessment model includes a feature extraction unit, a first predictor and a second predictor. The feature extraction unit extracts features from the input image to obtain local features and global features respectively. The first predictor performs quality assessment on multiple regions in the image according to the local features to obtain quality scores of the multiple regions, generates a quality score map according to the quality scores of the multiple regions, and the second predictor performs quality assessment on the image according to the global features to obtain a global quality score. Finally, the quality assessment model outputs the quality score map and the global quality score as the quality assessment result of the compressed video. Therefore, the quality assessment of the compressed video can be achieved based on a quality assessment model, which greatly simplifies the complexity of the quality assessment and improves the processing efficiency. Moreover, the results output by the model include local scores and global scores, which improve the perception ability of the model and the accuracy of the quality assessment.

[0066] Figure 6 The hardware structure diagram of an electronic device for implementing various embodiments of the present invention is shown in FIG. The electronic device 600 includes but is not limited to: a radio frequency unit 601, a network module 602, an audio output unit 603, an input unit 604, a sensor 605, a display unit 606, a user input unit 607, an interface unit 608, a memory 609, a processor 610, and a power supply 611. Those skilled in the art will appreciate that Figure 6The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and the electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. In the embodiments of the present invention, the electronic device includes but is not limited to a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted terminal, a wearable device, and a pedometer.

[0067] Among them, the processor 610 is used to input the image included in the compressed video into a quality assessment model, and the quality assessment model includes a feature extraction unit, a first predictor and a second predictor; the feature extraction unit performs feature extraction on the input image to obtain local features and global features respectively; the first predictor performs quality assessment on multiple regions in the image according to the local features, obtains quality scores of the multiple regions, and generates a quality score map according to the quality scores of the multiple regions; the second predictor performs quality assessment on the image according to the global features to obtain a global quality score; the quality assessment model outputs the quality score map and the global quality score as the quality assessment result of the compressed video.

[0068] An embodiment of the present invention provides an electronic device, which inputs an image included in a compressed video into a quality assessment model, wherein the quality assessment model includes a feature extraction unit, a first predictor, and a second predictor. The feature extraction unit extracts features from the input image to obtain local features and global features respectively. The first predictor performs quality assessment on multiple regions in the image according to the local features to obtain quality scores of the multiple regions, generates a quality score map according to the quality scores of the multiple regions, and the second predictor performs quality assessment on the image according to the global features to obtain a global quality score. Finally, the quality assessment model outputs a quality score map and a global quality score as a quality assessment result of the compressed video. Therefore, the quality assessment of the compressed video can be achieved based on a quality assessment model, which greatly simplifies the complexity of the quality assessment and improves the processing efficiency. Moreover, the results output by the model include local scores and global scores, which improve the perception ability of the model and the accuracy of the quality assessment.

[0069] It should be understood that in the embodiment of the present invention, the radio frequency unit 601 can be used for receiving and sending signals during information transmission or communication. Specifically, after receiving downlink data from the base station, it is sent to the processor 610 for processing; in addition, uplink data is sent to the base station. Generally, the radio frequency unit 601 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, etc. In addition, the radio frequency unit 601 can also communicate with the network and other electronic devices through a wireless communication system.

[0070] The electronic device provides users with wireless broadband Internet access through the network module 602, such as helping users to send and receive emails, browse web pages, and access streaming media.

[0071] The audio output unit 603 can convert the audio data received by the RF unit 601 or the network module 602 or stored in the memory 609 into an audio signal and output it as sound. Moreover, the audio output unit 603 can also provide audio output related to a specific function performed by the electronic device 600 (for example, a call signal reception sound, a message reception sound, etc.). The audio output unit 603 includes a speaker, a buzzer, a receiver, etc.

[0072] The input unit 604 is used to receive audio or video signals. The input unit 604 may include a graphics processor (GPU) 6041 and a microphone 6042. The graphics processor 6041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The processed image frame can be displayed on the display unit 606. The image frame processed by the graphics processor 6041 can be stored in the memory 609 (or other storage medium) or sent via the radio frequency unit 601 or the network module 602. The microphone 6042 can receive sound and can process such sound into audio data. The processed audio data can be converted into a format output that can be sent to a mobile communication base station via the radio frequency unit 601 in the case of a telephone call mode.

[0073] The electronic device 600 also includes at least one sensor 605, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor includes an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 6061 according to the brightness of the ambient light, and the proximity sensor can turn off the display panel 6061 and / or the backlight when the electronic device 600 is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary, which can be used to identify the posture of the electronic device (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; the sensor 605 can also include a fingerprint sensor, a pressure sensor, an iris sensor, a molecular sensor, a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, etc., which will not be repeated here.

[0074] The display unit 606 is used to display information input by the user or information provided to the user. The display unit 606 may include a display panel 6061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0075] The user input unit 607 can be used to receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device. Specifically, the user input unit 607 includes a touch panel 6071 and other input devices 6072. The touch panel 6071, also known as a touch screen, can collect user touch operations on or near it (such as operations performed by users using fingers, styluses, or any other suitable objects or accessories on or near the touch panel 6071). The touch panel 6071 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact point coordinates, and then sends it to the processor 610, receives the command sent by the processor 610 and executes it. In addition, the touch panel 6071 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 6071, the user input unit 607 may also include other input devices 6072. Specifically, other input devices 6072 may include but are not limited to a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be described in detail here.

[0076] Furthermore, the touch panel 6071 may be overlaid on the display panel 6061. When the touch panel 6071 detects a touch operation on or near it, it is transmitted to the processor 610 to determine the type of touch event. Then, the processor 610 provides corresponding visual output on the display panel 6061 according to the type of touch event. Figure 6 In the figure, the touch panel 6071 and the display panel 6061 are used as two independent components to realize the input and output functions of the electronic device. However, in some embodiments, the touch panel 6071 and the display panel 6061 can be integrated to realize the input and output functions of the electronic device, which is not limited here.

[0077] The interface unit 608 is an interface for connecting an external device to the electronic device 600. For example, the external device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, etc. The interface unit 608 may be used to receive input (e.g., data information, power, etc.) from an external device and transmit the received input to one or more elements within the electronic device 600 or may be used to transmit data between the electronic device 600 and an external device.

[0078] The memory 609 can be used to store software programs and various data. The memory 609 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 609 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0079] The processor 610 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 609 and calling data stored in the memory 609, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. The processor 610 may include one or more processing units; preferably, the processor 610 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 610.

[0080] The electronic device 600 may also include a power supply 611 (such as a battery) for supplying power to various components. Preferably, the power supply 611 may be logically connected to the processor 610 via a power management system, thereby managing functions such as charging, discharging, and power consumption through the power management system.

[0081] Preferably, an embodiment of the present invention further provides an electronic device, comprising a processor 610, a memory 609, and a computer program stored in the memory 609 and executable on the processor 610. When the computer program is executed by the processor 610, the various processes of the above-mentioned method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0082] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the various processes of the above-mentioned method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0083] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0084] The above-mentioned computer-readable storage medium provided by the embodiment of the present invention inputs the image included in the compressed video into the quality assessment model, and the quality assessment model includes a feature extraction unit, a first predictor and a second predictor. The feature extraction unit extracts features from the input image to obtain local features and global features respectively. The first predictor performs quality assessment on multiple regions in the image according to the local features to obtain quality scores of the multiple regions, generates a quality score map according to the quality scores of the multiple regions, and the second predictor performs quality assessment on the image according to the global features to obtain a global quality score. Finally, the quality assessment model outputs the quality score map and the global quality score as the quality assessment result of the compressed video. Therefore, the quality assessment of the compressed video can be achieved based on a quality assessment model, which greatly simplifies the complexity of the quality assessment and improves the processing efficiency. Moreover, the results output by the model include local scores and global scores, which improve the perception ability of the model and the accuracy of the quality assessment.

[0085] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0086] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0087] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0088] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0089] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0090] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0091] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0092] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0093] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A compressed video quality assessment method, characterized in that: The method comprises: Inputting an image included in the compressed video into a quality assessment model, the quality assessment model including a feature extraction unit, a first predictor, and a second predictor; The feature extraction unit performs feature extraction on the input image to obtain local features and global features respectively; The first predictor performs quality assessment on multiple regions in the image according to the local features to obtain quality scores of the multiple regions, and generates a quality score map according to the quality scores of the multiple regions; The second predictor performs quality assessment on the image according to the global feature to obtain a global quality score; The quality assessment model outputs the quality score map and the global quality score as a quality assessment result of the compressed video.

2. The method according to claim 1, characterized in that The feature extraction unit includes a multi-layer network structure, each layer of the network structure includes a first convolutional layer and a second convolutional layer, the scale of the second convolutional layer in each layer of the network structure is smaller than the scale of the first convolutional layer, and the number of output channels of the multi-layer network structure increases sequentially.

3. The method according to claim 1, characterized in that The first predictor includes a convolutional layer, and the second predictor includes an adaptive average pooling layer and a plurality of fully connected layers.

4. The method according to claim 1, characterized in that: The quality assessment model further includes a first correction unit. Before the quality assessment model outputs the quality score map, it further includes: The first correction unit generates pixel-by-pixel affine parameter pairs using convolution, and uses the affine parameter pairs to apply affine transformation spatially to correct the quality score map.

5. The method according to claim 4, characterized in that The using the affine parameter to apply an affine transformation spatially to correct the quality score map comprises: The quality score map is corrected using the following formula: ； in, is the mass fraction map, is the corrected quality score map, is the affine parameter pair.

6. The method according to claim 1, characterized in that The quality assessment model further includes a second correction unit. Before the quality assessment model outputs the global quality score, it further includes: The second correction unit corrects the global quality score by using a channel attention method.

7. The method according to claim 6, characterized in that The second correction unit corrects the global quality score by using a channel attention method, including: The global quality score is input into a convolutional layer to generate a vector containing the number of output channels, and the vector is used to correct the global quality score along the channel direction.

8. The method according to any one of claims 1 to 7, characterized in that The quality assessment model is trained by using compressed videos pre-encoded using a plurality of different quantization parameters.

9. A compressed video quality assessment device, characterized in that: The apparatus comprises a quality assessment model, the quality assessment model comprising a feature extraction unit, a first predictor, and a second predictor; The feature extraction unit is used to extract features from images included in the input compressed video to obtain local features and global features respectively; The first predictor is used to respectively perform quality assessment on multiple regions in the image according to the local features to obtain quality scores of the multiple regions, and generate a quality score map according to the quality scores of the multiple regions; The second predictor is used to perform quality assessment on the image according to the global feature to obtain a global quality score; The input of the quality assessment model is the image included in the compressed video, and the output of the quality assessment model is the quality score map and the global quality score as the quality assessment result of the compressed video.

10. An electronic device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the compressed video quality assessment method according to any one of claims 1 to 8 when executed by the processor.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the compressed video quality assessment method described in any one of claims 1 to 8 are implemented.

12. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the steps of the compressed video quality assessment method according to any one of claims 1 to 8.

Images