No-reference image evaluation method, device, electronic device and computer storage medium

Abstract

The present application discloses a method, device, electronic device and computer storage medium for image evaluation without reference, the method comprising: preprocessing a target file to determine at least one input image corresponding to an image block in the target file; extracting features from the input image based on a feature extraction network and a deep self-attention transformation neural network in a pre-trained image evaluation model to determine a feature map corresponding to the input image, and calculating and analyzing the feature point positions and feature vectors in the feature map to determine the quality evaluation score of the input image; and determining the quality evaluation score of the target file according to the quality evaluation score of each input image. Thus, by processing the input image through the feature extraction network and the deep self-attention transformation neural network, the input image information involved in the image quality evaluation process is more complete, and the accuracy of the image quality evaluation can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method, device, electronic equipment and computer storage medium for evaluating a reference-free image. Background Art

[0002] Video is our main medium for obtaining external information and a tool for communication. The video processing process includes acquisition, processing, compression, transmission and reconstruction, etc. Due to the limitations of equipment conditions and processing technology, the video quality will inevitably be damaged during the processing process. Therefore, it is particularly important to establish an effective quality assessment system. In the current popular evaluation system, the main evaluation methods include subjective evaluation and objective evaluation. Among them, subjective evaluation based on human eye observation is the most accurate, but it requires a lot of time and energy and cannot be processed in batches. Objective evaluation can be divided into three different types according to whether the original video information is used: full reference, semi-reference and no reference. In many cases, the specific information of the original video cannot be obtained, which limits the application of full reference and semi-reference evaluation methods in actual production and life. Therefore, it is of great practical significance to study the video quality evaluation method without reference.

[0003] However, in the existing no-reference quality assessment methods, large-size images cannot be directly input into the network for processing. The quality changes caused by scaling, the semantic loss caused by cropping, and the deficiencies in processing image detail texture information mean that the accuracy still needs to be improved. Summary of the invention

[0004] In view of this, an object of the embodiments of the present invention is to provide a method, device, electronic device and computer storage medium for evaluating images without reference, so as to improve the accuracy of image quality evaluation.

[0005] In a first aspect, an embodiment of the present invention is to provide a method for evaluating a reference-free image, the method comprising:

[0006] Obtaining a target file, wherein the target file is an image or a video;

[0007] Preprocessing the target file to determine at least one corresponding input image, wherein the input image corresponds to an image block in the target file, and the image block corresponds to a local image in the target file;

[0008] Performing feature extraction on the input image based on a feature extraction network in a pre-trained image evaluation model to determine a feature map corresponding to the input image;

[0009] Based on the deep self-attention transformation neural network in the pre-trained image evaluation model, the feature point positions and feature vectors in the feature map are calculated and analyzed to determine the quality evaluation score of the input image;

[0010] The quality evaluation score of the target file is determined according to the quality evaluation score of each of the input images.

[0011] Furthermore, the method further comprises:

[0012] Acquire a training data set, wherein the training data set includes a pre-training data set and a fine-tuning data set, wherein the pre-training data set includes a plurality of input images with objective labels, and the fine-tuning data set includes a plurality of input images with subjective labels;

[0013] Inputting the input images in the pre-training data set into a preset image evaluation model for training, and determining a pre-training model;

[0014] The input images in the fine-tuning dataset are input into the pre-trained model for training to determine the pre-trained image evaluation model.

[0015] Further, the preprocessing of the target file to determine the corresponding at least one input image includes:

[0016] Preprocessing the target file to determine at least one image block corresponding to the target file;

[0017] Electro-optical conversion calculation is performed on each of the image blocks to determine a corresponding input image.

[0018] Further, performing electro-optical conversion calculation on the image block to determine the corresponding input image includes:

[0019] Determine an electro-optical conversion formula under the corresponding color gamut and dynamic range specifications based on the image format of the target file;

[0020] The image block is converted based on the electro-optical conversion formula to determine a corresponding input image.

[0021] Furthermore, when the target file is a video, the preprocessing includes:

[0022] Extracting frames from the target file to determine a frame image corresponding to the target file;

[0023] Each of the frame images is cropped to determine a corresponding image block.

[0024] Furthermore, when the target file is a video, the target file includes at least one frame image, and determining the quality evaluation score of the target file according to the quality evaluation scores of each of the input images includes:

[0025] Performing statistical calculation on the quality evaluation scores of the input images corresponding to the frame images to determine the quality evaluation scores of the corresponding frame images, wherein the statistical calculation includes taking an average value, taking a maximum value, taking a minimum value, taking a median, or taking a mode;

[0026] The quality evaluation score of each frame image is statistically calculated to determine the quality evaluation score of the target file.

[0027] Furthermore, the feature extraction network in the pre-trained image evaluation model extracts features from the input image to determine a feature map corresponding to the input image, including:

[0028] Extract features from the input image based on a feature extraction network in a pre-trained image evaluation model to determine a feature map under a preset number of channels;

[0029] Feature mapping is performed on the feature map under the preset number of channels to determine the feature map under the target number of channels.

[0030] In a second aspect, an embodiment of the present invention is intended to provide a non-reference image processing device, the device comprising:

[0031] A file acquisition unit, used to acquire a target file, wherein the target file is an image or a video;

[0032] An image processing unit, configured to pre-process the target file to determine at least one corresponding input image, wherein the input image corresponds to an image block in the target file, and the image block corresponds to a local image in the target file;

[0033] A feature processing unit is used to extract features from the input image based on a feature extraction network in a pre-trained image evaluation model to determine a feature map corresponding to the input image; and to calculate and analyze feature point positions and feature vectors in the feature map based on a deep self-attention transformation neural network in the pre-trained image evaluation model to determine a quality evaluation score of the input image;

[0034] The quality evaluation unit is used to determine the quality evaluation score of the target file according to the quality evaluation score of each of the input images.

[0035] In a third aspect, an embodiment of the present invention is to provide an electronic device, comprising a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement any of the methods described above.

[0036] In a fourth aspect, an embodiment of the present invention aims to provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps as described in any one of the above items are implemented.

[0037] The technical solution of this embodiment pre-processes the acquired target file, determines at least one input image corresponding to the image block in the target file, extracts features from the input image based on the feature extraction network and deep self-attention transformation neural network in the pre-trained image evaluation model, determines the feature map corresponding to the input image, calculates and analyzes the feature point positions and feature vectors in the feature map, determines the evaluation score of the input image, and then determines the quality evaluation score of the target file based on the quality evaluation score of each input image. Thus, by processing the input image through the feature extraction network and the deep self-attention transformation neural network, the input image information involved in the image quality evaluation process is more complete, which can improve the accuracy of the image quality evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0039] Figure 1 is a flow chart of an image evaluation method according to an embodiment of the present invention;

[0040] Figure 2 is a schematic diagram of a target file preprocessing process according to an embodiment of the present invention;

[0041] Figure 3 is a flow chart of determining an input image according to an embodiment of the present invention;

[0042] Figure 4 is a schematic diagram of a processing process of an image quality assessment network according to an embodiment of the present invention;

[0043] Figure 5 is a flow chart of an image evaluation network training method according to an embodiment of the present invention;

[0044] Figure 6 is a schematic diagram of an image evaluation device according to an embodiment of the present invention;

[0045] Figure 7 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The present application is described below based on embodiments, but the present application is not limited to these embodiments. In the detailed description of the present application below, some specific details are described in detail. It is possible for those skilled in the art to fully understand the present application without the description of these details. In order to avoid confusing the essence of the present application, known methods, processes, flows, components and circuits are not described in detail.

[0047] In addition, persons of ordinary skill in the art will appreciate that the drawings provided herein are for illustration purposes and are not necessarily drawn to scale.

[0048] Unless the context clearly requires otherwise, the words "include", "comprising" and similar words throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, the meaning is "including but not limited to".

[0049] In the description of this application, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, the meaning of "plurality" is two or more.

[0050] The solutions described in this specification and in the examples, if they involve the processing of personal information, will be processed on the premise of having a legal basis (such as obtaining the consent of the subject of personal information, or being necessary for the performance of a contract, etc.), and will only be processed within the scope of regulations or agreements. If a user refuses to process personal information other than the necessary information for basic functions, it will not affect the user's use of basic functions.

[0051] Compared with other computer vision tasks, the main processing difficulties in the no-reference image quality assessment task are as follows: 1. Large-size images cannot be directly input into the network, and the images are often processed by scaling or cropping. In the two methods, cropping will lose semantic information, while scaling will not lose semantic information but will affect the image quality. Therefore, cropping is generally selected when adjusting the image size. However, if cropping is performed based on the RGB channel, more voice information will be lost; 2. Image quality assessment needs to pay attention to semantic information, and the network needs to learn detailed texture information. According to the current level of encoding and decoding, the quality of the image is mainly reflected in the processing of the edge details of the image by the encoding system, especially the difference in brightness information. The RGB image mainly shows semantic features and cannot reflect the perceptual features of the human eye well; 3. The no-reference image quality assessment task does not have the original video sequence as a reference comparison, and cannot accurately locate the damaged area and the degree of damage. In addition, the degree of damage is less related to the semantic features, which increases the learning difficulty of the network and often results in the non-convergence of the network.

[0052] At the same time, with the improvement of hardware level, larger-sized images have been allowed to be input into deep learning networks, but high-definition and ultra-high-definition videos still need to be scaled. The current common method is to input RGB images into the network, use a deep network to extract semantic features and fuse deep features with shallow features, but this still does not get rid of the disadvantage that the network mainly relies on semantic information, and features of different scales can be understood as the result of different scales of the image passing through the network. There will be differences in the quality of the scaled image and the original image, which will seriously affect the effect of quality evaluation. In view of this, an embodiment of the present invention aims to provide an image evaluation method to use more image information when evaluating image quality, thereby improving the accuracy of image quality evaluation.

[0053] Figure 1 FIG. 1 is a flow chart of an image evaluation method according to an embodiment of the present invention. Figure 1 As shown, the image evaluation method in this embodiment includes the following steps.

[0054] In step S100, a target file is obtained.

[0055] In this embodiment, the target file is a file with damaged or distorted quality to be evaluated, and the target file may be in the form of an image or a video.

[0056] In step S200, the target file is preprocessed to determine at least one corresponding input image, wherein the input image corresponds to an image block in the target file, and the image block corresponds to a local image in the target file.

[0057] In this embodiment, in an optional implementation, when the target file is an image, the target file is directly cropped to be cropped into a plurality of image blocks, namely, image blocks corresponding to the target file.

[0058] Optionally, in this embodiment, when cropping the target file in the form of an image, the target file is cropped with a preset sliding window, and the format of the preset window can be set to a size of H×W. Further, in this embodiment, starting from the upper left corner of the target file, the target file can be cropped with a preset window to obtain at least one image block of a size of H×W as the input image corresponding to the target file.

[0059] In another optional implementation, when the target file is a video image, the following Figure 2 The preprocessing process shown processes the target file in video form, including extracting frames from the target file A, determining frame images A1, A2, ..., An corresponding to the target file, and then cropping each frame image to determine the corresponding image blocks Pn1, Pn2, ..., Pni.

[0060] Optionally, in this embodiment, when performing frame extraction operation on a video, in order to improve image processing efficiency, the target file is extracted according to a preset frame interval. For example, the frame interval can be set to 20 frames or 50 frames, that is, a frame is extracted every 20 frames or 50 frames. Another option is that in this embodiment, when cropping a frame image in a target file in the form of a video, each frame image is cropped separately from the upper left corner of the frame image using a preset sliding window. The format of the preset window can be set to a size of H×W. Thus, by processing the target file and the frame image separately using the above method, at least one image block of a size of H×W can be obtained, and the image block is used as the input image of the corresponding frame image.

[0061] Furthermore, since all videos are composed of frame images of different frames, the processing process of the target file in video form will cover the processing process of the target file in image form. Therefore, the following content of this embodiment is mainly explained by taking the target file as a video as an example.

[0062] Optionally, in order to better adapt to the task of video image quality evaluation, focus on the characteristics of detail texture information, and fully consider the perceptual characteristics of human eyes when watching videos, this embodiment will also perform electro-optical conversion calculations on the original video signal in the target file, and visualize the colors in the image through the CIE Yxy color model (i.e., chromaticity diagram), so that the image information in the target file can be more accurately presented and used for subsequent quality evaluation analysis, further improving the accuracy of image evaluation. Correspondingly, in this embodiment, based on Figure 3 The processing procedure shown is to determine the input image corresponding to the target file.

[0063] In step S210, the target file is preprocessed to determine at least one image block corresponding to the target file.

[0064] In this embodiment, the preprocessing process of the target file includes frame extraction and cropping, wherein frame extraction is used to extract a frame image under a specific video frame from the target file, and cropping is used to separate an image block from the frame image. Optionally, in this embodiment, the aforementioned frame extraction method and cropping method are used to process the target file, which will not be described in detail here.

[0065] In step S220, electro-optical conversion calculation is performed on each image block to determine the corresponding input image.

[0066] In this embodiment, electro-optical conversion calculations are performed on each image block according to the video format corresponding to the target file, and different electro-optical conversion formulas correspond to different video formats. Furthermore, when determining the input image corresponding to the image block, this embodiment determines the electro-optical conversion formula under the corresponding color gamut and dynamic range specifications based on the image format of the target file, and then converts the image block based on the electro-optical conversion formula to determine the corresponding input image. For example, for videos in SDR / BT.709 format, the EOTF formula specified in ITU-R BT.709 is used for conversion; for videos in HDR / BT.2020 format, the EOTF formula specified in ITU-RBT.2100 is used for conversion. It should be noted that the conversion results of different electro-optical conversion formulas have different effects on the accuracy of the test method.

[0067] In step S300, the input image is processed based on the pre-trained image evaluation model to determine the quality evaluation score of the corresponding input image.

[0068] In this embodiment, the image evaluation model includes a feature extraction network and a deep self-attention transformation neural network. When determining the quality evaluation score of the input image, firstly, based on the feature extraction network in the pre-trained image evaluation model, the input image is feature extracted to determine the feature map corresponding to the input image, and then based on the deep self-attention transformation neural network in the pre-trained image evaluation model, the feature point positions and feature vectors in the feature map are calculated and analyzed to determine the quality evaluation score of the input image.

[0069] Optionally, since the number of channels corresponding to the feature map output by the feature extraction network may be different from the number of channels processed in the deep self-attention transform neural network, in order to ensure the smooth extraction and processing of relevant features in the input image, in this embodiment, when the feature extraction network in the pre-trained image evaluation model is used to extract features from the input image to determine the feature map corresponding to the input image, the feature extraction network in the pre-trained image evaluation model is used to extract features from the input image to determine the feature map under the preset number of channels; feature mapping is performed on the feature map under the preset number of channels to determine the feature map under the target number of channels. Furthermore, feature mapping can be achieved through a two-dimensional convolution operation.

[0070] Furthermore, the feature extraction network in this embodiment adopts ResNet-50, and the deep self-attention transformation neural network adopts the Transformer network. Among them, ResNet-50 is a deep residual network in the ResNet series, which is usually composed of 50 convolutional layers, including a 7×7 convolutional layer and a global average pooling layer. The core of ResNet-50 is the residual module, which is composed of two convolutional layers and a shortcut connection. The shortcut connection allows information to jump and pass in the network, thereby avoiding the problems of gradient disappearance and gradient explosion, so that the network can be deeper and easier to train. The training process of ResNet-50 usually uses the stochastic gradient descent (SGD) optimizer and the cross entropy loss function, and also uses data enhancement and batch normalization techniques to improve the performance and robustness of the model, and can show good performance in tasks such as image recognition, object detection and image segmentation. Transformer is a neural network model based on the self-attention mechanism (Self-Attention), which is mainly composed of two parts: encoder and decoder. They are all stacked by multiple layers, and each layer contains a multi-head self-attention mechanism and a fully connected feedforward network. In the encoder, the input sequence is processed through multiple layers and converted into a set of high-level features that can be used for various natural language processing tasks. In the decoder, these high-level features are used to generate sequences in the target language. The self-attention mechanism is the core of the Transformer, which constructs feature representations by calculating the correlation between different positions in the input sequence, rather than relying on a fixed window size or sequence order like traditional RNN or CNN. This attention mechanism can more flexibly capture the dependencies between different positions in the sequence and can be calculated in parallel, so the Transformer has better performance when processing long sequences. Transformer has been proven to be an efficient, flexible, and scalable model that has been widely used in the field of natural language processing and has also been explored and applied in other fields such as computer vision. Therefore, in this embodiment, two excellent neural network models, ResNet-50 and deep self-attention transform neural network, are used to extract feature information in the target file image block, which can optimize the image evaluation model so that the image evaluation model has a feature extraction capability far exceeding that of the convolutional neural network, and can extract more image information for calculation and analysis of feature point positions and feature vectors, as well as quality evaluation, while reducing the impact of information loss and incomplete information on image quality evaluation, thereby improving the accuracy of image evaluation.

[0071] Figure 4 Schematic diagram of the processing process of the image quality assessment network of the embodiment of the present invention. Figure 4As shown, in this embodiment, the image block P' obtained after the electro-optical conversion of the image block P corresponding to the target file is input into the feature extraction network. Among them, the image format of the image block P is YUV or RGB, the image size is H×W, and the number of corresponding image pixel channels is 3. After electro-optical conversion, an image block P' that can better describe human visual perception is generated, and the image size and the number of channels remain unchanged compared to the image block P. The feature extraction network performs multi-layer feature extraction on the image block P' to obtain a feature map B. The image size of the feature map B becomes 1 / 32 of the image block P', and the number of channels is also enriched from the original 3 channels to 2048 channels. Afterwards, feature mapping is performed on the feature map B through a two-dimensional convolution layer. The output channel of the convolution is 32, and the convolution kernel size is 1×1, so that a feature map B' with unchanged size and 32 channels is obtained. Finally, the feature map B' is input into the converter network, that is, the position information of each pixel on the feature map B' and all channel information are input into the encoder of the converter network structure to learn the relationship between each pixel position and output the quality evaluation score of the corresponding image block obtained by regression based on the multi-head attention mechanism.

[0072] Optionally, in this embodiment, based on Figure 5 The steps shown are used to train the image evaluation model to further optimize the image evaluation model and improve the overall performance of the image evaluation model.

[0073] In step S310, a training data set is obtained.

[0074] In this embodiment, the training data set includes a pre-training data set and a fine-tuning data set, wherein the pre-training data set includes a plurality of input images with objective labels, and the fine-tuning data set includes a plurality of input images with subjective labels.

[0075] Optionally, in this embodiment, when obtaining a training data set, a distorted video is first obtained, and the distorted video is subjected to frame extraction processing to obtain a frame extraction image sequence, and then each frame image in the frame extraction image sequence is subjected to image block cropping according to a preset cropping method (for example, a picture of size H×W is obtained by center cropping) to obtain a corresponding image block. Afterwards, the cropped image block is subjected to corresponding electro-optical conversion according to the video format, and the electro-optically converted image block is used as an input image in the training data set. Further, after the image preprocessing is completed, the objective label and subjective label of each video are respectively assigned to the image block obtained by the video after frame extraction and cropping, and a pre-training data set and a fine-tuning data set in the training data set are generated. Specifically, the objective label of each input image is the objective label of the video to which the image block corresponding to the input image belongs, and each input image and the corresponding objective label together constitute the data in the pre-training data set. The subjective label of each input image is the subjective label of the video to which the image block corresponding to the input image belongs, and each input image and the corresponding subjective label together constitute the data in the fine-tuning data set.

[0076] Optionally, the objective label in this embodiment is determined according to the value of the objective indicator. The objective indicator is an objective indicator of quality without reference, which is used to evaluate the visual quality of digital images, including but not limited to NIQE, BRISQUE, etc. At the same time, the subjective label in this embodiment is determined according to the value of the subjective indicator. The subjective indicator is a subjective indicator of quality without reference, which is used to evaluate the visual quality of digital images, and can be MOS or other indicators. Furthermore, the objective label in this embodiment can be a label formed by the objective indicator and the numerical value of the objective indicator, for example, NIQE=6; it can also be determined according to the label attribute represented by the value range to which the value of the objective indicator belongs. For example, since the NIQE value is generally between 0-10, it represents good image quality. Therefore, when the NIQE value is between 0-10, the objective label of the corresponding image is set to "good image quality", and when the NIQE value exceeds 10, the objective label of the corresponding image is set to "poor image quality". At the same time, the subjective label can be a label formed by the subjective index and the value of the subjective index, such as MOS=5; it can also be determined according to the label attribute represented by the value range of the subjective index value, for example, when the subjective index value MOS is greater than 4.5, the subjective label of the corresponding image is set to "good image quality". Therefore, the setting of objective labels and subjective labels is realized through the above method, making the setting of objective labels and subjective labels more flexible and convenient.

[0077] Further, the value of the objective indicator in this embodiment can be determined according to the corresponding evaluator. For example, the objective indicators NIQE and BRISQUE can be determined by the corresponding NIQE evaluator and BRISQUE evaluator. Among them, the NIQE evaluator (Natural Image Quality Evaluator) is an evaluator that evaluates image quality based on image statistical features. It quantifies the quality of the image by calculating the local statistical features of the image, including contrast, sharpness, and color saturation. The lower the NIQE score, the higher the visual quality of the image. It is generally believed that images with a NIQE score between 0 and 10 have good quality, and scores exceeding 10 indicate poor image quality. The BRISQUE evaluator (Blind / Referenceless Image Spatial Quality Evaluator) is also an evaluator based on image statistical features, which uses the spatial information of the image to extract features, and then uses these features to calculate the quality score of the image. At the same time, the subjective indicator MOS (Mean Opinion Score) is a label for subjective image and audio quality assessment, which determines the standard of image or audio quality through subjective assessment by human reviewers. In MOS evaluation, a group of reviewers are asked to evaluate one or more images or audio files and give a score from 0 to 100 to represent their overall impression of the quality. These scores are averaged to obtain a MOS score, which is generally regarded as an objective evaluation of the quality of the file. Therefore, in this embodiment, the objective label or subjective label of each image in the training data set is determined by the above method, and then the pre-training data set and the fine-tuning data set in the training data set are determined. The processing process is simple and convenient, and the results are reliable, which is conducive to improving the training efficiency of the image evaluation model and the accuracy of the output results, thereby improving the image evaluation efficiency of the target file and the accuracy of the image evaluation results.

[0078] Optionally, in order to further improve the accuracy of the image evaluation model, the number of input images corresponding to the training data set in this embodiment and the typical scenes covered by the input image content should be as large as possible. For example, the training data set covers at least 10 typical scenes, including indoors, outdoors, buildings, people, natural scenery, etc. At the same time, the number of input images in the pre-training data set in this embodiment is at least 10,000. The number of input images corresponding to the fine-tuning data set is at least 300. However, it should be noted that the above values ​​are only examples. On the basis of ensuring the processing efficiency of image evaluation accuracy, they can be adjusted according to the actual application scenario, and this is not limited here.

[0079] In step S320, the input image in the pre-training data set is input into a preset image evaluation model for training to determine the pre-training model.

[0080] In this embodiment, after determining the pre-training data set and the image evaluation model structure (the image evaluation model structure has been introduced before and will not be repeated here), the input image in the pre-training data set is input into the preset image evaluation model for training, and the image evaluation model with certain model parameters obtained through training is determined as the pre-training model.

[0081] Optionally, in this embodiment, the preset image evaluation model is trained using the Adam update strategy and the L1 loss function (used to measure the difference between the model prediction score and the target score), the learning rate adopts the gradient decay method, the initial learning rate of the feature extraction network in the image evaluation model is set to 0.01, and the initial learning rate of the converter network is set to 0.0001. Therefore, in this embodiment, the preset image evaluation model is trained by the above method to determine the pre-trained model, so that the image evaluation model can extract and use more image features for image evaluation, which is conducive to improving the robustness of image evaluation.

[0082] In step S330, the input image in the fine-tuning data set is input into the pre-training model for training to determine the pre-trained image evaluation model.

[0083] In this embodiment, after determining the fine-tuning data set and the pre-training model, the pre-training model parameters are first loaded, and then the input image in the fine-tuning data set is input into the pre-training model for training, so as to optimize and adjust the model parameters in the pre-training model, and the optimized and adjusted pre-training model is determined as the pre-trained image evaluation model, which is applied in the image evaluation process. Therefore, in this embodiment, the image evaluation model is trained and optimized by combining pre-training and fine-tuning, which can further improve the overall performance of the image evaluation model.

[0084] Optionally, in this embodiment, the Adam update strategy and L1 loss function are used to train the preset image evaluation model, the learning rate adopts the gradient decay method, the initial learning rate of the feature extraction network in the image evaluation model is set to 0.01, and the initial learning rate of the converter network is set to 0.00001. Therefore, in this embodiment, the pre-trained model that has been trained is optimized and trained by the above method, and then the image evaluation model that can be used in the image evaluation process is determined, which can improve the generalization performance and reliability of the image evaluation model.

[0085] Optionally, in this embodiment, after determining the pre-trained image evaluation model, the model will also be verified. The verification process is completed using test videos, which include 300 4K ultra-high-definition videos with different distortion levels. When testing the effect of the image evaluation model, the test videos can be randomly divided into 5 groups, numbered as Group 1, Group 2, Group 3, Group 4, and Group 5, with 60 videos in each group, and one of the groups is designated as the test set, and the remaining 4 groups are fine-tuning training sets. Five rounds of testing are performed to obtain the output results of the image evaluation model after each round of testing (that is, the objective evaluation results), and after determining the correlation coefficients (such as PLCC, SRCC, etc.) between the objective evaluation results (image evaluation model output results) after each round of testing and the actual subjective evaluation results, the average value of each correlation coefficient is taken, and the average value is determined as the model effect of the currently trained image evaluation model.

[0086] Furthermore, in order to improve the measurement accuracy of the model effect of the image evaluation model under different forms of input images, in this embodiment, when testing the pre-trained image evaluation model, the model can also be fine-tuned and trained based on other groups of videos outside the test set in the test video, and then tested based on the test set after the training is completed. For example, when Group 5 is used as the test set, Group 1, Group 2, Group 3, and Group 4 are first combined into a training set, including 240 videos. The pre-trained image evaluation model is fine-tuned and trained again with the images in the training set, and the training strategy can adopt the same method as described above. After the image evaluation model training is completed, the image evaluation model is tested with the videos in Group 5. In this way, one round of testing of the image evaluation model is completed, and the composition of the training set and the test machine can be adjusted to complete multiple rounds of testing, and then the overall effect of the image evaluation model is determined according to the model output results of each round of testing.

[0087] Compared with the NIQE model used in the prior art, the quality evaluation score obtained by the image evaluation model obtained through training in this embodiment for quality evaluation of the input image has a higher correlation coefficient with the actual subjective evaluation result, that is, the image evaluation result obtained by the image evaluation model in this embodiment is more accurate. In addition, compared with the model output results when the image block in the YUV input form and the RGB input form is used as the input image, the image evaluation result corresponding to the image generated after the electro-optical conversion of the image block is used as the input image is more accurate, that is, the use of the input image form after electro-optical conversion is conducive to further improving the accuracy of image evaluation.

[0088] In step S400, the quality evaluation score of the target file is determined according to the quality evaluation score of each input image.

[0089] In this embodiment, after the quality evaluation score of each input image in the target file is determined by the image evaluation model obtained by pre-training, the quality evaluation score of the corresponding target file is determined according to the quality evaluation score of each input image. In an optional implementation, when the target file is a video, the target file includes multiple frame images. When determining the quality evaluation score of the target file, in this embodiment, the quality evaluation score of the corresponding frame image is first determined according to the quality evaluation score of each image block corresponding to each frame image, and then the quality evaluation score corresponding to the target file is determined according to the quality evaluation score of each frame image.

[0090] Optionally, in this embodiment, the quality evaluation scores of each input image corresponding to the frame image are first statistically calculated to determine the quality evaluation score of the corresponding frame image; then the quality evaluation score of each frame image is statistically calculated to determine the quality evaluation score of the target file. Among them, the statistical calculation includes taking the average value, taking the maximum value, taking the minimum value, taking the median or taking the mode. Specifically, the statistical calculation in this embodiment adopts the method of taking the average value. Assume that the target file A includes frame images A1, A2, ..., An, the quality evaluation score of the target file is S, the quality evaluation scores corresponding to each frame image are S1, S2, ..., Sn, and the image blocks corresponding to each frame image An include Pn1, Pn2, ..., Pni, and the quality evaluation scores corresponding to each image block are Sn1, Sn2, ..., Sni; thus, it can be determined that the quality evaluation score Sn of each frame image is (Sn1+Sn2+...+Sni) / i, and the quality evaluation score S corresponding to the target file is (S1+S2+...+Sn) / n.

[0091] In another optional implementation, when the target file is an image, since the target file is cropped into multiple input images, the quality evaluation score of each input image in the target file can be statistically calculated, and the statistical calculation result is determined as the quality evaluation score of the target file. The statistical calculation in this embodiment adopts the same method as above, which will not be repeated here.

[0092] The technical solution of this embodiment pre-processes the acquired target file to determine at least one input image corresponding to the image block in the target file, extracts features from the input image based on the feature extraction network and deep self-attention transformation neural network in the pre-trained image evaluation model to determine the feature map corresponding to the input image, calculates and analyzes the feature point position and feature vector in the feature map, determines the evaluation score of the input image, and then determines the quality evaluation score of the target file according to the quality evaluation score of each input image. Thus, the input image is processed by the feature extraction network and the deep self-attention transformation neural network, so that the input image information involved in the image quality evaluation process is more complete, and the accuracy of the image quality evaluation can be improved. At the same time, by adopting a training method combining pre-training and fine-tuning to train the image evaluation model, while ensuring the accuracy of the output result of the image evaluation model, the image evaluation model can have stronger generalization performance, and improve the applicability of the image evaluation model and the corresponding image evaluation method. In addition, in this embodiment, by performing electro-optical conversion calculation on the image block of the target file, the input image processed by the image evaluation model can contain more texture information in the target file, which is conducive to further improving the accuracy of the image evaluation result.

[0093] Figure 6 Schematic diagram of an image processing device according to an embodiment of the present invention. Figure 6 As shown, the image processing device in this embodiment includes a file acquisition unit 1, an image processing unit 2, a feature processing unit 3 and a quality evaluation unit 4. Among them, the file acquisition unit 1 is used to acquire a target file, and the target file is an image or a video. The image processing unit 2 is used to pre-process the target file and determine at least one corresponding input image, and the input image corresponds to the image block in the target file. The feature processing unit 3 is used to extract features of the input image based on the feature extraction network in the pre-trained image evaluation model to determine the feature map corresponding to the input image; and to calculate and analyze the feature point positions and feature vectors in the feature map based on the deep self-attention transformation neural network in the pre-trained image evaluation model to determine the quality evaluation score of the input image. The quality evaluation unit 4 is used to determine the quality evaluation score of the target file according to the quality evaluation score of each of the input images.

[0094] Alternatively, if Figure 6As shown, the image processing device in this embodiment also includes a training unit 5. The training unit 5 is used to obtain a training data set, input the input image in the pre-training data set into a preset image evaluation model for training, and determine the pre-training model; input the input image in the fine-tuning data set into the pre-training model for training to determine the image evaluation model obtained by pre-training. The training data set includes a pre-training data set and a fine-tuning data set, the pre-training data set includes a plurality of input images with objective labels, and the fine-tuning data set includes a plurality of input images with subjective labels.

[0095] Optionally, the image processing unit 2 in this embodiment is also used to preprocess the target file, determine at least one image block corresponding to the target file, perform electro-optical conversion calculation on each image block, and determine the corresponding input image. Wherein, when the target file is a video, the preprocessing includes extracting frames from the target file to determine the frame image corresponding to the target file; and cropping each frame image to determine the corresponding image block. Further, when performing electro-optical conversion calculation on each image block to determine the corresponding input image, it is specifically used to determine the electro-optical conversion formula under the corresponding color gamut and dynamic range specifications based on the image format of the target file, convert the image block based on the electro-optical conversion formula, and determine the corresponding input image.

[0096] Optionally, the feature processing unit 3 in this embodiment is also used to perform feature extraction on the input image based on the feature extraction network in the pre-trained image evaluation model to determine the feature map under a preset number of channels; and to perform feature mapping on the feature map under the preset number of channels to determine the feature map under the target number of channels.

[0097] Optionally, the quality evaluation unit 4 in this embodiment is specifically used to perform statistical calculations on the quality evaluation scores of each input image corresponding to the frame image to determine the quality evaluation score of the corresponding frame image; and perform statistical calculations on the quality evaluation scores of each frame image to determine the quality evaluation score of the target file.

[0098] Figure 7 Schematic diagram of an electronic device according to an embodiment of the present invention. Figure 7 As shown, Figure 7The electronic device shown is a general address query device, which includes a general computer hardware structure, which at least includes a processor 71 and a memory 72. The processor 71 and the memory 72 are connected via a bus 73. The memory 72 is suitable for storing instructions or programs executable by the processor 71. The processor 71 can be an independent microprocessor or a collection of one or more microprocessors. Thus, the processor 71 executes the instructions stored in the memory 72, thereby executing the method flow of the embodiment of the present invention as described above to realize the processing of data and the control of other devices. The bus 73 connects the above-mentioned multiple components together, and at the same time connects the above-mentioned components to the display controller 74 and the display device and the input / output (I / O) device 75. The input / output (I / O) device 75 can be a mouse, a keyboard, a modem, a network interface, a touch input device, a somatosensory input device, a printer, and other devices known in the art. Typically, the input / output device 75 is connected to the system via an input / output (I / O) controller 76.

[0099] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices (equipment) or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may adopt a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0100] The present application is described with reference to flowcharts of methods, apparatuses (devices) and computer program products according to embodiments of the present application. It should be understood that each process in the flowchart can be implemented by computer program instructions.

[0101] These computer program instructions may be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device that implements the process Figure 1 A function specified in a process or multiple processes.

[0102] These computer program instructions may also be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce the instructions for implementing the process Figure 1 A device that specifies functions in a process or multiple processes.

[0103] Another embodiment of the present invention relates to a non-volatile storage medium for storing a computer-readable program, wherein the computer-readable program is used for a computer to execute part or all of the above method embodiments.

[0104] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by specifying relevant hardware through a program, and the program is stored in a storage medium, including several instructions for a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0105] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for evaluating a reference-free image, characterized in that: The method comprises: Obtaining a target file, wherein the target file is an image or a video; Preprocessing the target file to determine at least one corresponding input image, wherein the input image corresponds to an image block in the target file, and the image block corresponds to a local image in the target file; Performing feature extraction on the input image based on a feature extraction network in a pre-trained image evaluation model to determine a feature map corresponding to the input image, wherein the feature extraction network adopts ResNet-50; Based on a deep self-attention transform neural network in a pre-trained image evaluation model, the positions of feature points and feature vectors in the feature map are calculated and analyzed to determine the quality evaluation score of the input image, wherein the deep self-attention transform neural network adopts a Transformer network; Determining the quality evaluation score of the target file according to the quality evaluation score of each of the input images; The preprocessing of the target file to determine at least one corresponding input image comprises: preprocessing the target file to determine at least one image block corresponding to the target file; performing electro-optical conversion calculation on each of the image blocks to determine the corresponding input image; The feature extraction of the input image based on the feature extraction network in the pre-trained image evaluation model to determine the feature map corresponding to the input image comprises: extracting features from the input image based on the feature extraction network in the pre-trained image evaluation model to determine the feature map under a preset number of channels; performing feature mapping on the feature map under the preset number of channels to determine the feature map under a target number of channels; The performing electro-optical conversion calculation on the image block to determine the corresponding input image includes: Determine an electro-optical conversion formula under the corresponding color gamut and dynamic range specifications based on the image format of the target file; convert the image block based on the electro-optical conversion formula to determine the corresponding input image; Wherein, the method further comprises: Acquire a training data set, wherein the training data set includes a pre-training data set and a fine-tuning data set, wherein the pre-training data set includes a plurality of input images with objective labels, and the fine-tuning data set includes a plurality of input images with subjective labels; Inputting the input images in the pre-training data set into a preset image evaluation model for training, and determining a pre-training model; The input images in the fine-tuning dataset are input into the pre-trained model for training to determine the pre-trained image evaluation model.

2. The method according to claim 1, characterized in that When the target file is a video, the preprocessing includes: Extracting frames from the target file to determine a frame image corresponding to the target file; Each of the frame images is cropped to determine a corresponding image block.

3. The method according to claim 1, characterized in that When the target file is a video, the target file includes at least one frame image, and determining the quality evaluation score of the target file according to the quality evaluation scores of each of the input images includes: Performing statistical calculation on the quality evaluation scores of the input images corresponding to the frame images to determine the quality evaluation scores of the corresponding frame images, wherein the statistical calculation includes taking an average value, taking a maximum value, taking a minimum value, taking a median, or taking a mode; The quality evaluation score of each frame image is statistically calculated to determine the quality evaluation score of the target file.

4. A non-reference image evaluation device, characterized in that: The device comprises: A file acquisition unit, used to acquire a target file, wherein the target file is an image or a video; An image processing unit, configured to pre-process the target file to determine at least one corresponding input image, wherein the input image corresponds to an image block in the target file, and the image block corresponds to a local image in the target file; A feature processing unit is used to extract features from the input image based on a feature extraction network in a pre-trained image evaluation model to determine a feature map corresponding to the input image; and to calculate and analyze feature point positions and feature vectors in the feature map based on a deep self-attention transformation neural network in the pre-trained image evaluation model to determine a quality evaluation score of the input image, wherein the feature extraction network uses ResNet-50 and the deep self-attention transformation neural network uses a Transformer network; A quality evaluation unit, configured to determine a quality evaluation score of the target file according to the quality evaluation score of each of the input images; The image processing unit is further used to pre-process the target file to determine at least one image block corresponding to the target file; perform electro-optical conversion calculation on each image block to determine the corresponding input image; The feature processing unit is further used to extract features from the input image based on a feature extraction network in a pre-trained image evaluation model to determine a feature map under a preset number of channels; perform feature mapping on the feature map under the preset number of channels to determine a feature map under a target number of channels; Wherein, when performing electro-optical conversion calculation on each image block to determine the corresponding input image, the image processing unit is specifically used to determine the electro-optical conversion formula under the corresponding color gamut and dynamic range specifications based on the image format of the target file, and convert the image block based on the electro-optical conversion formula to determine the corresponding input image; The device also includes a training unit, which is used to obtain a training data set, wherein the training data set includes a pre-training data set and a fine-tuning data set, the pre-training data set includes a plurality of input images with objective labels, and the fine-tuning data set includes a plurality of input images with subjective labels; inputting the input images in the pre-training data set into a preset image evaluation model for training to determine the pre-training model; and inputting the input images in the fine-tuning data set into the pre-training model for training to determine the image evaluation model obtained by pre-training.

5. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps of any one of claims 1 to 3 are implemented.

Images