Image quality assessment method, image quality assessment model training method and device
By using sample sets with preset multiple quality levels in the intelligent security system to train the image quality evaluation model, generate image quality label distribution and adjust model weight parameters, the problem of uncertainty and low accuracy of monitoring image quality evaluation is solved, and the problem of improved The image analysis effect of the intelligent security system.
Patent Information
- Application Number
- CN202010786237.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-08-06
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-08-06
AI Technical Summary
The clarity and low resolution of the monitoring images in the intelligent security system lead to poor uncertainty and accuracy of image quality evaluation results, affecting the intelligent security effect.
By obtaining the images to be processed and inputting them into the pre-trained image quality evaluation model, machine learning training is performed using a sample set containing preset multiple quality levels, image quality label distribution is generated, and the probability that each image belongs to a different quality level is calculated. value, adjust the model weight parameters to improve the stability and accuracy of the evaluation results.
It improves the accuracy and stability of image quality evaluation, can more accurately evaluate the monitoring image quality, and enhance the effect of the intelligent security system.
Smart Images

Figure CN111915595B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to an image quality assessment method, and a training method and device for an image quality assessment model. Background Art
[0002] Smart security is to analyze the images captured by surveillance cameras through artificial intelligence technologies such as computer vision, and promptly detect specific people, objects and events from the images. In the smart security system, due to the long distance between the objects and the surveillance cameras, the poor lighting conditions and the poor imaging effect of the cameras themselves, the surveillance images captured by the cameras are likely to have poor clarity and low resolution, thus affecting the surveillance effect of smart security. Therefore, it is necessary to accurately evaluate the quality of the surveillance images, and automatically call the corresponding image enhancement algorithm to enhance the surveillance images according to the evaluation results, so that the smart security system can process the enhanced images accordingly to improve the effect of smart security.
[0003] In the related art, the quality of monitoring images can be evaluated through the trained deep learning model. When training the deep learning model, it is necessary to annotate each image in the training set with a quality grade label. Usually, images of one category correspond to the same quality grade label. Then, the deep learning model is trained through the training set. After the training is completed, the monitoring image is input into the model to obtain the quality grade result corresponding to the monitoring image. However, since image quality evaluation is highly subjective, different people generally give different image quality evaluations for the same image, resulting in uncertainty in the image quality evaluation itself. This method attributes a category of images to one quality grade, which is too absolute, resulting in poor stability and accuracy of the quality evaluation results obtained by this method. Summary of the invention
[0004] The purpose of the present invention is to provide an image quality assessment method, an image quality assessment model training method and a device to improve the stability and accuracy of image quality assessment results.
[0005] In a first aspect, an embodiment of the present invention provides an image quality assessment method, the method comprising: obtaining an image to be processed; inputting the image to be processed into a pre-trained image quality assessment model to obtain an evaluation result of the image to be processed at a preset plurality of quality levels; wherein the image quality assessment model is obtained by machine learning training using a sample set, each sample in the sample set comprising a sample image and a quality label distribution corresponding to the sample image; the quality label distribution comprising a preset plurality of quality levels, and a probability value of the sample image belonging to each quality level.
[0006] In an optional embodiment, for each sample image in the sample set, the quality label distribution of the current sample image is determined in the following manner: obtaining multiple annotation results corresponding to the current sample image; the annotation results are used to represent the quality level of the current sample image; based on the multiple annotation results and the preset multiple quality levels, generating the quality label distribution of the current sample image.
[0007] In an optional embodiment, the above-mentioned step of generating a quality label distribution of the current sample image based on multiple annotation results and preset multiple quality levels includes: calculating the average value of the quality levels corresponding to the multiple annotation results to obtain the level mean; based on the level mean and the preset multiple quality levels, determining the probability value that the current sample image belongs to each quality level in the preset multiple quality levels; and combining the probability values of each quality level to obtain the quality label distribution of the current sample image.
[0008] In an optional implementation, the step of determining the probability value of the current sample image belonging to each of the preset multiple quality levels based on the level mean and the preset multiple quality levels includes: calculating the probability value of the current sample image belonging to each of the preset multiple quality levels by the following formula:
[0009]
[0010]
[0011] Among them, p j It represents the probability value that the current sample image belongs to the jth quality level among the preset multiple quality levels; j represents the level value of the jth quality level; y represents the level mean; Z represents the normalization factor; N represents the total number of the preset multiple quality levels; σ represents the preset standard deviation; exp represents the exponential function operation with the natural constant e as the base; ∑ represents the summation operation.
[0012] In an optional embodiment, the weight parameters of the above-mentioned image quality assessment model are determined according to the loss amount during the machine learning training process; wherein the loss amount represents the difference between the quality assessment result obtained after the sample is input into the image quality assessment model and the quality label distribution of the input sample.
[0013] In an optional embodiment, the above loss amount is determined by the following formula:
[0014]
[0015] Where L represents the loss amount; p j Indicates the probability value of the sample image belonging to the jth quality level among the preset multiple quality levels in the quality label distribution; It indicates the probability value that the sample image belongs to the jth quality level among the preset multiple quality levels in the quality evaluation result; N indicates the total number of the preset multiple quality levels; log is used to indicate logarithmic operation; ∑ indicates summation operation.
[0016] In second aspect, an embodiment of the present invention provides a training method for an image quality assessment model, the method comprising: obtaining a sample set; each sample in the sample set comprises a sample image and a quality label distribution corresponding to the sample image; each quality label distribution comprises a plurality of preset quality levels, and a probability value of the sample image belonging to each quality level; performing machine learning training on an initial model based on the sample set to obtain an image quality assessment model.
[0017] In an optional embodiment, for each sample image in the sample set, the quality label distribution of the current sample image is determined in the following manner: multiple annotation results corresponding to the current sample image are obtained; the annotation results are used to represent the quality level of the current sample image; based on the multiple annotation results and the preset multiple quality levels, the quality label distribution of the current sample image is generated.
[0018] In an optional embodiment, the above-mentioned step of performing machine learning training on the initial model based on the sample set to obtain the image quality evaluation model includes: determining training samples based on the sample set; inputting sample images in the training samples into the initial model to obtain output results; obtaining the loss amount based on the output results and the quality label distribution of the training samples; adjusting the weight parameters of the initial model based on the loss amount; and continuing to execute the step of determining training samples based on the sample set until the loss amount converges or reaches a preset number of training times to obtain the image quality evaluation model.
[0019] In a third aspect, an embodiment of the present invention provides an image quality evaluation device, which includes: an image acquisition module, used to acquire an image to be processed; a quality evaluation module, used to input the image to be processed into a pre-trained image quality evaluation model to obtain evaluation results of the image to be processed at multiple preset quality levels; wherein the image quality evaluation model is obtained by machine learning training using a sample set, and each sample in the sample set includes a sample image and a quality label distribution corresponding to the sample image; the quality label distribution includes multiple preset quality levels, and the probability value of the sample image belonging to each quality level.
[0020] In a fourth aspect, an embodiment of the present invention provides a training device for an image quality assessment model, the device comprising: a sample set acquisition module, used to acquire a sample set; each sample in the sample set comprises a sample image and a quality label distribution corresponding to the sample image; each quality label distribution comprises a plurality of preset quality levels, and a probability value of the sample image belonging to each quality level; a model training module, used to perform machine learning training on an initial model based on the sample set to obtain an image quality assessment model.
[0021] In a fifth aspect, an embodiment of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above-mentioned image quality assessment method or the above-mentioned image quality assessment model training method.
[0022] In a sixth aspect, an embodiment of the present invention provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the above-mentioned image quality assessment method or the above-mentioned image quality assessment model training method.
[0023] The embodiments of the present invention bring the following beneficial effects:
[0024] An image quality evaluation method, an image quality evaluation model training method and a device provided by an embodiment of the present invention first obtain an image to be processed; then input the image to be processed into a pre-trained image quality evaluation model to obtain the evaluation results of the image to be processed under multiple preset quality levels. This method uses an image quality evaluation model to perform quality evaluation on the image to be processed to obtain the quality level of the image. The image quality evaluation model is obtained by machine learning training using a sample set including sample images and quality label distributions corresponding to the sample images, and the quality label distribution includes multiple preset quality levels and the probability value of the sample image belonging to each quality level. Therefore, during the training process, the image quality evaluation model can learn the inherent uncertainty of the image quality from the quality label distribution of the sample image, thereby improving the accuracy and stability of the model quality evaluation.
[0025] Other features and advantages of the present invention will be set forth in the following description, or some features and advantages may be inferred or unambiguously determined from the description, or may be learned by implementing the above-mentioned technology of the present invention.
[0026] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are specifically listed below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0028] Figure 1 A flowchart of an image quality evaluation method provided by an embodiment of the present invention;
[0029] Figure 2 A flowchart of another image quality evaluation method provided by an embodiment of the present invention;
[0030] Figure 3 A schematic diagram of a network structure of an image quality assessment model provided by an embodiment of the present invention;
[0031] Figure 4 A flowchart of a method for training an image quality assessment model provided by an embodiment of the present invention;
[0032] Figure 5 A schematic diagram of the structure of an image quality evaluation device provided by an embodiment of the present invention;
[0033] Figure 6 A schematic diagram of the structure of a training device for an image quality assessment model provided by an embodiment of the present invention;
[0034] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0035] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.
[0036] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0037] With the popularity of surveillance cameras and the development of artificial intelligence technologies represented by deep learning, intelligent security systems have played an important role in maintaining social stability and protecting people's lives and property. Intelligent security is a technology that uses artificial intelligence technologies such as computer vision to analyze images captured by surveillance cameras and promptly detect specific people, objects, and events. In order to analyze surveillance images more accurately, high-resolution surveillance image acquisition is essential. However, in actual intelligent security systems, due to the long distance between objects and surveillance cameras, poor lighting conditions, and poor imaging effects of the cameras themselves, the surveillance images captured by the cameras are likely to have poor clarity and low resolution. These low-quality surveillance images have brought great difficulties to current computer vision technology.
[0038] In related technologies, there are usually two methods for image quality assessment. The first is the traditional image quality assessment method, which is to estimate image quality through methods such as peak signal-to-noise ratio, structural similarity, and wavelet transform. However, this method lacks high-level semantic information of the image, resulting in low accuracy of the quality assessment results obtained by this method.
[0039] The second is a method for evaluating the quality of surveillance images based on a deep learning model. When training a deep learning model, this method needs to annotate a quality grade label for each image in the training set. Usually, images of one category correspond to the same quality grade label, and then the deep learning model is trained through the training set; but since image quality evaluation is highly subjective, that is, for the same image, different people generally give different image quality evaluations, resulting in uncertainty in the image quality evaluation itself. Therefore, compared with traditional image quality evaluation algorithms, this method can consider the high-level semantic information of the image, but does not take into account the uncertainty of the image quality evaluation task itself when training the deep learning model, resulting in poor generalization and stability of the obtained model, which makes the quality evaluation results obtained by this method less accurate.
[0040] Based on the above problems, the embodiments of the present invention provide an image quality assessment method, an image quality assessment model training method and a device. The technology can be applied to image analysis scenarios in various monitoring scenarios, especially to image quality analysis of monitoring images. To facilitate understanding of the present embodiment, an image quality assessment method disclosed in the embodiment of the present invention is first described in detail. Figure 1 As shown, the method comprises the following steps:
[0041] Step S102, obtaining an image to be processed.
[0042] The image can be acquired by shooting with a camera or other device connected by communication and then transmitting the image, or by acquiring the image from a storage device storing the shot images to be processed.
[0043] The above-mentioned image to be processed may be an image in various monitoring scenes, and the image may be taken by a camera or a camera. The image to be processed may contain specific people, objects or events, for example, a specific person may be a target person (such as a suspect) that the user needs to monitor; a specific object may be an animal, a building or an item carried by a person; a specific event may be a gathering, a fight or other specific activities.
[0044] Step S104: input the above-mentioned image to be processed into a pre-trained image quality evaluation model to obtain evaluation results of the image to be processed at multiple preset quality levels.
[0045] The above-mentioned multiple preset quality levels are multiple quality levels set by R&D personnel according to requirements. The range and number of the quality levels are also set according to R&D requirements. For example, 10 quality levels can be set, and the quality levels are integers between 1 and 10; 100 quality levels can also be set, and the quality levels are integers between 1 and 100. The above-mentioned image quality evaluation model is obtained by machine learning training using a sample set, and each sample in the sample set includes a sample image and a quality label distribution corresponding to the sample image; the quality label distribution includes multiple preset quality levels and the probability value of the sample image belonging to each quality level.
[0046] The above sample set contains a large number of samples, each of which contains a sample image and a quality label distribution corresponding to the sample image. The quality label distribution can be a vector, and each element in the vector represents the probability value of the sample image belonging to each quality level under the preset multiple quality levels. For example, 10 quality levels can be set, and the quality level is an integer between 1 and 10. At this time, the quality label distribution corresponds to a 10-dimensional vector, and the first element in the vector represents the probability value of the sample image's quality level being 1, and the second element represents the probability value of the sample image's quality level being 2. According to this rule, the tenth element represents the probability value of the sample image's quality level being 10. Generally, the higher the quality level, the better the quality of the sample image.
[0047] The above-mentioned image quality assessment model can adopt a deep learning model or a neural network model, etc. In the process of machine learning training of the image quality assessment model, it is necessary to select samples from the sample set, and then input the sample images in the sample into the initial model of the image quality assessment model. The initial model can perform image quality assessment on the sample image and output the evaluation results of the sample image under multiple preset quality levels, and then obtain the loss amount through the evaluation result and the quality label distribution corresponding to the sample image, adjust the network parameters of the initial model (that is, the weight parameters of each layer of the network) based on the loss amount, continue to select samples from the sample set and input them into the adjusted initial model, obtain the loss amount and adjust the adjusted initial model based on the loss amount, until the loss amount converges, and obtain the image quality assessment model.
[0048] Unlike other image recognition problems, image quality evaluation is highly subjective. For the same image, different people generally give different image quality judgment results. This is reflected in the fact that in order to annotate the quality level of an image, an image usually requires n (n>10) people to independently annotate, and finally the average of the annotation results of n people is used as the final annotation result. Therefore, it is too absolute to use a final identification result as the quality level label of each image, and it is difficult to obtain a high-precision model through training with the quality level label. Therefore, the quality label distribution adopted in this application can well characterize the uncertainty of the image quality level, that is, it can characterize the possibility that the image belongs to different quality levels. Then, the image quality evaluation model is trained through the quality label distribution, which can enable the image quality evaluation model to learn the inherent uncertainty of the image quality, thereby improving the accuracy and stability of the model prediction.
[0049] An image quality assessment method provided by an embodiment of the present invention first obtains an image to be processed; then the image to be processed is input into a pre-trained image quality assessment model to obtain the assessment results of the image to be processed under multiple preset quality levels. This method uses an image quality assessment model to perform quality assessment on the image to be processed to obtain the quality level of the image. The image quality assessment model is obtained by machine learning training using a sample set including sample images and quality label distributions corresponding to the sample images, and the quality label distribution includes multiple preset quality levels and the probability value of the sample image belonging to each quality level. Therefore, during the training process, the image quality assessment model can learn the inherent uncertainty of the image quality from the quality label distribution of the sample images, thereby improving the accuracy and stability of the model quality assessment.
[0050] The embodiment of the present invention also provides another image quality assessment method, which is implemented on the basis of the above-mentioned embodiment method; the method focuses on describing the specific process of determining the weight parameters of the image quality assessment model before obtaining the image to be processed (implemented by the following steps S202-S206); Figure 2 As shown, the method comprises the following steps:
[0051] Step S202: Input an input sample determined from the sample set into the image quality assessment model to obtain a quality assessment result corresponding to the input sample. The input sample is any sample in the sample set, and the input sample includes a sample image and a quality label distribution corresponding to the sample image, wherein the quality label distribution includes a plurality of preset quality levels and a probability value of the sample image belonging to each quality level.
[0052] In the specific implementation, before model training, it is necessary to determine the quality label distribution corresponding to each sample image in the sample set. For each sample image in the sample set, the quality label distribution of the current sample image is determined by the following steps 10-11:
[0053] Step 10: Obtain multiple annotation results corresponding to the current sample image; the annotation results are used to indicate the quality level of the current sample image.
[0054] The above-mentioned current sample image is any sample image in the sample set. The sample images in the sample set can be used as the current sample image in turn. The multiple annotation results corresponding to the current sample image can be n annotation results obtained after n preset people respectively annotate the quality level of the current sample image. The quality level annotated by these n people is the level value among the preset multiple quality levels.
[0055] Step 11: Generate a quality label distribution of the current sample image based on the multiple annotation results and the preset multiple quality levels.
[0056] In the specific implementation, the average values of the quality levels corresponding to the multiple annotation results are calculated to obtain the level mean; based on the level mean and the preset multiple quality levels, the probability value of the current sample image belonging to each quality level in the preset multiple quality levels is determined; the probability values of each quality level are combined to obtain the quality label distribution of the current sample image.
[0057] The quality levels corresponding to multiple annotation results are added and averaged to obtain the mean value. For example, suppose the preset multiple quality levels are integers between 1 and 10, 10 represents the highest image quality, 1 represents the worst image quality, and n people annotate the quality level of the current sample image, and n annotation results are obtained. The value range of k is from 1 to n. represents the labeling result of the kth person on the current sample image. According to the n labeling results, the average level can be obtained as:
[0058]
[0059] Among them, y represents the level mean of the current sample image, Indicates rounding down *.
[0060] After obtaining the level mean of the current sample image, the probability value of the current sample image belonging to each of the preset multiple quality levels can be calculated by the following formula:
[0061]
[0062]
[0063] Among them, p j It represents the probability value that the current sample image belongs to the jth quality level among the preset multiple quality levels; j represents the level value of the jth quality level; y represents the level mean; Z represents the normalization factor, which is to ensure that the sum of the probability values corresponding to the preset multiple levels is 1; N represents the total number of the preset multiple quality levels; σ represents the preset standard deviation, which is a value set manually according to experimental results and experience; exp represents the exponential function operation with the natural constant e as the base; ∑ represents the sum operation.
[0064] In specific implementation, the obtained probability values of each quality level of the current sample image are combined to obtain the quality label distribution of the current sample image. For example, assuming that the preset multiple quality levels are integers between 1 and 10, the quality label distribution P is P = [p 1 ,p 2 ,…,p j ,…,p 10 ], j = 1,…,10; where p 1 Indicates the probability value that the quality level of the current sample image is 1, p 2 Indicates the probability value that the quality level of the current sample image is 2, p 10 Indicates the probability value that the quality level of the current sample image is 10.
[0065] In this embodiment, Gaussian distribution is used to define the quality label distribution. In some embodiments, the quality label distribution can also be defined by other distribution methods, such as Poisson distribution. The quality label distribution describes the uncertainty of the quality judgment of the sample image to a certain extent. Usually, the probability of the quality level of the sample image being the mean level is the largest, and the probability of the quality level close to the mean level is also large, but less than the probability value corresponding to the mean level. This description of the uncertainty of the quality level of the image is consistent with the image quality task itself, which is conducive to improving the accuracy of subsequent model training.
[0066] Step S204, determining the loss amount according to the quality evaluation result corresponding to the input sample and the quality label distribution corresponding to the input sample; wherein the loss amount represents the difference between the quality evaluation result corresponding to the input sample and the quality label distribution of the input sample.
[0067] The quality evaluation result corresponding to the above input sample can be the probability value of the input sample belonging to each quality level under the preset multiple quality levels. The probability value in the quality evaluation result can be a normalized probability value or an unnormalized probability value. If you want to transform the unnormalized probability value into a normalized probability value, you can add a softmax function at the end of the image quality evaluation model. For example, assuming that the unnormalized quality evaluation result is Will Each element in is sent to the softmax function, and the quality evaluation result after the probability value is normalized can be obtained.
[0068]
[0069] in, Represents the probability value of the jth quality level in the normalized quality evaluation result; represents the probability value of the jth quality level in the unnormalized quality evaluation result; Represents the probability value of the mth quality level in the unnormalized quality evaluation result.
[0070] In specific implementation, the above loss amount is determined by the following formula:
[0071]
[0072] Where L represents the loss amount; p j Indicates the probability value of the sample image belonging to the jth quality level among the preset multiple quality levels in the quality label distribution; It indicates the probability value that the sample image belongs to the jth quality level among the preset multiple quality levels in the quality evaluation result; N indicates the total number of the preset multiple quality levels; log is used to indicate logarithmic operation; ∑ indicates summation operation.
[0073] The calculation formula for the above loss is the Cross Entropy Loss in machine learning. From the formula for calculating the loss L, it can be seen that the closer the probability value corresponding to each quality level in the quality label distribution is to the probability value of the corresponding quality level in the quality evaluation result, the smaller the loss. The loss describes the distance between the probability value in the quality label distribution and the probability value in the quality evaluation result. The smaller the distance between the two, the closer the two are.
[0074] Step S206, during the machine learning training process, the weight parameters of the image quality assessment model are determined according to the above loss amount; the image quality assessment model after the weight parameters are determined is determined as the trained image quality assessment model. The weight parameters are usually the weights corresponding to each network parameter in the network structure of the image quality assessment model.
[0075] In the specific implementation, it is necessary to continuously adjust the weight parameters of the image quality evaluation model according to the loss amount until the loss amount converges or reaches the specified number of training times; the weight parameters corresponding to when the weight parameter adjustment stops are determined as the final weight parameters of the image quality evaluation model.
[0076] Step S208: If an image to be processed is obtained, the image to be processed is input into the trained image quality evaluation model to obtain an evaluation result of the image to be processed.
[0077] The above-mentioned trained image quality assessment model may include a feature extraction layer and an output layer; wherein the feature extraction layer is used to extract features of the image to be processed to obtain feature data of the image to be processed; and the output layer is used to output the evaluation result of the image to be processed based on the feature data. The feature extraction layer includes a convolution layer and an activation function layer connected in sequence. The feature extraction layer can extract the image features of the image to be processed to obtain high-level semantic information of the image features. In order to improve the performance of the feature extraction layer, the feature extraction layer usually includes multiple groups of convolution layers and activation function layers connected in sequence. The activation function layer in the feature extraction layer can perform a function transformation on the image features output by the convolution layer. The transformation process can break the linear combination of the convolution layer input. The activation function layer can specifically be a Sigmoid function, a tanh function, a Relu function, etc.
[0078] The output layer includes at least one fully connected layer, which can obtain the evaluation result of the image quality of the specified dimension. In some embodiments, in order to normalize the evaluation result, an activation function layer can be connected to the last fully connected layer in the output layer, and a softmax function can be used in the activation function layer.
[0079] In specific implementation, the number of feature extraction layers included in the image quality assessment model, the number of groups of sequentially connected convolutional layers and activation function layers included in each feature extraction layer, and the number of fully connected layers in the output layer can be determined according to the speed and accuracy of data processing. Generally, the more numbers or groups, the deeper the overall network structure of the model will be and the better the performance will be, but the calculation speed will decrease.
[0080] like Figure 3 The figure shows a schematic diagram of the network structure of an image quality assessment model that includes 4 feature extraction layers and 2 fully connected layers. Figure 3 The four feature extraction layers are Block1, Block2, Block3 and Block4; the two fully connected layers are FC1 and FC2. The image to be processed is input into Block1, and the feature data is output through Bloc2, Block3 and Block4 in turn. The feature data is input into FC1 to obtain a feature vector with a dimension of c (the value of c is set according to the task requirements, and the larger the C value, the better the effect). The feature vector is also the image feature of the image to be processed. The image feature is input into FC2 to obtain the evaluation result of the image to be processed. The dimension of the evaluation result can be consistent with the number of preset multiple quality levels. Among them, the dimension of the data output by Block4 is usually large, and FC1 can be understood as a dimensionality reduction process, that is, reducing the data output by Block4 to c dimension.
[0081] In specific implementation, the above step S208 can be implemented by the following steps 20-21:
[0082] Step 20: input the image to be processed into the trained image quality assessment model, and output the assessment results of the image to be processed at multiple preset quality levels.
[0083] Step 21 : Select the quality level corresponding to the evaluation result with the highest probability from the evaluation results of the image to be processed at multiple preset quality levels, and use the quality level as the final evaluation result of the image to be processed.
[0084] The above-mentioned image quality assessment method uses quality label distribution to characterize the uncertainty of image quality. Different from the related art method of using deep learning models to simply use quality levels as image labels, this method uses label distribution as image quality labels to train the image quality assessment model, so that the image quality assessment model can learn the inherent uncertainty of image quality, thereby improving the accuracy and stability of the image quality assessment model in evaluating image quality.
[0085] In view of the above-mentioned embodiment of the image quality assessment method, the embodiment of the present invention also provides a training method for an image quality assessment model, such as Figure 4 As shown, the method comprises the following steps:
[0086] Step S402, obtaining a sample set; each sample in the sample set includes a sample image and a quality label distribution corresponding to the sample image; each quality label distribution includes a plurality of preset quality levels and a probability value of the sample image belonging to each quality level.
[0087] The above sample set contains a large number of samples, each of which contains a sample image and a quality label distribution corresponding to the sample image. The quality label distribution can be a vector, and each element in the vector represents the probability value of the sample image belonging to each quality level under multiple preset quality levels.
[0088] In specific implementation, the quality label distribution corresponding to each sample image in the sample set can be obtained through the above steps 10-11, which will not be repeated here.
[0089] Step S404: Perform machine learning training on the initial model based on the sample set to obtain an image quality assessment model.
[0090] In specific implementation, the above step S304 can be implemented by the following steps 30-33:
[0091] Step 30: determine training samples based on the sample set.
[0092] Step 31: Input the sample images in the training samples into the initial model to obtain an output result. The initial model may be a deep learning model or a neural network model.
[0093] Step 32, obtaining the loss amount according to the output result and the quality label distribution of the training samples.
[0094] In specific implementation, the above loss amount can be determined by the following formula:
[0095]
[0096] Where L represents the loss amount; p j Indicates the probability value of the sample image belonging to the jth quality level among the preset multiple quality levels in the quality label distribution; It indicates the probability value that the sample image in the output result belongs to the jth quality level among the preset multiple quality levels; N indicates the total number of the preset multiple quality levels; log is used to indicate logarithmic operation; ∑ indicates summation operation.
[0097] Step 33, adjusting the weight parameters of the initial model based on the loss amount; continuing to perform the step of determining the training sample based on the sample set until the loss amount converges or reaches a preset number of training times, to obtain an image quality evaluation model.
[0098] In a specific implementation, the above step 33 can be obtained through the following steps 40-43:
[0099] Step 40, calculate the derivative of the loss with respect to the weight parameter to be updated in the initial model Where L is the loss amount; W is the weight parameter to be updated; the weight parameter to be updated can be all the parameters in the initial model, or some parameters randomly determined from the initial model; the updated weight parameter is also the weight of each layer of the network in the initial model. Usually, the derivative of the weight parameter to be updated can be solved according to the back propagation algorithm; if the loss amount is large, it means that the recognition result of the current initial model is much different from the expected result, then the derivative of the loss amount to the weight parameter to be updated in the initial model is calculated, and the derivative can be used as the basis for updating the weight parameter to be updated.
[0100] Step 41: Update the weight parameter to be updated to obtain the updated weight parameter to be updated Among them, α is a preset coefficient, which is a hyperparameter set manually in advance, and can be 0.01, 0.001, etc. This process can also be called a stochastic gradient descent algorithm; the derivative of each weight parameter to be updated can also be understood as the direction in which the loss decreases fastest relative to the current parameter. By adjusting the parameters in this direction, the loss can be quickly reduced, so that the weight parameter converges.
[0101] Step 42, determining whether the parameters of the updated initial model have all converged, if so, executing the step of determining the training sample based on the sample set; otherwise, executing step 43.
[0102] If the parameters of the updated initial model are not all converged, new training samples are determined based on the sample set, and steps 30-33 are continued until the parameters of the updated initial model are all converged.
[0103] Step 43: determine the initial model after parameter update as the trained image quality assessment model.
[0104] In addition, when the initial model is trained once and the loss amount is obtained, one or more parameters can be randomly selected from the various weight parameters of the initial model to perform the above-mentioned update process. This method has a shorter model training time and a faster algorithm. Of course, the above-mentioned update process can also be performed on all parameters in the initial model. This method of model training is more accurate.
[0105] In specific implementation, the images in the sample set can be divided into a sample set for training the model and a sample set for verifying the model according to a preset ratio (for example, 10:1). The recognition accuracy of the trained image quality evaluation model can be determined by the sample set used to verify the model; usually, a test sample can be determined from the sample set used to verify the model, and the test sample includes a sample image and a quality label distribution corresponding to the sample image. The quality evaluation result can be obtained by inputting the test sample into the trained image quality evaluation model, and the quality evaluation result is compared with the quality label distribution to determine whether the quality evaluation result is correct. The test sample is continuously determined from the sample set used to verify the model until all samples in the sample set used to verify the model are selected; the correctness of the test result corresponding to each test sample is counted to obtain the prediction accuracy of the trained image quality evaluation model.
[0106] The training method of the above-mentioned image quality assessment model first obtains a sample set; then performs machine learning training on the initial model based on the sample set to obtain an image quality assessment model. The image quality assessment model in this method can automatically learn multi-level semantic features related to image quality, and this method can learn the inherent uncertainty of image quality from the quality label distribution of sample images, thereby improving the accuracy and stability of the image quality assessment model in evaluating image quality.
[0107] Corresponding to the above-mentioned embodiment of the image quality evaluation method, the embodiment of the present invention further provides an image quality evaluation device, such as Figure 5 As shown, the device comprises:
[0108] The image acquisition module 50 is used to acquire the image to be processed.
[0109] The quality evaluation module 51 is used to input the image to be processed into a pre-trained image quality evaluation model to obtain evaluation results of the image to be processed at multiple preset quality levels.
[0110] Among them, the image quality evaluation model is obtained through machine learning training using a sample set, and each sample in the sample set includes a sample image and a quality label distribution corresponding to the sample image; the quality label distribution includes multiple preset quality levels and the probability value of the sample image belonging to each quality level.
[0111] The above-mentioned image quality evaluation device first obtains the image to be processed; then inputs the image to be processed into a pre-trained image quality evaluation model to obtain the evaluation results of the image to be processed under multiple preset quality levels. This method uses the image quality evaluation model to evaluate the quality of the image to be processed to obtain the quality level of the image. The image quality evaluation model is obtained by machine learning training using a sample set containing sample images and quality label distributions corresponding to the sample images, and the quality label distribution includes multiple preset quality levels and the probability value of the sample image belonging to each quality level. Therefore, during the training process, the image quality evaluation model can learn the inherent uncertainty of the image quality from the quality label distribution of the sample image, thereby improving the accuracy and stability of the model quality evaluation.
[0112] Furthermore, the above-mentioned device also includes a label distribution generation module, which is used to: for each sample image in the sample set, obtain multiple annotation results corresponding to the current sample image; the annotation result is used to represent the quality level of the current sample image; based on the multiple annotation results and the preset multiple quality levels, generate the quality label distribution of the current sample image.
[0113] Specifically, the above-mentioned label generation module is used to: calculate the average value of the quality levels corresponding to multiple annotation results to obtain the level mean; based on the level mean and the preset multiple quality levels, determine the probability value of the current sample image belonging to each quality level in the preset multiple quality levels; combine the probability values of each quality level to obtain the quality label distribution of the current sample image.
[0114] In a specific implementation, the label generation module may calculate the probability value of the current sample image belonging to each of the preset multiple quality levels by the following formula:
[0115]
[0116]
[0117] Among them, p j It represents the probability value that the current sample image belongs to the jth quality level among the preset multiple quality levels; j represents the level value of the jth quality level; y represents the level mean; Z represents the normalization factor; N represents the total number of the preset multiple quality levels; σ represents the preset standard deviation; exp represents the exponential function operation with the natural constant e as the base; ∑ represents the summation operation.
[0118] Furthermore, the above-mentioned device also includes a model parameter training module, which is used to: determine the weight parameters of the image quality assessment model according to the loss amount during the machine learning training process; wherein the loss amount represents the difference between the quality assessment result obtained after the sample is input into the image quality assessment model and the quality label distribution of the input sample.
[0119] In specific implementation, the above loss amount is determined by the following formula:
[0120]
[0121] Where L represents the loss amount; p j Indicates the probability value of the sample image belonging to the jth quality level among the preset multiple quality levels in the quality label distribution; It indicates the probability value that the sample image belongs to the jth quality level among the preset multiple quality levels in the quality evaluation result; N indicates the total number of the preset multiple quality levels; log is used to indicate logarithmic operation; ∑ indicates summation operation.
[0122] The image quality assessment device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned image quality assessment method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.
[0123] Corresponding to the above-mentioned training method embodiment of the image quality assessment model, the embodiment of the present invention also provides a training device for the image quality assessment model, such as Figure 6 As shown, the device comprises:
[0124] The sample set acquisition module 60 is used to acquire a sample set; each sample in the sample set includes a sample image and a quality label distribution corresponding to the sample image; each quality label distribution includes a plurality of preset quality levels and a probability value of the sample image belonging to each quality level.
[0125] The model training module 61 is used to perform machine learning training on the initial model based on the sample set to obtain an image quality evaluation model.
[0126] The training model of the above-mentioned image quality assessment model first obtains a sample set; then performs machine learning training on the initial model based on the sample set to obtain an image quality assessment model. The image quality assessment model in this method can automatically learn multi-level semantic features related to image quality, and this method can learn the inherent uncertainty of image quality from the quality label distribution of sample images, thereby improving the accuracy and stability of the image quality assessment model in evaluating image quality.
[0127] Furthermore, the above-mentioned device also includes a label generation module, which is used to: obtain multiple annotation results corresponding to the current sample image; the annotation results are used to indicate the quality level of the current sample image; based on the multiple annotation results and the preset multiple quality levels, generate the quality label distribution of the current sample image.
[0128] Furthermore, the above-mentioned model training module 61 is used to: determine training samples based on the sample set; input the sample images in the training samples into the initial model to obtain output results; obtain the loss amount based on the output results and the quality label distribution of the training samples; adjust the weight parameters of the initial model based on the loss amount; continue to execute the steps of determining training samples based on the sample set until the loss amount converges or reaches a preset number of training times, and obtain an image quality evaluation model.
[0129] The training device for the image quality assessment model provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned training method embodiment for the image quality assessment model. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.
[0130] The embodiment of the present invention further provides an electronic device, see Figure 7 As shown, the electronic device includes a processor 101 and a memory 100, wherein the memory 100 stores machine executable instructions that can be executed by the processor 101, and the processor executes the machine executable instructions to implement the above-mentioned image quality assessment method or the above-mentioned image quality assessment model training method.
[0131] Further, Figure 7 The electronic device shown further includes a bus 102 and a communication interface 103 , and the processor 101 , the communication interface 103 and the memory 100 are connected via the bus 102 .
[0132] The memory 100 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 103 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 102 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0133] The processor 101 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 101. The above processor 101 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 100, and the processor 101 reads the information in the memory 100 and completes the steps of the method of the above embodiment in combination with its hardware.
[0134] An embodiment of the present invention also provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the above-mentioned image quality assessment method or the training method of the above-mentioned image quality assessment model. The specific implementation can be found in the method embodiment, which will not be repeated here.
[0135] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and / or electronic device can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.
[0136] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims
1. A method for evaluating image quality, characterized in that: The method comprises: Get the image to be processed; Inputting the image to be processed into a pre-trained image quality evaluation model to obtain evaluation results of the image to be processed at multiple preset quality levels; The image quality evaluation model is obtained by machine learning training using a sample set, each sample in the sample set includes a sample image and a quality label distribution corresponding to the sample image; the quality label distribution includes a plurality of preset quality levels and a probability value of the sample image belonging to each of the quality levels; For each of the sample images in the sample set, the quality label distribution of the current sample image is determined in the following manner: Acquire multiple annotation results corresponding to the current sample image; the annotation results are used to indicate the quality level of the current sample image; Based on the multiple annotation results and the preset multiple quality levels, generating a quality label distribution of the current sample image; The step of generating the quality label distribution of the current sample image based on the multiple annotation results and the preset multiple quality levels includes: Calculating an average value of the quality grades corresponding to the multiple annotation results to obtain a grade mean; Based on the level mean and the preset multiple quality levels, determining a probability value that the current sample image belongs to each quality level of the preset multiple quality levels; The probability values of each quality level are combined to obtain the quality label distribution of the current sample image; wherein the probability that the quality level of the current sample image is the mean of the levels is the largest; and the quality label distribution is defined by Gaussian distribution.
2. The method according to claim 1, characterized in that The step of determining the probability value of the current sample image belonging to each of the preset multiple quality levels based on the level mean and the preset multiple quality levels comprises: The probability value of the current sample image belonging to each of the preset multiple quality levels is calculated by the following formula: Among them, p j It represents the probability value that the current sample image belongs to the jth quality level among the preset multiple quality levels; j represents the level value of the jth quality level; y represents the mean value of the level; Z represents the normalization factor; N represents the total number of the preset multiple quality levels; σ represents the preset standard deviation; exp represents the exponential function operation with the natural constant e as the base; ∑ represents the summation operation.
3. The method according to claim 1, characterized in that: The weight parameters of the image quality assessment model are determined according to the loss amount during the machine learning training process; wherein the loss amount represents the difference between the quality assessment result obtained after the sample is input into the image quality assessment model and the quality label distribution of the input sample.
4. The method according to claim 3, characterized in that: The loss amount is determined by the following formula: Wherein, L represents the loss amount; p j Indicates the probability value of the sample image belonging to the jth quality level among the preset multiple quality levels in the quality label distribution; It indicates the probability value that the sample image belongs to the jth quality level among the preset multiple quality levels in the quality evaluation result; N indicates the total number of the preset multiple quality levels; log is used to indicate logarithmic operation; ∑ indicates summation operation.
5. A training method for an image quality assessment model, characterized in that: The method comprises: Acquire a sample set; each sample in the sample set includes a sample image and a quality label distribution corresponding to the sample image; each of the quality label distributions includes a plurality of preset quality levels and a probability value of the sample image belonging to each of the quality levels; Performing machine learning training on the initial model based on the sample set to obtain an image quality evaluation model; For each of the sample images in the sample set, the quality label distribution of the current sample image is determined in the following manner: Acquire multiple annotation results corresponding to the current sample image; the annotation results are used to indicate the quality level of the current sample image; Based on the multiple annotation results and the preset multiple quality levels, generating a quality label distribution of the current sample image; The step of generating the quality label distribution of the current sample image based on the multiple annotation results and the preset multiple quality levels includes: Calculating an average value of the quality grades corresponding to the multiple annotation results to obtain a grade mean; Based on the level mean and the preset multiple quality levels, determining a probability value that the current sample image belongs to each quality level of the preset multiple quality levels; The probability values of each quality level are combined to obtain the quality label distribution of the current sample image; wherein the probability that the quality level of the current sample image is the mean of the levels is the largest; and the quality label distribution is defined by Gaussian distribution.
6. The method according to claim 5, characterized in that The step of performing machine learning training on the initial model based on the sample set to obtain an image quality evaluation model comprises: Determining training samples based on the sample set; Inputting the sample images in the training samples into the initial model to obtain output results; Obtaining a loss amount according to the output result and the quality label distribution of the training samples; The weight parameters of the initial model are adjusted based on the loss amount; and the step of determining training samples based on the sample set is continued until the loss amount converges or a preset number of training times is reached, thereby obtaining an image quality evaluation model.
7. An image quality evaluation device, characterized in that: The device comprises: An image acquisition module, used for acquiring an image to be processed; A quality evaluation module, used for inputting the image to be processed into a pre-trained image quality evaluation model to obtain evaluation results of the image to be processed at multiple preset quality levels; The image quality evaluation model is obtained by machine learning training using a sample set, each sample in the sample set includes a sample image and a quality label distribution corresponding to the sample image; the quality label distribution includes a plurality of preset quality levels and a probability value of the sample image belonging to each of the quality levels; The device further comprises: a label distribution generating module; for obtaining, for each of the sample images in the sample set, a plurality of annotation results corresponding to the current sample image; the annotation results are used to indicate the quality level of the current sample image; Based on the multiple annotation results and the preset multiple quality levels, generating a quality label distribution of the current sample image; The label distribution generating module is also used for: Calculating an average value of the quality grades corresponding to the multiple annotation results to obtain a grade mean; Based on the level mean and the preset multiple quality levels, determining a probability value that the current sample image belongs to each quality level of the preset multiple quality levels; Combining the probability values of each quality level to obtain the quality label distribution of the current sample image; The probability that the quality level of the current sample image is the mean value of the level is the highest; and the quality label distribution is defined by Gaussian distribution.
8. A training device for an image quality assessment model, characterized in that: The device comprises: A sample set acquisition module, used to acquire a sample set; each sample in the sample set includes a sample image and a quality label distribution corresponding to the sample image; each of the quality label distributions includes a plurality of preset quality levels and a probability value of the sample image belonging to each of the quality levels; A model training module, used for performing machine learning training on the initial model based on the sample set to obtain an image quality evaluation model; The device also includes a label generation module, which is used to: Acquire multiple annotation results corresponding to the current sample image; the annotation results are used to indicate the quality level of the current sample image; Based on the multiple annotation results and the preset multiple quality levels, generating a quality label distribution of the current sample image; The label generation module is also used for: Calculating an average value of the quality grades corresponding to the multiple annotation results to obtain a grade mean; Based on the level mean and the preset multiple quality levels, determining a probability value that the current sample image belongs to each quality level of the preset multiple quality levels; The probability values of each quality level are combined to obtain the quality label distribution of the current sample image; wherein the probability that the quality level of the current sample image is the mean of the levels is the largest; and the quality label distribution is defined by Gaussian distribution.
9. An electronic device, characterized in that: It includes a processor and a memory, the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the image quality assessment method described in any one of claims 1 to 4 or the image quality assessment model training method described in any one of claims 5 to 6.
10. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are called and executed by the processor, the machine-executable instructions prompt the processor to implement the image quality assessment method described in any one of claims 1 to 4 or the training method of the image quality assessment model described in any one of claims 5 to 6.
