Image sharpness evaluation methods, devices and electronic equipment

Abstract

This application discloses an image sharpness evaluation method, apparatus, and electronic device, applied in the field of communication technology. The method includes: acquiring image data of a first image; obtaining perceptual data of the first image for each of at least one distortion item based on the image data; acquiring feature data of the image data through a feature extraction network; fusing the feature data with the perceptual data to obtain fused feature data; and performing regression processing on the fused feature data through a regression network to obtain a sharpness score of the first image.

Description

Technical Field

[0001] This application belongs to the field of communication technology, and more specifically, relates to an image sharpness evaluation method, apparatus and electronic device. Background Technology

[0002] Due to the limitations of camera sensor hardware, when people take photos with a mobile phone camera, the camera performs a series of image enhancement processes after obtaining the raw image data from the sensor, aiming to present users with a clearer image. To achieve better enhancement results, those skilled in the art often use algorithms to derive evaluation metrics to assess the sharpness of the enhanced mobile phone image. Then, based on these metrics, the enhancement process is optimized to obtain an even clearer image. However, currently widely used evaluation metrics such as PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index) both require a corresponding distortion-free image as a reference when evaluating image sharpness, making them difficult to apply to evaluating photos taken with mobile phones. Furthermore, research has shown that the evaluation results of PSNR and SSIM do not consistently align with subjective human evaluations. Summary of the Invention

[0003] The purpose of this application is to provide an image sharpness evaluation method, apparatus, and electronic device that can solve the problems in related technologies where sharpness evaluation indicators cannot be applied to photos taken by mobile phones, and where existing sharpness evaluation indicators are inconsistent with human subjective evaluation results.

[0004] In a first aspect, embodiments of this application provide an image sharpness evaluation method, the method comprising:

[0005] Obtain the image data of the first image;

[0006] Based on the image data, the perceptual data of the first image for each distortion item in at least one distortion item is obtained;

[0007] Feature data of the image data is obtained through a feature extraction network;

[0008] The feature data is fused with the perceived data to obtain fused feature data;

[0009] The sharpness score of the first image is obtained by performing regression processing on the fused feature data through a regression network.

[0010] Secondly, embodiments of this application provide an image sharpness evaluation device, the device comprising:

[0011] The first acquisition module is used to acquire image data of the first image;

[0012] The perception module is used to obtain the perception data of the first image for each distortion item in at least one distortion item, based on the image data.

[0013] The feature extraction module is used to obtain feature data of the image data through a feature extraction network;

[0014] The fusion module is used to fuse the feature data with the perceived data to obtain fused feature data;

[0015] The regression module is used to perform regression processing on the fused feature data through a regression network to obtain the sharpness score of the first image.

[0016] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0017] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0018] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0019] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0020] In this embodiment of the application, by fusing the feature data with the perceived data to obtain fused feature data, and by performing regression processing on the fused feature data through a regression network to obtain the sharpness score of the first image, the sharpness evaluation index in related technologies can be solved, which can solve the problems that the sharpness evaluation index cannot be applied to photos taken by mobile phones, and that the existing sharpness evaluation index is inconsistent with the subjective evaluation results of humans. Attached Figure Description

[0021] Figure 1 A flowchart of the image sharpness evaluation method provided in the embodiments of this application;

[0022] Figure 2 This is a schematic diagram of the feature extraction network in an embodiment of this application;

[0023] Figure 3 This is a schematic diagram of a regression network according to an embodiment of this application;

[0024] Figure 4 This is a schematic diagram of an image sharpness evaluation device according to an embodiment of this application;

[0025] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0026] Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0028] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0029] The image sharpness evaluation method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0030] like Figure 1 As shown, this is an image sharpness evaluation method provided in an embodiment of this application. The method includes the following steps 101 to 105, which will be described in detail below.

[0031] Step 101: Obtain the image data of the first image.

[0032] The first image is the image whose clarity the user needs to evaluate. This first image can be a photo taken by the user using an electronic device. For example, the first image can be a photo taken by the user using a mobile phone. The first image can also be a picture saved on the electronic device. For example, the first image can be a picture downloaded by the user through a browser. Another example is a picture pre-stored by the user using a picture-based application; yet another example is a picture sent by another user to the user using a social media application. Those skilled in the art should understand that the specific source of the first image is not limited here.

[0033] In this embodiment, when the user needs to evaluate the sharpness of the first image, the electronic device can automatically acquire the image data of the first image. The image data may include the brightness or grayscale values ​​of all pixels in the first image.

[0034] Step 102: Based on the image data, obtain the perceptual data of the first image for each distortion item in at least one distortion item.

[0035] In this embodiment, after acquiring the image data of the first image, distortion perception detection is performed on the image data to obtain perceptual data. For example, distortion perception detection such as noise detection, compression artifact detection, blur detection, and over / underexposure can be performed on the first image.

[0036] In one embodiment, the image data includes the grayscale values ​​of pixels in the first image, and obtaining perceptual data of the first image for each of at least one distortion item based on the image data includes:

[0037] Based on the grayscale values ​​of the pixels in the first image, the perceptual data of the first image for each distortion item is obtained.

[0038] In this embodiment, the image data includes the grayscale values ​​of the pixels in the first image. After acquiring the image data of the first image, perceptual data for each distortion item in the first image is obtained based on the grayscale values ​​of the pixels in the first image.

[0039] In one embodiment, the at least one distortion item includes a noise distortion item, and the perceptual data of the first image for the noise distortion item is the first perceptual data. The step of obtaining the first perceptual data of the first image for the noise distortion item based on the gray values ​​of the pixels in the first image includes: for a pixel in the first image, obtaining a first gray value difference between the gray value of the pixel and the average gray value of the other pixels based on the gray value of the pixel and the gray values ​​of other pixels adjacent to the pixel.

[0040] Based on the first grayscale difference, the noise value corresponding to the pixel is determined; wherein, the noise value indicates whether the pixel is a noise point;

[0041] The first perceptual data is obtained based on the noise value corresponding to each pixel in the first image;

[0042] In this embodiment, since image sharpness is mainly affected by four aspects: noise, compression artifacts, blur, and over / underexposure, when performing distortion perception detection on image data, at least one of the above four distortion aspects can be detected to obtain the perception data of each distortion aspect.

[0043] In the noise distortion detection process, the perceived data of the first image for the noise distortion is called the first perceived data. The step of obtaining the first perceived data of the first image for the noise distortion based on the grayscale values ​​of pixels in the first image includes:

[0044] Step 201: For a pixel in the first image, based on the gray value of the pixel and the gray values ​​of other pixels adjacent to the pixel, obtain a first gray value difference between the gray value of the pixel and the average gray value of the other pixels.

[0045] In this embodiment, noise distortion in the image is mainly caused by the following two aspects. First, the color sensors in the camera sensor are numerous and small, making them easily damaged and unable to display. In this case, they appear as dark spots in the image. Second, when shooting in dark scenes, due to insufficient exposure time, the camera sensor cannot accurately display certain pixels in the image. In this case, they will appear as white spots or dark spots in the image.

[0046] As can be seen from the causes of noise distortion described above, when performing noise distortion detection on the first image, the main focus is on detecting the grayscale values ​​of the pixels in the first image. When performing noise distortion detection on the first image, step 201 is executed first. That is, for each pixel in the first image, the grayscale value of that pixel is first obtained, then the average grayscale value of the other pixels adjacent to that pixel is calculated, and finally, based on the grayscale value of the pixel and the average grayscale value of the other pixels adjacent to that pixel, a first grayscale difference value is obtained between the grayscale value of the pixel and the average grayscale value of the other pixels.

[0047] Wherein, the average grayscale value of the other pixels adjacent to the pixel can be the average grayscale value of the other 8 pixels centered on the pixel, and the pixel and the other 8 pixels can form a 3x3 pixel array. In this case, the first grayscale difference can be calculated by the following formula:

[0048] D n (i)=|I(i)-M g (i)|, (1)

[0049] Where I(i) is the gray value of pixel i, M g (i) represents the average gray value of the other 8 pixels centered at i, D n (i) is the gray value difference between pixel i and the average gray value of the other 8 pixels centered at i, i.e., the first gray value difference. |.| indicates taking the absolute value of the element.

[0050] Step 202: Determine the noise value corresponding to the pixel based on the first grayscale difference; wherein the noise value indicates whether the pixel is a noise point.

[0051] In this embodiment, after obtaining the first grayscale difference value through formula (1), it can be determined whether the pixel is a noise point based on the first grayscale difference value and the average grayscale value of other pixels adjacent to the pixel.

[0052] The noise value can be used to identify whether a pixel is noisy. For example, a pixel can be marked with a value of 1 if it is noisy, and a pixel can be marked with a value of 0 if it is not noisy. Those skilled in the art should understand that the noise value here can also be other forms of characters, which are not specifically limited here.

[0053] In one example of this application, the first grayscale difference M of pixel i is obtained by formula (1). g (i) after which, if D n (i) greater than M g If (i) is half of the value of pixel i, then pixel i is determined to be noise and marked with the value 1. If D n (i) Less than or equal to M g If (i) is half of the value of pixel i, then pixel i is determined not to be noise and is marked with the value 0.

[0054] Step 203: Obtain the first perceptual data based on the noise value corresponding to each pixel in the first image.

[0055] In this embodiment, steps 201 and 202 are performed for each pixel in the first image to obtain the noise value corresponding to each pixel in the first image. Then, based on the noise value corresponding to each pixel in the first image, the first perceptual data is obtained. The first perceptual data accurately reflects the distribution location of noise points in the first image. That is, by using formula (1) to calculate pixel by pixel in the first image I, the perceptual data of the first image for the noise distortion term is finally obtained, i.e., the first perceptual data, denoted by I...n This indicates that the first perceived data can accurately reflect the distribution location of noise points in the first image I.

[0056] According to the embodiments of this application, by calculating the first gray-level difference between the gray-level value of the pixel and the average gray-level value of other pixels adjacent to the pixel, it is determined whether the pixel is a noise point, thereby obtaining the first perception data. This can realize noise detection based on local gray-level difference comparison, making it easier to perform noise detection on the image.

[0057] In one embodiment, the at least one distortion term includes a compression artifact distortion term, and the perceptual data of the first image for the compression artifact distortion term is second perceptual data. The step of obtaining the second perceptual data of the first image for the compression artifact distortion term based on the grayscale values ​​of pixels in the first image includes:

[0058] Step 301: For a pixel in the first image, based on the gray value of the pixel, the gray value of the first group of pixels adjacent to the pixel in the row direction, and the gray value of the second group of pixels adjacent to the pixel in the column direction, a second gray value difference between the pixel and the first group of pixels and a third gray value difference between the pixel and the second group of pixels are obtained.

[0059] In this embodiment, to save storage space, the original image can typically be compressed using JPEG to obtain the first image. However, if the compression level is too high, it can cause compression block artifacts, thus affecting the clarity of the first image. Furthermore, given the characteristics of JPEG compressed images—"compressing the image after dividing it into multiple compression blocks"—it is known that when using JPEG compression to obtain the first image, not all pixels in the first image will exhibit compression artifacts; these artifacts often appear at the junctions of multiple compression blocks. Therefore, when detecting compression artifacts in the first image, candidate points can be selected first, and then compression artifact detection can be performed on these candidate points.

[0060] For example, if the first image is obtained by JPEG compression of the original image into multiple 8x8 compressed blocks, candidate points can be selected according to position coordinates that are multiples of (8,8). Then, steps 301 to 303 are performed on the candidate points.

[0061] After selecting candidate points in the first image, step 301 is performed on the candidate points. That is, for the candidate points in the first image, a second gray-level difference between the candidate points and the first group of pixels and a third gray-level difference between the candidate points and the second group of pixels are obtained based on the gray-level value of the candidate points, the gray-level value of the first group of pixels adjacent to the candidate points in the row direction, and the gray-level value of the second group of pixels adjacent to the candidate points in the column direction.

[0062] The second grayscale difference can be calculated using the following formula (2), and the third grayscale difference can be calculated using the following formula (3).

[0063] C x (x,y)=(|I(x,y)-I(x+1,y)|+|I(x,y)-I(x-1,y)|) / 2, (2)

[0064] C y (x,y)=(|I(x,y)-I(x,y+1)|+|I(x,y)-I(x,y-1)|) / 2, (3)

[0065] Where I(x,y) is the gray value of the pixel with coordinates (x,y), and C x It is the grayscale value of the first group of adjacent pixels in the X direction of this pixel, C y It is the grayscale value of the second group of pixels adjacent to this pixel in the Y direction. If C x C y If any of the terms is greater than half of I(x,y), then the pixel block corresponding to that pixel is considered to have a compression block effect.

[0066] Step 302: Based on the second grayscale difference, the third grayscale difference, and the grayscale value of the pixel, obtain the grayscale loss value of the pixel relative to the surrounding pixels; wherein, the grayscale loss value reflects the degree of compression of the pixel.

[0067] In this embodiment, after determining that the pixel block corresponding to the pixel point has a compression block effect, the degree of compression artifact effect is further calculated by formula (4).

[0068] D c (x,y)=(max(C x (x,y),C y (x,y))-I(x,y) / 2) / max(C x (x,y),C y (x,y), (4)

[0069] Where max(..) represents taking the maximum of the two, D cThis represents the grayscale loss value, which reflects the degree of compression of the pixel. (D) c The value of D ranges from 0 to 1. c A higher value indicates a more severe compression artifact effect. Furthermore, the grayscale loss value for non-candidate points is set to 0.

[0070] Step 303: Obtain the second perceptual data based on the grayscale loss value of the pixels in the first image.

[0071] In this embodiment, after performing steps 301 to 303 on the candidate points of the first image, the grayscale loss value of the pixels in the first image is obtained. Based on the grayscale loss value of the pixels in the first image, the perceptual data of the first image for the compression artifact distortion term, i.e., the second perceptual data, is obtained, denoted as I. c .

[0072] According to the embodiments of this application, by pre-screening candidate points and then performing compression artifact detection on the candidate points to reflect whether there is a compression artifact effect in the compressed block corresponding to the candidate point, the amount of computation can be reduced and the power consumption can be lowered.

[0073] In one embodiment, the at least one distortion item includes a blur distortion item, and the perceptual data of the first image for the blur distortion item is third perceptual data. Obtaining the third perceptual data of the first image for the blur distortion item based on the grayscale values ​​of pixels in the first image includes:

[0074] Step 401: For a pixel in the first image, obtain the local high-frequency energy value of the pixel based on the gray value of the pixel and the gray values ​​of other pixels adjacent to the pixel.

[0075] In this embodiment, when performing blur detection on the first image, candidate points that may cause motion blur can be screened based on the optical flow of two adjacent frames. In one example, when taking a photo, the lens can quickly acquire two frames, denoted as frame T and frame T+1, where frame T is the final result returned to the user, which is the original image of the first image. The optical flow of frame T and frame T+1 is calculated using existing optical flow calculation methods. The amplitude of pixel i is denoted as L(i), and then the average value of L is calculated, with the result denoted as Li. m If there exists L(i) greater than L m This indicates that the pixel is moving significantly and is marked as a candidate point for potential motion blur.

[0076] After identifying candidate points that may experience motion blur, step 401 is executed. That is, for the candidate point, the local high-frequency energy value of the candidate point is obtained based on the gray value of the candidate point and the gray values ​​of other pixels adjacent to the candidate point. The local high-frequency energy value of the candidate point can be calculated using the following formula (5).

[0077] h(x,y)=(s(x,y)-s σ (x,y)) 2 (5)

[0078] Where h(x,y) is the local high-frequency energy value of the candidate point (x,y), s is the sum of the gray levels of the neighboring pixels centered at the candidate point (x,y), and s σ The result is obtained by Gaussian filtering with a standard deviation of σ, denoted by s. The local grayscale and s are calculated as follows:

[0079]

[0080] Where U(x,y) is the local region of the candidate point (x,y), j is the pixel coordinate within the local region, and f is the gray level. The local region of U(x,y) can be a local region defined by any one of the pixel arrays of 3x3, 5x5, or 7x7.

[0081] Using formulas (5) and (6), the local high-frequency energy values ​​h(x,y), h(x-1,y), h(x+1), h(x,y-1), and h(x,y+1) of the candidate point (x,y) and its adjacent pixels in the X and Y axes can be calculated respectively.

[0082] It should be noted that, in order to reduce the amount of computation, the local high-frequency energy value of non-candidate points can be set to 0.

[0083] Step 402: Based on the proportional relationship between the local high-frequency energy value of the pixel and the local high-frequency energy value of the image region where the pixel is located, obtain the blur degree value corresponding to the pixel; wherein, the image region includes the pixel and other pixels adjacent to the pixel.

[0084] In this embodiment, the local high-frequency energies of the candidate point (x,y) and its neighboring pixels can be compared. First, the mean of the five values ​​h(x,y), h(x-1,y), h(x+1,y), h(x,y-1), and h(x,y+1) is calculated and denoted as h. m (x,y). Next, calculate h(x,y) and h... m The difference D between (x, y) h (x,y). Specifically, h m (x,y) and D hThe calculation of (x,y) is as follows:

[0085] h m (x,y)=(h(x,y)+h(x-1,y)+h(x+1,y)+h(x,y-1)+h(x,y+1) / 5), (7)

[0086] D h (x,y)=|(h m (x,y)-h(x,y)|, (8)

[0087] Among them, D h (x,y) represents the blur value corresponding to the pixel.

[0088] If D h (x,y) is less than h m If a candidate point (x, y) is found to be 1 / 10 of a given value, then that point is determined to be a blurred pixel. Otherwise, the candidate point (x, y) is determined to be a non-blurred pixel.

[0089] Step 403: Obtain the third perceptual data based on the blur level values ​​of the pixels in the first image.

[0090] In this embodiment, blur detection is performed on all candidate points of the first image to obtain the perceptual data of the first image for the blur distortion item, which is denoted as the third perceptual data I. b .

[0091] According to the embodiments of this application, by pre-screening candidate points, then performing fuzzy detection on the candidate points, and setting the fuzziness level of non-candidate points to 0, the amount of computation can be reduced.

[0092] In one embodiment, the at least one distortion item includes an exposure distortion item, and the perceptual data of the first image for the exposure distortion item is fourth perceptual data. Obtaining the fourth perceptual data of the first image for the exposure distortion item based on the grayscale values ​​of pixels in the first image includes:

[0093] Step 501: For each pixel in the first image, the exposure distortion value of the pixel is obtained based on the grayscale value of the pixel and the grayscale range corresponding to the standard exposure.

[0094] Step 502: Obtain the fourth sensing data based on the exposure distortion values ​​of the pixels in the first image.

[0095] In this embodiment, when capturing the original image of the first image, due to the sensitivity limitations of the camera sensor, overexposed or underexposed areas may easily occur when shooting scenes that are too bright or too dark, affecting the image clarity. When detecting overexposed / underexposed areas in the first image, a grayscale range corresponding to a standard exposure is preset. For example, an overexposure threshold of 250 and an underexposure threshold of 50 can be set, with the grayscale range corresponding to the standard exposure being greater than 50 and less than 250. After presetting the grayscale range corresponding to the standard exposure, for each pixel in the first image, the exposure distortion value of the pixel is obtained based on its grayscale value and the grayscale range corresponding to the standard exposure. That is, when the grayscale value of a pixel is not within the grayscale range corresponding to the standard exposure, the pixel is determined to be an exposure distortion point. The exposure distortion value can be used to mark whether a pixel in the first image is an exposure distortion point. For example, 1 can represent that the pixel is an exposure distortion point, and 0 can represent that the pixel is not an exposure distortion point. Those skilled in the art should understand that the exposure distortion value can also be other forms of symbols, which are not specifically limited here.

[0096] Based on the exposure distortion values ​​of the pixels in the first image, the perceptual data of the first image for the exposure distortion term is obtained, that is, the fourth perceptual data, denoted as I. o .

[0097] According to the embodiments of this application, by presetting the grayscale range corresponding to the standard exposure, and then comparing whether the grayscale of the pixels of the first image is within the range, it is easier to determine the overexposed / underexposed areas of the image.

[0098] Step 103: Obtain feature data of the image data through a feature extraction network.

[0099] In this embodiment, after obtaining the perceptual data of the first image for each of at least one distortion item, feature data of the image data is obtained through a feature extraction network. In this embodiment, a feature extraction network model can be trained first based on training samples, and then the feature data of the image data can be obtained based on this feature extraction network model.

[0100] In one embodiment, before obtaining feature data of the first image through a feature extraction network, the method further includes:

[0101] Step 601: Obtain a training sample set; wherein each training sample in the training sample set includes a sample image and a target sharpness score of the sample image as a sample label.

[0102] In this embodiment, the training sample set includes multiple training samples. Each training sample in the training sample set includes a sample image and a target sharpness score of the sample image as a sample label. The target sharpness score of the sample image as a sample label is the ideal sharpness score of the sample image, denoted by q.

[0103] Step 602: Using the training sample set, jointly train the feature extraction network and the regression network to obtain the parameter values ​​of the first set of network parameters of the feature extraction network and the parameter values ​​of the second set of network parameters of the regression network.

[0104] In this embodiment, after obtaining the training sample set, the feature extraction network and the regression network are jointly trained to obtain the parameter values ​​of a first set of network parameters for the feature extraction network and the parameter values ​​of a second set of network parameters for the regression network. The parameter values ​​of the first set of network parameters can be represented by θ. d This means that the parameter values ​​of the second set of network parameters can be represented by θ. r If so, the training process can be represented by the following formula (9):

[0105]

[0106] Where, θ d Let θ be the parameter value of the first set of network parameters. r These are the parameter values ​​for the second set of network parameters.

[0107] f r () represents the sensory fusion process, G d () and G r () represent the feature extraction network and the regression network, respectively. The argmin() function represents the function that minimizes the loss function.

[0108] During joint training, initial values ​​for the first set of network parameters and the second set of network parameters can be randomly provided first. Then, these initial values ​​are substituted into N samples from the training sample set to calculate the predicted sharpness score. Then calculate The distance between q and the initial loss value is obtained, corresponding to the initial values ​​of the first and second sets of network parameters. Then, based on the existing algorithm, the initial values ​​of the first and second sets of network parameters are changed to make the loss value smaller. Finally, the parameter values ​​of the first and second sets of network parameters with smaller corresponding loss values ​​are obtained.

[0109] It should be noted that the parameter values ​​of the first set of network parameters and the parameter values ​​of the second set of network parameters can be the parameter values ​​corresponding to the 100th training iteration after N samples have been trained 100 times. Those skilled in the art should understand that there is no limit to the specific number of training iterations.

[0110] Step 603: Update the feature extraction network according to the parameter values ​​of the first set of network parameters, and update the regression network according to the parameter values ​​of the second set of network parameters.

[0111] In this embodiment, the parameter values ​​of the first set of network parameters obtained from training are applied to the feature extraction network, and the parameter values ​​of the second set of network parameters obtained from training are applied to the regression network.

[0112] According to the embodiments of this application, by jointly training to obtain the parameter values ​​of the first set of network parameters of the feature extraction network and the parameter values ​​of the second set of network parameters of the regression network, the accuracy of image sharpness evaluation can be improved.

[0113] In one embodiment, both the feature extraction network and the regression network are convolutional neural networks based on convolutional kernels. The feature extraction network includes N concatenated first convolutional blocks, each first convolutional block comprising a first convolutional layer, an activation layer, and a second convolutional layer connected in series, wherein N is greater than 1; the acquisition of feature data from the image data through the feature extraction network includes:

[0114] Step 701: Obtain the i-th level feature data of the i-th level input data through the i-th first convolutional block; where i is an integer from 1 to N. When i = 1, the i-th level input data is the image data. When i is greater than 1, the i-th level input data is the (i-1)-th level output data of the (i-1)-th first convolutional block.

[0115] Step 702: Add the i-th level feature data to the i-th level input data to obtain the i-th level output data of the i-th first convolutional block; wherein, the N-th level output data is the feature data of the image data.

[0116] In this embodiment, the feature extraction network includes N first convolutional blocks connected in series, where N can be four first convolutional blocks. Each first convolutional block includes a first convolutional layer, an activation layer, and a second convolutional layer connected in series. The first and second convolutional layers can each be dual-channel. Alternatively, one of the first and second convolutional layers can be four-channel; no specific limitation is made here. Those skilled in the art should understand that the first and second convolutional layers can also have other numerical values ​​of channels; no specific limitation is made here.

[0117] The i-th level feature data of the i-th level input data is obtained through the i-th first convolutional block. Specifically, the first-level feature data of the first-level input data is obtained through the first first convolutional block. The first-level input data is the image data of the first image. After obtaining the first-level feature data, it is added to the first-level input data to obtain the first-level output data of the first first convolutional block. Then, the first-level output data is used as the second-level input data, and the second-level feature data of the second-level input data is obtained through the second first convolutional block. After obtaining the second-level feature data, it is added to the second-level input data to obtain the second-level output data of the second first convolutional block. This process continues, with the i-th level input data being the (i-1)-th level output data of the (i-1)-th first convolutional block. When there are N first convolutional blocks, the output data of the Nth first convolutional block is the feature data of the image data of the first image.

[0118] For example, with Figure 2 For example, the feature extraction network G d It includes four first convolutional blocks. Each convolutional block consists of a first convolutional layer, an activation layer, and a second convolutional layer. The number of channels in each convolutional layer varies. When image data is input into the first first convolutional block, the input channel is 1 and the output channel is 4. After the second first convolutional block, the input channel becomes 4 and the output channel becomes 8. After the third first convolutional block, the input channel becomes 8 and the output channel becomes 16. After the fourth first convolutional block, the input channel becomes 16 and the output channel becomes 16. The final feature data obtained is 16-channel feature data F. d .

[0119] Step 104: Fuse the feature data with the perceived data to obtain fused feature data.

[0120] In this embodiment, the first, second, third, and fourth sensing data are fused with feature data to obtain fused feature data. Feature data F has been explained above. d The number of channels is 16. When fusing the first, second, third, and fourth sensory data with the feature data, that is, when fusing F... d The features are sequentially multiplied with the first, second, third, and fourth sensory data, and then stacked according to the channel dimension. Therefore, the final fused feature F is obtained. r Channel is F d 4 times, that is, F r The number of channels is 64.

[0121] Step 105: Perform regression processing on the fused feature data using a regression network to obtain the sharpness score of the first image.

[0122] In this embodiment, the regression network includes M second convolutional blocks connected in series, and a fully connected layer connected to the Mth convolutional block. Each second convolutional block includes a third convolutional layer, an activation layer, a max-pooling layer, and a fourth convolutional layer connected in series, wherein M is greater than 1. The regression processing of the fused feature data through the regression network to obtain the sharpness score of the first image includes:

[0123] Step 801: Obtain the j-th level feature data of the j-th level input data through the j-th second convolutional block; where j is an integer from 1 to M, when j = 1, the j-th level input data is the fused feature data, and when j is greater than 1, the j-th level input data is the (j-1)-th level output data of the (j-1)-th second convolutional block.

[0124] Step 802: Add the j-th level feature data to the j-th level input data to obtain the j-th level output data of the j-th second convolutional block.

[0125] Step 803: Regress the M-th level output data through the fully connected layer to obtain the sharpness score of the first image.

[0126] In this embodiment, the regression network includes M second convolutional blocks connected in series, and a fully connected layer connected to the Mth convolutional block. Each second convolutional block includes a third convolutional layer, an activation layer, a max-pooling layer, and a fourth convolutional layer connected in series. For example, when M is 4, the regression network can be as follows: Figure 3 The structure shown.

[0127] After obtaining the fused feature data F r Next, the first-level feature data of the first-level input data is obtained through the first second convolutional block. Then, the first-level feature data is added to the first-level input data to obtain the first-level output data of the first second convolutional block. The first-level output data is then used as the second-level input data, and the second-level feature data of the second-level input data is obtained through the second second convolutional block. This process is repeated, with the second-level feature data being added to the second-level input data to obtain the second-level output data of the second second convolutional block. This process continues, with the j-th level input data becoming the (j-1)-th level output data of the (j-1)-th second convolutional block. Finally, the M-th level output data of the M-th second convolutional block is obtained.

[0128] The sharpness score of the first image is obtained by regressing the M-th level output data through a fully connected layer.

[0129] For example, when M=4, the structural diagram of the regression network is as follows: Figure 3 As shown, the first-level input data is the fused feature data F. r After passing through the first convolutional block, the number of input channels is 64 and the number of output channels is 64. After passing through the second convolutional block, the number of input channels is 64 and the number of output channels is 128. After passing through the third convolutional block, the number of input channels is 128 and the number of output channels is 256. After passing through the fourth convolutional block, the number of input channels is 256 and the number of output channels is 256. Then, after passing through a fully connected layer, the fourth-level output data is reduced to single-channel data. This is the sharpness score of the first image.

[0130] According to the embodiments of this application, by fusing the feature data with the perceived data to obtain fused feature data, and by performing regression processing on the fused feature data through a regression network to obtain the sharpness score of the first image, the sharpness evaluation index in related technologies cannot be applied to photos taken by mobile phones, and the existing sharpness evaluation index is inconsistent with the subjective evaluation results of humans.

[0131] The image sharpness evaluation method provided in this application can be executed by an image sharpness evaluation device. This application uses an image sharpness evaluation device to perform the image sharpness evaluation method as an example to illustrate the image sharpness evaluation device provided in this application.

[0132] like Figure 4 As shown, an image sharpness evaluation device 900 provided in an embodiment of this application is included. The device 900 includes:

[0133] The acquisition module 901 is used to acquire the image data of the first image;

[0134] The perception module 902 is used to obtain the perception data of the first image for each distortion item in at least one distortion item based on the image data.

[0135] Feature extraction module 903 is used to obtain feature data of the image data through a feature extraction network;

[0136] The fusion module 904 is used to fuse the feature data with the perceived data to obtain fused feature data;

[0137] The regression module 905 is used to perform regression processing on the fused feature data through a regression network to obtain the sharpness score of the first image.

[0138] According to the embodiments of this application, by fusing the feature data with the perceived data to obtain fused feature data, and by performing regression processing on the fused feature data through a regression network to obtain the sharpness score of the first image, the sharpness evaluation index in related technologies cannot be applied to photos taken by mobile phones, and the existing sharpness evaluation index is inconsistent with the subjective evaluation results of humans.

[0139] In one embodiment, the image data includes the grayscale values ​​of pixels in the first image; the perception module 902 is used to obtain the perception data of the first image for each distortion item based on the grayscale values ​​of pixels in the first image.

[0140] In one embodiment, the at least one distortion item includes a noise distortion item, and the perceived data of the first image for the noise distortion item is the first perceived data. The perception module 902 is used to, for a pixel in the first image, obtain a first grayscale difference between the grayscale value of the pixel and the average grayscale value of other pixels adjacent to the pixel; determine the noise value corresponding to the pixel based on the first grayscale difference; wherein the noise value indicates whether the pixel is a noise point; and obtain the first perceived data based on the noise value corresponding to each pixel in the first image.

[0141] According to the embodiments of this application, by calculating the first gray-level difference between the gray-level value of the pixel and the average gray-level value of other pixels adjacent to the pixel, it is determined whether the pixel is a noise point, thereby obtaining the first perception data. This can realize noise detection based on local gray-level difference comparison, making it easier to perform noise detection on the image.

[0142] In one embodiment, the at least one distortion item includes a compression artifact distortion item, and the perceived data of the first image for the compression artifact distortion item is the second perceived data. The perception module 902 is used to, for a pixel in the first image, obtain a second grayscale difference between the pixel and the first group of pixels, and a third grayscale difference between the pixel and the second group of pixels, based on the grayscale value of the pixel, the grayscale value of a first group of pixels adjacent to the pixel in the row direction, and the grayscale value of a second group of pixels adjacent to the pixel in the column direction; obtain a grayscale loss value of the pixel relative to its surrounding pixels based on the second grayscale difference value, the third grayscale difference value, and the grayscale value of the pixel; wherein, the grayscale loss value reflects the degree of compression of the pixel; and obtain the second perceived data based on the grayscale loss value of the pixel in the first image.

[0143] According to the embodiments of this application, by pre-screening candidate points and then performing compression artifact detection on the candidate points to reflect whether there is a compression artifact effect in the compressed block corresponding to the candidate point, the amount of computation can be reduced and the power consumption can be lowered.

[0144] In one embodiment, the at least one distortion item includes a blur distortion item, and the perceptual data of the first image for the blur distortion item is third perceptual data. The perceptual module 902 is used to, for a pixel in the first image, obtain the local high-frequency energy value of the pixel based on the gray value of the pixel and the sum of the gray values ​​of other pixels adjacent to the pixel; obtain the blur degree value corresponding to the pixel based on the proportional relationship between the local high-frequency energy value of the pixel and the local high-frequency energy value of the image region where the pixel is located; wherein, the image region includes the pixel and other pixels adjacent to the pixel; and obtain the third perceptual data based on the blur degree value of the pixel in the first image.

[0145] According to the embodiments of this application, by pre-screening candidate points, then performing fuzzy detection on the candidate points, and setting the fuzziness level of non-candidate points to 0, the amount of computation can be reduced.

[0146] In one embodiment, the at least one distortion item includes an exposure distortion item, and the perceived data of the first image for the exposure distortion item is fourth perceived data. The perception module 902 is used to obtain the exposure distortion value of a pixel for a pixel in the first image based on the gray value of the pixel and the gray range corresponding to the standard exposure; and to obtain the fourth perceived data based on the exposure distortion value of the pixel in the first image.

[0147] According to the embodiments of this application, by presetting the grayscale range corresponding to the standard exposure, and then comparing whether the grayscale of the pixels of the first image is within the range, it is easier to determine the overexposed / underexposed areas of the image.

[0148] In one embodiment, both the feature extraction network and the regression network are convolutional neural networks based on convolutional kernels.

[0149] In one embodiment, the feature extraction network includes N first convolutional blocks connected in series. Each first convolutional block includes a first convolutional layer, an activation layer, and a second convolutional layer connected in series, wherein N is greater than 1. The feature extraction module 903 is used to obtain the i-th level feature data of the i-th level input data through the i-th first convolutional block; wherein i is an integer from 1 to N, when i = 1, the i-th level input data is the image data, and when i is greater than 1, the i-th level input data is the (i-1)-th level output data of the (i-1)-th first convolutional block; the i-th level feature data is added to the i-th level input data to obtain the i-th level output data of the i-th first convolutional block; wherein the N-th level output data is the feature data of the image data.

[0150] In one embodiment, the regression network includes M second convolutional blocks connected in series, and a fully connected layer connected to the Mth convolutional block. The second convolutional block includes a third convolutional layer, an activation layer, a max pooling layer, and a fourth convolutional layer connected in series, wherein M is greater than 1. The regression module 905 is used to obtain the j-th level feature data of the j-th level input data through the j-th second convolutional block; wherein j is an integer from 1 to M, when j = 1, the j-th level input data is the fused feature data, and when j is greater than 1, the j-th level input data is the (j-1)-th level output data of the (j-1)-th second convolutional block; the j-th level feature data is added to the j-th level input data to obtain the j-th level output data of the j-th second convolutional block; the fully connected layer is used to regress the M-th level output data to obtain the sharpness score of the first image.

[0151] In one embodiment, the device further includes a training module 906.

[0152] The training module 906 is used to acquire a training sample set; wherein each training sample in the training sample set includes a sample image and a target sharpness score of the sample image as a sample label; the feature extraction network and the regression network are jointly trained using the training sample set to obtain parameter values ​​of a first set of network parameters for the feature extraction network and parameter values ​​of a second set of network parameters for the regression network; the feature extraction network is updated according to the parameter values ​​of the first set of network parameters, and the regression network is updated according to the parameter values ​​of the second set of network parameters.

[0153] According to the embodiments of this application, by jointly training to obtain the parameter values ​​of the first set of network parameters of the feature extraction network and the parameter values ​​of the second set of network parameters of the regression network, the accuracy of image sharpness evaluation can be improved.

[0154] The image sharpness evaluation device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0155] The image sharpness evaluation device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0156] The image sharpness evaluation device provided in this application embodiment can achieve... Figures 1 to 3 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0157] Optionally, such as Figure 5 As shown, this application embodiment also provides an electronic device 10, including a processor 11 and a memory 12. The memory 12 stores a program or instructions that can run on the processor 11. When the program or instructions are executed by the processor 11, they implement the various steps of the above-described image sharpness evaluation method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0158] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0159] Figure 6 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0160] The electronic device 1000 includes, but is not limited to, components such as: radio frequency unit 1001, network module 1002, audio output unit 1003, input unit 1004, sensor 1005, display unit 1006, user input unit 1007, interface unit 1008, memory 1009, and processor 1010.

[0161] Those skilled in the art will understand that the electronic device 1000 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 1010 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 6 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0162] The processor 1010 is used to acquire the image data of the first image.

[0163] The processor 1010 is configured to obtain, based on the image data, perceptual data of the first image for each of at least one distortion item;

[0164] Processor 1010 is used to acquire feature data of the image data through a feature extraction network;

[0165] The processor 1010 is used to fuse the feature data with the perceived data to obtain fused feature data;

[0166] The processor 1010 is used to perform regression processing on the fused feature data through a regression network to obtain the sharpness score of the first image.

[0167] According to the embodiments of this application, by fusing the feature data with the perceived data to obtain fused feature data, and by performing regression processing on the fused feature data through a regression network to obtain the sharpness score of the first image, the sharpness evaluation index in related technologies cannot be applied to photos taken by mobile phones, and the existing sharpness evaluation index is inconsistent with the subjective evaluation results of humans.

[0168] Optionally, the image data includes the grayscale values ​​of pixels in the first image; the processor 1010 is used to obtain the perceptual data of the first image for each distortion item based on the grayscale values ​​of pixels in the first image.

[0169] Optionally, the at least one distortion item includes a noise distortion item, and the perceived data of the first image for the noise distortion item is the first perceived data. The processor 1010 is configured to, for a pixel in the first image, obtain a first grayscale difference between the grayscale value of the pixel and the average grayscale value of other pixels adjacent to the pixel; determine the noise value corresponding to the pixel based on the first grayscale difference; wherein the noise value indicates whether the pixel is a noise point; and obtain the first perceived data based on the noise value corresponding to each pixel in the first image.

[0170] According to the embodiments of this application, by calculating the first gray-level difference between the gray-level value of the pixel and the average gray-level value of other pixels adjacent to the pixel, it is determined whether the pixel is a noise point, thereby obtaining the first perception data. This can realize noise detection based on local gray-level difference comparison, making it easier to perform noise detection on the image.

[0171] Optionally, the at least one distortion item includes a compression artifact distortion item, and the perceptual data of the first image for the compression artifact distortion item is the second perceptual data. The processor 1010 is configured to, for a pixel in the first image, obtain a second grayscale difference between the pixel and the first group of pixels, and a third grayscale difference between the pixel and the second group of pixels, based on the grayscale value of the pixel, the grayscale value of a first group of pixels adjacent to the pixel in the row direction, and the grayscale value of a second group of pixels adjacent to the pixel in the column direction; obtain a grayscale loss value of the pixel relative to its surrounding pixels based on the second grayscale difference value, the third grayscale difference value, and the grayscale value of the pixel; wherein the grayscale loss value reflects the degree of compression of the pixel; and obtain the second perceptual data based on the grayscale loss value of the pixel in the first image.

[0172] According to the embodiments of this application, by pre-screening candidate points and then performing compression artifact detection on the candidate points to reflect whether there is a compression artifact effect in the compressed block corresponding to the candidate point, the amount of computation can be reduced and the power consumption can be lowered.

[0173] Optionally, the at least one distortion item includes a blur distortion item, and the perceptual data of the first image for the blur distortion item is third perceptual data. The processor 1010 is used to, for a pixel in the first image, obtain the local high-frequency energy value of the pixel based on the gray value of the pixel and the sum of the gray values ​​of other pixels adjacent to the pixel; obtain the blur degree value corresponding to the pixel based on the proportional relationship between the local high-frequency energy value of the pixel and the local high-frequency energy value of the image region where the pixel is located; wherein, the image region includes the pixel and other pixels adjacent to the pixel; and obtain the third perceptual data based on the blur degree value of the pixel in the first image.

[0174] According to the embodiments of this application, by pre-screening candidate points, then performing fuzzy detection on the candidate points, and setting the fuzziness level of non-candidate points to 0, the amount of computation can be reduced.

[0175] Optionally, the at least one distortion item includes an exposure distortion item, and the perceptual data of the first image for the exposure distortion item is fourth perceptual data. The processor 1010 is used to obtain the exposure distortion value of a pixel for a pixel in the first image based on the gray value of the pixel and the gray range corresponding to the standard exposure; and to obtain the fourth perceptual data based on the exposure distortion value of the pixel in the first image.

[0176] According to the embodiments of this application, by presetting the grayscale range corresponding to the standard exposure, and then comparing whether the grayscale of the pixels of the first image is within the range, it is easier to determine the overexposed / underexposed areas of the image.

[0177] Optionally, both the feature extraction network and the regression network are convolutional neural networks based on convolutional kernels.

[0178] Optionally, the feature extraction network includes N first convolutional blocks connected in series. Each first convolutional block includes a first convolutional layer, an activation layer, and a second convolutional layer connected in series, wherein N is greater than 1. The processor 1010 is used to obtain the i-th level feature data of the i-th level input data through the i-th first convolutional block; wherein i is an integer from 1 to N, when i = 1, the i-th level input data is the image data, and when i is greater than 1, the i-th level input data is the (i-1)-th level output data of the (i-1)-th first convolutional block; the i-th level feature data is added to the i-th level input data to obtain the i-th level output data of the i-th first convolutional block; wherein the N-th level output data is the feature data of the image data.

[0179] Optionally, the regression network includes M second convolutional blocks connected in series, and a fully connected layer connected to the Mth convolutional block. The second convolutional block includes a third convolutional layer, an activation layer, a max pooling layer, and a fourth convolutional layer connected in series, wherein M is greater than 1. The processor 1010 is used to obtain the j-th level feature data of the j-th level input data through the j-th second convolutional block; wherein j is an integer from 1 to M, when j = 1, the j-th level input data is the fused feature data, and when j is greater than 1, the j-th level input data is the (j-1)-th level output data of the (j-1)-th second convolutional block; the j-th level feature data is added to the j-th level input data to obtain the j-th level output data of the j-th second convolutional block; regression is performed on the M-th level output data through the fully connected layer to obtain the sharpness score of the first image.

[0180] Optionally, the processor 1010 is used to acquire a training sample set; wherein each training sample in the training sample set includes a sample image and a target sharpness score of the sample image as a sample label; the feature extraction network and the regression network are jointly trained using the training sample set to obtain parameter values ​​of a first set of network parameters for the feature extraction network and parameter values ​​of a second set of network parameters for the regression network; the feature extraction network is updated according to the parameter values ​​of the first set of network parameters, and the regression network is updated according to the parameter values ​​of the second set of network parameters.

[0181] According to the embodiments of this application, by jointly training to obtain the parameter values ​​of the first set of network parameters of the feature extraction network and the parameter values ​​of the second set of network parameters of the regression network, the accuracy of image sharpness evaluation can be improved.

[0182] It should be understood that, in this embodiment, the input unit 1004 may include a graphics processing unit (GPU) 10041 and a microphone 10042. The GPU 10041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 1006 may include a display panel 10061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0183] The memory 1009 can be used to store software programs and various data. The memory 1009 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 1009 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 1009 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0184] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into the processor 1010.

[0185] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described image sharpness evaluation method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0186] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0187] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described image sharpness evaluation method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0188] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0189] This application provides a computer program product stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the image sharpness evaluation method embodiment described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0190] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0191] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0192] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for evaluating image sharpness, characterized in that, The method includes: Obtain the image data of the first image; Based on the image data, the perceptual data of the first image for each of at least one distortion item is obtained; wherein, the at least one distortion item is at least one of noise distortion, compression artifact distortion, blur distortion, and exposure distortion. Feature data of the image data is obtained through a feature extraction network; The feature data is fused with the perceived data to obtain fused feature data; The sharpness score of the first image is obtained by performing regression processing on the fused feature data through a regression network. The at least one distortion item includes a noise distortion item, and the perceptual data of the first image for the noise distortion item is the first perceptual data. Obtaining the first perceptual data of the first image for the noise distortion item based on the image data includes: For a pixel in the first image, a first grayscale difference between the grayscale value of the pixel and the average grayscale value of the other pixels adjacent to the pixel is obtained based on the grayscale value of the pixel and the average grayscale value of the other pixels. Based on the first grayscale difference, the noise value corresponding to the pixel is determined; wherein, the noise value indicates whether the pixel is a noise point; The first perceptual data is obtained based on the noise value corresponding to each pixel in the first image.

2. The method according to claim 1, characterized in that, The at least one distortion term includes a compression artifact distortion term, and the perceptual data of the first image for the compression artifact distortion term is the second perceptual data. Obtaining the second perceptual data of the first image for the compression artifact distortion term based on the image data includes: For a pixel in the first image, a second grayscale difference between the pixel and the first group of pixels and a third grayscale difference between the pixel and the second group of pixels are obtained based on the grayscale value of the pixel, the grayscale value of the first group of pixels adjacent to the pixel in the row direction, and the grayscale value of the second group of pixels adjacent to the pixel in the column direction. Based on the second grayscale difference, the third grayscale difference, and the grayscale value of the pixel, a grayscale loss value of the pixel relative to its surrounding pixels is obtained; wherein, the grayscale loss value reflects the degree of compression of the pixel. The second perceptual data is obtained based on the grayscale loss values ​​of the pixels in the first image.

3. The method according to claim 1, characterized in that, The at least one distortion item includes a blur distortion item, and the perceptual data of the first image for the blur distortion item is third perceptual data. Obtaining the third perceptual data of the first image for the blur distortion item based on the image data includes: For a pixel in the first image, the local high-frequency energy value of the pixel is obtained based on the gray value of the pixel and the sum of the gray values ​​of other pixels adjacent to the pixel. The blur level value corresponding to the pixel is obtained based on the proportional relationship between the local high-frequency energy value of the pixel and the local high-frequency energy value of the image region where the pixel is located; wherein, the image region includes the pixel and other pixels adjacent to the pixel. The third perceptual data is obtained based on the blur level of the pixels in the first image.

4. The method according to claim 1, characterized in that, The at least one distortion item includes an exposure distortion item, and the perceptual data of the first image for the exposure distortion item is fourth perceptual data. Obtaining the fourth perceptual data of the first image for the exposure distortion item based on the image data includes: For each pixel in the first image, the exposure distortion value of the pixel is obtained based on the grayscale value of the pixel and the grayscale range corresponding to the standard exposure. The fourth sensing data is obtained based on the exposure distortion values ​​of the pixels in the first image.

5. The method according to claim 1, characterized in that, Both the feature extraction network and the regression network are convolutional neural networks based on convolutional kernels.

6. The method according to claim 5, characterized in that, The feature extraction network includes N first convolutional blocks connected in series. Each first convolutional block includes a first convolutional layer, an activation layer, and a second convolutional layer connected in series, wherein N is greater than 1. The feature data obtained from the image data through the feature extraction network includes: The i-th level feature data of the i-th level input data is obtained through the i-th first convolutional block; where i is an integer from 1 to N. When i=1, the i-th level input data is the image data, and when i is greater than 1, the i-th level input data is the (i-1)-th level output data of the (i-1)-th first convolutional block. The i-th level feature data is added to the i-th level input data to obtain the i-th level output data of the i-th first convolutional block; wherein, the N-th level output data is the feature data of the image data.

7. The method according to claim 5, characterized in that, The regression network includes M second convolutional blocks connected in series, and a fully connected layer connected to the Mth convolutional block. Each second convolutional block includes a third convolutional layer, an activation layer, a max-pooling layer, and a fourth convolutional layer connected in series, wherein M is greater than 1. The regression processing of the fused feature data through the regression network to obtain the sharpness score of the first image includes: The j-th level feature data of the j-th level input data is obtained through the j-th second convolutional block; where j is an integer from 1 to M, when j=1, the j-th level input data is the fused feature data, and when j is greater than 1, the j-th level input data is the (j-1)-th level output data of the (j-1)-th second convolutional block; The j-th level feature data is added to the j-th level input data to obtain the j-th level output data of the j-th second convolutional block; The sharpness score of the first image is obtained by regressing the M-th level output data through the fully connected layer.

8. The method according to any one of claims 1 to 7, characterized in that, Before obtaining the feature data of the first image through the feature extraction network, the method further includes: Obtain a training sample set; wherein each training sample in the training sample set includes a sample image and a target sharpness score of the sample image as a sample label; Using the training sample set, the feature extraction network and the regression network are jointly trained to obtain the parameter values ​​of the first set of network parameters of the feature extraction network and the parameter values ​​of the second set of network parameters of the regression network. The feature extraction network is updated based on the parameter values ​​of the first set of network parameters, and the regression network is updated based on the parameter values ​​of the second set of network parameters.

9. An image sharpness evaluation device, characterized in that, The device includes: The first acquisition module is used to acquire image data of the first image; The perception module is configured to obtain, based on the image data, the perception data of the first image for each of at least one distortion item; wherein the at least one distortion item is at least one of noise distortion, compression artifact distortion, blur distortion, and exposure distortion. The feature extraction module is used to obtain feature data of the image data through a feature extraction network; The fusion module is used to fuse the feature data with the perceived data to obtain fused feature data; The regression module is used to perform regression processing on the fused feature data through a regression network to obtain the sharpness score of the first image; The at least one distortion item includes a noise distortion item, and the perceived data of the first image for the noise distortion item is the first perceived data. The perception module is used to, for a pixel in the first image, obtain a first grayscale difference between the grayscale value of the pixel and the average grayscale value of other pixels adjacent to the pixel; determine the noise value corresponding to the pixel based on the first grayscale difference; wherein the noise value indicates whether the pixel is a noise point; and obtain the first perceived data based on the noise value corresponding to each pixel in the first image.

10. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the image sharpness evaluation method as described in any one of claims 1-8.

Images