Image processing method, device, equipment, and storage medium

Abstract

The present application discloses an image processing method and apparatus, a device, and a storage medium, wherein, based on clarity indicators of at least two dimensions, at least two clarity features of an image to be evaluated are extracted, different clarity features correspond to different clarity indicators, and different clarity indicators represent image information of different frequencies; at least two clarity features of the image to be evaluated are input into a clarity evaluation model to obtain evaluation information output by the clarity evaluation model.

Description

Technical Field

[0001] The present application relates to image processing technology, and in particular to an image processing method and apparatus, device, and storage medium. Background Art

[0002] In the related art, the solution for evaluating the clarity of an image includes: using a clarity evaluation tool to calculate a specified picture card (such as a dead leaves picture) to obtain indicators related to image clarity: texture and sharpness. However, the clarity evaluation tool calculates clarity-related indicators through a specified picture card, and this indicator cannot reflect whether the user likes it, and the calculation of the indicator requires a specified picture card. Summary of the invention

[0003] The embodiments of the present application provide an image processing method, apparatus, device, and storage medium, which do not require special image cards and professionally trained personnel, and can reduce the implementation complexity of clarity evaluation.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising:

[0006] Based on the clarity index of at least two dimensions, at least two clarity features of the image to be evaluated are extracted, different clarity features correspond to different clarity indexes, and different clarity indexes represent image information of different frequencies;

[0007] At least two clarity features of the image to be evaluated are input into a clarity evaluation model to obtain evaluation information output by the clarity evaluation model.

[0008] In a second aspect, an embodiment of the present application provides an image processing device, including:

[0009] A feature extraction module, used to extract at least two clarity features of the image to be evaluated based on clarity indicators of at least two dimensions, where different clarity features correspond to different clarity indicators, and different clarity indicators represent image information of different frequencies;

[0010] The evaluation module is used to input at least two clarity features of the image to be evaluated into a clarity evaluation model to obtain evaluation information output by the clarity evaluation model.

[0011] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps in the image processing method implemented by the above electronic device are implemented.

[0012] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps in the image processing method implemented by the above-mentioned terminal device are implemented.

[0013] The image processing method, apparatus, device and storage medium provided in the embodiments of the present application extract at least two clarity features of the image to be evaluated based on clarity indicators of at least two dimensions, and different clarity indicators represent image information of different frequencies; the at least two clarity features of the image to be evaluated are input into a clarity evaluation model to obtain evaluation information output by the clarity evaluation model, thereby using the clarity evaluation model to evaluate the clarity of the image to be evaluated from the clarity features representing the image information of different frequencies, and obtaining evaluation information of the clarity of the image to be evaluated. No special charts and professionally trained personnel are required, and the implementation complexity of the clarity evaluation can be reduced. At the same time, the clarity is evaluated based on image information of different frequencies. The image information of different frequencies is information of different granularities of the image, and plays different roles in the image structure. Therefore, the clarity of the image to be evaluated is evaluated by information of different roles, thereby improving the accuracy of the evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is an optional structural diagram of an image processing system provided by an embodiment of the present application;

[0015] Figure 2 It is an optional flowchart of the image processing method provided in the embodiment of the present application;

[0016] Figure 3 It is an optional flowchart of the image processing method provided in the embodiment of the present application;

[0017] Figure 4 It is an optional flow chart of feature extraction provided in an embodiment of the present application;

[0018] Figure 5 is an optional training schematic diagram of the initial clarity assessment model provided in the embodiment of the present application;

[0019] Figure 6 This is an optional effect diagram of image rendering provided by an embodiment of the present application;

[0020] Figure 7 This is an optional effect diagram of the pairwise comparison provided in the embodiment of the present application;

[0021] Figure 8 This is an optional effect diagram of the clarity feature extraction provided in the embodiment of the present application;

[0022] Fig. 9 is an optional effect diagram of a rendered image provided in an embodiment of the present application;

[0023] Fig.10 It is an optional evaluation effect schematic diagram provided in an embodiment of the present application;

[0024] Fig.11 is an optional structural diagram of an image processing device provided in an embodiment of the present application;

[0025] Fig.12 It is an optional structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0027] The embodiments of the present application can be provided as an image processing method, device and storage medium. In practical applications, the image processing method can be implemented by an image processing device, and each functional entity in the image processing device can be collaboratively implemented by hardware resources of electronic devices (such as wristbands, tablets, mobile phones, televisions and other electronic devices), such as computing resources such as processors, and communication resources (such as for supporting various communication methods such as optical cables and cellular).

[0028] Of course, the embodiments of the present application are not limited to being provided as methods and hardware, and there may be multiple implementation methods, such as being provided as a storage medium (storing instructions for executing the image processing method provided by the embodiments of the present application).

[0029] The image processing method provided in the embodiment of the present application can be applied to Figure 1 The image processing system shown in Figure 1 As shown, the image processing system may include a client 101 and a server 102, wherein the client 101 and the server 102 may be located on the same physical entity or on different physical entities. Figure 1 As shown, the client 101 and the server 102 can interact through the network 103.

[0030] The client 101 receives the image to be evaluated and sends the received image to be evaluated to the server 102. The image to be evaluated may include: image content such as people, animals, cars, and scenery. The embodiment of the present application does not impose any limitation on the image content of the image to be evaluated. Here, the client 101 may collect the first image through the image collector, or may receive the image to be evaluated sent by other electronic devices through the network.

[0031] After receiving the image to be evaluated sent by the client, the server 102 extracts at least two clarity features of the image to be evaluated based on clarity indicators of at least two dimensions, where different clarity indicators represent image information of different frequencies of the image to be evaluated; and inputs the at least two clarity features of the image to be evaluated into a clarity evaluation model to obtain evaluation information output by the clarity evaluation model.

[0032] After the server determines the evaluation information of the image to be evaluated, it can send the evaluation information of the image to be evaluated to the client, so that the client can output it to other electronic devices or display screens. When the client outputs the evaluation information of the image to be evaluated to the display, it can simultaneously output the image to be evaluated, or it can display the evaluation information corresponding to the image identifier of the image to be evaluated. Among them, the image identifier is an identifier for distinguishing different images, such as: image name, image ID, etc.

[0033] Next, combine Figure 1 The image processing system shown illustrates various embodiments of the image processing method, device, and storage medium provided in the embodiments of the present application.

[0034] The present application provides an image processing method, which is applied to electronic equipment. Figure 2 FIG. 1 is a schematic diagram of an implementation flow of an image processing method according to an embodiment of the present application. Figure 2 As shown, the method may include the following steps:

[0035] S201. The electronic device extracts at least two clarity features of the image to be evaluated based on clarity indicators in at least two dimensions, where different clarity indicators represent image information of different frequencies.

[0036] Different clarity indices in the at least two clarity indices represent indices of different frequencies. The at least two clarity indices may include a high-frequency index, a medium-frequency index, and a low-frequency index, and the high-frequency index represents high-frequency image information, the medium-frequency index represents medium-frequency image information, and the low-frequency index represents low-frequency image information.

[0037] Image frequency refers to the indicator of the intensity of grayscale changes in an image. The main component of an image is low-frequency image information, which forms the basic grayscale level of the image and has a small role in determining the image structure; medium-frequency image information determines the basic structure of the image and forms the main edge structure of the image; high-frequency image information forms image details and is an enhancement of the image content based on medium-frequency image information.

[0038] At least two clarity indicators may include: a first clarity indicator corresponding to high frequency, a second clarity indicator corresponding to medium frequency, and a third clarity indicator corresponding to low frequency, wherein the first clarity indicator may include: noise (nosie), resolution and other indicators representing high-frequency image information, the second clarity indicator may include: texture contrast, local contrast and other indicators representing medium-frequency image information, and the third clarity indicator may include: edge (edge), structure (structure) and other indicators representing low-frequency image information.

[0039] For different clarity indices, the electronic device extracts image information representing the clarity indices in the image to be evaluated, and obtains corresponding clarity features based on the image information. Here, the electronic device may perform statistics on the extracted image information and determine the statistical information as the clarity feature, or may compare the statistical information with a normal distribution to obtain a kurtosis value representing the distribution of the statistical information, and use the obtained kurtosis value as the clarity feature.

[0040] In one example, the clarity index includes: noise, texture contrast, and edge, and the following clarity features of the image to be evaluated are obtained: noise statistical information corresponding to noise, texture contrast statistical information corresponding to texture contrast, and edge statistical information corresponding to edge.

[0041] S202: The electronic device inputs at least two clarity features of the image to be evaluated into a clarity evaluation model to obtain evaluation information output by the clarity evaluation model.

[0042] When the electronic device extracts at least two clarity features, the at least two clarity features are input into the clarity evaluation model, so that the clarity evaluation model obtains evaluation information based on the at least two input clarity features, and the evaluation information is used to evaluate the clarity of the evaluation image to characterize the user's preference or interest in the clarity of the image to be evaluated. The evaluation information can be a score or a level information.

[0043] In one example, the clarity index includes: noise, texture contrast and edge, and the clarity evaluation model can be expressed as formula (1):

[0044] IQ 清晰度 =f(Vaule Noise ,Vaule Texture ,VauleEdge ) formula (1);

[0045] Among them, IQ 清晰度 Indicates evaluation information, Vaule Noise Represents the clarity characteristics corresponding to the noise, Vaule Texture Represents the clarity feature corresponding to the texture contrast, Vaule Edge It indicates the clarity feature corresponding to the edge. From formula (1), we can know that the clarity evaluation model can be understood as the input parameter Vaule Noise , Vaule Texture , Vaule Edge The function is a function of , and the output is evaluation information.

[0046] like Figure 3 As shown, three clarity features 301 - 1 to 301 - 3 are input to a clarity evaluation model 302 , and the clarity evaluation model 302 outputs evaluation information 303 .

[0047] In an embodiment of the present application, the user's preference score for each sample image in the sample image set may be used as a label to train the initial clarity assessment model to obtain a clarity assessment model.

[0048] The image processing method provided in the embodiment of the present application extracts at least two clarity features of the image to be evaluated based on clarity indicators of at least two dimensions, and different clarity indicators characterize image information of different frequencies of the image to be evaluated; the at least two clarity features of the image to be evaluated are input into a clarity evaluation model to obtain evaluation information output by the clarity evaluation model, thereby using the clarity evaluation model to evaluate the clarity of the image to be evaluated from the clarity features that characterize image information of different frequencies, and obtaining evaluation information of the clarity of the image to be evaluated. No special charts or professionally trained personnel are required, and the implementation complexity of the clarity evaluation can be reduced. At the same time, the clarity is evaluated based on image information of different frequencies. The image information of different frequencies is information of different granularities of the image, and plays different roles in the image structure. Therefore, the clarity of the image to be evaluated is evaluated through information of different roles, thereby improving the accuracy of the evaluation.

[0049] In some embodiments, S201 extracts at least two clarity features of the image to be evaluated based on clarity indicators of at least two dimensions, including: processing the image to be evaluated based on clarity indicators of at least two dimensions to obtain at least two feature maps corresponding to the image to be evaluated, different feature maps corresponding to different clarity indicators; for each feature map of the at least two feature maps, comparing the image information included in the feature map with a normal distribution to obtain the kurtosis value of the clarity indicator corresponding to the feature map.

[0050] For each of the at least two clarity indicators, the electronic device extracts image information of the clarity indicator in the image to be evaluated to obtain a feature map corresponding to the clarity indicator. For a feature map corresponding to a clarity indicator, the feature map does not include image information of other clarity indicators.

[0051] In one example, at least two clarity indicators include: noise, texture contrast, and edge. Figure 4 As shown, the electronic device extracts the image information corresponding to the noise, texture contrast, and edge in the image to be evaluated 401, respectively, to obtain a noise feature map 402, a texture contrast feature map 403, and an edge feature map 404. The noise feature map includes the noise information in the image to be evaluated, the texture contrast feature map includes the texture contrast information in the image to be evaluated, and the edge feature map includes the edge information in the image to be evaluated.

[0052] After the electronic device determines a feature map corresponding to a clarity index, it performs statistics on the image information included in the feature map to obtain statistical information, and compares the obtained statistical information with a normal distribution to obtain a kurtosis value corresponding to the clarity index, namely a kurtosis value.

[0053] Kurtosis is a statistic that describes the steepness of the distribution of all values ​​in the population data. The plugging needs to be compared with the normal distribution. A kurtosis value of 0 means that the distribution of the population data is as steep as the normal distribution; a kurtosis value greater than 0 means that the distribution of the population data is steeper than the normal distribution, with a sharp peak; a kurtosis value less than 0 means that the distribution of the population data is flatter than the normal distribution, with a flat peak.

[0054] In practical applications, before the electronic device determines the statistical information of a feature map, it may first filter out the background information of the feature map and only retain the image information of the corresponding clarity index included in the feature map.

[0055] In an embodiment of the present application, the input of the clarity assessment model is the kurtosis value of at least two clarity indicators, so that the clarity of the image to be evaluated is evaluated by the kurtosis values ​​corresponding to at least two clarity indicators, thereby reducing the amount of data input to the clarity assessment model and further reducing the structural complexity of the clarity assessment model.

[0056] In some embodiments, the electronic device further implements the following steps: based on clarity indicators in at least two dimensions, obtaining at least two clarity features of each first image in a first image set, where different first images have different clarity; based on at least two clarity features of each first image in the first image set and labels of each first image in the first image set, training an initial clarity model to obtain the clarity model, where the label is evaluation information annotated by the corresponding first image.

[0057] In the embodiment of the present application, the roughness or clarity model is trained by the first image set and the label set corresponding to the first image set to obtain the clarity model. Wherein, the clarity of different first images in the first image set is different, and it is understandable that at least one clarity index of different first images is different.

[0058] In one example, the first image set includes: images 1 to 10, where images 1 to 5 are 5 images with different texture contrasts, and images 6 to 10 are 5 images with different noises.

[0059] In one example, the first image set includes: images 1 to 14, where images 1 to 5 are 5 images with different texture contrasts, images 6 to 10 are 5 images with different noises, and images 11 to 15 are 5 images with different edges.

[0060] In the embodiment of the present application, the images in the first image set may be images with the same image content but different sharpness.

[0061] In an embodiment of the present application, for each first image in the first image set, the electronic device may extract at least two clarity features of the first image based on at least two clarity indicators, and input the at least two clarity features of the first image into an initial clarity assessment model.

[0062] In an embodiment of the present application, the electronic device extracts at least two clarity features of each first image in the first image set to obtain a clarity feature set, and inputs the clarity feature set into an initial clarity assessment model to obtain assessment information of each first image output by the initial clarity assessment model, and the assessment information of each first image constitutes a predicted assessment information set. In the case where the initial clarity assessment model has not converged, the electronic device determines the current loss based on a label set composed of labels of each first image in the first image set and the predicted assessment information set, and adjusts the parameters of the initial clarity assessment model based on the determined loss to obtain a new initial clarity assessment model, and continues to input the clarity feature set into the new initial clarity assessment model to obtain assessment information of each first image output by the new initial clarity assessment model, and the assessment information of each first image constitutes a new predicted assessment information set. When the initial clarity assessment model has not converged, the electronic device determines the current loss based on a label set consisting of labels of each first image in the first image set and a predicted evaluation information set, and adjusts the parameters of the initial clarity assessment model based on the determined loss to obtain a new initial clarity assessment model until the initial clarity assessment model converges, thereby completing the training of the clarity assessment model through continuous iterations.

[0063] In the embodiment of the present application, for one iteration in the training process, the following can be done: Figure 5 As shown, the clarity feature set 501 corresponding to the first image set 500 is input into the initial clarity evaluation model 502, the initial clarity evaluation model 502 outputs a predicted evaluation information set 503, a loss 505 is obtained based on the label set 504 corresponding to the first image set 501 and the predicted evaluation information set 503, and the parameters of the initial object detection model 502 are adjusted based on the loss 505.

[0064] In some embodiments, the electronic device further implements the following steps: for each clarity indicator among at least two clarity indicators, rendering a reference image based on the clarity indicator to obtain a first image subset corresponding to the clarity indicator; and obtaining the first image set based at least on the first image subset corresponding to each clarity indicator among the at least two clarity indicators.

[0065] In an embodiment of the present application, a reference image can be obtained as an anchor image, and a first image set can be obtained by rendering different clarity indicators of the reference image, wherein for each clarity indicator, the reference image can be rendered based on different rendering amounts, thereby obtaining multiple first images with different corresponding clarity characteristics in the dimension of one clarity indicator.

[0066] In one example, if Figure 6As shown, the reference image is image 600, and at least two clarity indexes include: clarity index 1, clarity index 2 and clarity index 3. When image 600 is rendered based on clarity index 1, images 601-1 to 601-3 are obtained, and the rendering amounts of clarity index 1 of images 601-1 to 601-3 are different; when image 600 is rendered based on clarity index 2, images 602-1 to 602-3 are obtained, and the rendering amounts of clarity index 2 of images 602-1 to 602-3 are different; when image 600 is rendered based on clarity index 3, images 603-1 to 603-3 are obtained, and the rendering amounts of clarity index 3 of images 603-1 to 603-3 are different.

[0067] Here, the values ​​of the rendering amount corresponding to each definition index may be different. Here, the rendering amount corresponding to a definition index may be understood as the value that the definition feature corresponding to the definition index needs to be rendered.

[0068] In an embodiment of the present application, anchor images are rendered from different dimensions based on different clarity indicators, so that the generation of the first image set for clarity assessment model training is completed through the control variable method, thereby effectively controlling the training samples and improving the training efficiency.

[0069] In some embodiments, rendering the reference image based on the clarity index to obtain a first image subset corresponding to the clarity index includes: obtaining a rendering range corresponding to the clarity index; determining a first number of rendering amounts based on the rendering range, the first number of rendering amounts including a second number of rendering amounts lower than a reference clarity feature and a third number of rendering amounts higher than the reference clarity feature, the reference clarity feature being a clarity feature of the reference image corresponding to the clarity index; rendering the reference image respectively based on the first number of rendering amounts to obtain a first image subset corresponding to the clarity index.

[0070] Here, in the process of generating the first image set, for each clarity index, when rendering the reference image, the rendering amount is determined based on the rendering range corresponding to the clarity index and the clarity feature of the reference image corresponding to the clarity index, wherein the rendering range represents the value range of the clarity feature of the image.

[0071] For each definition index, the number of rendered first images is a first number, and the rendering amount is determined based on the first number, the reference definition feature, and the rendering range.

[0072] In the embodiment of the present application, different rendering amount intervals are the same or different. Here, the rendering amount interval is the difference between two adjacent rendering amounts.

[0073] In one example, for the clarity index 1, its rendering range is the clarity feature 1 with values ​​of 1 to 5, the first quantity is 5, the value of the clarity feature 1 corresponding to the clarity index 1 of the reference image is 3, then the 5 rendering amounts corresponding to the clarity index 1 are: clarity feature 1 with values ​​of 4.5, 4, 3.5, 2.5, and 2, respectively, where the first quantity is 2 and the second quantity is 3.

[0074] In one example, for the clarity index 1, its rendering range is the clarity feature 1 with values ​​of 2 to 10, the first quantity is 5, the value of the clarity feature 1 corresponding to the clarity index 1 of the reference image is 5, then the 5 rendering quantities corresponding to the clarity index 1 are: clarity feature 1 with values ​​of 7, 6, 4, 3, and 2 respectively, among which the first quantity is 3 and the second quantity is 4.

[0075] In some embodiments, the electronic device further implements the following steps:

[0076] For each definition index of the at least two definition indicators, the following processing is performed:

[0077] Based on the clarity index, determine the clarity features of each second image in the second image set; determine a first clarity feature with the lowest value and a second clarity feature with the highest value among the clarity features of each second image in the second image set; based on the first clarity feature and the second clarity feature, determine a rendering range corresponding to the clarity index.

[0078] In the embodiment of the present application, the second image set is a set of images collected by the electronic device, wherein the second images in the second image set may include images collected by different mobile phones.

[0079] After acquiring the second image set, the electronic device determines, for any clarity index, a clarity feature corresponding to the clarity index of each second image in the second image set based on the clarity index, and determines a maximum value and a minimum value of the clarity feature.

[0080] Optionally, after determining the lowest value, i.e., the value of the first clarity feature, and the highest value, i.e., the value of the second clarity feature, of a clarity feature corresponding to a clarity index based on the second image, the electronic device directly determines the determined first clarity feature and second clarity feature as the highest clarity feature and the lowest clarity feature of the rendering range of the clarity index.

[0081] In some embodiments, determining the rendering range corresponding to the clarity index based on the first clarity feature and the second clarity feature includes: obtaining a first coefficient and a second coefficient corresponding to the clarity index; obtaining a minimum clarity feature of the rendering range based on the first coefficient and the first clarity feature; and obtaining a maximum clarity feature of the rendering range based on the second coefficient and the second clarity feature.

[0082] At this time, after determining the highest and lowest values ​​of the clarity feature corresponding to a clarity index based on the second image, the electronic device determines the rendering range of the clarity index based on the first coefficient, the second coefficient, and the determined first and second clarity features. Among them, the first coefficient is used to determine the lowest clarity feature of the rendering range with the first clarity feature, and the first coefficient is a positive number less than 1, and the second coefficient is used to determine the highest clarity feature of the rendering range with the second clarity feature, and the second coefficient is a positive number greater than 1. Here, the second coefficient may be less than 2. The size of the first coefficient and the second coefficient can be set according to actual needs. In one example, the first coefficient is 0.8 and the second coefficient is 1.2. In one example, the first coefficient is 0.6 and the second coefficient is 1.5.

[0083] In some embodiments, in the image processing method provided by the embodiments of the present application, the electronic device further implements the following steps: obtaining comparison information of pairwise comparison of the first images included in the first image set collected; and determining, for each first image in the first image set, evaluation information for labeling the first image based on the comparison information related to the first image.

[0084] In the embodiment of the present application, the label of each first image in the first image set can be obtained by collecting feedback information of users on the first image.

[0085] Here, the electronic device can display the first images in the first image set to the user in pairs, so that the user selects an image he likes from the two displayed first images. The electronic device receives the image selected by the user from the two displayed images. Here, the image selected by the user is the image that the user likes more from the two displayed images.

[0086] In one example, if Figure 7 As shown, the first image set includes image 1, image 2 and image 3, then image 1 and image 2 are compared, image 2 and image 3 are compared, and image 1 and image 3 are compared.

[0087] The electronic device may obtain comparison information of first images in the first image set that are compared with each other, wherein the comparison information includes: two compared images, the number of times the two images are compared, and the number of times an image is selected.

[0088] In one example, the first image set includes image 1, image 2 and image 3, then the comparison information of image 1, image 2 and image 3 compared pairwise includes: the number of comparisons when comparing image 1 and image 2, the number of times image 2 is selected, the number of times image 2 is selected, the number of comparisons when comparing image 2 and image 3, the number of times image 2 is selected, the number of times image 3 is selected, the number of comparisons when comparing image 1 and image 3, the number of times image 1 is selected, the number of times image 3 is selected.

[0089] After determining the comparison information for pairwise comparison, the electronic device determines, for a first image, the comparison information related to the first image in the pairwise comparison information, where the comparison information related to the first image includes the comparison information of the compared image including the first image, and determines the evaluation information, i.e., the label, of the first image based on the comparison information related to the first image.

[0090] The first image includes image 1, image 2 and image 3, and the first image set includes image 1, image 2 and image 3. For image 1, the label of image 1 is determined based on comparison information of image 1 compared with image 2 and comparison information of image 1 compared with image 3; for image 2, the label of image 2 is determined based on comparison information of image 1 compared with image 2 and comparison information of image 2 compared with image 3; for image 3, the label of image 3 is determined based on comparison information of image 2 compared with image 3 and comparison information of image 1 compared with image 3.

[0091] For a first image, a score of the first image compared with any other first image is determined based on the comparison information of the first image with the other first image, and the scores of the first image compared with all other first images are added to obtain the evaluation information of the first image, wherein the evaluation information is regarded as a score or a grade.

[0092] In one example, image 1 is compared with image 2 and image 3 respectively. The corresponding score when image 1 is compared with image 2 is score 1, and the corresponding score when image 1 is compared with image 3 is score 2. Then the corresponding score of image 1 is the sum of score 1 and score 2.

[0093] Here, the score of the first image compared with another first image is determined based on the probability of the first image being selected more when compared with the other first image. In one example, when comparing image A with image B, the number of comparisons is 100, and image A is selected 60 times, and image B is selected 40 times, then image A is liked 20 times more than image B, and the probability of image A being liked more is 20% of the ratio of 20 to 100, and at this time, the score of image A relative to image B is 20.

[0094] In the image processing method provided in the embodiment of the present application, labels are determined based on the user's selection results of the first images in the first image set when comparing them pairwise, so that the labels can represent the user's degree of liking for each of the first images, thereby allowing the evaluation information output by the clarity evaluation model trained based on the labels of the first images to represent the user's degree of liking for the image to be evaluated of the current clarity, thereby realizing the prediction of the user's preference for the image through the network model.

[0095] Below, the image processing method provided in the embodiment of the present application is further explained by taking the clarity indicators including noise, texture contrast, and edge as an example.

[0096] In the related art, the solutions for image clarity evaluation include the following solutions:

[0097] The clarity evaluation tool is used to calculate the specified picture card (such as the dead leaves picture) to obtain indicators related to image clarity: texture and sharpness.

[0098] For the above solution, the clarity evaluation tool calculates clarity-related indicators by specifying a picture card. This indicator cannot reflect whether the user likes it, and the calculation of the indicator requires a specified picture card.

[0099] The image processing method provided in the embodiment of the present application includes the following contents:

[0100] 1. Simulate images of different clarity levels based on the control variable method;

[0101] 2. Use images of different clarity levels to conduct user research;

[0102] 3. Using the survey images and survey results, we build a regression model, i.e., a clarity assessment model, through machine learning methods, and finally use the model to predict the user preference score of the image.

[0103] For the controlled variable method, the clarity indicators of three different frequency domains, i.e., different dimensions, of the same natural image, i.e., the reference image, are enhanced or weakened to varying degrees. The clarity indicators of these three dimensions include: noise belonging to the high-frequency area, texture contrast belonging to the medium-frequency area, and edge belonging to the low-frequency area.

[0104] At this time, the natural image is enhanced or weakened to different degrees from three different frequency domains, i.e., three different clarity indicators, including:

[0105] 1. Render the noise of natural images, thereby enhancing or weakening the high-frequency areas to varying degrees;

[0106] 2. Render the texture contrast of natural images, thereby enhancing or weakening the mid-frequency area to varying degrees;

[0107] 3. Render the edges of natural images to enhance or weaken the low-frequency areas to varying degrees.

[0108] In the embodiment of the present application, when a natural image is rendered based on image features in different frequency domains, the rendering range is determined based on the following method: the kurtosis of image features in different frequency domains of each image in the reference image set is calculated.

[0109] Kurtosis is a statistic that describes the steepness of the distribution of all values ​​in the population data. The plugging needs to be compared with the normal distribution. A kurtosis value of 0 means that the distribution of the population data is as steep as the normal distribution; a kurtosis value greater than 0 means that the distribution of the population data is steeper than the normal distribution, with a sharp peak; a kurtosis value less than 0 means that the distribution of the population data is flatter than the normal distribution, with a flat peak.

[0110] like Figure 8 As shown, the edge, texture contrast and noise of image 801 are extracted respectively to obtain edge feature map 802, texture feature map 803 and noise feature map 804, and background filtering is performed on edge feature map 802, texture feature map 803 and noise feature map 804 respectively to obtain edge statistical information 805, texture contrast statistical information 806 and noise statistical information 807 of image 801. The kurtosis of edge statistical information 805, texture contrast statistical information 806 and noise statistical information 807 is calculated respectively to obtain edge kurtosis 808, texture contrast kurtosis 809 and noise kurtosis 810.

[0111] The reference image set used to determine the rendering range, that is, the second image set, includes images captured by electronic devices of multiple brands.

[0112] In one example, the reference image set includes images collected by mobile phones of six brands, namely, brand 1 to brand 6. The kurtosis value of texture contrast, the kurtosis value of noise, and the kurtosis value of edge of each brand are shown in Table 1.

[0113] Table 1. Examples of kurtosis values ​​of clarity indicators of different dimensions of images in the reference image set

[0114] Phone brand The kurtosis value of the texture Noise kurtosis value The kurtosis value of the edge Brand 1 <![CDATA[Texture h ]]> <![CDATA[Noise h ]]> <![CDATA[Edge h ]]> Brand 2 <![CDATA[Texture m ]]> <![CDATA[Noise m ]]> <![CDATA[Edge m > Brand 3 <![CDATA[Texture o ]]> <![CDATA[Noise o ]]> <![CDATA[Edge o ]]> Brand 4 <![CDATA[Texture v ]]> <![CDATA[Noise v ]]> <![CDATA[Edge v ]]> Brand 5 <![CDATA[Texture a ]]> <![CDATA[Noise a ]]> <![CDATA[Edge a ]]> Brand 6 <![CDATA[Texture s ]]> <![CDATA[Noise S ]]> <![CDATA[Edge s ]]>

[0115] From Table 1, the following information can be determined:

[0116] The maximum kurtosis value of the edge max and the minimum kurtosis value Edge min ,

[0117] Maximum kurtosis value of texture contrast Texturemax and minimum kurtosis value Texture min ,

[0118] Maximum kurtosis value of noise max and minimum kurtosis value Noise min .

[0119] The rendering range of the clarity index of different dimensions can be determined based on the above information and the buffer value as follows:

[0120] Edge rendering range: max *1.2,Edge min *0.8);

[0121] Texture contrast rendering range: max *1.2, Texture min *0.8);

[0122] Noise rendering range: max *1.2,Noise min *0.8).

[0123] After determining the rendering range, an image with an intermediate value in each dimension is selected as the anchor image, i.e., the reference image. According to the rendering range of the three dimensions of edge, texture contrast, and noise, the first number of images are rendered in each dimension to obtain simulation results of different dimensions.

[0124] In one example, if Fig. 9 As shown, image 901 is rendered with texture contrast according to the rendering range of texture contrast, and images 902, 903 and 904 are obtained, wherein the kurtosis value of the texture contrast of image 902 increases, the kurtosis value of the texture contrast of image 903 decreases, and the kurtosis value of the texture contrast of image 904 is Texture min .

[0125] After rendering the anchor image based on the clarity index of different dimensions, rendered images of different clarity levels are obtained. At this time, the rendered images of different clarity levels, i.e., the first images, are used for user research. The rendered images can be compared in pairs, and users can be asked to select the image they like more online, and the user preference score of each rendered image can be determined based on the user's selection result.

[0126] In an example, the rendered images include image 1, image 2, and image 3, and the comparison times of image 1, image 2, and image 3 are shown in Table 2.

[0127] Table 2. Example of pairwise comparisons of images 1, 2, and 3

[0128] Image 1 Image 2 Image 3 Image 1 0 116 96 Image 2 116 0 114 Image 3 96 114 0

[0129] Based on Table 1, we can see that image 1 is compared with image 2 116 times, image 1 is compared with image 3 96 times, and image 2 is compared with image 3 96 times.

[0130] The comparison results of Image 1, Image 2 and Image 3 are shown in Table 3.

[0131] Table 3. Example of comparison results of Image 1, Image 2 and Image 3

[0132] Image 1 Image 2 Image 3 Image 1 0 -8.62069 -14.58333 Image 2 8.62069 0 -9.69123 Image 3 14.58333 9.69123 0 Like score 23.2 1.02 -24.22

[0133] As shown in Table 3, the comparison result between Image 1 and Image 2 is that 8.62% of the 100 people prefer Image 1. The comparison result between Image 1 and Image 3 is that 14.5% of the 100 people prefer Image 1. The comparison result between Image 2 and Image 3 is that 9.65% of the 100 people prefer Image 2. The statistical data of the pairwise comparison results of user preferences are used as the preference score of each image. In Table 3, the preference score of Image 1 is 23.2, the preference score of Image 2 is 1.02, and the preference score of Image 3 is -24.22.

[0134] After determining the preference score of each rendered image rendered based on the anchor image, the preference score is used as a label of each rendered image to train a clarity assessment model.

[0135] In an embodiment of the present application, for rendered image to rendered image, the edge kurtosis value, texture contrast kurtosis value, and noise kurtosis value of each rendered image are determined, and regression training is performed based on the edge kurtosis value, texture contrast kurtosis value, noise kurtosis value of each rendered image, and a preference score set consisting of the preference scores of each rendered image from rendered image to rendered image to obtain a clarity evaluation model.

[0136] Among them, the clarity evaluation model can be expressed as a function based on the edge kurtosis value, the texture contrast kurtosis value, and the noise kurtosis value. The value output by the clarity evaluation model is the preference score of the evaluation result that characterizes the image clarity.

[0137] The image processing method provided in the embodiment of the present application uses the User Image Assessment (UIA) method to take the real user scene preference score as the data benchmark (Ground Truth, GT) and establish a new subjective user scene data set; through the UIA method, the objective measurement index of the image can be used to quantify the user preference score, and the image capability is based on the user's clear preference. During the implementation of the solution, a clear preference score is generated through a set of pairwise comparisons.

[0138] The image processing method provided in the embodiment of the present application uses the real user's preference statistics as the subjective image preference score, and completely takes the user's preference and feelings as the most important yardstick in the image clarity evaluation, and the obtained image clarity preference score will be more realistic. Among them, because the user's acceptance of clarity has a range, as long as the image clarity reaches Fig.10 The user's approval is achieved within the shown range 1001, wherein the range 1001 is determined based on noise, texture contrast, and edges.

[0139] In addition, the image processing method provided in the embodiment of the present application does not require special charts and professionally trained personnel for evaluation. Once a scene image enters the system, the score of the scene image can be automatically generated, thereby improving evaluation efficiency.

[0140] Fig.11 FIG. 1 is a schematic diagram of an implementation flow of an image processing in an embodiment of the present application, which is applied to electronic devices such as Fig.11 As shown, the image processing device 1100 includes:

[0141] A feature extraction module 1101 is used to extract at least two clarity features of the image to be evaluated based on clarity indicators of at least two dimensions, where different clarity features correspond to different clarity indicators, and different clarity indicators represent image information of different frequencies;

[0142] The evaluation module 1102 is used to input at least two clarity features of the image to be evaluated into a clarity evaluation model to obtain evaluation information output by the clarity evaluation model.

[0143] In some embodiments, the feature extraction module 1101 is further used to:

[0144] Processing the image to be evaluated based on clarity indicators of at least two dimensions to obtain at least two feature maps corresponding to the image to be evaluated, where different feature maps correspond to different clarity indicators;

[0145] For each feature map of the at least two feature maps, the image information included in the feature map is compared with a normal distribution to obtain a kurtosis value of a clarity index corresponding to the feature map.

[0146] In some embodiments, the apparatus 1100 further includes: a training module,

[0147] The feature extraction module 1100 is further used to obtain at least two clarity features of each first image in the first image set based on clarity indicators of at least two dimensions, where different first images have different clarity;

[0148] A training module is used to train an initial clarity model based on at least two clarity features of each first image in the first image set and a label of each first image in the first image set to obtain the clarity model, wherein the label is evaluation information annotated by the corresponding first image.

[0149] In some embodiments, the apparatus 1100 further includes a rendering module, configured to:

[0150] For each clarity index of at least two clarity indexes, rendering a reference image based on the clarity index to obtain a first image subset corresponding to the clarity index;

[0151] The first image set is obtained based at least on a first image subset corresponding to each clarity index of the at least two clarity indexes.

[0152] In some embodiments, the rendering module is further configured to:

[0153] Obtaining a rendering range corresponding to the clarity index;

[0154] Based on the rendering range, determining a first number of rendering amounts, the first number of rendering amounts including a second number of rendering amounts lower than a reference clarity feature and a third number of rendering amounts higher than the reference clarity feature, the reference clarity feature being a clarity feature of the reference image corresponding to the clarity index;

[0155] Based on the first number of rendering amounts, the reference images are rendered respectively to obtain a first image subset corresponding to the clarity index.

[0156] In some embodiments, the rendering module is further configured to:

[0157] Based on the clarity index, determining a clarity feature of each second image in the second image set;

[0158] Determine a first clarity feature with the lowest value and a second clarity feature with the highest value among the clarity features of each second image in the second image set;

[0159] Based on the first clarity feature and the second clarity feature, a rendering range corresponding to the clarity index is determined.

[0160] In some embodiments, the rendering module is further configured to:

[0161] Obtaining a first coefficient and a second coefficient corresponding to the clarity index;

[0162] Obtaining a minimum definition feature of the rendering range based on the first coefficient and the first definition feature;

[0163] A highest definition feature of the rendering range is obtained based on the second coefficient and the second definition feature.

[0164] In some embodiments, the apparatus 1100 further includes: a tag determination module, configured to:

[0165] Acquire comparison information of comparing two first images included in the first image set;

[0166] For each first image in the first image set, based on comparison information related to the first image, evaluation information for labeling the first image is determined.

[0167] It should be noted that the functions implemented by the modules included in the image processing device provided in the embodiment of the present application can be implemented by a processor in an electronic device; of course, they can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0168] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0169] It should be noted that in the embodiment of the present application, if the above-mentioned image processing method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0170] Correspondingly, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps in the image processing method provided in the above embodiment are implemented. The electronic device may be a terminal device.

[0171] Correspondingly, an embodiment of the present application provides a storage medium, that is, a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the image processing method provided in the above embodiment are implemented.

[0172] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0173] It should be noted that Fig.12 A hardware entity diagram of an electronic device implemented as a terminal device in an embodiment of the present application is shown in FIG. Fig.12 As shown, the electronic device 1200 includes: a processor 1201, at least one communication bus 1202, at least one external communication interface 1204 and a memory 1205. The communication bus 1202 is configured to achieve connection and communication between these components. In an example, the electronic device 1200 also includes: a user interface 1203, wherein the user interface 1203 may include a display screen, and the external communication interface 1204 may include a standard wired interface and a wireless interface.

[0174] The memory 1205 is configured to store instructions and applications executable by the processor 1201, and can also cache data to be processed or processed by the processor 1201 and various modules in the electronic device (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0175] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in some embodiments" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The above-mentioned sequence numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments.

[0176] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0177] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0178] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0179] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0180] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0181] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0182] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, It is characterized in that The method comprises: Based on the clarity index of at least two dimensions, at least two clarity features of the image to be evaluated are extracted, different clarity features correspond to different clarity indexes, and different clarity indexes represent image information of different frequencies; Inputting at least two clarity features of the image to be evaluated into a clarity evaluation model to obtain evaluation information output by the clarity evaluation model; The step of extracting at least two clarity features of the image to be evaluated based on the clarity index of at least two dimensions includes: Processing the image to be evaluated based on clarity indicators of at least two dimensions to obtain at least two feature maps corresponding to the image to be evaluated, where different feature maps correspond to different clarity indicators; For each feature map of the at least two feature maps, the image information included in the feature map is compared with a normal distribution to obtain a kurtosis value of a clarity index corresponding to the feature map as a clarity feature.

2. The method according to claim 1, It is characterized in that The method further comprises: Based on the clarity index of at least two dimensions, obtaining at least two clarity features of each first image in the first image set, where different first images have different clarity; Based on at least two clarity features of each first image in the first image set and a label of each first image in the first image set, an initial clarity model is trained to obtain the clarity model, wherein the label is evaluation information annotated by the corresponding first image.

3. The method according to claim 2, It is characterized in that The method further comprises: For each clarity index of at least two clarity indexes, rendering a reference image based on the clarity index to obtain a first image subset corresponding to the clarity index; The first image set is obtained based at least on a first image subset corresponding to each clarity index of the at least two clarity indexes.

4. The method according to claim 3, It is characterized in that The rendering of the reference image based on the definition index to obtain a first subset of images corresponding to the definition index includes: Obtaining a rendering range corresponding to the clarity index; Based on the rendering range, determining a first number of rendering amounts, the first number of rendering amounts including a second number of rendering amounts lower than a reference clarity feature and a third number of rendering amounts higher than the reference clarity feature, the reference clarity feature being a clarity feature of the reference image corresponding to the clarity index; Based on the first number of rendering amounts, the reference images are rendered respectively to obtain a first image subset corresponding to the clarity index.

5. The method according to claim 4, It is characterized in that The method further comprises: For each definition index of the at least two definition indicators, the following processing is performed: Based on the clarity index, determining a clarity feature of each second image in the second image set; Determine a first clarity feature with the lowest value and a second clarity feature with the highest value among the clarity features of each second image in the second image set; Based on the first clarity feature and the second clarity feature, a rendering range corresponding to the clarity index is determined.

6. The method according to claim 5, It is characterized in that The determining, based on the first clarity feature and the second clarity feature, a rendering range corresponding to the clarity index includes: Obtaining a first coefficient and a second coefficient corresponding to the clarity index; Obtaining a minimum definition feature of the rendering range based on the first coefficient and the first definition feature; A highest definition feature of the rendering range is obtained based on the second coefficient and the second definition feature.

7. The method according to claim 2, It is characterized in that The method further comprises: Acquire comparison information of comparing two first images included in the first image set; For each first image in the first image set, based on comparison information related to the first image, evaluation information for labeling the first image is determined.

8. An image processing device, It is characterized in that include: A feature extraction module, used to extract at least two clarity features of the image to be evaluated based on clarity indicators of at least two dimensions, where different clarity features correspond to different clarity indicators, and different clarity indicators represent image information of different frequencies; An evaluation module, used for inputting at least two clarity features of the image to be evaluated into a clarity evaluation model to obtain evaluation information output by the clarity evaluation model; The feature extraction module is also used to process the image to be evaluated based on clarity indicators in at least two dimensions to obtain at least two feature maps corresponding to the image to be evaluated, and different feature maps correspond to different clarity indicators; for each feature map in the at least two feature maps, the image information included in the feature map is compared with a normal distribution to obtain the kurtosis value of the clarity indicator corresponding to the feature map as a clarity feature.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the steps in the image processing method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing an executable program, It is characterized in that When the executable program is executed by a processor, the steps in the image processing method according to any one of claims 1 to 7 are implemented.

Images