Image processing method and device, electronic equipment and storage medium

Abstract

The invention relates to an image processing method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a first definition index of a first image, wherein the first definition index is used for representing the definition of the first image; based on the first definition index and a target resolution, definition improvement processing is conducted on the first image, a second image corresponding to the first image is obtained, the resolution of the second image is the target resolution, and the target resolution is the resolution of a screen of electronic equipment used for displaying the second image. According to the method, the resolution of the second image is enabled to be the resolution of the screen of the electronic equipment for displaying the second image while the definition of the first image is improved, so that the image which is matched with the resolution of the screen and has relatively high definition is displayed to a user, and the visual effect of image display is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] When users use electronic devices, they will receive videos and images from applications. Since the sources of images and videos are relatively wide, the resolution, clarity, contrast, etc. of different images and videos are also different. When displaying videos and images on electronic devices, in order to match the resolution of the electronic device's screen, the images and videos are enlarged, resulting in reduced clarity. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present disclosure provides an image processing method, device, electronic device and storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, the method comprising:

[0005] Acquire a first clarity index of a first image, where the first clarity index is used to characterize the clarity of the first image;

[0006] Based on the first clarity index and the target resolution, the first image is processed for clarity enhancement to obtain a second image corresponding to the first image, wherein the resolution of the second image is the target resolution, and the target resolution is the resolution of the screen of the electronic device used to display the second image.

[0007] In some embodiments, obtaining a first clarity index of the first image includes:

[0008] A clarity prediction model is called to perform clarity prediction on the first image to obtain the first clarity index, wherein the clarity prediction model is used to predict the clarity of an image.

[0009] In some embodiments, the clarity prediction model includes a feature extraction layer, a feature screening layer, and an activation function layer, and calling the clarity prediction model to perform clarity prediction on the first image to obtain the first clarity index includes:

[0010] Inputting the first image into the feature extraction layer to obtain a first image feature of the first image output by the feature extraction layer;

[0011] Inputting the first image feature into the feature screening layer to obtain the filtered first image feature output by the feature screening layer, wherein the filtered first image feature is related to the clarity of the image;

[0012] The filtered first image feature is input into the activation function layer to obtain the first clarity index output by the activation function layer.

[0013] In some embodiments, the training process of the clarity prediction model includes:

[0014] Acquire a first sample image, a first sample clarity index of the first sample image, and a second sample clarity index, wherein the first sample clarity index is determined based on manual annotation, and the second sample clarity index is determined based on a pixel average value and a pixel preset threshold value corresponding to the first sample image, wherein the pixel average value is a ratio between a sum of pixel values ​​of a plurality of pixel points of the first sample image and the number of the plurality of pixel points;

[0015] Determining a sample clarity index of the first sample image based on the first sample clarity index and the second sample clarity index;

[0016] Calling the clarity prediction model to be trained, performing clarity prediction on the first sample image, and obtaining a predicted clarity index of the first sample image;

[0017] Based on the difference between the predicted clarity index and the sample clarity index, a model parameter of the clarity prediction model to be trained is adjusted.

[0018] In some embodiments, performing definition enhancement processing on the first image based on the first definition index and the target resolution to obtain a second image corresponding to the first image includes:

[0019] A clarity enhancement model is called to perform clarity enhancement processing on the first image based on the first clarity index and the target resolution to obtain the second image, wherein the clarity enhancement model is used to enhance the clarity of the image and to modify the resolution of the image to the target resolution.

[0020] In some embodiments, calling the definition enhancement model to perform definition enhancement processing on the first image based on the first definition index and the target resolution to obtain the second image includes:

[0021] Inputting the first image and the first definition index into the definition enhancement model to obtain the second image output by the definition enhancement model, wherein the hyperparameters of the definition enhancement model include the target resolution; or

[0022] The first image, the first clarity index and the target resolution are input into the clarity improvement model to obtain the second image output by the clarity improvement model.

[0023] In some embodiments, the definition enhancement model includes a residual network and an upsampling layer; and inputting the first image and the first definition index into the definition enhancement model to obtain the second image output by the definition enhancement model includes:

[0024] Inputting the first image and the first clarity index into the residual network to obtain a second image feature output by the residual network, wherein the residual network includes a plurality of residual blocks and a connecting line between every two residual blocks in the plurality of residual blocks, the connecting line from the first residual block to the second residual block has corresponding first model parameters and second model parameters, the first model parameter is a weight of a feature output by the first residual block when input to the second residual block, the second model parameter is a maximum allowed number of times the feature output by the first residual block is input to the second residual block, and the first residual block and the second residual block are any two residual blocks in the plurality of residual blocks;

[0025] The second image feature is input into the upsampling layer to obtain the second image output by the upsampling layer.

[0026] In some embodiments, the training process of the clarity enhancement model includes:

[0027] Acquire a second sample image and a clarity index of the second sample image;

[0028] Calling the definition enhancement model to be trained, and performing definition enhancement processing on the second sample image based on the definition index and target resolution of the second sample image to obtain a predicted image corresponding to the second sample image;

[0029] Obtaining a clarity index of the predicted image;

[0030] Based on the difference between the definition index of the predicted image and the preset definition index, the model parameters of the definition enhancement model to be trained are adjusted.

[0031] In some embodiments, the clarity enhancement model to be trained includes a weight prediction network and a residual network to be trained, and the weight prediction network is used to predict the first model parameter corresponding to each connecting line in the residual network to be trained during the training process.

[0032] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, the apparatus comprising:

[0033] A clarity acquisition module is configured to acquire a first clarity index of a first image, where the first clarity index is used to characterize the clarity of the first image;

[0034] The clarity enhancement module is configured to perform clarity enhancement processing on the first image based on the first clarity index and the target resolution to obtain a second image corresponding to the first image, wherein the resolution of the second image is the target resolution, and the target resolution is the resolution of the screen of the electronic device used to display the second image.

[0035] In some embodiments, the clarity acquisition module is configured to:

[0036] A clarity prediction model is called to perform clarity prediction on the first image to obtain the first clarity index, wherein the clarity prediction model is used to predict the clarity of an image.

[0037] In some embodiments, the clarity prediction model includes a feature extraction layer, a feature screening layer and an activation function layer, and the clarity acquisition module is configured as follows:

[0038] Inputting the first image into the feature extraction layer to obtain a first image feature of the first image output by the feature extraction layer;

[0039] Inputting the first image feature into the feature screening layer to obtain the filtered first image feature output by the feature screening layer, wherein the filtered first image feature is related to the clarity of the image;

[0040] The filtered first image feature is input into the activation function layer to obtain the first clarity index output by the activation function layer.

[0041] In some embodiments, the device further includes a model training module, wherein the model training module is configured to:

[0042] Acquire a first sample image, a first sample clarity index of the first sample image, and a second sample clarity index, wherein the first sample clarity index is determined based on manual annotation, and the second sample clarity index is determined based on a pixel average value and a pixel preset threshold value corresponding to the first sample image, wherein the pixel average value is a ratio between a sum of pixel values ​​of a plurality of pixel points of the first sample image and the number of the plurality of pixel points;

[0043] Determining a sample clarity index of the first sample image based on the first sample clarity index and the second sample clarity index;

[0044] Calling the clarity prediction model to be trained, performing clarity prediction on the first sample image, and obtaining a predicted clarity index of the first sample image;

[0045] Based on the difference between the predicted clarity index and the sample clarity index, a model parameter of the clarity prediction model to be trained is adjusted.

[0046] In some embodiments, the definition enhancement module is configured to:

[0047] A clarity enhancement model is called to perform clarity enhancement processing on the first image based on the first clarity index and the target resolution to obtain the second image, wherein the clarity enhancement model is used to enhance the clarity of the image and to modify the resolution of the image to the target resolution.

[0048] In some embodiments, the definition enhancement module is configured to:

[0049] Inputting the first image and the first definition index into the definition enhancement model to obtain the second image output by the definition enhancement model, wherein the hyperparameters of the definition enhancement model include the target resolution;

[0050] The first image, the first clarity index and the target resolution are input into the clarity improvement model to obtain the second image output by the clarity improvement model.

[0051] In some embodiments, the definition enhancement model includes a residual network and an upsampling layer; the definition enhancement module is configured to:

[0052] Inputting the first image and the first clarity index into the residual network to obtain a second image feature output by the residual network, wherein the residual network includes a plurality of residual blocks and a connecting line between every two residual blocks in the plurality of residual blocks, the connecting line from the first residual block to the second residual block has corresponding first model parameters and second model parameters, the first model parameter is a weight of a feature output by the first residual block when input to the second residual block, the second model parameter is a maximum allowed number of times the feature output by the first residual block is input to the second residual block, and the first residual block and the second residual block are any two residual blocks in the plurality of residual blocks;

[0053] The second image feature is input into the upsampling layer to obtain the second image output by the upsampling layer.

[0054] In some embodiments, the device further includes a model training module, wherein the model training module is configured to:

[0055] Acquire a second sample image and a clarity index of the second sample image;

[0056] Calling the definition enhancement model to be trained, and performing definition enhancement processing on the second sample image based on the definition index and target resolution of the second sample image to obtain a predicted image corresponding to the second sample image;

[0057] Obtaining a clarity index of the predicted image;

[0058] Based on the difference between the definition index of the predicted image and the preset definition index, the model parameters of the definition enhancement model to be trained are adjusted.

[0059] In some embodiments, the clarity enhancement model to be trained includes a weight prediction network and a residual network to be trained, and the weight prediction network is used to predict the first model parameter corresponding to each connecting line in the residual network to be trained during the training process.

[0060] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0061] processor;

[0062] a memory for storing processor-executable instructions;

[0063] The processor is configured to execute the method as described in the first aspect of the embodiment of the present disclosure.

[0064] According to a fourth aspect of an embodiment of the present disclosure, a non-temporary computer-readable storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method described in the first aspect of the embodiment of the present disclosure.

[0065] The above method of the present disclosure has the following beneficial effects:

[0066] The method provided by the embodiment of the present disclosure obtains a first clarity index of a first image, where the first clarity index is used to characterize the clarity of the first image. Then, with the first clarity index and the target resolution as reference, the first image is subjected to clarity enhancement processing to obtain a second image corresponding to the first image. While improving the clarity of the first image, the resolution of the second image is made to be the resolution of the screen of an electronic device used to display the second image, thereby ensuring that an image that matches the resolution of the screen and has a higher clarity is displayed to the user, thereby improving the visual effect of the image display and improving the user experience.

[0067] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0069] Figure 1 is a flow chart of an image processing method according to an exemplary embodiment;

[0070] Figure 2 is a flow chart showing another image processing method according to an exemplary embodiment;

[0071] Figure 3 is a schematic diagram of a clarity prediction model according to an exemplary embodiment;

[0072] Figure 4 is a schematic diagram of a definition improvement model according to an exemplary embodiment;

[0073] Figure 5 is a flowchart of a training process of a clarity prediction model according to an exemplary embodiment;

[0074] Figure 6 is a schematic diagram showing a training process of a clarity prediction model according to an exemplary embodiment;

[0075] Figure 7 is a flow chart showing a training process of a clarity enhancement model according to an exemplary embodiment;

[0076] Figure 8 is a schematic diagram of a training process of a clarity enhancement model according to an exemplary embodiment;

[0077] Fig. 9 is a block diagram of an image processing device according to an exemplary embodiment;

[0078] Fig.10 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0079] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0080] The method provided in the embodiment of the present disclosure is executed by an electronic device, which is a mobile phone, a tablet computer, a laptop computer, a wearable device, etc.

[0081] The image processing method provided by the embodiment of the present disclosure can be applied in a variety of scenarios. For example, in the scenario of converting the resolution of a received image, when a user uses an electronic device, for an image that needs to be displayed by the electronic device, the electronic device uses the image processing method provided by the embodiment of the present disclosure to convert the resolution of the image to be displayed into the resolution of the screen of the electronic device, and improves the clarity of the image during the conversion process, and then displays the processed image, thereby presenting to the user an image that matches the resolution of the screen and has a higher clarity. For another example, in the scenario of converting the resolution of a received video, each frame of the video is processed using the image processing method provided by the embodiment of the present disclosure, and then the processed image is displayed, thereby presenting to the user a video that matches the resolution of the screen and has a higher clarity.

[0082] Figure 1 is a flowchart of an image processing method according to an exemplary embodiment, which is executed by an electronic device, see Figure 1 , the method comprises the following steps:

[0083] Step S101, obtaining a first clarity index of a first image.

[0084] The first image is any image, for example, the first image is an image received by any application, or the first image is any frame image in any video, or the first image is the current display screen of the electronic device. The first clarity index is used to characterize the clarity of the first image. The larger the first clarity index is, the higher the clarity of the first image is, and the smaller the first clarity index is, the lower the clarity of the first image is. For example, the clarity index can be any value between 0 and 1. When the clarity index is 1, it indicates that the image is the clearest, and when the clarity index is 0, it indicates that the image is the least clear.

[0085] Step S102, based on the first clarity index and the target resolution, the first image is processed for clarity enhancement to obtain a second image corresponding to the first image, wherein the resolution of the second image is the target resolution, and the target resolution is the resolution of the screen of the electronic device used to display the second image.

[0086] In the embodiment of the present disclosure, the definition of the first image is improved based on the target resolution, so that the resolution of the obtained second image can be the target resolution, that is, the resolution of the first image is converted into the target resolution, so that when the second image is displayed, the resolution of the second image matches the resolution of the screen of the electronic device. At the same time, the definition of the first image is improved based on the first definition index, and the first definition index is used as a reference. When improving the definition of the first image, the definition of the obtained image is higher than the definition represented by the first definition index, and the second image with improved definition is obtained.

[0087] It should be noted that, in some embodiments, the original resolution of the first image is first obtained. When the original resolution is different from the target resolution, steps S101 and S102 are executed. When the original resolution is the same as the target resolution, steps S101 and S102 are not executed. Alternatively, when the original resolution is the same as the target resolution or different from the target resolution, steps S101 and S102 are executed.

[0088] It should be noted that, in some embodiments, after obtaining the second image, the electronic device displays the second image.

[0089] The method provided by the embodiment of the present disclosure obtains a first clarity index of a first image, where the first clarity index is used to characterize the clarity of the first image. Then, with the first clarity index and the target resolution as reference, the first image is subjected to clarity enhancement processing to obtain a second image corresponding to the first image. While improving the clarity of the first image, the resolution of the second image is made to be the resolution of the screen of an electronic device used to display the second image, thereby ensuring that an image that matches the resolution of the screen and has a higher clarity is displayed to the user, thereby improving the visual effect of the image display and improving the user experience.

[0090] Figure 2 is a flowchart of another image processing method according to an exemplary embodiment, which is executed by an electronic device, see Figure 2 , the method comprises the following steps:

[0091] Step S201, calling a clarity prediction model to perform clarity prediction on a first image to obtain a first clarity index.

[0092] The clarity prediction model is used to predict the clarity of an image. The image is input into the clarity prediction model, and the clarity prediction model outputs a clarity index of the image.

[0093] In some embodiments, see Figure 3The schematic diagram of the clarity prediction model shown in the figure includes a feature extraction layer, a feature screening layer and an activation function layer. The clarity prediction model is called to perform clarity prediction on the first image to obtain a first clarity index, including: inputting the first image into the feature extraction layer to obtain the first image feature of the first image output by the feature extraction layer; inputting the first image feature into the feature screening layer to obtain the filtered first image feature output by the feature screening layer; inputting the filtered first image feature into the activation function layer to obtain the first clarity index output by the activation function layer. The feature extraction layer is used to extract the first image feature of the first image, and the first image feature includes image color feature, image contrast feature, image clarity feature, etc.; the feature screening layer is used to screen features related to the clarity of the image, and the filtered first image feature is related to the clarity of the image, for example, the filtered first image feature is an image clarity feature; the activation function layer is a sigmoid or other activation function.

[0094] Before using the clarity prediction model, you need to train the clarity prediction model. The training process of the clarity prediction model is as follows: Figure 5 The embodiments shown will not be described in detail here.

[0095] It should be noted that the embodiment of the present disclosure is only described by calling the clarity prediction model to obtain the first clarity index as an example. In another embodiment, the first clarity index may be obtained in other ways, for example, the difference between the pixel average value corresponding to the first image and the pixel preset threshold value is obtained, and the corresponding first clarity index is determined according to the size of the difference. The pixel average value is the ratio between the sum of the pixel values ​​of multiple pixels of the first image and the number of multiple pixels. For example, the clarity index is any value between 0 and 1. When the difference is greater than 0, it indicates that the clarity is high, and when the difference is less than 0, it indicates that the clarity is low. The difference is proportional to the clarity index, that is, the larger the difference, the larger the clarity index, and the smaller the difference, the smaller the clarity index.

[0096] Step S202 , calling a definition enhancement model, and performing definition enhancement processing on the first image based on the first definition index and the target resolution to obtain a second image.

[0097] The definition enhancement model is used to enhance the definition of the image and modify the resolution of the image to the target resolution. The second definition index of the second image is higher than the first definition index.

[0098] In some embodiments, the hyperparameters of the clarity enhancement model include a target resolution, that is, the clarity enhancement model can only output an image with a resolution of the target resolution. In this case, the first image and the first clarity index are input into the clarity enhancement model to obtain a second image output by the clarity enhancement model. The first clarity index characterizes the clarity of the first image, and the first clarity index is input into the clarity enhancement model. The clarity enhancement model uses the first clarity index as a reference to obtain an image with a higher clarity as a goal, and processes the first image to obtain a second image. The clarity enhancement model is specifically used to process an image with a resolution of the target resolution, and is more targeted, so that the image quality of the obtained second image is better.

[0099] In some embodiments, the clarity enhancement model can output an image of any resolution. In this case, the first image, the first clarity index and the target resolution are input into the clarity enhancement model to obtain a second image output by the clarity enhancement model. The first clarity index characterizes the clarity of the first image. The first clarity index and the target resolution are input into the clarity enhancement model. The clarity enhancement model uses the first clarity index and the target resolution as references to obtain an image with higher clarity and resolution of the target resolution. The first image is processed to obtain a second image. The clarity enhancement model can process images of any resolution while improving the clarity of the image, and is more versatile.

[0100] In some embodiments, see Figure 4 The schematic diagram of the clarity enhancement model shown in FIG. 1 includes a residual network and an upsampling layer. The first image and the first clarity index are input into the residual network to obtain the second image feature output by the residual network; the second image feature is input into the upsampling layer to obtain the second image output by the upsampling layer.

[0101] Among them, the residual network includes multiple residual blocks and connecting lines between every two residual blocks in the multiple residual blocks. The connecting line from the first residual block to the second residual block has corresponding first model parameters and second model parameters. The first model parameter is the weight of the feature output by the first residual block when it is input to the second residual block. When the weight is 0, it means that the connecting line from the first residual block to the second residual block does not actually exist. When the weight is not 0, it means that the connecting line from the first residual block to the second residual block exists, and the feature transferred from the first residual block to the second residual block needs to be multiplied by the weight to ensure that the feature output by each residual block is backward with minimal loss. Transfer, the second model parameter is the maximum allowed number of times the feature output by the first residual block is input to the second residual block. The maximum allowed number is the number of times the feature can be transferred. For example, the maximum allowed number of times the connection line from the first residual block to the second residual block is 3. In the process of calling the model, after the first residual block processes the feature for the first time, the feature is transferred to the second residual block. After the first residual block processes the feature for the second time, the feature is transferred to the second residual block. After the first residual block processes the feature for the third time, the feature is transferred to the second residual block. After the first residual block processes the feature for the fourth time, it will no longer be transferred to the second residual block. Among them, the first residual block and the second residual block are any two residual blocks among the multiple residual blocks.

[0102] In the disclosed embodiment, the residual block used in the clarity enhancement model can process input images of any size. Compared with the upsampling and downsampling structures in the related art, the clarity enhancement model has a wider range of applicability and has lower requirements on the input image itself.

[0103] Before using the clarity prediction model, you need to train the clarity prediction model. The training process of the clarity prediction model is as follows: Figure 7 The embodiments shown will not be described in detail here.

[0104] The method provided by the embodiment of the present disclosure calls a clarity prediction model to obtain a first clarity index of a first image, and then calls a clarity improvement model to perform clarity improvement processing on the first image with the first clarity index and a target resolution as a reference to obtain a second image corresponding to the first image. The clarity prediction model is coupled with the clarity improvement model to improve the clarity of the first image while making the resolution of the second image the resolution of a screen of an electronic device used to display the second image, thereby ensuring that an image that matches the resolution of the screen and has a higher clarity is displayed to the user, thereby improving the visual effect of the image display and improving the user experience.

[0105] Figure 5 is a flowchart of a training process of a clarity prediction model according to an exemplary embodiment, which is executed by an electronic device, see Figure 5, the method comprises the following steps:

[0106] Step S501: acquiring a first sample image.

[0107] The first sample image is any image.

[0108] Step S502: determining a first sample clarity index of a first sample image based on manual annotation.

[0109] The clarity of the first sample image is manually observed, and then the clarity of the first sample image is scored to obtain a first sample clarity index. The first sample image is manually labeled to form a data format in which the image and the label correspond one to one, and the label is the first sample clarity index.

[0110] Step S503: determining a second sample clarity index of the first sample image based on a pixel average value and a pixel preset threshold corresponding to the first sample image.

[0111] The pixel average value is the ratio between the sum of the pixel values ​​of the multiple pixel points of the first sample image and the number of the multiple pixel points.

[0112] In the embodiment of the present disclosure, the clarity of the first sample image is determined by using a spectrum response method. First, the first sample image is Fourier transformed to obtain high-frequency information and low-frequency information in the first sample image, and then the pixel average is calculated according to the center of the spectrum, that is, the ratio between the sum of the pixel values ​​of multiple pixel points of the first sample image and the number of multiple pixel points is determined as the pixel average. Then, the second sample clarity index is determined according to the difference between the pixel average and the pixel preset threshold.

[0113] Optionally, a plurality of images with higher definition are obtained, a pixel average value of each image is obtained, and then the pixel average values ​​of the plurality of images are averaged to obtain a pixel preset threshold value.

[0114] Step S504: Determine a sample clarity index of the first sample image based on the first sample clarity index and the second sample clarity index.

[0115] The first sample clarity index is a subjective judgment result, and the second sample clarity index is an objective judgment result. The subjective judgment result and the objective judgment result can be comprehensively considered, that is, the first sample clarity index and the second sample clarity index are averaged to obtain the sample clarity index of the first sample image.

[0116] Step S505 , calling the definition prediction model to be trained, performing definition prediction on the first sample image, and obtaining a predicted definition index of the first sample image.

[0117] Step S506: adjusting the model parameters of the clarity prediction model to be trained based on the difference between the predicted clarity index and the sample clarity index.

[0118] Based on the preset loss function, predicted clarity index and sample clarity index, the loss value is calculated, and the optimization model parameters are updated according to the loss value.

[0119] It should be noted that steps S501-S506 are described based on a training process. When training the clarity prediction model, the clarity prediction model can be iteratively trained multiple times until the number of iterations reaches a preset number, and the training is stopped to obtain a trained clarity prediction model.

[0120] Figure 6 FIG. 1 is a schematic diagram showing a training process of another definition prediction model according to an exemplary embodiment. Figure 6 , input the sample image into the clarity prediction model to be trained, output the predicted clarity index, and then use the cross entropy loss function to calculate the loss value corresponding to the predicted clarity index and the sample clarity index, and then update the optimization model parameters according to the loss value.

[0121] In the disclosed embodiment, when training the clarity prediction model, the sample clarity index in the training data is obtained after comprehensively considering the subjective judgment results and the objective judgment results. The sample clarity index is more accurate, and thus the trained clarity prediction model is also more accurate.

[0122] Figure 7 is a flowchart of a training process of a definition enhancement model according to an exemplary embodiment, which is executed by an electronic device, see Figure 7 , the method comprises the following steps:

[0123] Step S701: Obtain a second sample image and a clarity index of the second sample image.

[0124] The second sample image is any image, and the second sample image may be the same as or different from the first sample image in the above embodiment. The clarity index of the second sample image may be obtained by using the above clarity prediction model or other methods, which is not limited in the present embodiment.

[0125] Step S702 , calling the definition enhancement model to be trained, and performing definition enhancement processing on the second sample image based on the definition index and target resolution of the second sample image to obtain a predicted image corresponding to the second sample image.

[0126] When the hyperparameters of the clarity improvement model to be trained include the target resolution, the second sample image and the clarity index of the second sample image are input into the clarity improvement model to be trained to obtain an output predicted image; when the hyperparameters of the clarity improvement model to be trained do not include the target resolution, the second sample image, the clarity index of the second sample image and the target resolution are input into the clarity improvement model to be trained to obtain an output predicted image.

[0127] Step S703, obtaining the clarity index of the predicted image.

[0128] The method for obtaining the clarity index of the predicted image may be: obtaining it by calling a clarity prediction model, or obtaining it by other methods, which is not limited in the embodiments of the present disclosure.

[0129] Step S704: adjusting the model parameters of the definition enhancement model to be trained based on the difference between the definition index of the predicted image and the preset definition index.

[0130] Based on the preset loss function, the definition index of the predicted image and the preset definition index, the loss value is calculated, and the optimization model parameters are updated according to the loss value. The preset definition index is a higher definition index, for example, the preset definition index is 1.

[0131] In some embodiments, the definition enhancement model to be trained includes a weight prediction network and a residual network to be trained, and the weight prediction network is used to predict the first model parameter corresponding to each connection line in the residual network to be trained during the training process. That is, the weight prediction network is used to automatically learn the features output by each residual block and the weights passed down. This method can enable the network to automatically update the parameters of each residual block and select a more reasonable residual block connection method to obtain more refined image features, provide a basis for changing the image resolution, achieve higher-precision definition enhancement, and bring better visual display effects.

[0132] It should be noted that steps S701-S704 are described by using one second sample image for training in each training process. In another embodiment, multiple second sample images may be used for training at the same time in each training process.

[0133] It should be noted that steps S701-S704 are described based on a training process. When training the clarity improvement model, the clarity improvement model can be iteratively trained multiple times until the number of iterations reaches a preset number, and the training is stopped to obtain the trained clarity improvement model, or until the clarity index of the output predicted image reaches the highest clarity index, and the training is stopped to obtain the trained clarity improvement model.

[0134] Figure 8 FIG. 1 is a schematic diagram showing a training process of another definition enhancement model according to an exemplary embodiment. Figure 8 , multiple sample images and the clarity index of each sample image are input into the clarity enhancement model to obtain the predicted image corresponding to each sample image, and then each predicted image is input into the clarity prediction model to obtain the clarity index of each predicted image, and the Euclidean distance between the clarity index of multiple predicted images and the preset clarity index 1 is calculated, and the model parameters are updated and optimized according to the Euclidean distance.

[0135] In the disclosed embodiment, in the clarity enhancement model, the original method of extracting features by up and down sampling is abandoned, and multiple residual blocks are used to extract image features. The residual blocks are cascaded according to certain weights, and the resolution is changed in the last layer of the model. The up and down sampling modules are discarded, so that the model can accept input of any resolution and achieve output of any resolution, and can be compatible and implemented on screens of different resolutions, with high application flexibility.

[0136] Moreover, in common image generation tasks, pixel-level losses such as MSE (Mean Square Error) are usually used for model training. The generated images are often too smooth to achieve high-quality image reconstruction. The conventional pixel loss calculation method is abandoned in the present disclosure, and a clarity prediction model is used to predict the clarity index. The loss is then calculated based on the predicted clarity index to optimize the model parameters in the generation model, so that the images generated by the clarity enhancement model are more in line with the subjective standards of users, thereby improving the subjective visual effect.

[0137] Fig. 9 is a block diagram of an image processing device according to an exemplary embodiment;

[0138] The clarity acquisition module 901 is configured to acquire a first clarity index of the first image, where the first clarity index is used to characterize the clarity of the first image;

[0139] The clarity enhancement module 902 is configured to perform clarity enhancement processing on the first image based on the first clarity index and the target resolution to obtain a second image corresponding to the first image, wherein the resolution of the second image is the target resolution, and the target resolution is the resolution of the screen of the electronic device used to display the second image.

[0140] The device provided by the embodiment of the present disclosure obtains a first clarity index of a first image, where the first clarity index is used to characterize the clarity of the first image, and then performs clarity enhancement processing on the first image with the first clarity index and the target resolution as reference to obtain a second image corresponding to the first image. While improving the clarity of the first image, the resolution of the second image is made to be the resolution of the screen of an electronic device used to display the second image, thereby ensuring that an image that matches the resolution of the screen and has a higher clarity is displayed to the user, thereby improving the visual effect of the image display and improving the user experience.

[0141] In some embodiments, the clarity acquisition module 901 is configured to:

[0142] A clarity prediction model is called to perform clarity prediction on the first image to obtain a first clarity index. The clarity prediction model is used to predict the clarity of the image.

[0143] In some embodiments, the clarity prediction model includes a feature extraction layer, a feature screening layer, and an activation function layer, and the clarity acquisition module 901 is configured as follows:

[0144] Inputting the first image into the feature extraction layer, and obtaining the first image feature of the first image output by the feature extraction layer;

[0145] Inputting the first image feature into the feature screening layer to obtain the filtered first image feature output by the feature screening layer, wherein the filtered first image feature is related to the clarity of the image;

[0146] The filtered first image feature is input into the activation function layer to obtain the first clarity index output by the activation function layer.

[0147] In some embodiments, the device further includes a model training module, the model training module being configured to:

[0148] Obtaining a first sample image, a first sample clarity index of the first sample image, and a second sample clarity index, wherein the first sample clarity index is determined based on manual annotation, and the second sample clarity index is determined based on a pixel average value corresponding to the first sample image and a pixel preset threshold, wherein the pixel average value is a ratio between a sum of pixel values ​​of a plurality of pixel points of the first sample image and the number of the plurality of pixel points;

[0149] Determining a sample clarity index of a first sample image based on the first sample clarity index and the second sample clarity index;

[0150] Calling the clarity prediction model to be trained, performing clarity prediction on the first sample image, and obtaining a predicted clarity index of the first sample image;

[0151] Based on the difference between the predicted clarity index and the sample clarity index, a model parameter of the clarity prediction model to be trained is adjusted.

[0152] In some embodiments, the definition enhancement module 902 is configured to:

[0153] The clarity enhancement model is called, and based on the first clarity index and the target resolution, the clarity enhancement processing is performed on the first image to obtain a second image. The clarity enhancement model is used to enhance the clarity of the image and modify the resolution of the image to the target resolution.

[0154] In some embodiments, the definition enhancement module 902 is configured to:

[0155] Inputting the first image and the first clarity index into a clarity enhancement model to obtain a second image output by the clarity enhancement model, wherein the hyperparameters of the clarity enhancement model include a target resolution;

[0156] The first image, the first definition index and the target resolution are input into a definition enhancement model to obtain a second image output by the definition enhancement model.

[0157] In some embodiments, the definition enhancement model includes a residual network and an upsampling layer; the definition enhancement module 902 is configured to:

[0158] Inputting the first image and the first clarity index into a residual network to obtain a second image feature output by the residual network, wherein the residual network includes a plurality of residual blocks and a connecting line between every two residual blocks in the plurality of residual blocks, the connecting line from the first residual block to the second residual block has corresponding first model parameters and second model parameters, the first model parameter is a weight of a feature output by the first residual block when input to the second residual block, the second model parameter is a maximum allowed number of times the feature output by the first residual block is input to the second residual block, and the first residual block and the second residual block are any two residual blocks in the plurality of residual blocks;

[0159] The second image feature is input to the upsampling layer to obtain a second image output by the upsampling layer.

[0160] In some embodiments, the device further includes a model training module, the model training module being configured to:

[0161] Acquire a second sample image and a clarity index of the second sample image;

[0162] Calling the definition enhancement model to be trained, and performing definition enhancement processing on the second sample image based on the definition index and target resolution of the second sample image to obtain a predicted image corresponding to the second sample image;

[0163] Obtain the clarity index of the predicted image;

[0164] Based on the difference between the clarity index of the predicted image and the preset clarity index, a model parameter of the clarity enhancement model to be trained is adjusted.

[0165] In some embodiments, the clarity enhancement model to be trained includes a weight prediction network and a residual network to be trained, and the weight prediction network is used to predict the first model parameter corresponding to each connecting line in the residual network to be trained during the training process.

[0166] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0167] An embodiment of the present disclosure further provides an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the image processing method in the above embodiment.

[0168] Fig.10 is a block diagram of an electronic device 1000 according to an exemplary embodiment.

[0169] Reference Fig.10 , the electronic device 1000 may include one or more of the following components: a processing component 1002 , a memory 1004 , a power component 1006 , a multimedia component 1008 , an audio component 1010 , an input / output (I / O) interface 1012 , a sensor component 1014 , and a communication component 1016 .

[0170] The processing component 1002 generally controls the overall operation of the electronic device 1000, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 1002 may include one or more processors 1020 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 1002 may include one or more modules to facilitate the interaction between the processing component 1002 and other components. For example, the processing component 1002 may include a multimedia module to facilitate the interaction between the multimedia component 1008 and the processing component 1002.

[0171] The memory 1004 is configured to store various types of data to support operations on the electronic device 1000. Examples of such data include instructions for any application or method operating on the electronic device 1000, contact data, phone book data, messages, pictures, videos, etc. The memory 1004 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0172] The power supply component 1006 provides power to various components of the electronic device 1000. The power supply component 1006 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 1000.

[0173] The multimedia component 1008 includes a screen that provides an output interface between the electronic device 1000 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 1008 includes a front camera and / or a rear camera. When the electronic device 1000 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0174] The audio component 1010 is configured to output and / or input audio signals. For example, the audio component 1010 includes a microphone (MIC), and when the electronic device 1000 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 1004 or sent via the communication component 1016. In some embodiments, the audio component 1010 also includes a speaker for outputting audio signals.

[0175] I / O interface 1012 provides an interface between processing component 1002 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0176] The sensor assembly 1014 includes one or more sensors for providing various aspects of status assessment for the electronic device 1000. For example, the sensor assembly 1014 can detect the open / closed state of the electronic device 1000, the relative positioning of components, such as the display and keypad of the electronic device 1000, and the sensor assembly 1014 can also detect the position change of the electronic device 1000 or a component of the electronic device 1000, the presence or absence of user contact with the electronic device 1000, the orientation or acceleration / deceleration of the electronic device 1000, and the temperature change of the electronic device 1000. The sensor assembly 1014 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 1014 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1014 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0177] The communication component 1016 is configured to facilitate wired or wireless communication between the electronic device 1000 and other devices. The electronic device 1000 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 1016 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1016 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0178] In an exemplary embodiment, the electronic device 1000 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0179] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1004 including instructions, and the instructions can be executed by the processor 1020 of the electronic device 1000 to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0180] The embodiments of the present disclosure also provide a non-temporary computer-readable storage medium. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the image processing method in the above embodiments.

[0181] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this disclosure. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0182] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a first clarity index of a first image, where the first clarity index is used to characterize the clarity of the first image; Based on the first clarity index and the target resolution, the first image is processed for clarity enhancement to obtain a second image corresponding to the first image, wherein the resolution of the second image is the target resolution, and the target resolution is the resolution of the screen of the electronic device used to display the second image.

2. The method according to claim 1, characterized in that The step of obtaining a first clarity index of the first image includes: A clarity prediction model is called to perform clarity prediction on the first image to obtain the first clarity index, wherein the clarity prediction model is used to predict the clarity of an image.

3. The method according to claim 2, characterized in that The clarity prediction model includes a feature extraction layer, a feature screening layer, and an activation function layer. The calling of the clarity prediction model to perform clarity prediction on the first image to obtain the first clarity index includes: Inputting the first image into the feature extraction layer to obtain a first image feature of the first image output by the feature extraction layer; Inputting the first image feature into the feature screening layer to obtain the filtered first image feature output by the feature screening layer, wherein the filtered first image feature is related to the clarity of the image; The filtered first image feature is input into the activation function layer to obtain the first clarity index output by the activation function layer.

4. The method according to claim 2 or 3, characterized in that: The training process of the clarity prediction model includes: Acquire a first sample image, a first sample clarity index of the first sample image, and a second sample clarity index, wherein the first sample clarity index is determined based on manual annotation, and the second sample clarity index is determined based on a pixel average value and a pixel preset threshold value corresponding to the first sample image, wherein the pixel average value is a ratio between a sum of pixel values ​​of a plurality of pixel points of the first sample image and the number of the plurality of pixel points; Determining a sample clarity index of the first sample image based on the first sample clarity index and the second sample clarity index; Calling the clarity prediction model to be trained, performing clarity prediction on the first sample image, and obtaining a predicted clarity index of the first sample image; Based on the difference between the predicted clarity index and the sample clarity index, a model parameter of the clarity prediction model to be trained is adjusted.

5. The method according to claim 1, characterized in that The performing definition enhancement processing on the first image based on the first definition index and the target resolution to obtain a second image corresponding to the first image includes: A clarity enhancement model is called to perform clarity enhancement processing on the first image based on the first clarity index and the target resolution to obtain the second image, wherein the clarity enhancement model is used to enhance the clarity of the image and to modify the resolution of the image to the target resolution.

6. The method according to claim 5, characterized in that The calling of the definition improvement model to perform definition improvement processing on the first image based on the first definition index and the target resolution to obtain the second image includes: Inputting the first image and the first definition index into the definition enhancement model to obtain the second image output by the definition enhancement model, wherein the hyperparameters of the definition enhancement model include the target resolution; or The first image, the first clarity index and the target resolution are input into the clarity improvement model to obtain the second image output by the clarity improvement model.

7. The method according to claim 6, characterized in that The definition enhancement model includes a residual network and an upsampling layer; the first image and the first definition index are input into the definition enhancement model to obtain the second image output by the definition enhancement model, including: Inputting the first image and the first clarity index into the residual network to obtain a second image feature output by the residual network, wherein the residual network includes a plurality of residual blocks and a connecting line between every two residual blocks in the plurality of residual blocks, the connecting line from the first residual block to the second residual block has corresponding first model parameters and second model parameters, the first model parameter is a weight of a feature output by the first residual block when input to the second residual block, the second model parameter is a maximum allowed number of times the feature output by the first residual block is input to the second residual block, and the first residual block and the second residual block are any two residual blocks in the plurality of residual blocks; The second image feature is input into the upsampling layer to obtain the second image output by the upsampling layer.

8. The method according to claim 7, characterized in that The training process of the clarity enhancement model includes: Acquire a second sample image and a clarity index of the second sample image; Calling the definition enhancement model to be trained, and performing definition enhancement processing on the second sample image based on the definition index and target resolution of the second sample image to obtain a predicted image corresponding to the second sample image; Obtaining a clarity index of the predicted image; Based on the difference between the definition index of the predicted image and the preset definition index, the model parameters of the definition enhancement model to be trained are adjusted.

9. The method according to claim 8, characterized in that The clarity enhancement model to be trained includes a weight prediction network and a residual network to be trained, and the weight prediction network is used to predict the first model parameter corresponding to each connecting line in the residual network to be trained during the training process.

10. An image processing device, characterized in that: The device comprises: A clarity acquisition module is configured to acquire a first clarity index of a first image, where the first clarity index is used to characterize the clarity of the first image; The clarity enhancement module is configured to perform clarity enhancement processing on the first image based on the first clarity index and the target resolution to obtain a second image corresponding to the first image, wherein the resolution of the second image is the target resolution, and the target resolution is the resolution of the screen of the electronic device used to display the second image.

11. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the method as claimed in any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method as claimed in any one of claims 1 to 9.

Images