Image processing method, device, apparatus and storage medium

Abstract

The application discloses an image processing method and device, equipment and storage medium, and relates to the field of image processing. The method comprises the following steps: processing an image through an image segmentation model to obtain a probability distribution diagram of a sky region in the image, that is, a confidence degree of a pixel point in the sky region in the image; determining a high-confidence region with a confidence degree higher than a first confidence threshold from the sky region; and replacing the sky region with a target sky material when the pixel point in the high-confidence region does not meet a preset condition to obtain a target image. The method solves the problem that at least seven steps are required to manually replace the sky region in the related art, achieves the purpose of automatically replacing the original sky region in the image with the target sky material, does not require a user to perform post-editing, reduces the manual operation steps of the user, and improves the human-computer interaction efficiency. Even if the user does not have a good foundation in editing, the user can also obtain a sky photo that can be shot by a professional single-lens reflex camera.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing, and in particular to an image processing method, apparatus, device, and storage medium. Background Art

[0002] Typically, users use their devices to capture objects of interest. When the captured image includes the sky, the quality of the captured image is affected by ambient lighting. For example, when capturing images, users may encounter poor weather conditions or low lighting, resulting in the captured image not meeting their desired quality. For example, on a cloudy day, if a user captures an image with the sky as the background, the sky area in the resulting image will appear gray and hazy, failing to achieve the desired effect.

[0003] In related technologies, users use photo editing software to replace the sky area in a captured image with an ideal sky material. The relevant steps are as follows:

[0004] 1. After taking an image using the terminal, the user uploads the image to the computer;

[0005] 2. If the computer has photo editing software installed (if the computer does not have photo editing software installed, you need to install the software first), open the photo editing software;

[0006] 3. Open the image and the ideal sky material in the above-mentioned photo editing software;

[0007] 4. Set the sky material as background and the image as layer 1;

[0008] 5. Use the cutout tool to select the non-sky area in the image, copy the non-sky area, and set the non-sky area as layer 2;

[0009] 6. Delete layer 1 and feather layer 2;

[0010] 7. Save the modified image.

[0011] The user needs to go through at least seven steps to obtain the image after replacing the sky area. This method is complex to operate, has cumbersome steps, and has low human-computer interaction efficiency. Summary of the Invention

[0012] The embodiments of the present application provide an image processing method, apparatus, device, and storage medium that can solve the problem of requiring many manual operations, complex steps, and low human-computer interaction efficiency when replacing sky materials in the sky area of ​​a captured image. The technical solution is as follows:

[0013] According to a first aspect of the present application, there is provided an image processing method, the method comprising:

[0014] Processing the image using an image segmentation model to obtain a probability distribution map of a sky area in the image; the probability distribution map is the confidence level of the pixels in the sky area;

[0015] Determine a high confidence region from the sky region, where the confidence level is higher than a first confidence threshold;

[0016] When the pixel points in the high confidence area do not meet the preset conditions, the sky area is replaced with the target sky material to obtain the target image.

[0017] In some embodiments, replacing the sky area with a target sky material to obtain a target image includes:

[0018] Segmenting the edge of the sky area;

[0019] The segmented sky area is replaced with the target sky material to obtain the target image.

[0020] In some embodiments, replacing the segmented sky area with the target sky material to obtain the target image includes:

[0021] Obtaining attribute parameters of the image;

[0022] Filtering the target sky material from the candidate sky materials according to the attribute parameters;

[0023] The attribute parameters include at least one of a shooting time and a shooting location of the image.

[0024] In some embodiments, the preset conditions include:

[0025] The average primary color value of the blue channel of the pixel points in the high confidence area is less than the first primary color value threshold;

[0026] Alternatively, the average grayscale value of the pixels in the high confidence area is less than the grayscale value threshold;

[0027] Alternatively, the average primary color value of the blue channel of the pixel points in the high confidence area is less than the first primary color value threshold, and the average grayscale value of the pixel points in the high confidence area is less than the grayscale value threshold.

[0028] In some embodiments, before processing the image using the image segmentation model to obtain a probability distribution map of the sky area in the image, the method includes:

[0029] Obtaining the average original color value of the dark channel in the image;

[0030] When the average primary color value of the dark channel is less than the second primary color value threshold, the step of processing the image using the image segmentation model to obtain a probability distribution map of the sky area in the image is performed.

[0031] In some embodiments, before replacing the sky area with the target sky material to obtain the target image, the process includes:

[0032] Determine a first ratio of the pixels of the sky area to the total pixels of the image;

[0033] When the first ratio is greater than a first ratio threshold, the step of replacing the sky area with a target sky material to obtain a target image is performed.

[0034] In some embodiments, before replacing the sky area with the target sky material to obtain the target image, the method further includes:

[0035] Determine an area in the non-sky area where the confidence level of the pixel points is higher than the second confidence threshold and lower than the third confidence threshold as a first blurred area; and determine an area in the sky area where the confidence level of the pixel points is higher than the third confidence threshold and lower than the fourth confidence threshold as a second blurred area;

[0036] determining a second ratio of pixels of the first blurred area and the second blurred area in the image;

[0037] When the second ratio is less than a second ratio threshold, performing the step of replacing the sky area with a target sky material to obtain a target image;

[0038] The non-sky area refers to an area in the image other than the sky area.

[0039] In some embodiments, the method further comprises:

[0040] When the pixel points in the high confidence area meet the preset conditions, the image is processed using a filter; the filter is used to enhance the display effect of the image.

[0041] According to a second aspect of the present application, there is provided an image processing apparatus, the apparatus comprising:

[0042] a processing module configured to process the image using an image segmentation model to obtain a probability distribution map of a sky area in the image; the probability distribution map includes confidence scores of pixels in the sky area;

[0043] a determination module configured to determine, from the sky area, a high confidence area having a confidence level higher than a first confidence threshold;

[0044] The replacement module is configured to replace the sky area with a target sky material to obtain a target image when the pixel points in the high confidence area do not meet the preset conditions.

[0045] In some embodiments, the replacement module includes:

[0046] a segmentation submodule, configured to perform segmentation processing on the edge of the sky area;

[0047] The replacement submodule is configured to replace the segmented sky area with the target sky material to obtain the target image.

[0048] In some embodiments, the replacement submodule is configured to obtain attribute parameters of the image; and filter out the target sky material from the candidate sky materials based on the attribute parameters; wherein the attribute parameters include at least one of the shooting time and shooting location of the image.

[0049] In some embodiments, the preset conditions include:

[0050] The average primary color value of the blue channel of the pixel points in the high confidence area is less than the first primary color value threshold;

[0051] Alternatively, the average grayscale value of the pixels in the high confidence area is less than the grayscale value threshold;

[0052] Alternatively, the average primary color value of the blue channel of the pixel points in the high confidence area is less than the first primary color value threshold, and the average grayscale value of the pixel points in the high confidence area is less than the grayscale value threshold.

[0053] In some embodiments, the apparatus comprises:

[0054] an acquisition module, configured to acquire an average original color value of a dark channel in the image;

[0055] The determination module is configured to execute the step of processing the image through the image segmentation model to obtain a probability distribution map of the sky area in the image when the average primary color value of the dark channel is less than a second primary color value threshold.

[0056] In some embodiments, the determination module is configured to determine a first proportion of the pixel points of the sky area in the total pixel points of the image; when the first proportion is greater than a first proportion threshold, the step of replacing the sky area with the target sky material to obtain the target image is executed.

[0057] In some embodiments, the determination module is configured to determine an area in the non-sky area where the confidence level of pixels is higher than a second confidence threshold and lower than a third confidence threshold as a first blurred area; and determine an area in the sky area where the confidence level of pixels is higher than the third confidence threshold and lower than a fourth confidence threshold as a second blurred area;

[0058] The determination module is configured to determine a second ratio of pixels of the first blurred area and the second blurred area in the image; when the second ratio is less than a second ratio threshold, perform the step of replacing the sky area with a target sky material to obtain a target image; wherein the non-sky area refers to the area in the image other than the sky area.

[0059] In some embodiments, the processing module is further configured to process the image using a filter when the pixel points in the high confidence area meet the preset conditions; the filter is used to enhance the display effect of the image.

[0060] According to a third aspect of the present application, a terminal is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the image processing method as described in any one of the first aspects above.

[0061] According to the fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by the processor to implement the image processing method as described in any one of the first aspects above.

[0062] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0063] The image is processed using an image segmentation model to obtain a probability distribution map of the sky area in the image; the probability distribution map shows the confidence levels of the pixels in the sky area; a high-confidence region with a confidence level above a first confidence threshold is determined within the sky area; when a pixel in the high-confidence region does not meet a preset condition, the sky area is replaced with a target sky material to obtain a target image. This solves the problem in related technologies where at least seven steps are required to obtain a target image by replacing the sky area with the target sky material. This achieves the goal of automatically replacing the original sky area in the image with the target sky material, eliminating the need for post-processing by the user, reducing manual steps and improving human-computer interaction efficiency. Even users without extensive photo editing skills can still obtain sky photos comparable to those captured with professional SLR lenses. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0065] Figure 1 is a structural block diagram of a terminal provided by an exemplary embodiment of the present application;

[0066] Figure 2 is a structural block diagram of a terminal provided by another exemplary embodiment of the present application;

[0067] Figure 3 is a flowchart of an image processing method provided by an exemplary embodiment of the present application;

[0068] Figure 4 is a schematic diagram of an interface of an image processing method provided by an exemplary embodiment of the present application;

[0069] Figure 5 is a schematic diagram of an interface of an image processing method provided by another exemplary embodiment of the present application;

[0070] Figure 6 is a flowchart of an image processing method provided by another exemplary embodiment of the present application;

[0071] Figure 7 is a flowchart of an image processing method provided by another exemplary embodiment of the present application;

[0072] Figure 8 is a flowchart of an image processing method provided by another exemplary embodiment of the present application;

[0073] Figure 9is a flowchart of an image processing method provided by another exemplary embodiment of the present application;

[0074] Figure 10 is a block diagram of an image processing apparatus provided by an exemplary embodiment of the present application;

[0075] Figure 11 is a block diagram of an image processing device provided by another exemplary embodiment of the present application. DETAILED DESCRIPTION

[0076] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0077] First, several terms involved in the embodiments of this application are explained:

[0078] Neural network model: an artificial neural network formed by interconnecting n neurons, where n is a positive integer. In the present application, the neural network model is an artificial network model that can identify the sky area in the image. The neural network model can be divided into an input layer, a hidden layer, and an output layer. The terminal inputs the image into the input layer of the neural network model, and the hidden layer downsamples the input image, that is, performs convolution calculation on the pixels in the image, and finally outputs the recognition result through the output layer. Neural network models include CNN (Convolutional Neural Network, Convolutional Neural Network) model, FCN (Fully Convolutional Networks, Full Convolutional Neural Network) model, DNN (Deep Neural Network, Deep Neural Network) model, RNN (Recurrent Neural Network, Recurrent Neural Network) model, embedding model, GBDT (Gradient Boosting Decision Tree, Gradient Boosting Decision Tree) model, LR (Logistic Regression, Logistic Regression) model, etc.

[0079] CNN model: It is a deep feedforward artificial neural network. CNN includes but is not limited to the following three parts: an input layer, a combination of n convolutional layers and pooling layers, and a fully connected multi-layer perceptron, where n is a positive integer. CNN contains a feature extractor consisting of a convolutional layer and a pooling layer. The feature extractor extracts features from the samples input by the input layer, obtains model parameters, and completes the final model training in the perceptron based on the model parameters. In recent years, CNN models have been widely used in speech recognition, general object recognition, face recognition, image recognition, motion analysis, natural language processing, brain wave analysis, etc. This application takes the application of CNN models in image recognition as an example, and the specific content is to identify the sky area in the image through the CNN model.

[0080] The FCN model is a deep feedforward artificial neural network. Unlike the aforementioned CNN model, the FCN model uses a convolutional output layer, while the CNN model uses a fully connected output layer. The FCN model uses convolution, pooling, and deconvolution to output the recognized image.

[0081] A DNN model is a deep learning framework. It consists of an input layer, at least one hidden layer (or intermediate layer), and an output layer. Optionally, each of the input layer, at least one hidden layer (or intermediate layer), and the output layer includes at least one neuron, which processes the received data. The number of neurons in different layers can be the same or different.

[0082] The RNN model is a neural network with a feedback structure. In the RNN model, the output of a neuron can directly act on itself at the next timestamp. That is, the input of the i-th layer neuron at time m includes not only the output of the (i-1) layer neuron at that time, but also its own output at time (m-1).

[0083] The embedding model is based on a distributed vector representation of entities and relations, treating the relations in each triple instance as a translation from the entity head to the entity tail. A triple instance consists of a subject, a relation, and an object. A triple instance can be represented as (subject, relation, object), with the subject being the entity head and the object being the entity tail. For example, if Xiao Zhang's father is Da Zhang, the triple instance can be represented as (Xiao Zhang, father, Da Zhang).

[0084] The GBDT model is an iterative decision tree algorithm consisting of multiple decision trees, with the results of all trees accumulated as the final result. Each node in the decision tree receives a predicted value. For example, for age, the predicted value is the average age of all people belonging to the node corresponding to that age.

[0085] LR model: refers to a model established by applying a logistic function based on linear regression.

[0086] Image segmentation model: A model built based on at least one of the CNN model, FCN model, DNN model, RNN model, embedding model, GBDT model, and LR model. This model is used to identify the sky area in the image and obtain a probability distribution map of the sky area in the image, where the above probability distribution map is the confidence level of the pixels in the sky area.

[0087] Schematically, the image segmentation model adopts a deep convolutional neural network model and adds a trainable guided filter to participate in the training to improve the accuracy of the image segmentation model and increase detail information.

[0088] Confidence: The confidence interval of a sample is an interval estimate of the parameter in that sample. The probability that the parameter in the sample falls within that confidence interval is the confidence level. For example, the confidence level of a pixel in the sky region in this application is used as an example. The sky region corresponds to a confidence interval, and the probability that a pixel in the sky region falls within that confidence interval is the confidence level corresponding to that pixel.

[0089] Viewfinder image: Image data captured by the photosensitive device, used to display the image in the shooting preview interface. If a shutter signal is triggered by the user, the viewfinder image can be processed and saved as the captured image.

[0090] Captured image: The image obtained by saving the viewfinder image according to the shutter signal.

[0091] Attribute parameters: After capturing an image, the image format file is generated. The format file includes file header information. The file header information includes information such as aperture, shutter speed, white balance, ISO, focal length, date, time, and location when the image was captured. The file header information is used as attribute parameters.

[0092] Candidate sky clips: These are pre-stored sky clips stored in memory. These clips include sky clips from different times, geographic regions, and / or weather conditions. These can be high-quality, manually selected clips or clips captured with a high-performance SLR camera.

[0093] Target sky material: a sky material corresponding to the attribute parameters of the image determined from multiple candidate sky materials.

[0094] Target image: is the image obtained after replacing the sky area in the image with the target sky material.

[0095] Please refer to Figure 1, which shows a block diagram of a terminal provided by an exemplary embodiment of the present application. The terminal includes: a photosensitive device 101, an ISP (Image Signal Processing) module 102, a processor 103 and a memory 104.

[0096] The photosensitive device 101 is configured to sense the shooting environment and obtain a view image. The photosensitive device 101 can be a CCD (charge coupled device) image sensor or a CMOS (complementary metal oxide semiconductor) image sensor.

[0097] The ISP module 102 is electrically connected to the photosensitive device 101. Optionally, the ISP module 102 and the photosensitive device 101 are connected via a bus, or the ISP module 102 and the photosensitive device 101 are integrated into the same electrical package or chip. The photosensitive device 101 transmits the collected image data to the ISP module 102 for processing.

[0098] The ISP module 102 is configured to obtain the viewfinder image captured by the photosensitive device 101, and when a shutter signal is received, it captures the viewfinder image to obtain a captured image. In some embodiments, the ISP module 102 is also configured to perform functions such as automatic exposure, automatic focus, and automatic white balance adjustment.

[0099] The processor 103 is electrically connected to the ISP module 102. Optionally, the processor 103 and the ISP module 102 are connected via a bus, or the processor 103 and the ISP module 102 are integrated into the same electrical package or chip. The processor 103 may include one or more processing cores for transmitting a shutter signal to the ISP module 102, and for acquiring and storing images captured by the ISP module 102. Optionally, the processor 103 includes an image segmentation model; the processor 103 loads and executes executable commands to implement the image processing method provided by the present application. Schematically, the processor 103 identifies the sky area in the image through the image segmentation model and obtains a probability distribution map of the sky area, wherein the probability distribution map is the confidence of the pixels in the sky area; secondly, a high confidence area with a confidence higher than a first confidence threshold is determined from the sky area; thirdly, when the pixels in the high confidence area do not meet the preset conditions, the target sky material is obtained from the sky material stored in the memory 104, and the sky area in the image is replaced with the target sky material to obtain the target image.

[0100] The memory 104 is electrically connected to the processor 103. Optionally, the memory 104 and the processor 103 are connected via a bus. The memory 120 may include RAM (Random Access Memory) or ROM (Read-Only Memory). The memory 104 is used to store pre-set sky materials and images processed by the processor 103. The memory 104 is also used to store programs that are loaded and executed by the processor 103 to implement the image processing method provided in this application.

[0101] In some embodiments, the terminal further includes: an AI (Artificial Intelligence) chip 105. Figure 2 , the AI ​​chip 105 is electrically connected to the ISP module 102. Optionally, the AI ​​chip 105 and the ISP module 102 are connected via a bus. The AI ​​chip includes an image segmentation model; in some embodiments, the AI ​​chip 105 uses the image segmentation model to identify the sky area in the image and obtain a probability distribution map of the sky area.

[0102] The AI ​​chip 105 is also electrically connected to the memory 104. Optionally, the AI ​​chip 105 is connected to the memory 104 via a bus. The AI ​​chip is further configured to determine a high-confidence region in the sky region whose confidence level is higher than a first confidence threshold; secondly, when a pixel in the high-confidence region does not meet a preset condition, obtain a target sky material from the sky material stored in the memory 104, replace the sky region in the image with the target sky material, and obtain a target image; and thirdly, store the target image in the memory 104.

[0103] Please refer to Figure 3 , shows a flowchart of an image processing method provided by an exemplary embodiment of the present application, in which the method is applied to Figure 1 or Figure 2 Taking the terminal shown as an example, the method includes:

[0104] In step 201 , the terminal processes the image through an image segmentation model to obtain a probability distribution map of the sky area in the image.

[0105] The probability distribution diagram of the sky area is the confidence level of the pixels in the sky area.

[0106] Optionally, after the terminal captures the image, it opens the image in the album and uses the image segmentation model to identify the sky area in the image and a probability distribution map of the sky area.

[0107] Optionally, the above-mentioned image segmentation model includes but is not limited to at least one of an FCN model, a CNN model, a DNN model, an RNN model, an embedding model, a GBDT model, and an LR model.

[0108] Schematically, the image segmentation model adopts a deep convolutional neural network model and adds a trainable guided filter to participate in the training to improve the accuracy of the image segmentation model and increase detail information.

[0109] Optionally, the sky in the sky area can be the sky photographed at different times, and / or different locations, and / or different weather types; for example, it can be photographed in the early morning, at noon, at sunrise, at sunset, in the city, on the grassland, on the ocean, on a sunny day, on cloudy weather, and so on.

[0110] Step 202: The terminal determines a high-confidence area in the sky area, the confidence level of which is higher than a first confidence threshold.

[0111] The terminal selects an area in the sky area where the confidence level of pixels is higher than a first confidence threshold and determines it as a high-confidence area. The first confidence threshold indicates the degree to which a pixel in the sky area is a sky-corresponding pixel. When the confidence level of a pixel is higher than the first confidence threshold, the pixel is determined to be a sky-corresponding pixel.

[0112] In step 203 , the terminal determines whether the pixel points in the high confidence area meet the preset conditions.

[0113] The above preset condition is that the pixel in the high confidence region belongs to a dark scene. When the pixel in the high confidence region belongs to a dark scene, the terminal executes step 205; when the pixel in the high confidence region belongs to a bright scene, the terminal executes step 204. A bright scene indicates that the image was taken during the day, which is suitable for sky replacement processing; a dark scene indicates that the image was taken at night, which is not suitable for sky replacement processing.

[0114] Optionally, the above-mentioned preset conditions include:

[0115] The average original color value of the blue channel of the pixel points in the high confidence area is less than the first original color value threshold;

[0116] Alternatively, the average gray value of the pixels in the high confidence area is less than the gray value threshold;

[0117] Alternatively, the average primary color value of the blue channel of the pixels in the high confidence area is less than the first primary color value threshold, and the average grayscale value of the pixels in the high confidence area is less than the grayscale value threshold.

[0118] The first primary color value threshold is used to determine whether a pixel in a high confidence area of ​​the sky area belongs to a bright scene or a dark scene; the gray value threshold is also used to determine whether a pixel in a high confidence area of ​​the sky area belongs to a bright scene or a dark scene.

[0119] Illustratively, when the average primary color value of the blue channel of the pixel points in the high confidence area is less than the first primary color value threshold, the terminal determines that the image is taken at night;

[0120] Alternatively, when the grayscale value of the pixel in the high confidence area is less than the grayscale value threshold, the terminal determines that the image was taken at night;

[0121] Alternatively, when the average primary color value of the blue channel of the pixel point in the high confidence area is less than the first primary color value threshold, and the grayscale value of the pixel point in the high confidence area is less than the grayscale value threshold, the terminal determines that the image is taken at night.

[0122] In step 204 , the terminal replaces the sky area with the target sky material to obtain the target image.

[0123] When pixels in the high-confidence region do not meet the preset conditions, the sky region is replaced with the target sky material to obtain the target image. If the pixels in the high-confidence region do not meet the preset conditions, that is, the pixels in the high-confidence region belong to a bright scene and the image was taken during the day, then the sky region is suitable for replacement processing.

[0124] Optionally, the replacement of the sky area includes the following two schematic steps:

[0125] 1) The terminal segments the edge of the sky area.

[0126] The edge is the connecting portion between the sky area and the non-sky area, and the non-sky area refers to the area other than the sky area in the image.

[0127] Optionally, the terminal segments the edge of the sky area using an edge optimization algorithm. The schematic steps are as follows:

[0128] a. The terminal obtains a histogram of the brightness values ​​of the blue channel of the pixels in the sky area, and reads the minimum brightness value of the blue channel from the histogram, which is recorded as min_blue.

[0129] b. The terminal divides the histogram into at least two regions according to the brightness value represented by the horizontal axis, and determines the target region with the largest number of pixels from the at least two regions. For example, if the histogram is divided into four regions, namely region 1, region 2, region 3, and region 4, and region 4 has the largest number of pixels, then region 4 is determined as the target region.

[0130] c. Determine the minimum brightness value in the target area, recorded as b_margin_min; and the maximum brightness value, recorded as b_margin_max.

[0131] d. Segment the sky area based on min_blue, b_margin_min, b_margin_max and the image.

[0132] Optionally, step d may include at least one of the following processing methods:

[0133] (1) Determine the first region in the image where the brightness value of the blue channel is less than min_blue, and adjust the confidence level in the segmented image corresponding to the first region to 1 / 2 of the original confidence level;

[0134] (2) Determine a second region in the segmented image whose confidence level is higher than a fifth confidence threshold and whose corresponding blue channel brightness value is greater than b_margin_min and less than b_margin_max, and adjust the confidence level in the segmented image corresponding to the second region to 1. The fifth confidence threshold is used to determine the edge region.

[0135] The above operation can sharpen the edges and reduce the probability of small sky areas being erased due to their small probability values.

[0136] Schematically, the edge area includes the edges of leaves, and the edges of the leaves are segmented by an edge optimization algorithm to obtain finely segmented edges; the edge optimization algorithm is used for edge segmentation between the sky area and the non-sky area, which improves the segmentation accuracy of the edges in the image and enables the segmented edge area to achieve a finer degree of segmentation.

[0137] 2) The terminal replaces the processed sky area with the target sky material to obtain the target image.

[0138] Optionally, the terminal obtains attribute parameters of the image and selects the target sky material from the candidate sky materials according to the attribute parameters, wherein the attribute parameters include at least one of the shooting time and shooting location of the image.

[0139] Optionally, the terminal reads the EXIF ​​information of the image as an attribute parameter. For most shooting scenes, an EXIF ​​format file of the image will be obtained after shooting and saving the image. The file header information of the EXIF ​​format file is the EXIF ​​information. The terminal reads the EXIF ​​information of the EXIF ​​format file corresponding to the image, and uses the EXIF ​​information as the attribute parameter. Among them, the EXIF ​​information includes at least one of the aperture, shutter, white balance, sensitivity, focal length, date, time, and location when the image was shot. Therefore, the attribute parameter includes at least one of the shooting time and shooting location of the image.

[0140] In some optional embodiments, the attribute parameters include: shooting time;

[0141] The terminal selects the target sky material corresponding to the time period from the candidate sky materials according to the time period to which the shooting time belongs.

[0142] In some optional embodiments, the attribute parameters include: shooting location;

[0143] The terminal selects a target sky material corresponding to the geographical area to which the shooting location belongs from the candidate sky materials.

[0144] In some optional embodiments, the attribute parameters include: shooting time and shooting location;

[0145] The terminal determines the corresponding weather type according to the shooting time and shooting location; and selects the target sky material corresponding to the weather type from the candidate sky materials.

[0146] Optionally, the candidate sky material may be sky images taken at different times and / or different locations and / or different weather types; for example, it may be taken in the early morning, at noon, at sunrise, at sunset, in a city, on a grassland, on the ocean, on a sunny day, on a cloudy day, and so on.

[0147] Step 205: The terminal saves the image.

[0148] When the pixels in the high confidence area meet the preset conditions, the image is not processed and is saved directly.

[0149] Optionally, before saving the image, the terminal may further process the image using a filter. The filter is used to enhance the display effect of the image, for example, to adjust the image color tone, improve the image texture, etc.

[0150] During the image capture process, many shooting parameters are stored as attribute parameters. Optionally, these attribute parameters can also serve as reference data for the terminal to obtain filters. Based on the attribute parameters, the terminal determines a target filter from candidate filters stored in memory and processes the image using the target filter.

[0151] The attribute parameters also include at least one of aperture, shutter, white balance, sensitivity, and focal length during image capture.

[0152] Optionally, the filter includes at least one of an internal filter, a built-in filter, and an external filter.

[0153] Schematic, reference Figure 4 In the photo album in the terminal, the user selects and opens an image 31, which includes a sky area, that is, the portion enclosed by a dotted line in the image 31; the user triggers the image processing function of replacing the sky area in the image in the photo album. Schematically, the user clicks the control button 32; Figure 4 As shown in the figure below, the terminal processes image 31 using a network segmentation model to obtain a sky region within the image and a probability distribution map of the sky region. This probability distribution map shows the confidence levels of the pixels in the sky region. The terminal then identifies high-confidence regions within the sky region whose confidence levels exceed a first confidence threshold. When a pixel in the high-confidence region represents a bright scene, the sky region in image 31 is automatically replaced with the target sky material to obtain target image 33. A comparison of the sky regions in images 31 and 33 reveals a clear difference. Optionally, the terminal creates a new image file and automatically saves image 33.

[0154] In some embodiments, the triggering method of the image processing function of replacing the sky area in the image in the album can be at least one of a long press operation, a pressure touch operation, a two-finger press operation, a knuckle double-click operation, and a multiple-click operation.

[0155] In other embodiments, a button control is provided on the image display interface for triggering the replacement of the sky area in the image, such as Figure 4 Control button 32 in.

[0156] Optionally, users can customize the target sky material. For example, refer to Figure 5 , the terminal displays image 35 in the album, and below image 35 are also displayed image 36 corresponding to candidate sky material 1 and image 37 corresponding to candidate sky material 2; the user selects candidate sky material 2 in image 37 as the target sky material, and the terminal replaces the sky area of ​​image 35 with candidate sky material 2 to obtain image 38, as shown in FIG. Figure 5 The sky area in image 35 is obviously different from the sky area in image 38; the user triggers the "Save" control, the terminal creates a new image file, and saves image 38.

[0157] In summary, the image processing method provided in the embodiment of the present application processes the image through an image segmentation model to obtain a probability distribution map of the sky area in the image; the probability distribution map shows the confidence of the pixels in the sky area; a high-confidence area with a confidence higher than a first confidence threshold is determined from the sky area; when the pixels in the high-confidence area do not meet the preset conditions, the sky area is replaced with the target sky material to obtain the target image. This solves the problem in the related art that at least 7 steps are required to obtain the target image in which the sky area is replaced with the target sky material, and achieves the purpose of automatically replacing the original sky area in the image with the target sky material. The user does not need to perform post-processing, which reduces the user's manual operation steps and improves the efficiency of human-computer interaction. Even if the user does not have a solid foundation in photo editing, he can still get a sky photo comparable to that taken with a professional SLR lens.

[0158] In addition, the image processing method provided in the embodiment of the present application allows users to filter images according to preset conditions, screen out images that are not suitable for sky area replacement, and enable the terminal to replace the sky area of ​​suitable images, thereby improving the success rate of replacing the sky area in the image and enhancing the user experience.

[0159] It should be noted that before processing the image, the terminal screens the image to exclude pictures that are not suitable for replacing the sky area. Schematically, the image screening methods include the following three:

[0160] First, the whitened image is filtered out using the dark channel algorithm;

[0161] Second, the images with a small proportion of sky area are filtered out;

[0162] Third, the images with blurred demarcation between sky and non-sky areas are filtered out.

[0163] In the first case, based on Figure 3 , add steps 301 to 302 before step 201, such as Figure 6 , the schematic steps are as follows:

[0164] In step 301 , the terminal obtains the average original color value of the dark channel in the image.

[0165] Each pixel in an image has three primary color channels: red, blue, and green. The smallest value among these three channels is the primary color value of the dark channel. For example, if the primary color value of the red channel corresponding to a pixel is 0.5, the primary color value of the blue channel is 0.2, and the primary color value of the green channel is 0.6, then the primary color value of the dark channel corresponding to this pixel is 0.2.

[0166] The terminal first obtains the primary color value of the dark channel corresponding to each pixel in the image. Second, it determines the average primary color value of the dark channel for the pixels in the image. In other words, if there are n pixels in the image, the primary color values ​​of the dark channels of the n pixels are added together to obtain the total. The sum of these primary color values ​​is then divided by n to obtain the average primary color value of the dark channels for the pixels in the image, where n is a positive integer.

[0167] In step 302 , the terminal determines whether the average primary color value of the dark channel is less than a second primary color value threshold.

[0168] Among them, the second primary color value threshold is used to determine an image with clear pixels; when the average primary color value of the dark channel is equal to or greater than the second primary color value threshold, the terminal determines that the image pixels are not clear, for example, the image is overexposed or the picture is white, etc.; when the average primary color value of the dark channel is less than the second primary color value threshold, the terminal determines that the image pixels are clear.

[0169] When the average primary color value of the dark channel is less than the second primary color value threshold, step 201 is executed to replace the sky area with the target sky material to obtain the target image; when the average primary color value of the dark channel is equal to or greater than the second primary color value threshold, step 205 is executed to save the image.

[0170] To sum up, the image processing method provided in this application filters images through the original color values ​​of the dark channel, and screens out images that are not suitable for replacing the sky area, such as snow scenes or overexposed images and other white images, so that the terminal replaces the sky area of ​​suitable images, thereby improving the success rate of replacing the sky area in the image and enhancing the user experience.

[0171] In the second case, based on Figure 3 , add step 401 to step 402 before step 204, such as Figure 7 , the schematic steps are as follows:

[0172] In step 401 , the terminal determines a first ratio of pixels in the sky area to the total pixels in the image.

[0173] Optionally, the terminal counts the number of pixels in the sky area to determine a first value; counts the number of pixels in the image to determine a second value; and divides the first value by the second value to obtain a first proportion of the pixels in the sky area to the total pixels in the image.

[0174] Optionally, the terminal determines the area of ​​the sky area as the first area; determines the area of ​​the image as the second area; and determines the ratio of the first area to the second area as the first proportion of the pixels of the sky area in the total pixels of the image.

[0175] Step 402: The terminal determines whether the first ratio is greater than a first ratio threshold.

[0176] Optionally, the first ratio threshold is used to filter out images that include sky areas; or the first ratio threshold is used to filter out images whose sky areas meet a ratio condition. When the first ratio is greater than the first ratio threshold, the terminal determines that the image includes a sky area, and the ratio of the sky area to the image is greater than the first ratio threshold, that is, the size of the sky area in the image is suitable for replacing the sky area. When the first ratio is less than or equal to the first ratio threshold, the terminal determines that the image does not include a sky area; or the terminal determines that the image includes a sky area, but the sky area in the image is too small and is not suitable for replacing the sky area.

[0177] When the first ratio is greater than the first ratio threshold, step 204 is executed to replace the sky area with the target sky material to obtain the target image; when the first ratio is less than or equal to the first ratio threshold, step 205 is executed to save the image.

[0178] To sum up, the image processing method provided by the present application filters images according to the proportion of the sky area in the image, and screens out images with too small a proportion; if the proportion of the sky area in the image is too small, it has little impact on the overall image, and it is not necessary or suitable to replace the sky area, so the terminal replaces the sky area of ​​suitable images, thereby improving the success rate of replacing the sky area in the image and enhancing the user experience.

[0179] In the third case, based on Figure 3 Add steps 501 to 503 before step 204, such as Figure 8 , the schematic steps are as follows:

[0180] In step 501, the terminal determines an area in the non-sky area where the confidence level of the pixel points is higher than the second confidence threshold and lower than the third confidence threshold as a first fuzzy area; and determines an area in the sky area where the confidence level of the pixel points is higher than the third confidence threshold and lower than the fourth confidence threshold as a second fuzzy area.

[0181] The terminal further stores a second confidence threshold, a third confidence threshold, and a fourth confidence threshold, wherein the second confidence threshold < the third confidence threshold < the fourth confidence threshold.

[0182] The third confidence threshold is used to determine pixels in the sky area and pixels in the non-sky area of ​​the image. When the confidence of a pixel is greater than the third confidence threshold, the terminal determines the pixel as a pixel in the sky area; when the confidence of a pixel is less than or equal to the third confidence threshold, the terminal determines the pixel as a pixel in the non-sky area.

[0183] The greater the confidence of the first pixel point is than the third confidence threshold, the greater the probability that the first pixel point is a pixel point in the sky area; conversely, the closer the confidence of the first pixel point is to the third confidence threshold, the smaller the probability that the first pixel point is a pixel point in the sky area, and the first pixel point is a blurred pixel point in the sky area.

[0184] The fourth confidence threshold is used to determine the blurred pixel points in the sky area; when the first pixel point is less than the fourth confidence threshold, the first pixel point is a blurred pixel point; otherwise, the first pixel point is a clear pixel point.

[0185] The smaller the confidence of the second pixel is than the third confidence threshold, the greater the probability that the second pixel is a pixel in the non-sky area; conversely, the closer the confidence of the second pixel is to the third confidence threshold, the smaller the probability that the pixel is a pixel in the sky area, and the second pixel is a blurred pixel in the sky area.

[0186] The second confidence threshold is used to determine the blurred pixel points in the non-sky area; when the second pixel point is greater than the second confidence threshold, the second pixel point is a blurred pixel point; otherwise, the second pixel point is a clear pixel point.

[0187] The first pixel point is a pixel point in the sky area of ​​the image; and the second pixel point is a pixel point in the non-sky area of ​​the image.

[0188] Optionally, the terminal determines an area where the confidence of the second pixel point is higher than the second confidence threshold and lower than the third confidence threshold as a first fuzzy area; the first fuzzy area is an area in the non-sky area where the fuzzy pixel point is located.

[0189] The terminal determines an area where the confidence of the first pixel point is higher than the third confidence threshold and lower than the fourth confidence threshold as a second fuzzy area; the second fuzzy area is the area where the fuzzy pixel point is located in the sky area.

[0190] In step 502 , the terminal determines a second ratio of pixels of the first fuzzy area and the second fuzzy area in the image.

[0191] Optionally, the terminal counts the total number of pixels in the first blurred area and the second blurred area to determine a third value; counts the number of pixels in the image to determine a second value; and divides the third value by the second value to obtain a second ratio, which is the ratio of the pixels in the first blurred area and the second blurred area in the image.

[0192] Step 503: The terminal determines whether the second ratio is less than a second ratio threshold.

[0193] The second ratio threshold is used to determine an image in which the sky area and the non-sky area are clearly divided; when the second ratio is less than the second ratio threshold, the terminal determines that the sky area and the non-sky area in the image are clearly divided, and the sky area can be clearly identified; when the second ratio is equal to or greater than the second ratio threshold, the terminal determines that the division between the sky area and the non-sky area in the image is blurred, and the sky area cannot be clearly identified.

[0194] When the second ratio is less than the second ratio threshold, step 204 is executed to replace the sky area with the target sky material to obtain the target image; when the second ratio is equal to or greater than the second ratio threshold, step 205 is executed to save the image.

[0195] To sum up, the image processing method provided by the present application determines the blurred area, judges the proportion of the blurred area in the image, and filters out images with large proportions, that is, filters out images with large blurred areas. Images with large blurred areas have too many blurred pixels in the sky area and non-sky area, which directly affects the clarity of the division between the sky area and the non-sky area, and is not conducive to the replacement of the sky area in the image; filtering out images with a large proportion of blurred areas in the image enables the terminal to replace the sky area of ​​suitable images, thereby improving the success rate of replacing the sky area in the image and enhancing the user experience.

[0196] In some embodiments, the terminal may combine at least two of the above three situations to filter the image. Schematically, the image processing method provided by the present application is illustrated by taking the terminal combining the above three situations as an example. Figure 9 , the schematic steps are as follows:

[0197] In step 601, the terminal determines whether the image is whitish by using a dark channel algorithm.

[0198] Optionally, the terminal obtains the original color value of the dark channel of each pixel in the image, divides the sum of the original color values ​​of the dark channels of the pixels in the image by the total number of pixels in the image, and obtains the average original color value of the dark channel in the image.

[0199] The terminal determines whether the average primary color value of the above-mentioned dark channel is less than the second primary color value threshold; when the average primary color value of the dark channel is less than the second primary color value threshold, it indicates that the image is not whitish, and step 602 is executed to replace the sky area with the target sky material to obtain the target image; when the average primary color value of the dark channel is equal to or greater than the second primary color value threshold, step 608 is executed to not replace the sky area in the image.

[0200] In step 602, the terminal performs image segmentation using an image segmentation model to obtain a probability distribution map of the sky area in the image.

[0201] Optionally, the image segmentation model adopts a deep convolutional neural network model and adds a trainable guided filter to participate in the training to improve the accuracy of the image segmentation model and increase detail information.

[0202] The probability distribution graph shows the confidence of the pixels in the sky area.

[0203] In step 603, the terminal selects an area with a high sky confidence from the sky area, and determines whether the shooting scene of the image is at night based on the area with a high sky confidence.

[0204] The terminal determines the area corresponding to the pixel points in the sky area whose confidence level is higher than the first confidence threshold as an area with high sky confidence level, that is, a high confidence area.

[0205] Optionally, for pixels in the high confidence area, the terminal obtains the average primary color value of the blue channel of the pixel and the average grayscale value of the pixel; when the average primary color value of the blue channel of the pixel in the high confidence area is less than the first primary color value threshold, and the average grayscale value of the pixel in the high confidence area is less than the grayscale value threshold, the terminal determines that the shooting scene of the image is night, that is, the image is taken at night.

[0206] Alternatively, the terminal obtains the average primary color value of the blue channel of the pixel point; when the average primary color value of the blue channel of the pixel point in the high confidence area is less than the first primary color value threshold, the terminal determines that the shooting scene of the image is night, that is, the image is taken at night.

[0207] Alternatively, the terminal obtains the average grayscale value of the pixels; when the average grayscale value of the pixels in the high confidence area is less than the grayscale value threshold, the terminal determines that the shooting scene of the image is night, that is, the image is shot at night.

[0208] When the shooting scene of the image is night, step 608 is executed; when the shooting scene of the image is not night, step 604 is executed.

[0209] In step 604, the terminal determines whether the proportion of the sky area in the image is less than 10%.

[0210] Optionally, the terminal determines the proportion of the sky area in the image by the ratio of the number of pixels in the sky area to the number of pixels in the image.

[0211] Alternatively, the terminal determines the proportion of the sky area in the image by calculating the ratio of the area of ​​the sky area to the area of ​​the image.

[0212] When the proportion of the sky area in the image is less than 10%, step 608 is executed; when the proportion of the sky area in the image is equal to or greater than 10%, step 605 is executed.

[0213] Step 605 , determining whether the proportion of the blurred area after image segmentation in the image is higher than a threshold.

[0214] The segmented image includes a sky area and a non-sky area, and the fuzzy area includes a first fuzzy area corresponding to the non-sky area and a second fuzzy area included in the sky area.

[0215] The terminal determines the area in the non-sky area where the confidence of the pixel points is higher than the second confidence threshold and lower than the third confidence threshold as the first blurred area; and determines the area in the sky area where the confidence of the pixel points is higher than the third confidence threshold and lower than the fourth confidence threshold as the second blurred area.

[0216] Optionally, the terminal determines the proportion of the blurred area in the image by the ratio of the number of pixels in the blurred area to the number of pixels in the image.

[0217] Alternatively, the terminal determines the proportion of the blurred area in the image by the ratio of the area of ​​the blurred area to the area of ​​the image.

[0218] When the proportion of the blurred area in the image is higher than the threshold, step 608 is executed; when the proportion of the uncertain area in the image is lower than or equal to the threshold, step 606 is executed.

[0219] In step 606, the terminal segments the connected portion of the sky area and the non-sky area.

[0220] The terminal uses an edge optimization algorithm to segment the connected parts of the sky area and the non-sky area.

[0221] Step 607: Replace the sky area with the target sky material to obtain the target image.

[0222] Optionally, the replacement of the sky area may include the following exemplary steps:

[0223] 1) Obtaining attribute parameters of the image;

[0224] 2) Filter out the target sky material from the candidate sky materials according to the attribute parameters.

[0225] The attribute parameters include at least one of the shooting time and shooting location of the image.

[0226] Step 608: Do not replace the sky area in the image.

[0227] In summary, the image processing method provided in the embodiment of the present application processes the image through an image segmentation model to obtain a probability distribution map of the sky area in the image; the probability distribution map shows the confidence of the pixels in the sky area; a high-confidence area with a confidence higher than a first confidence threshold is determined from the sky area; when the pixels in the high-confidence area do not meet the preset conditions, the sky area is replaced with the target sky material to obtain the target image. This solves the problem in the related art that at least 7 steps are required to obtain the target image in which the sky area is replaced with the target sky material, and achieves the purpose of automatically replacing the original sky area in the image with the target sky material. The user does not need to perform post-processing, which reduces the user's manual operation steps and improves the efficiency of human-computer interaction. Even if the user does not have a solid foundation in photo editing, he can still get a sky photo comparable to that taken with a professional SLR lens.

[0228] In addition, the image processing method provided in the embodiment of the present application allows users to filter images based on factors such as the average original color value of the dark channel, the proportion of the sky area in the image, and the proportion of the blurred area in the image, thereby filtering out images that are not suitable for replacing the sky area, allowing the terminal to replace the sky area of ​​suitable images, thereby improving the success rate of replacing the sky area in the image and enhancing the user experience.

[0229] Please refer to Figure 10 , shows a block diagram of an image processing device provided by an exemplary embodiment of the present application. The device can be implemented as part or all of a terminal through software, hardware, or a combination of both. The device includes:

[0230] The processing module 701 is configured to process the image using an image segmentation model to obtain a probability distribution map of the sky area in the image; the probability distribution map is the confidence level of the pixels in the sky area;

[0231] The determination module 702 is configured to determine a high confidence region having a confidence level higher than a first confidence threshold from the sky region;

[0232] The replacement module 703 is configured to replace the sky area with the target sky material to obtain the target image when the pixel points in the high confidence area do not meet the preset conditions.

[0233] In some embodiments, the replacement module 703 includes:

[0234] The segmentation submodule 7031 is configured to perform segmentation processing on the edge of the sky area;

[0235] The replacement submodule 7032 is configured to replace the segmented sky area with the target sky material to obtain a target image.

[0236] In some embodiments, the replacement submodule 7032 is configured to obtain attribute parameters of the image; filter out the target sky material from the candidate sky materials according to the attribute parameters; wherein the attribute parameters include at least one of the shooting time and shooting location of the image.

[0237] In some embodiments, the preset conditions include:

[0238] The average original color value of the blue channel of the pixel points in the high confidence area is less than the first original color value threshold;

[0239] Alternatively, the average gray value of the pixels in the high confidence area is less than the gray value threshold;

[0240] Alternatively, the average primary color value of the blue channel of the pixels in the high confidence area is less than the first primary color value threshold, and the average grayscale value of the pixels in the high confidence area is less than the grayscale value threshold.

[0241] In some embodiments, the apparatus comprises:

[0242] An acquisition module 704 is configured to acquire an average original color value of a dark channel in an image;

[0243] The determination module 702 is configured to execute the step of processing the image through the image segmentation model to obtain a probability distribution map of the sky area in the image when the average primary color value of the dark channel is less than the second primary color value threshold.

[0244] In some embodiments, the determination module 702 is configured to determine a first proportion of the pixels of the sky area in the total pixels of the image; when the first proportion is greater than a first proportion threshold, the step of replacing the sky area with the target sky material to obtain the target image is executed.

[0245] In some embodiments, the determination module 702 is configured to determine an area in the non-sky area where the confidence level of the pixel points is higher than the second confidence threshold and lower than the third confidence threshold as a first blurred area; and determine an area in the sky area where the confidence level of the pixel points is higher than the third confidence threshold and lower than the fourth confidence threshold as a second blurred area;

[0246] Determination module 702 is configured to determine a second ratio of pixels of the first blurred area and the second blurred area in the image; when the second ratio is less than a second ratio threshold, execute the step of replacing the sky area with the target sky material to obtain the target image; wherein the non-sky area refers to the area in the image other than the sky area.

[0247] In some embodiments, the processing module 701 is further configured to process the image using a filter when the pixel points in the high confidence area meet preset conditions; the filter is used to enhance the display effect of the image.

[0248] In summary, the image processing device provided in the embodiment of the present application processes the image through an image segmentation model to obtain a probability distribution map of the sky area in the image; the probability distribution map shows the confidence of the pixels in the sky area; a high-confidence area with a confidence higher than a first confidence threshold is determined from the sky area; when the pixels in the high-confidence area do not meet the preset conditions, the sky area is replaced with the target sky material to obtain the target image. This solves the problem in the related art that at least 7 steps are required to obtain the target image in which the sky area is replaced with the target sky material, and achieves the purpose of automatically replacing the original sky area in the image with the target sky material. The user does not need to perform post-processing, which reduces the user's manual operation steps and improves the efficiency of human-computer interaction. Even if the user does not have a solid foundation in photo editing, he can still get a sky photo comparable to that taken with a professional SLR lens.

[0249] In addition, the image processing device provided in the embodiment of the present application allows users to filter images according to preset conditions, screen out images that are not suitable for sky area replacement, and enable the terminal to replace the sky area of ​​suitable images, thereby improving the success rate of replacing the sky area in the image and enhancing the user experience.

[0250] Figure 11 FIG8 is a block diagram of an image processing apparatus 800 according to an exemplary embodiment of the present invention. For example, the apparatus 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0251] Reference Figure 11 , the device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0252] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 802 may include one or more processors 818 to execute instructions to complete all or part of the steps performed by the UE 20 in the above-described method embodiment. In addition, the processing component 802 may include one or more modules to facilitate interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

[0253] The memory 804 is configured to store various types of data to support the operations of the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0254] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device 800.

[0255] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have focal length and optical zoom capabilities.

[0256] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), which is configured to receive external audio signals when the device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0257] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include but are not limited to: a home button, volume buttons, a start button, and a lock button.

[0258] The sensor assembly 814 includes one or more sensors for providing various aspects of the status assessment of the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800. The sensor assembly 814 can also detect changes in the position of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and temperature changes of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0259] The communication component 816 is configured to facilitate wired or wireless communication between the apparatus 800 and other devices. The apparatus 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel.

[0260] In an exemplary embodiment, the device 800 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the image processing method in the above method embodiments.

[0261] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions. The instructions can be executed by the processor 818 of the apparatus 800 to perform the image processing method in the above method embodiment. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0262] In an exemplary embodiment, a computer-readable storage medium is also provided. The computer-readable storage medium is a non-volatile computer-readable storage medium, which stores a computer program. When the stored computer program is executed by a processing component, it can implement the image processing method provided in the above embodiment of the present disclosure.

[0263] The embodiment of the present disclosure further provides a computer program product, in which instructions are stored. When the computer program product is run on a computer, the computer is enabled to execute the image processing method provided by the embodiment of the present disclosure.

[0264] An embodiment of the present disclosure further provides a chip, which includes a programmable logic circuit and / or program instructions, and when the chip is running, it can execute the image processing method provided by the embodiment of the present disclosure.

[0265] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0266] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: The method comprises: Get the average original color value of the dark channel in the image; When the average primary color value of the dark channel is less than a second primary color value threshold, the image is processed using an image segmentation model to obtain a probability distribution map of a sky area in the image; the probability distribution map is the confidence level of the pixel points in the sky area; Determine a high confidence region from the sky region, where the confidence level is higher than a first confidence threshold; Acquire attribute parameters of the image, wherein the attribute parameters include shooting time and shooting location; determining a weather type corresponding to the image according to the shooting time and the shooting location; Filtering target sky materials from candidate sky materials according to the weather type; When the pixel in the high confidence area does not meet a preset condition, determining a histogram corresponding to the brightness value of the blue channel of the pixel in the sky area, wherein the preset condition is that the average primary color value of the blue channel of the pixel in the high confidence area is less than a first primary color value threshold, and the average grayscale value of the pixel in the high confidence area is less than a grayscale value threshold; Reading a first minimum value corresponding to the brightness value of the blue channel from the histogram; Dividing the histogram into at least two regions on average according to the brightness values ​​represented by the abscissa axis, and determining the target region with the largest number of pixels from the at least two regions; determining a second minimum value of the brightness value in the target area and a maximum value of the brightness value; Determine a first region in the image where the brightness value of the blue channel is less than the first minimum value, and adjust the confidence level in the segmented image corresponding to the first region to 1 / 2 of the original confidence level; determining a second region in the segmented image having a confidence level higher than a fifth confidence threshold and a brightness value of the blue channel in the segmented image greater than the second minimum value and less than the maximum value, and adjusting the confidence level in the segmented image corresponding to the second region to 1, wherein the fifth confidence threshold is used to determine an edge region; Replacing the segmented sky area with the target sky material to obtain a target image; Acquiring attribute parameters of the image, where the attribute parameters are determined according to shooting parameters of the image during shooting; According to the attribute parameters, a target filter is determined from preset stored candidate filters, and the target image is processed using the target filter, wherein the target filter is used to enhance the display effect of the target image.

2. The method according to claim 1, characterized in that The method comprises: Determine a first ratio of the pixels of the sky area to the total pixels of the image; When the first ratio is greater than a first ratio threshold, the step of replacing the sky area with a target sky material to obtain a target image is performed.

3. The method according to claim 1, characterized in that The method further comprises: Determine an area in the non-sky area where the confidence level of the pixel points is higher than the second confidence threshold and lower than the third confidence threshold as a first blurred area; and determine an area in the sky area where the confidence level of the pixel points is higher than the third confidence threshold and lower than the fourth confidence threshold as a second blurred area; determining a second ratio of pixels of the first blurred area and the second blurred area in the image; When the second ratio is less than a second ratio threshold, performing the step of replacing the sky area with a target sky material to obtain a target image; The non-sky area refers to an area in the image other than the sky area.

4. An image processing device, characterized in that The device comprises: An acquisition module is configured to obtain an average original color value of a dark channel in an image; a processing module configured to, when the average primary color value of the dark channel is less than a second primary color value threshold, process the image using an image segmentation model to obtain a probability distribution map of a sky area in the image; wherein the probability distribution map is a confidence level of a pixel point in the sky area; a determination module configured to determine, from the sky area, a high confidence area having a confidence level higher than a first confidence threshold; The segmentation submodule is configured to obtain attribute parameters of the image, wherein the attribute parameters include shooting time and shooting location; determine the weather type corresponding to the image according to the shooting time and the shooting location; filter out the target sky material from the candidate sky material according to the weather type; when the pixel point in the high confidence area does not meet the preset conditions, determine the histogram corresponding to the brightness value of the blue channel of the pixel point in the sky area, and the preset conditions are that the average original color value of the blue channel of the pixel point in the high confidence area is less than the first original color value threshold, and the average grayscale value of the pixel point in the high confidence area is less than the grayscale value threshold; read the first minimum value corresponding to the brightness value of the blue channel from the histogram; and divide the histogram into two groups according to the horizontal coordinates. The brightness value represented by the axis is evenly divided into at least two areas, and the target area with the largest number of pixels is determined from the at least two areas; the second minimum value and the maximum value of the brightness value in the target area are determined; a first area in which the brightness value of the blue channel in the image is less than the first minimum value is determined, and the confidence level in the segmented image corresponding to the first area is adjusted to 1 / 2 of the original value; a second area in which the confidence level in the segmented image is higher than a fifth confidence threshold and the brightness value of the blue channel in the segmented image is greater than the second minimum value and less than the maximum value is determined, and the confidence level in the segmented image corresponding to the second area is adjusted to 1, and the fifth confidence threshold is used to determine the edge area; a replacement submodule, configured to replace the segmented sky area with the target sky material to obtain a target image; a module for acquiring attribute parameters of the image, wherein the attribute parameters are determined according to shooting parameters of the image during shooting; The processing module is further configured to determine a target filter from preset stored candidate filters according to the attribute parameters, and process the target image using the target filter, wherein the target filter is configured to enhance the display effect of the target image.

5. The device according to claim 4, characterized in that The determination module is configured to determine a first proportion of the pixels of the sky area to the total pixels of the image; when the first proportion is greater than a first proportion threshold, execute the step of replacing the sky area with the target sky material to obtain the target image.

6. The device according to claim 4, characterized in that The determining module is configured to determine an area in the non-sky area where the confidence level of the pixel points is higher than the second confidence threshold and lower than the third confidence threshold as a first blurred area; Determine an area in the sky area where the confidence level of pixels is higher than the third confidence threshold and lower than the fourth confidence threshold as a second fuzzy area; The determination module is configured to determine a second ratio of pixels of the first blurred area and the second blurred area in the image; when the second ratio is less than a second ratio threshold, perform the step of replacing the sky area with a target sky material to obtain a target image; wherein the non-sky area refers to the area in the image other than the sky area.

7. A terminal, characterized in that: The terminal includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the image processing method according to any one of claims 1 to 3.

8. A computer-readable storage medium, characterized in that The readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the image processing method as described in any one of claims 1 to 3.

Images