Image generation method and apparatus, electronic device, and computer readable medium

Abstract

Embodiments of the present disclosure disclose image generation methods, devices, electronic equipment and computer readable media. A specific implementation of the method comprises: classifying pixels of a to-be-processed image to determine at least one pixel type; determining at least one type region through the at least one pixel type; for a type region in the at least one type region, adjusting the color of the type region to a target color to obtain a target color region corresponding to the type region; and combining at least one target color region corresponding to the at least one type region into a target image. This implementation improves the smoothness of the color of the target image, and is also conducive to reducing noise.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of image processing technology, and more specifically to image generation methods, apparatus, electronic devices, and computer-readable media. Background Technology

[0002] Images typically contain a variety of colors and content, conveying rich information. To further improve the relevance of images to scenes and meet scene requirements, the colors of existing images can be adjusted. Existing methods for image color adjustment often have the following shortcomings: they do not adjust colors in sections according to the actual characteristics of the image, which can easily lead to problems such as noise, abrupt and uneven colors after overall image adjustment. Summary of the Invention

[0003] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0004] Some embodiments of this disclosure provide image generation methods, apparatuses, electronic devices, and computer-readable media to address the technical problems mentioned in the background section above.

[0005] In a first aspect, some embodiments of this disclosure provide an image generation method, the method comprising: classifying pixels of an image to be processed to determine at least one pixel type; determining at least one type region based on the at least one pixel type; adjusting the color of a type region within the at least one type region to a target color to obtain a target color region corresponding to the type region; and combining the at least one target color region corresponding to the at least one type region into a target image.

[0006] Secondly, some embodiments of this disclosure provide an image generation apparatus, which includes: a pixel type classification unit configured to classify pixels of an image to be processed and determine at least one pixel type; a type region determination unit configured to determine at least one type region based on the at least one pixel type; a target color adjustment unit configured to adjust the color of a type region within the at least one type region to a target color to obtain a target color region corresponding to the type region; and a target image generation unit configured to combine at least one target color region corresponding to the at least one type region into a target image.

[0007] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0008] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0009] The above embodiments of this disclosure have the following beneficial effects: the target image obtained by the image generation method of some embodiments of this disclosure has improved smoothness. Specifically, the reason why the target image is not smooth enough is that the color is not adjusted specifically for the pixel type and type region of the image to be processed. Based on this, the image generation method of some embodiments of this disclosure first classifies the pixels of the image to be processed into multiple pixel types. Classification at the pixel level can greatly improve the effectiveness of color adjustment. Then, multiple type regions are determined based on multiple pixel types, and each type region is adjusted to the corresponding target color region. In this way, the color is adjusted independently for different type regions, improving the targeting of color adjustment. Also, because the color is adjusted independently for each type region, the final target image not only achieves overall color adjustment, but also achieves targeted adjustment for each type region. Thus, the smoothness of the target image color is greatly improved, and noise is also reduced. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0011] Figure 1 This is a schematic diagram illustrating application scenarios of some embodiments of the image generation method disclosed herein;

[0012] Figure 2 This is a flowchart of some embodiments of the image generation method according to the present disclosure;

[0013] Figure 3 This is a flowchart of some other embodiments of the image generation method according to the present disclosure;

[0014] Figure 4 This is a flowchart of yet another embodiment of the image generation method according to the present disclosure;

[0015] Figure 5 These are schematic diagrams illustrating the structure of some embodiments of the image generation apparatus according to the present disclosure;

[0016] Figure 6 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure;

[0017] Figure 7 It is a diagram of the color block structure within the changing area;

[0018] Figure 8 It corresponds Figure 7 A structural diagram of proximity relationships;

[0019] Figure 9 It is a structural diagram of the proximity relationships of a neighbor-matching image in a neighbor-matching image library. Detailed Implementation

[0020] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0021] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0022] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0023] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0024] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0025] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0026] Figure 1 This is a schematic diagram illustrating an application scenario of an image generation method according to some embodiments of the present disclosure.

[0027] After receiving the image 102 to be processed, the electronic device 101 can classify each pixel contained in the image 102 to obtain at least one pixel type 103. For example, pixel type 103 can be solid color type, gradient type, highlight type, shadow type, texture type, and noise type, etc. Among them, solid color type can be a pixel whose color is the same as the color of surrounding pixels; gradient type can be a pixel whose color is similar to the color of surrounding pixels; highlight type can be a pixel whose brightness exceeds a set first brightness threshold; shadow type can be a pixel whose brightness is below a set second brightness threshold; texture type can be a pixel that contains texture features; noise type can be a pixel that belongs to a pixel with a high probability of noise. Then, the electronic device 101 further analyzes each pixel type 103 to determine at least one type region 104 corresponding to at least one pixel type 103. For example, type region 104 can be a change region, a stable region, an adjustment region, and a deletion region. Among them, change region can be an image region used for color adjustment; stable region can be considered an image region that does not require color adjustment; adjustment region can be an image region whose color is adjusted according to the features of the image to be processed; deletion region can be an image region where the image content needs to be reduced. After adjusting each type region 104 to its corresponding target color region, the execution entity combines these target color regions to form the target image. In this way, the pixel type and type region of the image 102 to be processed are specifically considered, and color adjustment is performed independently for each type region. This reduces noise in the target image after color adjustment of the image 102 to be processed and improves the smoothness of colors within the target image.

[0028] It should be understood that Figure 1 The number of electronic devices 101 shown is merely illustrative. Any number of electronic devices 101 may be used depending on the implementation requirements.

[0029] Continue to refer to Figure 2 , Figure 2 A flow 200 of some embodiments of an image generation method according to the present disclosure is shown. The image generation method includes the following steps:

[0030] Step 201: Classify the pixels of the image to be processed and determine at least one pixel type.

[0031] In some embodiments, the execution subject of the image generation method (e.g. Figure 1The electronic device 101 shown can receive the image to be processed via a wired connection or a wireless connection. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G / 5G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultrawideband) connections, and other currently known or future wireless connection methods.

[0032] To analyze the image to be processed in a targeted manner, the executing entity can analyze the pixels of the image and classify them into various different pixel types. For example, the executing entity can classify pixels into adjustable pixels or non-adjustable pixels based on conditions such as brightness and contrast. This pixel classification enables accurate and effective adjustment of the color of the image to be processed at the pixel level.

[0033] Step 202: Determine at least one type of region using at least one of the above pixel types.

[0034] In some embodiments, image regions composed of pixels corresponding to different pixel types can be processed using different color methods. That is, image regions composed of pixels of different pixel types can be different types of regions. For example, an image region composed of adjustable pixels is an image region whose color can be adjusted; an image region composed of non-adjustable pixels is an image region whose color cannot be adjusted, and so on. This achieves region division of the image to be processed, improving the targeting and effectiveness of color adjustment.

[0035] Step 203: For a type region in at least one of the above type regions, adjust the color of the type region to the target color to obtain the target color region corresponding to the type region.

[0036] The executing entity can adjust the color of each type of region to the corresponding target color to obtain the corresponding target color region. In this way, the abruptness of the color after adjustment is greatly reduced.

[0037] Step 204: Combine at least one target color region corresponding to at least one of the above-mentioned types of regions into a target image.

[0038] The target color region is obtained by adjusting the color of the type region. Therefore, the target image, which is formed by combining the various target color regions, achieves local color adjustment of the image to be processed, thus improving the overall effect of color adjustment.

[0039] The target image obtained by the image generation method of some embodiments of this disclosure has improved smoothness. Specifically, the reason for the lack of smoothness in the target image is that the color is not specifically adjusted for the pixel type and type region of the image to be processed. Based on this, the image generation method of some embodiments of this disclosure first classifies the pixels of the image to be processed into multiple pixel types. Classification at the pixel level can greatly improve the effectiveness of color adjustment. Then, multiple type regions are determined based on multiple pixel types, and each type region is specifically adjusted to the corresponding target color region. In this way, the color is adjusted independently for different type regions, improving the targeting of color adjustment. Because the color is adjusted independently for each type region, the final target image not only achieves overall color adjustment, but also achieves targeted adjustment for each type region. Thus, the smoothness of the target image color is greatly improved, and noise is also reduced.

[0040] Continue to refer to Figure 3 , Figure 3 A flow 300 of some embodiments of the image generation method according to the present disclosure is shown. The image generation method includes the following steps:

[0041] Step 301: For the pixels of the image to be processed, the pixel is imported into a preset fully convolutional network to obtain the probability that the pixel belongs to each specified pixel type in the specified pixel type set, and the specified pixel type with the highest probability value is set as the pixel type of the pixel.

[0042] The execution body of the image generation method (e.g.) Figure 1 The electronic device 101 shown can import pixels of the image to be processed into a preset fully convolutional network. The fully convolutional network can be trained using sample pixels of various specified pixel types. After receiving a pixel, the fully convolutional network can calculate the probability that the pixel belongs to each specified pixel type in a set of specified pixel types. This set of specified pixel types can include: solid color, gradient, highlight, shadow, texture, and noise. The executing entity can set the specified pixel type with the highest probability value as the pixel type of the pixel. In this way, pixel classification is achieved.

[0043] Step 302: Determine at least one type of region using at least one of the above pixel types.

[0044] In some optional implementations of some embodiments, determining at least one type region by the above-described at least one pixel type may include: setting an image region composed of pixels of the above-described solid color type and gradient type as a change region; setting an image region composed of pixels of the above-described highlight type and shadow type as a stable region; setting an image region composed of pixels of the above-described highlight type, shadow type and texture type as an adjustment region; and setting an image region composed of pixels of the above-described noise type as a deletion region.

[0045] Solid color and gradient pixels are easily affected by the target color; the execution entity can designate the image region composed of these pixels as a variation region. Highlight and shadow pixels are not sensitive to color adjustments; the execution entity can designate the image region composed of these pixels as a stable region. Image regions composed of a mixture of highlight, shadow, and texture pixels are more likely to reflect the texture features of the image being processed during color adjustment; the execution entity can designate the image region composed of these pixels as an adjustment region. Noise pixels are prone to producing noise points after color adjustment; the execution entity can designate the image region composed of these noise pixels as a deletion region.

[0046] Step 303: For a type region in at least one of the above type regions, adjust the color of the type region to the target color to obtain the target color region corresponding to the type region.

[0047] In some optional implementations of some embodiments, adjusting the color of a type region to a target color to obtain a target color region corresponding to the type region for at least one type region includes: in response to the type region being a variable region and the target color being a monochrome, adjusting the color of the type region to the monochrome to obtain a first target color region.

[0048] When the type region is a variable region and the target color is a single color, the executing entity can directly adjust the color of the type region to the aforementioned single color to obtain the first target color region.

[0049] In some optional implementations of certain embodiments, adjusting the color of a type region within at least one type region to a target color to obtain a target color region corresponding to that type region includes:

[0050] The first step is to perform color clustering on the region of this type, which is a variable region and the target color is multi-colored, to obtain at least one color patch.

[0051] When the type region is a variable region and the target color is multi-colored, in order to ensure the smoothness of the adjusted color, the execution entity can perform color clustering on the type region to obtain at least one color patch. A color patch can be considered as a small image region composed of identical or similar pixels.

[0052] The second step is to determine the proximity relationship between at least one of the aforementioned color blocks.

[0053] The changing region consists of pixels of solid color and gradient type; therefore, the resulting multiple color patches have similarities. The execution entity can determine the proximity relationships between multiple color patches, on a per-patch basis. Proximity relationships can be considered as overlaps or inclusions between different color patches.

[0054] The third step is to set the color of the target image in the preset neighbor matching image library that has the same proximity relationship as the above-mentioned target image as the target color of the region of that type, so as to obtain the second target color region.

[0055] The pre-defined neighbor matching image library contains neighbor matching images with adjusted colors under various neighbor relationship conditions. The executing entity can set the color of a target image in the neighbor matching image library that has the same neighbor relationship as described above as the target color of that type of region, thus obtaining a second target color region. The target image may contain multiple colors. In this case, the second target color region may contain multiple different color regions. For example, if the target color is red, the second target color region may contain pink. This improves the targeting and effectiveness of color adjustment for type regions, which helps to improve the smoothness of the target image and reduce image abruptness.

[0056] Figure 9 This is a structural diagram of the proximity relationships of a proximity-matching image within a proximity-matching image database. The diagram includes multiple color blocks: a, b, c, d, e, f, g, and h. Each color block is represented by a circle, and the number in the lower right corner of the circle indicates the number of nodes that color block has. For example, the lower right corner number of color block b is 2, indicating that color block b has two nodes: color blocks f and color blocks g. Correspondingly... Figure 8 The color blocks are a, b, c, d, g, and h. Therefore, the colors of the target image corresponding to a, b, c, d, g, and h can be set to... Figure 8 The target color for the medium-sized region.

[0057] In some optional implementations of certain embodiments, determining the proximity relationship between the at least one color patch may include:

[0058] The first step is to generate the boundary line of the region to be processed in response to the aforementioned change region, which includes a region to be processed composed of pixels of solid color type.

[0059] When the changing region comprises pixels of a solid color, the executing agent can enhance the boundaries of the region to be processed, generating boundary lines for that region. For example, the executing agent can identify the pixel type to determine the boundary of the region to be processed, and then color the pixels at the boundary to obtain the boundary lines.

[0060] The second step is to determine the connected regions between the color blocks using the aforementioned boundary lines.

[0061] Boundary lines define the regions of color blocks. By defining boundary lines, we can determine the overlapping and inclusion relationships between color blocks, and thus determine the connected regions between color blocks.

[0062] For example, the changing area can contain 5 color blocks, and the corresponding color block structure diagram is as follows: Figure 7 As shown. Figure 7 In the diagram, "1" represents the area to be processed; "2" represents the first color block; "3" represents the second color block; "4" represents the third color block; "5" represents the fourth color block; and "6" represents the fifth color block.

[0063] The third step is to determine the proximity relationship between at least one color patch based on the connected regions mentioned above.

[0064] A connected region is associated with at least two color patches. By identifying connected regions, we can determine color patches that overlap or contain each other, and then set the color patches associated with the connected region as having a proximity relationship. This proximity relationship can be an overlap or containment relationship between color patches. Figure 7 It can be seen that color blocks 2, 3, and 6 are relatively independent. Color block 4 overlaps with both color blocks 2 and 3; color block 5 is located within color block 3. The corresponding proximity relationship structure diagram is shown below. Figure 8 As shown.

[0065] In some optional implementations of certain embodiments, adjusting the color of a type region within at least one type region to a target color to obtain a target color region corresponding to that type region includes:

[0066] The first step is to extract the texture features of the adjusted region in response to the region being an adjusted region.

[0067] The adjustment region consists of pixels of highlight, shadow, and texture types. When the type region is an adjustment region, the execution entity can extract the texture features of the aforementioned adjustment region. Texture features can be characterized by the RGB values ​​of the pixels.

[0068] The second step is to set the color of the target texture image in the preset texture image library that has the same texture features as the above as the target color of the region, thus obtaining the third target color region.

[0069] The executing entity can match texture features with texture images in a texture image library to determine the target texture image that matches successfully. Then, the executing entity can set the color of the target texture image to the target color of that type of region, obtaining a third target color region. Here, the target texture image is an image related to the target color of the image to be processed. This improves the targeting and effectiveness of color adjustment for type-regions, which helps to improve the smoothness of the target image and reduce abruptness.

[0070] Specifically, the executing entity can cluster the extracted texture features to obtain multiple different texture clusters. Then, the executing entity can search for the target texture image in the texture image library using the RGB values ​​corresponding to the texture features. The formula for representing the corresponding texture feature points using the Gaussian kernel function is as follows:

[0071]

[0072] The gradient of the Gaussian function can be further calculated using the above formula, as follows:

[0073]

[0074]

[0075]

[0076] Then, Gaussian kernel function accumulation can be performed, and the accumulation formula is as follows:

[0077]

[0078] Where C(r,g,b) is the Gaussian kernel function of r,g,b; r0 is R in the RGB color value of the nearest neighbor texture feature point; g0 is G in the RGB color value of the nearest neighbor texture feature point; and b0 is B in the RGB color value of the nearest neighbor texture feature point.

[0079] To search for texture features similar to texture feature points in a texture image library, the execution entity can update the rgh value corresponding to the texture feature points to find the target texture image. The formula for updating the rgh value is as follows:

[0080]

[0081]

[0082]

[0083] Where t represents the current iteration search; t+1 represents the next iteration search.

[0084] In this way, the texture images in the texture image library can be searched one by one until the target texture image is found.

[0085] In some optional implementations of certain embodiments, adjusting the color of a type region within at least one type region to a target color to obtain a target color region corresponding to that type region includes:

[0086] The first step is to fit the region of this type to obtain the fitted image region, in response to the region being a region to be deleted.

[0087] When the type region is a deletion region, since the deletion region is mainly composed of noise-type pixels, the color cannot be directly adjusted based on the deletion region. The execution entity can first fit the type region to obtain a fitted image region. The fitted image region can be considered as the image region after noise reduction.

[0088] The second step is to overlay the fitted image region with this type of region to obtain the fourth target color region.

[0089] The execution entity can overlay the fitted image region with this type of region, thereby reducing or eliminating noise in the fourth target color region.

[0090] In some optional implementations of certain embodiments, the above-described fitting of the type of region to obtain the fitted image region may include:

[0091] The first step is to obtain the noise probability of each pixel within the region of this type, thus obtaining the noise probability matrix corresponding to the region of this type.

[0092] The execution entity can obtain the noise probability of each pixel within the type region, and then construct a noise probability matrix according to the coordinates of the pixels within the type region or the positional relationship between pixels.

[0093] The second step is to sum the noise probabilities in the above noise probability matrix to obtain the noise probability field.

[0094] The executing entity can accumulate the noise probabilities within the noise probability matrix using methods such as the Gaussian kernel function to obtain the noise probability field. The Gaussian kernel function can be used to calculate the Euclidean distance between each noise pixel and the midpoint of the type region, which in this application can be used to characterize the difference between two noise pixels.

[0095] The formula for calculating the noise probability field is as follows:

[0096]

[0097] Where F(x,y) is the probability field of the pixel with x-coordinate and y-coordinate; NOISE (x,y) G(x,y) represents the probability that a pixel is noise, as determined by the preceding FCN network; G(x,y) is the Gaussian kernel function; σ(NOISE) represents the probability that a pixel is noise. (x,y) G(x,y)) is the activation function mapping function for all noise fields.

[0098] The formula for calculating the Gaussian kernel function is as follows:

[0099]

[0100] Where σ is the variance of the Gaussian function.

[0101] The third step is to calculate the image gradient of the pixels in this type of region to obtain the gradient field of this type of region.

[0102] The executing entity can further calculate the image gradient of the pixels in the type region to obtain the gradient field of that type region. The image gradient refers to the rate of change of a pixel in the image along the x and y directions (compared to neighboring pixels). It is a two-dimensional vector composed of two components: the change along the x-axis and the change along the y-axis.

[0103] The formula for calculating the image gradient is:

[0104]

[0105] Among them, G xy is the gradient value of the pixel with x as the horizontal coordinate and y as the vertical coordinate; thresh is the threshold value for gradient thresholding.

[0106] Calculate the image gradient for each pixel to obtain the gradient field of the type region.

[0107] The fourth step is to train the fitted network based on the noise probability field and gradient field described above.

[0108] After obtaining the noise probability field and gradient field, the executing agent can train the fitted network using the noise pixels corresponding to the noise probability field and gradient field. The obtained noise field corresponds to a non-noise field. The value of each pixel in the non-noise field is the difference between 1 and the corresponding pixel value in the noise field. Specifically, the executing agent can first train the attention function using the non-noise probability field and gradient field. The formula for calculating the attention function is as follows:

[0109] A(x,y)=w g ·g(x,y)+w n ·(1.0-F(x,y))

[0110] Where A(x,y) is the network training attention function for the pixels with x-coordinate and y-coordinate; w g The attention weights are defined by g(x,y), which is the edge attention function for the edge pixels with x as the x-coordinate and y as the y-coordinate; w n is the attention term for pixels that are not noise, and its value is the difference between 1 and noise; n represents the non-noise case of pixels.

[0111] The formula for calculating g(x,y) is as follows:

[0112]

[0113] Where g(x,y) is the edge attention calculation function; W N c represents the weights for edge connectivity. i Let be the category to which the gradient point belongs through clustering; i is the i-th category; This represents the total number of gradient points for the i-th category; W is the activation mapping for the number of gradient points; G Connectivity is used as the weight for the attention mechanism; σ(G) xy ) is the activation function for the gradient value; C i Let i be the i-th category after clustering is completed.

[0114] The input to the fitting network can be noisy pixels, and the output of the fitting network can be pixels that satisfy a set of noise probability fields and gradient fields. These set noise probability fields and gradient fields are related to the target color of that type of region. In this way, the fitting network can output pixels that satisfy the target color by using noise probability fields and gradient fields that meet the target color requirements.

[0115] The formula for calculating the loss function of the fitted network is as follows:

[0116]

[0117] Among them, A xy R represents the attention weights in the above equation. xy G is the R value in the RGB coordinates of the pixel with x-coordinate and y-coordinate. xy The G value in the RGB coordinates of the pixel with x-coordinate and y-coordinate; B xy R is the B value in the RGB of the pixel with x-coordinate and y-coordinate; G is the G value in the RGB of the pixel with x-coordinate and y-coordinate; and B is the B value in the RGB of the pixel with x-coordinate and y-coordinate.

[0118] The fifth step is to input the pixels of this type of region into the above fitting network to obtain the fitted image region corresponding to this type of region.

[0119] After obtaining the fitting network, the executing entity can input the pixels of the type region into the fitting network to obtain the fitting image region corresponding to the type region.

[0120] The formula for calculating each pixel in the fitted image region is as follows:

[0121] P out = (1-noise(x,y))·P original +noise(x,y)·P predict

[0122] Among them, P out The final output denoised image pixels; noise(x,y) is the probability that the pixel with x-coordinate and y-coordinate is noise; P original P represents the pixel value of the original primitive. predict These are the pixel values ​​predicted by the network.

[0123] This improves the targeting and effectiveness of color adjustments for different types of regions, which helps to improve the smoothness of the target image and reduce abruptness.

[0124] Step 304: Combine at least one target color region corresponding to at least one of the above-mentioned type regions into a target image.

[0125] The content of step 304 is the same as that of step 204, and will not be repeated here.

[0126] Further reference Figure 4 This illustrates a flow 400 of another embodiment of the image generation method. Flow 400 of the image generation method includes the following steps:

[0127] Step 401: Preprocess the image to be processed to remove noise from the image to be processed.

[0128] In some embodiments, the image generation method runs on an execution entity (e.g., Figure 1 The electronic device 101 shown can preprocess the image to be processed to eliminate obvious noise in the image. The preprocessing may include at least one of the following: sharpness adjustment, brightness adjustment, and contrast adjustment.

[0129] Step 402: Classify the pixels of the image to be processed and determine at least one pixel type.

[0130] Step 403: Determine at least one type of region using at least one of the above pixel types.

[0131] Step 404: For a type region in at least one of the above type regions, adjust the color of the type region to the target color to obtain the target color region corresponding to the type region.

[0132] Step 405: Combine at least one target color region corresponding to at least one of the above-mentioned type regions into a target image.

[0133] The content of steps 402 to 405 is the same as that of steps 201 to 204, and will not be repeated here.

[0134] from Figure 4 It can be seen from this that, with Figure 2 Compared to the description of some corresponding embodiments, Figure 4 In some corresponding embodiments, noise is removed before the image to be processed. This results in a smaller amount of data for subsequent processing of the image and a smoother target image.

[0135] Further reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of an image generation apparatus, which are similar to... Figure 2 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0136] like Figure 5 As shown, an image generation apparatus 500 in some embodiments includes: a pixel type classification unit 501, a type region determination unit 502, a target color adjustment unit 503, and a target image generation unit 504. The pixel type classification unit 501 is configured to classify pixels of the image to be processed and determine at least one pixel type; the type region determination unit 502 is configured to determine at least one type region based on the at least one pixel type; the target color adjustment unit 503 is configured to adjust the color of a type region within the at least one type region to a target color, thereby obtaining a target color region corresponding to that type region; and the target image generation unit 504 is configured to combine at least one target color region corresponding to the at least one type region into a target image.

[0137] In some optional implementations of the embodiments, the pixel type classification unit 501 may include: a pixel type classification subunit (not shown in the figure), configured to, for the pixels of the image to be processed, import the pixel into a preset fully convolutional network, obtain the probability that the pixel belongs to each specified pixel type in a specified pixel type set, and set the specified pixel type with the highest probability value as the pixel type of the pixel. The specified pixel type set includes: solid color type, gradient type, highlight type, shadow type, texture type and noise type.

[0138] In some optional implementations of embodiments, the type region determination unit 502 may include: a variable region setting subunit (not shown in the figure), a stable region setting subunit (not shown in the figure), an adjustment region setting subunit (not shown in the figure), and a deletion region setting subunit (not shown in the figure). The variable region setting subunit is configured to set the image region composed of pixels of the solid color type and the gradient type as a variable region; the stable region setting subunit is configured to set the image region composed of pixels of the highlight type and the shadow type as a stable region; the adjustment region setting subunit is configured to set the image region composed of pixels of the highlight type, the shadow type, and the texture type as an adjustment region; and the deletion region setting subunit is configured to set the image region composed of pixels of the noise type as a deletion region.

[0139] In some optional implementations of the embodiments, the target color adjustment unit 503 may include: a first target color region adjustment subunit (not shown in the figure), configured to adjust the color of the type region to the monochrome in response to the type region being a variable region and the target color being monochrome, thereby obtaining the first target color region.

[0140] In some optional implementations of embodiments, the target color adjustment unit 503 may include: a color patch acquisition subunit (not shown in the figure), a proximity relationship determination subunit (not shown in the figure), and a second target color region adjustment subunit (not shown in the figure). The color patch acquisition subunit is configured to perform color clustering on the type of region in response to the fact that the region is a changing region and the target color is multi-colored, to obtain at least one color patch; the proximity relationship determination subunit is configured to determine the proximity relationship between the at least one color patch; the second target color region adjustment subunit is configured to set the color of a target image in a preset proximity matching image library that has the same proximity relationship as the target color of the region, to obtain a second target color region; the target image contains the multi-colored region.

[0141] In some optional implementations of embodiments, the aforementioned proximity determination subunit may include: a boundary line generation module (not shown in the figure), a connected region determination module (not shown in the figure), and a proximity determination module (not shown in the figure). The boundary line generation module is configured to generate a boundary line for the region to be processed in response to the aforementioned changing region comprising a region to be processed composed of pixels of a solid color type; the connected region determination module is configured to determine connected regions between color patches through the boundary lines; and the proximity determination module is configured to determine the proximity relationship between the at least one color patch based on the connected regions.

[0142] In some optional implementations of embodiments, the target color adjustment unit may include a texture feature extraction subunit (not shown in the figure) and a third target color region adjustment subunit (not shown in the figure). The texture feature extraction subunit is configured to extract texture features of the adjustment region in response to the region being an adjustment region; the third target color region adjustment subunit is configured to set the color of a target texture image from a preset texture image library that has the same texture features as the target color of the region, thus obtaining a third target color region.

[0143] In some optional implementations of embodiments, the target color adjustment unit 503 may include: a fitted image region acquisition subunit (not shown in the figure) and a fourth target color region adjustment subunit (not shown in the figure). The fitted image region acquisition subunit is configured to, in response to the type of region being a deletion region, fit the type of region to obtain a fitted image region; the fourth target color region adjustment subunit is configured to superimpose the fitted image region with the type of region to obtain a fourth target color region.

[0144] In some optional implementations of embodiments, the above-mentioned image region acquisition subunit may include: a noise probability matrix acquisition module (not shown in the figure), a noise probability field acquisition module (not shown in the figure), a gradient field acquisition module (not shown in the figure), a fitting network training module (not shown in the figure), and a fitting image region acquisition module (not shown in the figure). Specifically, the noise probability matrix acquisition module is configured to acquire the noise probability of each pixel within the region of that type, obtaining a noise probability matrix corresponding to that region; the noise probability field acquisition module is configured to accumulate the noise probabilities within the noise probability matrix to obtain a noise probability field; the gradient field acquisition module is configured to calculate the image gradient of the pixels in the region of that type, obtaining a gradient field for that region; the fitting network training module is configured to train a fitting network based on the noise probability field and the gradient field; and the fitting image region acquisition module is configured to input the pixels of the region of that type into the fitting network to obtain a fitting image region corresponding to that region.

[0145] In some optional implementations of the embodiments, the image generation apparatus 500 may further include: a noise removal unit (not shown in the figure), configured to preprocess the image to be processed to remove noise from the image to be processed, wherein the preprocessing includes at least one of the following: sharpness adjustment, brightness adjustment, and contrast adjustment.

[0146] It is understandable that the units described in the device 500 are related to the reference. Figure 2The steps in the described method correspond accordingly. Therefore, the operations, features, and beneficial effects described above for the method also apply to the device 500 and the units contained therein, and will not be repeated here.

[0147] like Figure 6 As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0148] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 6 Each box shown can represent a device or multiple devices as needed.

[0149] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined above in the methods of some embodiments of this disclosure.

[0150] It should be noted that, in some embodiments of this disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0151] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0152] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: classify the pixels of the image to be processed to determine at least one pixel type; determine at least one type region based on the at least one pixel type; for each type region within the at least one type region, adjust the color of that type region to a target color to obtain a target color region corresponding to that type region; and combine the at least one target color region corresponding to the at least one type region into a target image.

[0153] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0154] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0155] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including a pixel type classification unit, a type region determination unit, a target color adjustment unit, and a target image generation unit. The names of these units do not necessarily limit the specific unit; for example, the target image generation unit may also be described as "a unit for combining acquired target color regions into a target image."

[0156] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0157] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. An image generation method, comprising: Classifying the pixels of the image to be processed and determining at least one pixel type includes: for a pixel of the image to be processed, importing the pixel into a preset fully convolutional network to obtain the probability that the pixel belongs to each specified pixel type in a specified pixel type set, and setting the specified pixel type with the highest probability value as the pixel type of the pixel. The specified pixel type set includes: solid color type, gradient type, highlight type, shadow type, texture type and noise type. Determining at least one type of region based on the at least one pixel type includes: setting an image region composed of pixels of the solid color type and the gradient type as a change region; setting an image region composed of pixels of the highlight type and the shadow type as a stable region; setting an image region composed of pixels of the highlight type, the shadow type and the texture type as an adjustment region; and setting an image region composed of pixels of the noise type as a deletion region. For a type region within the at least one type region, adjust the color of the type region to the target color to obtain the target color region corresponding to the type region; Combine at least one target color region corresponding to the at least one type region into a target image.

2. The method according to claim 1, wherein, For a type region within the at least one type region, adjusting the color of that type region to the target color to obtain a target color region corresponding to that type region includes: In response to the fact that the type of region is a variable region and the target color is a monochrome, the color of the type of region is adjusted to the monochrome to obtain the first target color region.

3. The method according to claim 1, wherein, For a type region within the at least one type region, adjusting the color of that type region to the target color to obtain a target color region corresponding to that type region includes: In response to the fact that the region of this type is a variable region and the target color is multi-colored, color clustering is performed on the region of this type to obtain at least one color patch; Determine the proximity relationship between the at least one color patch; The color of a target image in a preset neighbor matching image library that has the same proximity relationship as the target image is set as the target color of the region of that type, thus obtaining a second target color region; the target image contains the multicolor.

4. The method according to claim 3, wherein, Determining the proximity relationship between the at least one color patch includes: In response to the change region comprising a region to be processed composed of pixels of solid color type, a boundary line of the region to be processed is generated; The boundary lines define the connected regions between color blocks. The proximity relationship between the at least one color patch is determined based on the connected region.

5. The method according to claim 1, wherein, For a type region within the at least one type region, adjusting the color of that type region to the target color to obtain a target color region corresponding to that type region includes: In response to the fact that the region of this type is an adjustment region, the texture features of the adjustment region are extracted; The color of a target texture image with the same texture features as the preset texture image library is set as the target color of the region of that type, thus obtaining the third target color region.

6. The method according to claim 1, wherein, For a type region within the at least one type region, adjusting the color of that type region to the target color to obtain a target color region corresponding to that type region includes: In response to the fact that this type of region is a deleted region, a fitted image region is obtained by fitting this type of region. The fitted image region is superimposed with this type of region to obtain the fourth target color region.

7. The method according to claim 6, wherein, The process of fitting the region of this type to obtain the fitted image region includes: Obtain the noise probability of each pixel within this type of region to obtain the noise probability matrix corresponding to this type of region; The noise probabilities within the noise probability matrix are summed to obtain the noise probability field; Calculate the image gradient of the pixels in this type of region to obtain the gradient field of this type of region; The fitted network is trained based on the noise probability field and gradient field; The pixels of this type of region are input into the fitting network to obtain the fitted image region corresponding to this type of region.

8. The method according to claim 1, wherein, Before classifying the pixels of the image to be processed and determining at least one pixel type, the method further includes: The image to be processed is preprocessed to remove noise. The preprocessing includes at least one of the following: color adjustment, brightness adjustment, and contrast adjustment.

9. An image generation apparatus, comprising: A pixel type classification unit is configured to classify pixels of an image to be processed and determine at least one pixel type, including: for a pixel of the image to be processed, importing the pixel into a preset fully convolutional network to obtain the probability that the pixel belongs to each specified pixel type in a specified pixel type set, and setting the specified pixel type with the highest probability value as the pixel type of the pixel. The specified pixel type set includes: solid color type, gradient type, highlight type, shadow type, texture type and noise type. A type region determination unit is configured to determine at least one type region based on the at least one pixel type, including: setting an image region composed of pixels of the solid color type and the gradient type as a change region; setting an image region composed of pixels of the highlight type and the shadow type as a stable region; setting an image region composed of pixels of the highlight type, the shadow type and the texture type as an adjustment region; and setting an image region composed of pixels of the noise type as a deletion region. The target color adjustment unit is configured to adjust the color of a type region to a target color for a type region in the at least one type region, thereby obtaining a target color region corresponding to the type region. The target image generation unit is configured to combine at least one target color region corresponding to the at least one type region into a target image.

10. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 8.

11. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 8.

Images