Image processing method and device and readable storage medium

Abstract

The embodiment of the invention provides an image processing method and device and a readable storage medium. The method comprises the following steps: acquiring a first image and a second image, wherein the second image is an image obtained by performing style editing processing on the first image; performing Lab color space conversion on the first image and the second image to obtain a first brightness channel and a first color channel of the first image and a second brightness channel and a second color channel of the second image; fusing and filtering the first brightness channel and the second brightness channel to obtain a third brightness channel; based on the pixel correlation of the first brightness channel and the second brightness channel, reconstructing the second color channel to obtain a third color channel; and outputting a fused image based on the third brightness channel and the third color channel. The fused image keeps the style of the second image and keeps the texture features of the first image, and detail distortion is avoided.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more particularly to an image processing method, apparatus, and readable storage medium. Background Technology

[0002] With the development of artificial intelligence and deep learning technologies, a brand-new image editing technology—the diffusion model (such as InstructPix2Pix)—has emerged. InstructPix2Pix utilizes deep neural networks to transform natural language descriptions into image editing instructions, thereby automatically generating images that match the descriptions. The advent of InstructPix2Pix has greatly simplified the image editing process, enabling ordinary users to easily achieve complex image editing effects.

[0003] InstructPix2Pix proposes a method for editing images using text, preserving the spatial composition of the original image while achieving a new image style. The style of the original image can be modified using commands. For example... Figure 1 The diagram illustrates the image before and after modification of the original image using InstructPix2Pix. For example, inputting the original image (left image) and modification instructions, such as "Add fireworks to the sky," into the model will output a new image (right image). This new image retains the spatial composition of the original image and adds fireworks to the sky.

[0004] While InstructPix2Pix can provide a wider variety of styles for modifying images, the modified images may lose texture features, resulting in loss of detail. Summary of the Invention

[0005] This invention provides an image processing method, apparatus, and readable storage medium that can fuse a first image and a second image (such as an image modified using a diffusion model) to obtain a fused image, such that the fused image maintains the style of the second image and preserves the texture features of the first image, thus avoiding detail distortion.

[0006] In a first aspect, embodiments of the present invention disclose an image processing method, the method comprising:

[0007] Acquire a first image and a second image, wherein the second image is an image obtained after style editing of the first image;

[0008] The first image and the second image are converted to Lab color space respectively to obtain the first brightness channel and the first color channel of the first image, and the second brightness channel and the second color channel of the second image.

[0009] The first brightness channel and the second brightness channel are fused and filtered to obtain the third brightness channel;

[0010] Based on the pixel correlation between the first luminance channel and the second luminance channel, the second color channel is reconstructed to obtain the third color channel;

[0011] Based on the third luminance channel and the third color channel, a fused image is output.

[0012] In a second aspect, embodiments of the present invention disclose an image processing apparatus, the apparatus comprising:

[0013] An image acquisition module is used to acquire a first image and a second image, wherein the second image is an image obtained after style editing processing of the first image;

[0014] The image conversion module is used to convert the first image and the second image to Lab color space respectively to obtain the first brightness channel and the first color channel of the first image, and to obtain the second brightness channel and the second color channel of the second image.

[0015] The first fusion module is used to fuse and filter the first brightness channel and the second brightness channel to obtain the third brightness channel.

[0016] The second fusion module is used to reconstruct the second color channel based on the pixel correlation between the first brightness channel and the second brightness channel to obtain the third color channel;

[0017] An image output module is used to output a fused image based on the third luminance channel and the third color channel.

[0018] Thirdly, embodiments of the present invention disclose a machine-readable medium having instructions stored thereon, which, when executed by one or more processors of a device, cause the device to perform the image processing method as described above.

[0019] The embodiments of the present invention have the following advantages:

[0020] After modifying the style of the first image using a diffusion model to obtain the second image, this embodiment of the invention can perform a fusion process on the first and second images. During the fusion process, this embodiment separates the luminance channel (L channel) and the color channel (a / b channel), fusing the luminance and color channels separately. The luminance channel (L channel) represents luminance and texture, while the color channel (a / b channel) represents color (i.e., hue or style). This embodiment obtains color from the a / b channel and luminance from the L channel while preserving texture. Therefore, the resulting fused image can maintain the style of the second image and retain the texture features of the first image, avoiding detail distortion, and is suitable for image processing tasks with high resolution requirements. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a diagram illustrating the original image before and after modification using InstructPix2Pix;

[0023] Figure 2 This is a flowchart illustrating the steps of an embodiment of the image processing method of the present invention;

[0024] Figure 3 This is a schematic diagram of a pixel region of the present invention;

[0025] Figure 4 This is a schematic diagram of the input and output of a pre-set neural network according to the present invention;

[0026] Figure 5 This is a structural block diagram of an embodiment of an image processing device according to the present invention;

[0027] Figure 6 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] The terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, the first object can be one or more. Furthermore, the term "and / or" in the specification and claims is used to describe the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. In embodiments of this invention, the term "multiple" refers to two or more, and other quantifiers are similar.

[0030] Reference Figure 2 The diagram illustrates a flowchart of an embodiment of an image processing method according to the present invention, the method comprising the following steps:

[0031] Step 101: Obtain a first image and a second image, wherein the second image is an image obtained after style editing of the first image;

[0032] Step 102: Convert the first image and the second image to Lab color space to obtain the first luminance channel and the first color channel of the first image, and the second luminance channel and the second color channel of the second image.

[0033] Step 103: Perform fusion and filtering on the first brightness channel and the second brightness channel to obtain the third brightness channel;

[0034] Step 104: Based on the pixel correlation between the first luminance channel and the second luminance channel, reconstruct the second color channel to obtain the third color channel;

[0035] Step 105: Output the fused image based on the third luminance channel and the third color channel.

[0036] The image processing method provided by this invention can be applied to electronic devices, including but not limited to: headphones, voice recorders, smart home terminals (including: air conditioners, refrigerators, rice cookers, water heaters, etc.), smart business terminals (including: video phones, smart conference desktop terminals, etc.), wearable devices (including smartwatches, smart glasses, etc.), financial smart terminals, as well as smartphones, tablets, personal digital assistants (PDAs), in-vehicle devices, servers, etc.

[0037] After modifying the original image, for example using a diffusion model (such as instruct-pix2pix), the resulting new image retains the style specified by the modification instruction, but loses some texture features from the original image. By fusing the original image (first image) and the new image (second image) using the method provided in this invention, a fused image can be obtained. This fused image retains the style of the new image while preserving the texture features of the original image, thus avoiding the loss of texture features.

[0038] Specifically, a first image and a second image are first acquired. The second image can be an image obtained by style editing the first image using any method or image editing tool, while maintaining the same spatial composition as the first image. For example, the second image can be an image obtained by style editing the first image using a diffusion model.

[0039] This invention does not limit the type of diffusion model. For example, the diffusion model can be InstructPix2Pix, InstructAny2Pix, or MagicBrush, etc.

[0040] The first image and the second image are converted to Lab color space to obtain the first luminance channel and the first color channel of the first image, and the second luminance channel and the second color channel of the second image.

[0041] The Lab color space, also known as the CIELAB color space, is a device-independent color space used to describe color as perceived by human vision. The Lab color space includes three channels: the L channel, the a channel, and the b channel. The L channel represents lightness (or descriptiveness) and texture. The pixel values ​​in the L channel are called L channel values, ranging from [0, 100], that is, from pure black to pure white. The a and b channels are chromaticity coordinates, representing color (i.e., style). The pixel values ​​in the a channel are called a channel values. The a channel represents the range from magenta to green, with values ​​ranging from [127, -128]. The pixel values ​​in the b channel are called b channel values. The b channel represents the range from yellow to blue, with values ​​ranging from [127, -128].

[0042] It's important to note that the Lab color space is not an image file format like JPEG, PNG, or GIF. Instead, it's an intermediate color space used in image processing to aid in color correction, color adjustment, and other color-related operations. On a computer, the Lab color space is typically represented using floating-point values ​​for the L, a, and b channels.

[0043] In this embodiment of the invention, the luminance channel refers to the L channel, which contains luminance and texture features. The first luminance channel refers to the L channel of the first image. The second luminance channel refers to the L channel of the second image. Color channels include a channel and b channel, which contain color features. The first color channel refers to the a channel and b channel of the first image. The second color channel refers to the a channel and b channel of the second image.

[0044] In the process of fusing the first image and the second image in this embodiment of the invention, the luminance channel and the color channel are separated, and fusion processing is performed on the luminance channel and the color channel respectively. The luminance and texture of the fused image are obtained through the luminance channel, and the color (i.e., color style) of the fused image is obtained through the color channel, so that the final fused image has the style of the second image (new image) and retains the texture features of the first image (original image).

[0045] Step 103 is the process of fusing the luminance channel (L channel). This invention fuses and filters the first and second luminance channels to obtain a third luminance channel, which is the luminance channel of the fused image. For example, the luminance gain of the second luminance channel (i.e., the L channel of the second image) relative to the first luminance channel (i.e., the L channel of the first image) can be calculated first. After filtering the luminance gain, the first luminance channel is fused using the filtered luminance gain to obtain the third luminance channel. The luminance gain reflects the change in brightness of each pixel in the second luminance channel relative to pixels at the same position in the first luminance channel. The third luminance channel is the L channel of the fused image. The L channel of the fused image obtained by filtering and fusing based on the luminance gain has neighboring pixels with continuously increasing luminance gains, resulting in a smoother fused image. While maintaining the texture features of the first image, noise can be further reduced.

[0046] Furthermore, the specific method for fusing and filtering the first luminance channel and the second luminance channel in this invention can be as follows: first, calculate the luminance gain of the second luminance channel relative to the first luminance channel; then, sequentially perform Gaussian filtering and High Order Markov random field (HDMRF) processing on the luminance gain. After the Gaussian filtering and HDMRF processing are completed, multiply the filtered luminance gain by the first luminance channel to obtain the third luminance channel, which is the luminance channel of the fused image.

[0047] Step 104 is the process of fusing color channels (a channel and b channel), specifically referring to fusing the a channel of the first image with the a channel of the second image, and fusing the b channel of the first image with the b channel of the second image. Since the fusing of the a channel and the fusing of the b channel are done in the same way, this embodiment of the invention is uniformly described as fusing color channels.

[0048] Step 104 may include: reconstructing the second color channel (i.e., the a channel of the second image) based on the pixel correlation between the first luminance channel (i.e., the L channel of the first image) and the second luminance channel (i.e., the L channel of the second image) to obtain a third color channel (i.e., the a channel of the fused image); and reconstructing the second color channel (i.e., the b channel of the second image) based on the pixel correlation between the first luminance channel (i.e., the L channel of the first image) and the second luminance channel (i.e., the L channel of the second image) to obtain a third color channel (i.e., the b channel of the fused image).

[0049] Reconstructing the second color channel refers to reconstructing the a-channel value of each pixel in the a-channel of the second image, and reconstructing the b-channel value of each pixel in the b-channel of the second image. Reconstruction means redetermining the values; it can either keep the original values ​​or modify them.

[0050] This invention reconstructs the second color channel based on the pixel correlation between the first and second luminance channels to obtain a third color channel. The pixel correlation between the first and second luminance channels refers to the correlation between a pixel at coordinate (i,j) in the first luminance channel and every pixel in the neighborhood of the pixel at coordinate (i,j) in the second luminance channel for a given coordinate (i,j). The pixel correlation refers to the similarity of pixel values ​​in the L channel. In this embodiment, the first and second images are converted to the Lab color space to obtain the L, a, and b channels of the first image, and the L, a, and b channels of the second image. The coordinates of each pixel in the first, second, and merged images correspond one-to-one. For coordinate (i,j), the value of the pixel at coordinate (i,j) in the L channel of the first image refers to the L channel value of that pixel. The value of the pixel at coordinate (i,j) in the a channel of the first image refers to the a channel value of that pixel. The value of the pixel at coordinate (i,j) in the b channel of the first image refers to the b channel value of that pixel. The same applies to the second image.

[0051] Reference Figure 3 This diagram illustrates the pixel neighborhood of a pixel at coordinate (i,j), where a square represents a pixel. Figure 3As shown, coordinate (i,j) is the center pixel of its neighborhood, which contains eight other pixels centered at coordinate (i,j). The coordinates of these eight pixels are: (i-1,j-1), (i-1,j), (i-1,j+1), (i,j-1), (i,j+1), (i+1,j-1), (i+1,j+1), (i+1,j+1). The correlation between the pixel at coordinate (i,j) in the first luminance channel and each pixel in the neighborhood of the pixel at coordinate (i,j) in the second luminance channel refers to the similarity between the L-channel value of the pixel at coordinate (i,j) in the first luminance channel and the L-channel value of each of the eight pixels in the neighborhood of the pixel at coordinate (i,j) in the second luminance channel.

[0052] In this embodiment of the invention, the L channel (first luminance channel and second luminance channel) is used as a reference to obtain the correlation between pixels. Thus, when performing color channel fusion, for a given coordinate (i,j), the color of the pixel at that coordinate can be determined based on the correlation between the pixel at that coordinate and each of its neighboring pixels. This allows the color of the pixel at that coordinate in the second color channel to be reconstructed, so that the fused color comes from texture-related pixels. This can prevent misaligned pixels from introducing noise into the texture and further improve the fusion effect.

[0053] The image processing described in this invention allows the new image obtained after modification using a diffusion model to retain the texture features of the original image, avoiding detail distortion, and thus making it suitable for image processing tasks with high resolution requirements.

[0054] In an optional embodiment of the present invention, the step of fusing and filtering the first luminance channel and the second luminance channel to obtain the third luminance channel may include:

[0055] Step S11: Calculate the ratio between each pixel of the second brightness channel and the corresponding pixel of the first brightness channel to obtain a brightness gain map;

[0056] Step S12: Perform a preset filtering process on the brightness gain map to obtain the target smooth map;

[0057] Step S13: Multiply each pixel of the target smoothing map by the corresponding pixel of the first brightness channel to obtain the third brightness channel.

[0058] This invention filters the brightness gain of the second brightness channel relative to the first brightness channel to obtain a target smooth map. The target smooth map is then used to fuse the first brightness channel, thereby achieving fusion and filtering of the first and second brightness channels to obtain a third brightness channel. This results in neighboring pixels in the third brightness channel having numerically continuous brightness gains, preserving the texture features of the first image, and reducing noise.

[0059] Specifically, firstly, the ratio of each pixel in the second luminance channel to the corresponding pixel in the first luminance channel is calculated to obtain the luminance gain map. The specific calculation is expressed as follows:

[0060] L gain =L alt / L org (1)

[0061] Among them, L alt Indicates the second brightness channel, L org L represents the first brightness channel. gain The above equation (1) represents the brightness gain map. It states that the ratio of the L-channel value of each pixel in the second brightness channel to the L-channel value of the corresponding pixel (i.e., the pixel at the same position) in the first brightness channel is calculated to obtain the brightness gain map. The pixels in the brightness gain map correspond to the pixels in the first and second brightness channels. The L-channel value of each pixel in the brightness gain map is the ratio of the L-channel value of the corresponding pixel in the second brightness channel to the L-channel value of the corresponding pixel in the first brightness channel. The value of each pixel in the brightness gain map is the brightness gain of each pixel in the second brightness channel relative to the corresponding pixel in the first brightness channel. The value of the pixel at coordinate (i,j) in the brightness gain map is referred to as the brightness gain of the pixel at coordinate (i,j) in the second brightness channel. For example, if the brightness gain is 1.5, it means that the brightness of the pixel at coordinate (i,j) in the second brightness channel is increased by 1.5 times compared to the pixel at coordinate (i,j) in the first brightness channel. It is understood that the above 1.5 is only an example.

[0062] This invention uses a brightness gain map to adjust the editing style of fused images.

[0063] It is understood that in the embodiments of the present invention, calculations on pixels, such as ratio calculations, multiplication calculations, and correlation calculations, refer to calculations on the channel values ​​of the pixels. For example, calculating the ratio of each pixel in the second luminance channel (e.g., L2) with the corresponding pixel in the first luminance channel (e.g., L1) means calculating the ratio of the L2 channel value of each pixel in L2 with the L1 channel value of the corresponding pixel in L1. For example, for a pixel at coordinates (i,j) in the first image (and the second image), the ratio of the L2 channel value to the L1 channel value of that pixel is calculated. The L2 channel value of the pixel refers to the value of the pixel in the L2 channel. The L1 value of the pixel refers to the value of the pixel in the L1 channel.

[0064] The luminance gain map (also referred to as the gain map in this paper) reflects the second luminance channel L alt Each pixel relative to the first luminance channel L org The degree of change in brightness of pixels at the same position, such as a 1.5-fold increase in brightness.

[0065] Then, the brightness gain map is subjected to a preset filtering process, that is, the L... gain Perform preset filtering to obtain the target smooth map (e.g., denoted as L). gain ').

[0066] Next, each pixel of the target smoothing map is multiplied by the corresponding pixel of the first brightness channel to obtain the third brightness channel (e.g., denoted as L). fuse The specific calculation is as follows:

[0067] L fuse =L gain '*L org (2)

[0068] Based on the brightness gain of the second brightness channel relative to the first brightness channel, this invention fuses and filters the first and second brightness channels to obtain a third brightness channel. This results in neighboring pixels in the third brightness channel having numerically continuous brightness gains, which can preserve the texture features of the first image and reduce noise.

[0069] In an optional embodiment of the present invention, the step of performing a preset filtering process on the brightness gain map to obtain a target smooth map may include:

[0070] Step S21: Perform a first filtering process on the brightness gain map to obtain an initial smooth map;

[0071] Step S22: Perform a second filtering process on the initial smoothed map to obtain the target smoothed map.

[0072] After calculating the brightness gain map, the present invention performs a preset filtering process on the brightness gain map to remove noise and obtain the target smooth map.

[0073] This invention employs a sequential two-step filtering method for the preset filtering process. Specifically, a first filtering process is performed on the luminance gain map to initially smooth it, resulting in an initial smoothed map. In practice, if the luminance gain differences between neighboring pixels are significant, the texture after fusion will be destroyed, causing the fused image to fail to retain the texture of the original image or potentially carry noise. To address this issue, this embodiment of the invention performs a second filtering process on the initial smoothed map to further smooth it, eliminating the differences in luminance gain between neighboring pixels. This ensures that the final fused image retains the texture of the original image and eliminates noise.

[0074] It can be understood that the initial smoothed image and target smoothed image obtained by filtering the brightness gain image in this invention contain corresponding pixels to the brightness gain image. For example, a pixel value of 1.5 at coordinate (i,j) in the brightness gain image indicates that the brightness of the pixel at coordinate (i,j) in the second brightness channel is increased by 1.5 times compared to the pixel at coordinate (i,j) in the first brightness channel. Assuming that the initial smoothed image is obtained after the first filtering process, and the pixel value of (i,j) in the initial smoothed image is 1.4, it indicates that the brightness of the pixel at coordinate (i,j) in the second brightness channel is increased by 1.4 times compared to the pixel at coordinate (i,j) in the first brightness channel after the first filtering process. Assuming that the target smoothed image is obtained after the second filtering process, and the pixel value of (i,j) in the target smoothed image is 1.3, it indicates that the brightness of the pixel at coordinate (i,j) in the second brightness channel is increased by 1.3 times compared to the pixel at coordinate (i,j) in the first brightness channel after the second filtering process. It is understood that the coordinates (i,j) above represent the coordinates of a certain pixel in the first image and the second image, and the brightness gain mentioned above is only an example.

[0075] In an optional embodiment of the present invention, the first filtering process may include Gaussian filtering; the second filtering process is based on high-order Markov random field (HDMRF) filtering.

[0076] Gaussian filtering is a linear smoothing filter suitable for eliminating Gaussian noise. Gaussian filtering involves scanning the entire image using a sliding window, and at each scanned pixel, calculating a weighted average of the pixel's own value and the values ​​of its neighboring pixels.

[0077] Markov random fields (MRFs) are probabilistic graphical models. In image denoising, MRFs treat each pixel in an image as a random variable and model the interactions between pixels. These interactions can be represented by an energy function (or objective function). The goal of a MRF is to find a configuration that minimizes the energy function.

[0078] Furthermore, this embodiment of the invention extends MRF to implement a filtering method based on high-order dense Markov random field (HDMRF), and the second filtering process adopts the filtering process based on HDMRF.

[0079] In an optional embodiment of the present invention, the step of performing a second filtering process on the initial smoothed image to obtain a target smoothed image may include:

[0080] Step S31: Initialize the parameters of the preset neural network to the initial expected output brightness gain smoothing map; the input of the preset neural network is the initial smoothing map; the objective function of the preset neural network includes the following constraints: the square error between pixels at the same position in the image output by the preset neural network and the input image is minimized, and the square error between any two pixels in the neighborhood of any higher-order pixel in the image output by the preset neural network is minimized.

[0081] Step S32: Input the initial smoothing map into the preset neural network for the first optimization iteration to reduce the objective function;

[0082] Step S33: Update the parameters of the preset neural network as the expected output brightness gain smoothing map for the first optimization iteration;

[0083] Step S34: Input the initial smoothing map into the preset neural network for a second optimization iteration to further reduce the objective function;

[0084] Step S35: Update the parameters of the preset neural network as the expected output brightness gain smoothing map for the second optimization iteration;

[0085] Step S36: When the objective function satisfies the preset conditions, output the parameters of the preset neural network to obtain the target smooth map.

[0086] In one embodiment, the luminance gain map is subjected to Gaussian filtering to obtain an initial smooth map. The initial smooth map is then subjected to HDMRF-based filtering to obtain the expected output luminance gain smooth map (i.e., the target smooth map).

[0087] Furthermore, embodiments of the present invention use neural networks to solve the HDMRF. Solving the HDMRF refers to the process of performing HDMRF-based filtering on the initial smoothed image to obtain the target smoothed image.

[0088] Specifically, this invention defines a neural network (called a preset neural network) that processes the initial smoothed image (i.e., the smoothed image on L) into a smoothed image. gain The result after Gaussian filtering is input into the preset neural network, which is then used to solve the HDMRF and output the target smooth map L. gain '.

[0089] In this embodiment of the invention, the objective function of the preset neural network is defined according to the constraints of the HDMRF. The constraints of the HDMRF may include the following two aspects: first, the prior of the output of the preset neural network comes from the input; second, pixels that are spatially close to the output of the preset neural network have similar brightness gains.

[0090] The purpose of the first constraint is to ensure that the pre-defined neural network removes noise while maintaining consistency between the output and input. This first constraint can be achieved through the squared error between pixels at the same location in the output image and the input image. For example, the first constraint can be quantified as follows:

[0091] min Lgain′ ∑ i,j (Lgain′ i,j -Lgain i,j ) 2 (3)

[0092] Among them, Lgain′ i,j This represents the brightness gain of the pixel at coordinates (i,j) in the target smooth map. i,j Let (i,j) represent the brightness gain of the pixel at coordinate (i,j) in the initial smooth image. Equation (3) above indicates that after performing HDMRF filtering on Lgain, Lgain′ is obtained. The squared error between each pixel in Lgain′ and each pixel at the same position in Lgain is minimized, so that the features of the output Lgain′ of the preset neural network are consistent with those of the input Lgain.

[0093] The purpose of the second constraint is to ensure that neighboring pixels have similar brightness gains, resulting in a smooth gradient across the entire image. This second constraint can be achieved through the squared errors between pairwise pixels within the higher-order pixel neighborhood. For example, the second constraint can be quantized as follows:

[0094] min Lgain′ ∑ i,j ∑ n,m∈Nei(i,j)(Lgain′ i,j -Lgain′ n,m ) 2 (4)

[0095] Among them, Lgain′ i,j This represents the brightness gain of the pixel at coordinates (i,j) in the target smooth map. Lgain′ n,m Let represent the brightness gain of the pixel at coordinates (m,n) in the initial smooth image, where the range of coordinates (m,n) is Nei(i,j). Nei(i,j) represents the neighborhood of the pixel at coordinates (i,j). In this embodiment of the invention, Nei(i,j) in the above equation (4) represents a higher-order pixel neighborhood, which can represent any two adjacent pixels, not limited to the 8-neighborhood range of the traditional MRF. The above equation (4) indicates that Lgain′ is obtained after HDMRF filtering of Lgain. Within any higher-order pixel neighborhood of Lgain′, the squared error between any two pixels is minimized, so that the entire image achieves a smooth gradient. Lgain′ This refers to finding the minimum objective function for the output.

[0096] Based on the above two constraints, the objective function can be defined as follows:

[0097] min Lgain′ ∑ i,j (Lgain′ i,j -Lgain i,j ) 2 +min Lgain′ ∑ i,j ∑ n,m∈Nei(i,j) (Lgain′ i,j -Lgain′ n,m ) 2 (5)

[0098] The objective function includes the constraints mentioned above. Solving the HDMRF involves inputting Lgain into a pre-set neural network, outputting Lgain′ through the pre-set neural network, and minimizing the value of the objective function of the pre-set neural network. In other words, the process of solving the HDMRF is as follows:

[0099]

[0100] The process of solving the HDMRF using a pre-set neural network in this embodiment of the invention is essentially the process of training the pre-set neural network. When the pre-set neural network is trained (the value of the objective function meets the preset conditions), the parameters of the pre-set neural network are the target smooth graph Lgain′.

[0101] The process of training the pre-set neural network (that is, the process of solving the HDMRF by the pre-set neural network) is shown in steps S31 to S36.

[0102] First, a pre-set neural network is initialized, and its parameters are randomly initialized. Then, a first optimization iteration is performed. The initial smoothing map is input into the pre-set neural network. Based on the input of the first optimization iteration (i.e., the initial smoothing map) and the reduction of the objective function while updating the parameters (i.e., the expected output brightness gain smoothing map), the pre-set neural network obtains network parameters that reduce the objective function. These parameters are the expected output brightness gain smoothing map of the first optimization iteration. Next, a second optimization iteration is performed. The input for each optimization iteration is the same: the initial smoothing map. The initial smoothing map is input into the pre-set neural network. Based on the input of the second optimization iteration (i.e., the initial smoothing map) and the parameters (i.e., the expected output brightness gain smoothing map after the first optimization iteration), the pre-set neural network obtains network parameters that reduce the objective function. These parameters are the expected output brightness gain smoothing map after the second optimization iteration. Next, a third optimization iteration is performed. This process continues until the value of the objective function meets a preset condition (at which point the model converges). If the value of the objective function is less than a preset threshold, the training is complete. At this point, the parameters of the pre-set neural network are the final expected output brightness gain smoothing map (i.e., the target smoothing map).

[0103] This invention performs a second filtering process based on HDMRF, extending the constraints of the high-order dense Markov random field to all pixels in the neighborhood, so that the square error between any two pixels in the neighborhood of any high-order pixel in the target smooth map is minimized, rather than just the 8-neighborhood range specified in the traditional MRF, which can further improve the smoothing effect and reduce noise.

[0104] It should be noted that Nei(i,j) in equations (4) to (6) above all represent higher-order pixel neighborhoods, that is, Nei(i,j) includes any pixel spatially adjacent to the pixel with coordinates (i,j). (Refer to...) Figure 4 This is a schematic diagram illustrating the input and output of a preset neural network in an embodiment of the present invention. Figure 4 As shown, Lgain i,j Lgain′ is the pixel in the initial smoothed image of the input. i,j These are the pixels in the output target smoothed map. Solving the HDMRF yields the target smoothed map Lgain′. For example... Figure 4 As shown, this embodiment of the invention extends the constraints of HDMRF to higher-order pixel neighborhoods, so that the squared error between any two pixels in any higher-order pixel neighborhood in the target smooth map is minimized, rather than just the 8-neighborhood range specified in the traditional MRF.

[0105] In an optional embodiment of the present invention, the step of reconstructing the second color channel based on the pixel correlation between the first luminance channel and the second luminance channel to obtain the third color channel may include:

[0106] Step S41: For the target coordinates, calculate the correlation weight between the pixel at the target coordinates in the first brightness channel and each pixel in the neighborhood of the pixel at the target coordinates in the second brightness channel; the target coordinates include the coordinates of each pixel in the first brightness channel;

[0107] Step S42: Based on the correlation weight, perform a weighted average of all pixels in the neighborhood of the target coordinate in the second color channel to obtain the pixel of the target coordinate in the third color channel.

[0108] Target coordinates refer to the coordinates of each pixel in the merged image. Pixels in the first image, second image, and merged image have the same coordinates. For example, if the target coordinates are (i,j), then the first image, second image, and third image all contain pixels with coordinates (i,j). Therefore, the first luminance channel, second luminance channel, third luminance channel, first color channel, second color channel, and third color channel all contain pixels with coordinates (i,j).

[0109] For the target coordinates, calculate the correlation weight between the pixel at the target coordinates in the first brightness channel and each pixel in the neighborhood of the pixel at the target coordinates in the second brightness channel; the target coordinates include the coordinates of each pixel in the first brightness channel.

[0110] In an optional embodiment of the present invention, calculating the correlation weight between the pixel at the target coordinate in the first brightness channel and each pixel in the neighborhood of the pixel at the target coordinate in the second brightness channel may include:

[0111] Step S51: Calculate the product of the pixel at the target coordinate in the first brightness channel with each pixel in the neighborhood of the pixel at the target coordinate in the second brightness channel to obtain the target vector;

[0112] Step S52: Normalize the target vector to obtain the correlation weight of each pixel in the neighborhood of the target coordinate pixel in the first brightness channel and the target coordinate pixel in the second brightness channel.

[0113] For example, for target coordinates (i,j), the relevance weight w is calculated as follows:

[0114] w = softmax(repeat[lalt]) i,j ]*[lorg x,y ]x,y∈Nei(i,j) (7)

[0115] Where i,j represents the coordinates of a pixel in the second brightness channel, i.e., the target coordinates (i,j). Nei(i,j) represents the pixel neighborhood of the target coordinates (i,j), such as... Figure 3 As shown. It should be noted that Nei(i,j) in equation (7) above represents an 8-neighborhood. i,j This represents the L channel of the pixel at coordinates (i,j) in the second brightness channel. x,y This represents the L-channel value of the pixel at coordinates (x, y) in the first luminance channel. `repeat` indicates repeated execution. Since there are 8 pixels in Nei(i, j), the number of times `repeat` is used in equation (7) above is 8. The target vector obtained after repeating the calculation 8 times is as follows:

[0116] [lalt i,j *lorg i-1,j-1 ,lalt i,j *lorg i-1,j ,lalt i,j *lorg i-1,j+1 ,lalt i,j *lorg i,j-1 ,lalt i,j *lorg i,j+1 ,lalt i,j *lorg i+1,j-1 ,lalt i,j *lorg j+1,j ,lalt i,j *lorg i+1,j+1 ].

[0117] softmax is a normalization function used to normalize the target vector obtained above.

[0118] Taking the fused a channel as an example, the calculation process can be represented as follows:

[0119] a′ i,j =∑ x,y∈Nei(i,j) w x,y a i,j (8)

[0120] Among them, w x,y This represents the relevance weight between the pixel at target coordinates (i,j) and each pixel in its neighborhood. i,j This represents the a-channel value of the pixel at coordinates (i,j) in the second color channel. a′ i,j This indicates that a is weighted according to its relevance weight w. i,jThe value is reconstructed to obtain the a-channel value of the pixel at coordinate (i,j) in the third color channel.

[0121] In this embodiment of the invention, the correlation between each pixel and its neighboring pixels is calculated in the L channel, and then the softmax function is used to convert the correlation into a correlation weight w. Based on the correlation weight w, the a-channel values ​​of all pixels in the pixel's neighborhood in the a-channel of the second image are weighted and averaged to obtain the reconstructed a-channel value of the pixel. The fusion process of the b-channel is the same as that of the a-channel, and will not be described again here.

[0122] After fusing to obtain the third luminance channel and the third color channel, a fused image can be output based on the third luminance channel and the third color channel. For example, the fused L channel, a channel, and b channel can be converted to RGB to obtain an RGB image, which is the final fused image.

[0123] It should be noted that Nei(i,j) in equations (7) and (8) above both represent an 8-pixel neighborhood, that is, Nei(i,j) contains 8 pixels adjacent to the pixel with coordinates (i,j) (e.g., ...). Figure 3 (As shown).

[0124] In an optional embodiment of the present invention, the second image may include an image obtained by processing the first image using a diffusion model; the second image and the first image maintain the same spatial composition. The method may further include: using the obtained fused image as training data for an image processing task model, and training the image processing task model.

[0125] This invention can also be used to expand the training data for image processing task models. This invention does not limit the specific type of the image processing task model. The image processing task model can be an image classification model, an image detection model, an image denoising model, or an image segmentation model, etc.

[0126] For example, suppose an image processing task model is used for image classification, specifically for identifying the weather environment of captured images. To train an image processing task model capable of recognizing different weather environments, a large number of images taken under different weather conditions need to be collected as training data. Collecting a large number of images taken under different weather conditions presents certain difficulties. Using the method of this embodiment, it is only necessary to obtain images taken under ideal weather conditions. By using a diffusion model to modify these images (first images) in different ways, multiple images (second images) under different weather conditions can be obtained. For example, the ideal weather environment is a sunny daytime environment. Using a diffusion model, images taken under ideal weather conditions can be modified into daytime images under cloudy conditions, nighttime images under cloudy conditions, daytime images under rainy conditions, nighttime images under rainy conditions, and so on. Multiple images can be generated from a single image, expanding the training data while preserving the content and texture details of the original image, providing a foundation for training the accuracy of the image processing task model.

[0127] In summary, the image processing method provided by this invention, after modifying the style of a first image using a diffusion model to obtain a second image, can perform a fusion process on the first and second images. During the fusion process, this invention separates the luminance channel (L channel) and color channel (a / b channel) and fuses the luminance and color channels separately. The luminance channel (L channel) represents luminance and texture, while the color channel (a / b channel) represents color (i.e., hue or style). This invention obtains color from the a / b channel and luminance from the L channel while preserving texture. Therefore, the resulting fused image can maintain the style of the second image and retain the texture features of the first image, avoiding detail distortion, and is suitable for image processing tasks with high resolution requirements.

[0128] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0129] Reference Figure 5 The diagram illustrates a structural block diagram of an embodiment of an image processing apparatus according to the present invention, the apparatus comprising:

[0130] Image acquisition module 201 is used to acquire a first image and a second image, wherein the second image is an image obtained after style editing processing of the first image;

[0131] Image conversion module 202 is used to convert the first image and the second image to Lab color space respectively to obtain the first brightness channel and the first color channel of the first image, and to obtain the second brightness channel and the second color channel of the second image;

[0132] The first fusion module 203 is used to fuse and filter the first brightness channel and the second brightness channel to obtain a third brightness channel.

[0133] The second fusion module 204 is used to reconstruct the second color channel based on the pixel correlation between the first brightness channel and the second brightness channel to obtain the third color channel;

[0134] Image output module 205 is used to output a fused image based on the third luminance channel and the third color channel.

[0135] Optionally, the first fusion module includes:

[0136] The gain calculation submodule is used to calculate the ratio between each pixel of the second brightness channel and the corresponding pixel of the first brightness channel to obtain a brightness gain map.

[0137] The filtering submodule is used to perform preset filtering on the brightness gain map to obtain the target smooth map;

[0138] The first fusion submodule is used to multiply each pixel of the target smooth map with the corresponding pixel of the first brightness channel to obtain the third brightness channel.

[0139] Optionally, the filtering submodule includes:

[0140] The first filtering unit is used to perform a first filtering process on the brightness gain map to obtain an initial smooth map;

[0141] The second filtering unit is used to perform a second filtering process on the initial smoothed image to obtain the target smoothed image.

[0142] Optionally, the first filtering process includes Gaussian filtering; the second filtering process includes Markov random field-based filtering.

[0143] Optionally, the second filtering unit includes:

[0144] An initialization subunit is used to initialize the parameters of the preset neural network to the initial expected output brightness gain smoothing map; the input of the preset neural network is the initial smoothing map; the objective function of the preset neural network includes the following constraints: the square error between pixels at the same position in the image output by the preset neural network and the input image is minimized, and the square error between any two pixels in the neighborhood of any higher-order pixel in the image output by the preset neural network is minimized;

[0145] An iterative optimization subunit is used to input the initial smoothing map into the preset neural network for a first optimization iteration to reduce the objective function; update the parameters of the preset neural network as the expected output brightness gain smoothing map for the first optimization iteration; input the initial smoothing map into the preset neural network for a second optimization iteration to further reduce the objective function; update the parameters of the preset neural network as the expected output brightness gain smoothing map for the second optimization iteration; repeat the optimization iteration, and when the objective function meets a preset condition, output the parameters of the preset neural network to obtain the target smoothing map.

[0146] Optionally, the second fusion module includes:

[0147] The weight calculation submodule is used to calculate the correlation weight between the pixel at the target coordinate in the first brightness channel and each pixel in the neighborhood of the pixel at the target coordinate in the second brightness channel, for the target coordinate; the target coordinate includes the coordinates of each pixel in the first brightness channel;

[0148] The second fusion submodule is used to perform a weighted average of all pixels in the neighborhood of the target coordinate in the second color channel based on the correlation weight, so as to obtain the pixel of the target coordinate in the third color channel.

[0149] Optionally, the weight calculation submodule includes:

[0150] The vector calculation unit is used to calculate the product of the pixel of the target coordinate in the first brightness channel with each pixel in the neighborhood of the pixel of the target coordinate in the second brightness channel to obtain the target vector;

[0151] The weight calculation unit is used to perform normalization calculation on the target vector to obtain the correlation weight of each pixel in the neighborhood of the target coordinate pixel in the first brightness channel and the target coordinate pixel in the second brightness channel.

[0152] Optionally, the luminance channel includes the L channel of the Lab color space, and the color channel includes the a channel and b channel of the Lab color space.

[0153] Optionally, the second image includes an image obtained by processing the first image using a diffusion model; the second image and the first image maintain the same spatial composition; the apparatus further includes:

[0154] The model training module is used to train the image processing task model by using the obtained fused image as training data.

[0155] The image processing apparatus provided by this invention, after modifying the style of a first image using a diffusion model to obtain a second image, can perform a fusion process on the first and second images. During the fusion process, this invention separates the luminance channel (L channel) and the color channel (a / b channel) and fuses the luminance and color channels separately. The luminance channel (L channel) represents luminance and texture, while the color channel (a / b channel) represents color (i.e., hue or style). This invention obtains color from the a / b channel and luminance from the L channel while preserving texture. Therefore, the resulting fused image can maintain the style of the second image and retain the texture features of the first image, avoiding detail distortion, and is suitable for image processing tasks with high resolution requirements.

[0156] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0157] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0158] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0159] Reference Figure 6 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Figure 6 As shown, the electronic device includes: a processor, a memory, a communication interface, and a communication bus. The processor, the memory, and the communication interface communicate with each other through the communication bus. The memory is used to store at least one executable instruction, which causes the processor to perform the steps of the image processing method of the foregoing embodiment.

[0160] This invention also provides a non-transitory computer-readable storage medium, wherein when the instructions in the storage medium are executed by a processor of a device (server or terminal), the device is able to execute the foregoing text. Figure 1The description of the image processing method in the corresponding embodiments is already provided and will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the computer program products or computer program embodiments related to this application, please refer to the description of the method embodiments of this application.

[0161] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the aforementioned image processing methods.

[0162] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0163] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0164] These computer program instructions may also be stored in a computer-readable storage medium capable of directing a computer or other programmable data processing terminal device to operate in a predictive manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0165] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0166] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0167] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

[0168] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0169] The foregoing has provided a detailed description of an image processing method, apparatus, and machine-readable storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. An image processing method, characterized in that, The method includes: Acquire a first image and a second image, wherein the second image is an image obtained after style editing of the first image; The first image and the second image are converted to Lab color space respectively to obtain the first brightness channel and the first color channel of the first image, and the second brightness channel and the second color channel of the second image. The first brightness channel and the second brightness channel are fused and filtered to obtain the third brightness channel; Based on the pixel correlation between the first luminance channel and the second luminance channel, the second color channel is reconstructed to obtain the third color channel; Based on the third luminance channel and the third color channel, a fused image is output.

2. The method according to claim 1, characterized in that, The process of fusing and filtering the first luminance channel and the second luminance channel to obtain the third luminance channel includes: The ratio of each pixel in the second brightness channel to the corresponding pixel in the first brightness channel is calculated to obtain the brightness gain map. The brightness gain map is subjected to a preset filtering process to obtain the target smooth map; Each pixel of the target smooth map is multiplied by the corresponding pixel of the first brightness channel to obtain the third brightness channel.

3. The method according to claim 2, characterized in that, The step of performing a preset filtering process on the brightness gain map to obtain the target smooth map includes: The brightness gain map is subjected to a first filtering process to obtain an initial smooth map; The initial smoothed image is subjected to a second filtering process to obtain the target smoothed image.

4. The method according to claim 3, characterized in that, The first filtering process includes Gaussian filtering; the second filtering process includes Markov random field-based filtering.

5. The method according to claim 3, characterized in that, The step of performing a second filtering process on the initial smoothed map to obtain the target smoothed map includes: The parameters of the preset neural network are initialized as the initial expected output brightness gain smoothing map; the input of the preset neural network is the initial smoothing map; the objective function of the preset neural network includes the following constraints: the square error between pixels at the same position in the image output by the preset neural network and the input image is minimized, and the square error between any two pixels in the neighborhood of any higher-order pixel in the image output by the preset neural network is minimized. The initial smooth map is input into the preset neural network for the first optimization iteration to reduce the objective function. Update the parameters of the preset neural network to serve as the expected output brightness gain smoothing map for the first optimization iteration; The initial smooth map is input into the preset neural network for a second optimization iteration to further reduce the objective function. Update the parameters of the preset neural network to serve as the expected output brightness gain smoothing map for the second optimization iteration; The optimization iteration is repeated, and when the objective function meets the preset conditions, the parameters of the preset neural network are output to obtain the target smooth map.

6. The method according to claim 1, characterized in that, The step of reconstructing the second color channel based on the pixel correlation between the first luminance channel and the second luminance channel to obtain the third color channel includes: For the target coordinates, calculate the correlation weight between the pixel at the target coordinates in the first brightness channel and each pixel in the neighborhood of the pixel at the target coordinates in the second brightness channel; the target coordinates include the coordinates of each pixel in the first brightness channel; Based on the correlation weight, a weighted average is performed on all pixels in the neighborhood of the target coordinate in the second color channel to obtain the pixel of the target coordinate in the third color channel.

7. The method according to claim 6, characterized in that, The calculation of the correlation weight between the pixel at the target coordinate in the first brightness channel and each pixel in the neighborhood of the pixel at the target coordinate in the second brightness channel includes: The target vector is obtained by multiplying the pixel at the target coordinate in the first brightness channel with each pixel in the neighborhood of the pixel at the target coordinate in the second brightness channel. The target vector is normalized to obtain the correlation weight of each pixel in the neighborhood of the target coordinate pixel in the first brightness channel and the target coordinate pixel in the second brightness channel.

8. The method according to any one of claims 1 to 7, characterized in that, The luminance channel includes the L channel of the Lab color space, and the color channel includes the a channel and b channel of the Lab color space.

9. The method according to any one of claims 1 to 7, characterized in that, The second image comprises an image obtained by processing the first image using a diffusion model; the second image and the first image maintain the same spatial composition; the method further includes: The resulting fused image is used as training data for the image processing task model, and the image processing task model is trained.

10. An image processing apparatus, characterized in that, The device includes: An image acquisition module is used to acquire a first image and a second image, wherein the second image is an image obtained after style editing processing of the first image; The image conversion module is used to convert the first image and the second image to Lab color space respectively to obtain the first brightness channel and the first color channel of the first image, and to obtain the second brightness channel and the second color channel of the second image. The first fusion module is used to fuse and filter the first brightness channel and the second brightness channel to obtain the third brightness channel. The second fusion module is used to reconstruct the second color channel based on the pixel correlation between the first brightness channel and the second brightness channel to obtain the third color channel; An image output module is used to output a fused image based on the third luminance channel and the third color channel.

Images