Image processing method, electronic equipment and storage medium

Abstract

The invention relates to the technical field of data processing, and discloses an image processing method, electronic equipment and a storage medium, in the method, a to-be-processed image is converted into a to-be-processed image matrix containing image information, and a plurality of preset reference image matrixes are adopted to perform data adjustment on the to-be-processed image matrix. Therefore, by adopting different reference image matrixes, the to-be-processed image matrix can be adjusted into an image matrix with different illumination directions and / or illumination intensities. And then, reconstructing the image matrixes to obtain a plurality of images with different illumination directions and / or illumination intensities corresponding to the to-be-processed image. Therefore, the object in the to-be-processed image does not need to be modeled, and the workload of equipment can be reduced.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an image processing method, an electronic device and a storage medium. Background Art

[0002] Relighting is a computer graphics technology that aims to make different images look like they are under different lighting by changing the lighting direction and / or lighting intensity of each frame in the scene, making the scene look more realistic. For example, in the post-production of a movie, the lighting direction and / or lighting intensity of the objects in each frame can be adjusted to make the movie more realistic. In the game screen display, the lighting direction and / or lighting intensity of the objects in each frame can be adjusted in real time to make the game screen more delicate. In the secondary editing of photos, the lighting direction and / or lighting intensity of the objects in the photos can be adjusted so that the lighting of the objects is arranged according to user needs, which can reduce the waste of existing materials and reduce the cost of secondary shooting.

[0003] At present, the re-lighting technology first determines the objects in the image, then constructs the multi-dimensional model corresponding to the object, and then renders the multi-dimensional model corresponding to the object based on different lighting directions and / or lighting intensities. However, since the current re-lighting technology mainly adjusts the lighting direction and / or lighting intensity of the object, for scenes with complex backgrounds and diverse lighting, such as dynamic wallpapers, there are a large number of objects in the image, and the way of modeling the objects in the image will bring a huge workload to the device. Summary of the invention

[0004] In order to solve the problem that the way of modeling objects in an image will bring a huge workload to the device, the embodiments of the present application provide an image processing method, an electronic device and a storage medium.

[0005] A first aspect of an embodiment of the present application provides an image processing method, which is applied to an electronic device, comprising: calculating a first image matrix of a first image to be processed, wherein the data in the first image matrix represents the image information of each pixel in the first image; using a first reference image matrix, adjusting the data in the first image matrix to obtain a second image matrix, wherein the first reference image matrix corresponds to an illumination reference map having a first illumination parameter value; and obtaining a second image based on the second image matrix, wherein the illumination parameter value of the second image is different from the illumination parameter value of the first image.

[0006] Based on the above solution, by using different illumination reference images to process the first image, an image with different illumination directions and / or illumination intensities can be obtained from the first image. In this way, there is no need to model the objects in the image to be processed, which can reduce the workload of the device.

[0007] It can be understood that the first image can be an image to be processed, and the second image can be an image with different illumination direction and / or illumination intensity from the image to be processed. The illumination reference image can be a grayscale image, and the larger the grayscale value of the pixel in the grayscale image, the stronger the illumination intensity. Among them, the grayscale value of the pixel is 0, which can indicate that the pixel is black, and the grayscale value of the pixel is 255, which can indicate that the pixel is white. The illumination intensity of the pixel decreases as the distance between the pixel and the pixel corresponding to the light source increases, and reaches 0 within a specified range. That is, the grayscale value of the pixel corresponding to the light source in the grayscale image is greater than the grayscale value of any pixel corresponding to the light field, and the closer the grayscale value of the pixel to the pixel corresponding to the light source in the light field is, the greater the illumination intensity of the pixel is.

[0008] It can be understood that the illumination reference map may include an omnidirectional illumination reference map and a unidirectional illumination reference map.

[0009] In an optional example, the electronic device may determine the area circled by the user in the first image as the subject position, and based on the light source type selected by the user, and when the user chooses to generate an omnidirectional illumination reference map, generate an omnidirectional illumination reference map based on the subject position selected by the user. For example, when the user selects a spotlight light source and chooses to generate an omnidirectional illumination reference map, the light source position may be selected based on the subject position, and a multi-frame illumination reference map may be generated based on the light source position, in which the radiation direction of the light source has a certain angle with the horizontal direction.

[0010] In other optional instances, based on the type of light source selected by the user and when the user chooses to generate a unidirectional lighting guide map, the position and range of the light source selected by the user (i.e., the light field coverage range) are obtained to generate a unidirectional lighting reference map, that is, a frame of lighting reference map is generated in which the radiation direction of the light source has a certain angle with the horizontal direction.

[0011] In some optional instances, when determining the lighting reference map, part or all of the area in the omnidirectional lighting reference map may be used as the lighting reference map, or part or all of the area in the unidirectional lighting reference map may be used as the lighting reference map.

[0012] In an embodiment of the present application, by generating a lighting reference map for subsequent processing of the image to be processed, not only the workload of modeling each object in the image to be processed can be reduced, but also the types of new images with different lighting intensities and / or lighting directions that are subsequently generated can be enriched.

[0013] In some optional examples of the first aspect, the lighting parameter value includes at least one of a lighting direction and a lighting intensity.

[0014] It can be understood that the illumination directions of the first image and the second image may be different, the illumination intensities of the first image and the second image may be different, and both the illumination directions and illumination intensities of the first image and the second image may be different.

[0015] In some optional instances of the first aspect, the image information includes at least one of color, texture, and brightness.

[0016] In some optional instances of the first aspect, the first data element in the first image matrix includes first data, second data, and third data, wherein the first data represents the color value of the pixel corresponding to the first data element, the second data represents the texture value of the pixel, and the third data represents the brightness value of the pixel.

[0017] In some optional examples of the first aspect, the second data element in the first reference image matrix includes fourth data, and the fourth data represents a brightness value of a pixel point corresponding to the second data element.

[0018] In some optional instances of the first aspect, calculating a first image matrix of a first image to be processed includes: inputting the first image into a diffusion model, encoding the first image based on the diffusion model, and obtaining a first image matrix of the first image.

[0019] It can be understood that the diffusion model can have a diffusion parameter to control the degree of modification of the image during the diffusion process and the reconstruction process. The diffusion parameter can be recorded as denosing. The larger the diffusion parameter, the more noise is added to the image during the diffusion process, and more information needs to be regenerated during the reconstruction process, that is, the greater the difference between the generated reconstructed image and the image to be processed, and vice versa.

[0020] It can be understood that the working principle of the diffusion model is: in the inference calculation process of the diffusion model, noise is gradually added to the image, and then based on the noise added at each step, how to remove the noise is learned to restore the original image. Among them, the inference calculation process of the diffusion model can be: extracting the initial features such as depth, edge and texture of the sample image, and inputting the sample image and the sample illumination reference map into the diffusion model, that is, inputting the sample illumination reference map into the img2img process, inputting the sample image into the control net, and performing noise processing on the sample image based on the current diffusion parameter of the diffusion model (that is, adding the noise of the sample image) to obtain the image after noise processing. Denoising the image after noise processing to obtain a candidate image. Then, extract the current features such as depth, edge and texture of the candidate image, and compare the initial features with the current features for similarity. When the similarity is greater than the similarity threshold, the current diffusion parameter is determined as the diffusion parameter of the diffusion model, otherwise the current diffusion parameter of the diffusion model is adjusted, and the steps of extracting features, adding noise, denoising, and similarity comparison are repeated until the similarity is greater than the similarity threshold.

[0021] In the embodiment of the present application, a batch of reconstructed images with new illumination directions and / or illumination intensities can be generated through diffusion model processing, as well as the guidance and specification of img2img and control net.

[0022] In some optional instances of the first aspect, a first reference image matrix is ​​used to adjust data in the first image matrix to obtain a second image matrix, including: based on diffusion parameters of a diffusion model and the data in the first reference image matrix, adjusting the data in the first image matrix to obtain the second image matrix, wherein the diffusion parameters represent the weight of the data in the first reference matrix participating in the adjustment in the process of adjusting the data in the first image matrix to the second image matrix.

[0023] In some optional instances of the first aspect, based on diffusion parameters of a diffusion model and data in a first reference image matrix, the data in the first image matrix is ​​adjusted to obtain a second image matrix, including: multiplying the diffusion parameters by the first reference image matrix to obtain a third image matrix; and adding the third image matrix to the first image matrix to obtain the second image matrix.

[0024] In some optional instances of the first aspect, the method also includes: extracting a first feature of the first image, and extracting a second feature of the second image; comparing the similarity between the first feature and the second feature; and adjusting the diffusion coefficient of the diffusion model corresponding to the similarity between the first feature and the second feature being less than or equal to a similarity threshold.

[0025] In some optional examples of the first aspect, the first feature and the second feature include at least one of depth, edge, and texture.

[0026] In some optional examples of the first aspect, the diffusion coefficient of the diffusion model is adjusted by the following formula:

[0027]

[0028] Among them, P denoising It can be expressed as the diffusion parameter, P denoising The value range of P is (0,1). depth It can represent the depth similarity, P line It can represent the edge similarity, P texture Can represent texture similarity, P depth , P line , P texture The value range of is (10,100), α can represent the weight of depth similarity, β can represent the weight of edge similarity, γ can represent the weight of texture similarity, and 0<α, β, γ<1, and α+β+γ=1.

[0029] In a second aspect, the present application provides an electronic device, comprising: a memory for storing instructions executed by one or more processors of the electronic device, and a processor, which is one of the one or more processors of the electronic device, for executing the image processing method mentioned in the present application.

[0030] In a third aspect, the present application provides a readable storage medium, on which instructions are stored. When the instructions are executed on an electronic device, the electronic device executes the image processing method mentioned in the present application.

[0031] In a fourth aspect, an embodiment of the present application provides a computer program product, including: a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium containing a computer program code for executing an image processing method. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 According to some examples of the present application, a schematic diagram of an application scenario is shown;

[0033] Figure 2 According to some examples of the present application, a schematic diagram of a framework of an image processing system is shown;

[0034] Figure 3 According to some examples of the present application, a schematic flow chart of an image processing method is shown;

[0035] Figure 4 According to some examples of the present application, a schematic diagram of a framework of an image processing system is shown;

[0036] Figure 5 According to some examples of the present application, a structural schematic diagram of a guided light generation module is shown;

[0037] Figure 6 According to some examples of the present application, a schematic flow chart of a method for generating a lighting reference map is shown;

[0038] Figure 7 According to some examples of the present application, a schematic diagram of a point light source and a spotlight light source is shown;

[0039] Figure 8 According to some examples of the present application, a structural schematic diagram of an image processing and generation module is shown;

[0040] Fig. 9 According to some examples of the present application, a schematic flow chart of a method for generating a reconstructed image is shown;

[0041] Fig.10According to some examples of the present application, a schematic diagram of the difference comparison between a reconstructed image and an image to be processed as the diffusion parameter changes is shown;

[0042] Fig.11 According to some examples of the present application, a schematic diagram of the effect of controlling a network for processing an image is shown;

[0043] Fig.12 According to some examples of the present application, a schematic diagram of a processing flow of a diffusion model is shown;

[0044] Fig.13 According to some examples of the present application, a schematic diagram of the structure of an adaptive adjustment module is shown;

[0045] Fig.14 According to some examples of the present application, a schematic flow chart of a method for adjusting diffusion parameters to generate a reconstructed image is shown;

[0046] Fig.15 According to some examples of the present application, a schematic diagram of the relationship between the diffusion parameter and the similarity is shown;

[0047] Fig.16 According to some examples of the present application, a flowchart of another image processing method is shown as follows:

[0048] Fig.17 According to some examples of the present application, a schematic diagram of a lighting reference map is shown;

[0049] Fig.18 According to some examples of the present application, a hardware structure of an electronic device is shown. DETAILED DESCRIPTION

[0050] The illustrative embodiments of the present application include, but are not limited to, an image processing method, an electronic device, and a storage medium.

[0051] It can be understood that the electronic device mentioned in the embodiments of the present application can be an electronic device capable of displaying static images or dynamic images, such as any implementable electronic device such as a smart phone, a tablet computer, a desktop computer or a personal digital assistant.

[0052] It can be understood that the image processing method mentioned in the embodiments of the present application can be applied to scenes such as movie post-production, game screen rendering, photo secondary editing, static wallpaper display, dynamic wallpaper display, car display, and smart wearable device display.

[0053] Figure 1 A schematic diagram of an application scenario is shown, such as Figure 1As shown, the electronic device 100 can adjust the illumination direction and / or illumination intensity of the image to be processed 110 to obtain a new image 120. The image can be an image in a movie screen, an image in a game screen, a photo that needs secondary editing, an image in a static wallpaper, and an image in a dynamic wallpaper.

[0054] The image processing methods mentioned in some embodiments are described in detail below in conjunction with the image processing system.

[0055] Figure 2 A schematic diagram of the framework of an image processing system is shown in FIG. Figure 2 As shown, the image processing system 200 may include: a scene data acquisition module 210 , a lighting model generation module 220 , a new lighting condition calculation module 230 , a scene re-rendering module 240 and an image synthesis module 250 .

[0056] The scene data acquisition module 210 may be used to acquire geometric information, texture information, and depth information of objects in the image to be processed through a camera or a laser scanner or other device.

[0057] The illumination model generation module 220 can be used to construct a point cloud model (i.e., a multi-dimensional model of the object) based on the depth information of the object, and generate an illumination model based on the geometric information and texture information of the object, wherein the illumination model can include an illumination equation and a reflection equation, wherein the illumination equation can describe the propagation process of light from the light source to the surface of the object, and the reflection equation can describe the propagation process of light reflected from the surface of the object to the observer. Based on the illumination equation, it can be used to determine the light source position, light source intensity, material reflectivity, etc. of the object.

[0058] The module 230 for calculating new lighting conditions may be used to determine a new lighting direction and / or lighting intensity based on user input parameters or preset parameters and a lighting model.

[0059] The scene re-rendering module 240 may be used to re-render objects according to new lighting direction and / or lighting intensity to generate a new image.

[0060] The composite image module 250 may be used to composite a new image with an image to be processed to generate a final image.

[0061] However, for scenes with complex backgrounds and diverse lighting, such as dynamic wallpapers, modeling objects in images can be a huge workload.

[0062] In order to solve the above problems, an embodiment of the present application provides another image processing method. In this method, the image to be processed is first converted into an image matrix to be processed containing image information. For example, the data in the image matrix to be processed includes numerical values ​​representing the color, texture, brightness, etc. of each pixel in the image to be processed. Then, a plurality of pre-set reference image matrices are used to adjust the data of the image matrix to be processed, wherein the data in the reference image matrix includes the brightness value in the illumination reference map. The illumination reference map is a reference map with a certain illumination direction and a certain illumination intensity. In this way, by using different reference image matrices, the image matrix to be processed can be adjusted to an image matrix with different illumination directions and / or illumination intensities. Then, these image matrices are reconstructed to obtain multiple images with different illumination directions and / or illumination intensities corresponding to the image to be processed. In this way, there is no need to model the objects in the image to be processed, which can reduce the workload of the device.

[0063] It can be understood that in some embodiments of the application, the illumination reference map can be a grayscale map, and different illumination reference maps have different illumination parameters. Each pixel in the illumination guide map has a value representing the brightness value of the pixel, which can be a value obtained by converting the three values ​​of each pixel in the color image (i.e., an image with three values ​​of R, G, and B).

[0064] In addition, it can be understood that in some other embodiments of the present application, the image to be processed and various illumination reference images can be converted into corresponding image matrices through a diffusion model. The image to be processed is subjected to noise processing using the diffusion model, that is, the data in the preset matrix is ​​increased or decreased based on the diffusion parameters of the diffusion model, and added to the matrix of the image to be processed to obtain a new matrix, which can represent the image after noise processing. Then, the image after noise processing is reconstructed using the diffusion model to obtain a new image.

[0065] The image processing method mentioned in the embodiments of the present application is introduced below. Figure 3 A schematic diagram of a process flow of an image processing method is shown, and the method can be executed by an electronic device. Figure 3 As shown, the image processing method may include:

[0066] 301: Calculate a first image matrix of a first image to be processed.

[0067] It can be understood that the data in the first image matrix represents the image information of each pixel in the first image, wherein the image information includes color, texture, brightness, etc.

[0068] 302: Using the first reference image matrix, adjust the data in the first image matrix to obtain a second image matrix.

[0069] It can be understood that the first reference image matrix corresponds to a lighting reference map having a first lighting parameter value. The lighting reference map may be a grayscale image, and different lighting reference maps have different lighting parameters. Each pixel in the lighting guide map has a value representing the brightness value of the pixel, which may be a value obtained by converting the three values ​​of each pixel in a color image (i.e., an image having three values ​​of R, G, and B).

[0070] 303: Obtain a second image based on the second image matrix, wherein the illumination parameter value of the second image is different from the illumination parameter value of the first image.

[0071] In some optional examples, the first image can be input into a diffusion model, the first image can be denoised based on the diffusion parameters of the diffusion model, and the first image after the denoising can be reconstructed using the diffusion model, that is, the second image can be denoised to obtain a third image. Part of the information of the illumination reference image and part of the information of the first image can be balanced and retained based on the diffusion parameters of the diffusion model to obtain a third image having the illumination direction and / or illumination intensity in the illumination reference image.

[0072] The following is an introduction to the image processing system used in the image processing method mentioned in the embodiments of the present application.

[0073] Figure 4 A schematic diagram of the framework of an image processing system is shown in FIG. Figure 4 As shown, the image processing system 400 may include: a guided light generation module 410 , a picture processing and generation module 420 , and an adaptive adjustment module 430 .

[0074] The guiding light generating module 410 may be used to generate a lighting reference map, and input the lighting reference map into the image processing and generating module 420 .

[0075] In some optional examples, the guide light generation module 410 can generate one or more frames of illumination reference maps that the user wants. The guide light generation module 410 can generate the illumination reference map based on the drawing area and drawing graphics of the user on the image to be processed. For example, a circle drawn on the face area of ​​the portrait image based on the weaker illumination intensity selected by the user generates a frame of illumination reference map, and a triangle drawn on the hill image based on the stronger illumination intensity selected by the user (such as Figure 1 As shown in FIG. 4 , a frame of illumination reference map is generated. The guide light generation module 410 may also provide an illumination reference map library for the user to select a desired illumination reference map. The guide light generation module 410 may also provide an illumination reference map library for the user to modify to obtain the illumination reference map desired by the user.

[0076] The image processing and generation module 420 can use artificial intelligence generated content (AIGC) technology to generate multiple frames of new images with different lighting directions and / or lighting intensities from the image to be processed, that is, multiple frames of images with new lighting directions and / or lighting intensities, based on the image to be processed and the lighting reference map.

[0077] The adaptive adjustment module 430 can be used to perform similarity judgment on the image to be processed and each new frame of multiple frames of new images, and adjust the relevant parameters in the image processing and generation module 420 according to the similarity between the image to be processed and the new image, so that the generated new image has a natural lighting effect while retaining as much information of the image to be processed as possible (i.e., without deformation).

[0078] The image processing system mentioned above is introduced in detail below. Figure 5 A schematic diagram of the structure of a guided light generating module is shown. Figure 5 As shown, the guide light generation module 410 may include: a lighting reference map generation module 411 and a lighting reference map selection module 412 .

[0079] The illumination reference map generation module 411 may be used to generate an illumination reference map, wherein the illumination reference map may be drawn by the user, selected by the user from an illumination reference map library, or obtained by the user modifying an illumination reference map in the illumination reference map library.

[0080] The illumination reference map selection module 412 is used to obtain a final illumination reference map based on the user's further selection of the illumination reference map generated by the illumination reference map selection module. In some optional examples, the user can select part or all of the areas in the illumination reference map generated by the illumination reference map generation module 411 as the final illumination reference map.

[0081] In some optional instances, the guided light generation module 410 can implement a guided light generation process, whose main function is to generate an omnidirectional lighting reference map or a unidirectional lighting reference map to serve as an input of the image processing and generation module 420 together with the image to be processed.

[0082] The specific process of generating multiple frames of illumination reference images by the guide light generation module is introduced below. Figure 6 A schematic flow chart of a method for generating a lighting reference map is shown, which can be executed by an electronic device, and specifically by a guide light generation module in an image processing system of the electronic device. Figure 6 As shown, the method for generating a lighting reference map may include:

[0083] 601: Select the main position of the picture.

[0084] It can be understood that the picture can be an image to be processed, and the subject position can be determined based on the area in the image to be processed selected by the user where the lighting direction and / or lighting intensity is to be changed. For example, the area circled by the user in the image to be processed is determined as the subject position, so that the subsequently generated lighting reference map can change the lighting direction and / or lighting intensity of the subject position.

[0085] 602: Select the light source type.

[0086] It can be understood that the light source type may include point light sources and spotlight light sources, and may also include other types of light sources, which are not specifically limited in the embodiments of the present application. Figure 7 A schematic diagram of a point light source and a spotlight light source is shown, Figure 7 As shown, a point light source radiates a light field uniformly from the light source to the surroundings, and a spotlight light source radiates a light field from the light source to the light source, for example, from the upper left corner to the lower right corner. The entire light field diffuses and attenuates within a conical area (i.e., the light field) starting from the light source.

[0087] 603: Whether to generate an omnidirectional lighting reference map.

[0088] It can be understood that whether to generate an omnidirectional illumination reference map or a unidirectional illumination reference map can be determined based on whether the user selects to generate an omnidirectional illumination reference map or a unidirectional illumination reference map. When the user selects to generate an omnidirectional illumination reference map, that is, when the judgment result is yes, the process proceeds to step 604, that is, generating a surround illumination reference map around the subject position. When the user selects to generate a unidirectional illumination reference map, that is, when the judgment result is no, the process proceeds to step 605, that is, defining the light field coverage range and generating an illumination reference map.

[0089] 604: Generate an ambient lighting reference map around the subject position.

[0090] It can be understood that if the user chooses to generate an omnidirectional illumination reference map, the omnidirectional illumination reference map can be generated based on the subject position selected by the user. For example, when the user selects a point light source and chooses to generate an omnidirectional illumination reference map, the light source position can be selected based on the subject position, and an illumination reference map of a circular light field can be generated based on the light source position. When the user selects a spotlight light source and chooses to generate an omnidirectional illumination reference map, the light source position can be selected based on the subject position, and a multi-frame illumination reference map in which the radiation direction of the light source has a certain angle with the horizontal direction can be generated based on the light source position.

[0091] 605: Delineate the light field coverage and generate a lighting reference map.

[0092] It can be understood that if the user chooses to generate a unidirectional lighting reference map, the position and range of the light source selected by the user (i.e., the light field coverage range) can be obtained to generate a unidirectional lighting reference map, that is, a frame of lighting reference map in which the radiation direction of the light source has a certain angle with the horizontal direction.

[0093] 606: Adjust the distance of the generated light source from the subject position.

[0094] It can be understood that when the user chooses to generate an omnidirectional lighting reference map, the distance between the light source and the subject position can be adjusted, and the size of the light field can be adjusted, such as adjusting the radius of the circular light field and adjusting the length of the conical light field.

[0095] 607: Select the area of ​​the lighting reference map you want to use.

[0096] It can be understood that when the user chooses to generate a unidirectional lighting reference map, part or all of the area in the unidirectional lighting reference map selected by the user can be obtained as the lighting reference map area.

[0097] 608: Determine the lighting reference map.

[0098] In some optional examples, part or all of the areas in the omnidirectional illumination reference map may be used as the illumination reference map, and part or all of the areas in the unidirectional illumination reference map may also be used as the illumination reference map.

[0099] It can be understood that the illumination reference map can be a grayscale map. The larger the grayscale value of the pixel in the grayscale map, the stronger the intensity of the illumination. Among them, the grayscale value of the pixel is 0, which can indicate that the pixel is black, and the grayscale value of the pixel is 255, which can indicate that the pixel is white. The illumination intensity of the pixel decreases as the distance between the pixel and the pixel corresponding to the light source increases, and reaches 0 within the specified range. That is, the grayscale value of the pixel corresponding to the light source in the grayscale map is greater than the grayscale value of any pixel corresponding to the light field, and the closer the grayscale value of the pixel is to the pixel corresponding to the light source in the light field, the stronger the illumination intensity of the pixel.

[0100] In some optional examples, the illumination intensity of a pixel is inversely proportional to the square of the distance between the pixel and the pixel corresponding to the light source, that is, the grayscale value of the pixel in the grayscale image conforms to the law of decreasing distance. The specific relationship is shown in Formula 1:

[0101]

[0102] Among them, intensity can represent the light intensity represented by the pixel in the grayscale image, and distance can represent the distance between the pixel in the grayscale image and the pixel corresponding to the light source.

[0103] It can be understood that the light field in the illumination reference image can be directional or can be diffused in all directions. For example, the light field of a point light source diffuses in all directions, and the light field of a spotlight light source is directional.

[0104] In the embodiment of the present application, by guiding the light generation module to generate a lighting reference map for subsequent processing of the image to be processed, not only can the workload of modeling each object in the image to be processed be reduced, but also the types of new images with different lighting intensities and / or lighting directions that are subsequently generated can be enriched.

[0105] The following is a detailed introduction to the image processing system mentioned above. Figure 8 A schematic diagram of the structure of an image processing and generation module is shown, such as Figure 8 As shown, the image processing and generating module 420 may include: an image information creator 421 and an image decoder 422 .

[0106] The image information generator 420 is a main module for modifying the content of the image to be processed, and can generate lower-dimensional image information, also called implicit spatial information, that is, a set of vectors representing image information in the feature space.

[0107] The picture decoder 422 can be used to perform dimensionality-upgrading processing on the picture information of lower dimension generated by the picture information generator 421, and restore the complete picture information, that is, the picture information containing new illumination information.

[0108] In some optional instances, the image information generator 421 may be composed of U-Net (an image segmentation model) and a scheduler, and the image decoder may be an auto-encoder & decoder.

[0109] In some optional instances, the image processing and generation module 420 can implement the image processing and generation process, whose main function is to process the image to be processed according to the lighting reference map so that the image to be processed has the lighting direction and / or lighting intensity in the lighting reference map.

[0110] The following introduces the specific process of the image processing and generation module generating a new image. Fig. 9 A schematic flow chart of a method for generating a reconstructed image is shown, which can be executed by an electronic device, and specifically by an image processing and generation module in an image processing system of the electronic device. Fig. 9 As shown, the method for generating a reconstructed image may include:

[0111] 901: Get the lighting reference map.

[0112] It can be understood that the illumination reference map generated by the guide light generation module can be input into the image processing and generation module.

[0113] 902: Upload the lighting reference map to the image generation process.

[0114] It can be understood that the lighting reference map can be uploaded to the image-to-image (img2img) process.

[0115] 903: Set the diffusion parameters of the diffusion model.

[0116] It can be understood that the diffusion parameter of the diffusion model can be used to control the degree of modification of the image during the diffusion process and the reconstruction process. The diffusion parameter can be recorded as denosing. The larger the diffusion parameter, the more noise is added to the image during the diffusion process, and more information needs to be regenerated during the reconstruction process, that is, the greater the difference between the generated reconstructed image and the image to be processed, and vice versa. Fig.10 FIG. 4 shows a schematic diagram of the difference between the reconstructed image and the image to be processed as the diffusion parameter changes. Fig.10 As shown, when the diffusion parameter is 0.1, the difference between the reconstructed image and the image to be processed is smaller than when the diffusion parameter is 1.0. In the embodiment of the present application, by continuously adjusting the diffusion parameters of the diffusion model, fine control of the image can be achieved, thereby achieving high-quality image conversion.

[0117] 904: Obtain an image to be processed.

[0118] 905: Upload the image to be processed to the control network process.

[0119] It is understood that the image to be processed can be uploaded to a control network process (control net), wherein the control network process can gradually intervene in the processing of the diffusion model and control the information of the image through the guidance mode so as to further standardize the image generated by the diffusion model.

[0120] 906: Select boot mode.

[0121] It can be understood that the guidance mode of the control network process selected by the user can be obtained, wherein the guidance mode can include canny edge, depth, normal direction, line drawing, human skeleton, gesture and other modes. After obtaining the guidance mode selected by the user, the control network process can be started, and the image to be processed is subjected to control network processing based on the guidance mode to obtain an image processed by the control network. Fig.11 A schematic diagram of the effect of controlling the network for processing an image is shown. Fig.11As shown, under the guidance of line drawings, the main structure and relative positions of the buildings have not changed.

[0122] 907: The diffusion model enters iteration.

[0123] It can be understood that the working principle of the diffusion model is: in the inference calculation process of the diffusion model, noise is gradually added to the image, and then based on the noise added at each step, how to remove the noise is learned to restore the original image. Among them, the inference calculation process of the diffusion model can be: extracting the initial features such as depth, edge and texture of the sample image, and inputting the sample image and the sample illumination reference map into the diffusion model, that is, inputting the sample illumination reference map into the img2img process, inputting the sample image into the control net, and performing noise processing on the sample image based on the current diffusion parameter of the diffusion model (that is, adding the noise of the sample image) to obtain the image after noise processing. Denoising the image after noise processing to obtain a candidate image. Then, extract the current features such as depth, edge and texture of the candidate image, and compare the initial features with the current features for similarity. When the similarity is greater than the similarity threshold, the current diffusion parameter is determined as the diffusion parameter of the diffusion model, otherwise the current diffusion parameter of the diffusion model is adjusted, and the steps of extracting features, adding noise, denoising, and similarity comparison are repeated until the similarity is greater than the similarity threshold.

[0124] In some optional embodiments, Fig.12 A schematic diagram of the processing effect of a diffusion model is shown. During the processing of the diffusion model, the image to be processed can be subjected to noise processing based on the current diffusion parameters of the diffusion model, as shown in the processing flow in the first direction, to obtain a noise-processed image. Then, the noise-processed image is reconstructed through img2img and control net, as shown in the processing flow in the second direction, to obtain a denoised image, and based on the information of the guided mode directional control image of the control net, a reconstructed image with a new illumination direction and / or illumination intensity is obtained.

[0125] In the embodiment of the present application, a batch of reconstructed images with new illumination directions and / or illumination intensities can be generated through diffusion model processing, as well as the guidance and specification of img2img and control net.

[0126] 908: Determine similarity.

[0127] In some optional instances, after generating a reconstructed image, features such as depth, edge, and texture of the reconstructed image can be extracted, as well as features such as depth, edge, and texture of the image to be processed can be extracted, and the features of the reconstructed image can be compared with the features of the image to be processed for similarity to determine the similarity between the reconstructed image and the image to be processed, wherein the similarity can represent the difference between the reconstructed image and the image to be processed.

[0128] 909: Determine whether the output condition is met.

[0129] In some optional examples, when the similarity is greater than the similarity threshold, it can be determined that the reconstructed image meets the output condition, that is, the reconstructed image can be used as the final image, and the process goes to step 910, that is, entering the calculation in the next direction. When the similarity is less than or equal to the similarity threshold, it can be determined that the reconstructed image does not meet the output condition, and the process goes to step 911, that is, adjusting the diffusion parameter according to the strategy.

[0130] 910: Enter the calculation of the next direction.

[0131] It can be understood that a frame of illumination reference map can be re-acquired from the omnidirectional illumination reference map as a new round of illumination reference map, wherein the illumination direction of the re-acquired illumination reference map and the illumination reference map acquired in the previous round can be different.

[0132] 911: Adjust diffusion parameters according to the strategy.

[0133] It can be understood that the diffusion parameters of the diffusion model can be adjusted based on the adaptive adjustment module, and the specific adjustment method will be described in detail below.

[0134] In the embodiment of the present application, by using a diffusion model to continuously add noise and then remove the noise, and combining multiple methods to control the diffusion model to generate a reconstructed image, the reconstructed image generated by the diffusion model can be gradually approached to the expected value.

[0135] In some optional instances, when generating a reconstructed image, the ambient light of the image to be processed can be modified according to guide words. For example, when the radiation direction of the light source in the lighting reference image is from right to left, guide words such as "sunset", "dusk", and "sunset" can be added to change the ambient light of the image to be processed to a certain extent, thereby achieving a natural lighting effect.

[0136] In some optional examples, the description words of the image to be processed can be extracted based on CLIP (a pre-trained model based on contrastive text-image pairs), such as Fig.12 For the image to be processed in , descriptive words such as "kitten", "lying down", "indoor", and "brown" can be extracted, and the extracted descriptive words can be used as guide words for reconstructing the image to ensure that the difference between the image to be processed and the reconstructed image is small.

[0137] The following is a detailed introduction to the image processing system mentioned above. Fig.13 A schematic diagram of the structure of an adaptive adjustment module is shown. Fig.13As shown, the image processing and generating module 430 may include: a depth similarity calculation module 431 , an edge similarity calculation module 432 , a texture similarity calculation module 433 and a total similarity calculation and determination module 434 .

[0138] The depth similarity calculation module 431 may be used to calculate the similarity of depth information between the reconstructed image and the image to be processed through monocular depth estimation.

[0139] The edge similarity calculation module 432 may be used to detect edge information of the reconstructed image and the image to be processed through Hough transform.

[0140] The texture similarity calculation module 433 can be used to calculate the texture information of the reconstructed image and the image to be processed, that is, to compare the color and material of the image, through the local binary pattern (LBP) detection algorithm under the LAB channel.

[0141] The total similarity calculation and discrimination module 434 can be used to integrate the similarity calculation results of the depth similarity calculation module 431, the edge similarity calculation module 432 and the texture similarity calculation module 433, and adjust the diffusion parameters of the diffusion model to balance the lighting amount and the degree of retention of the original information.

[0142] In some optional instances, the image processing and generation module 430 can implement an adaptive parameter adjustment process, whose main function is to evaluate the reconstructed image. If the reconstructed image meets the preset conditions, it is directly output; otherwise, the diffusion parameters of the diffusion model are adjusted to regenerate the reconstructed image.

[0143] In some optional instances, the functions of the depth similarity calculation module 431, the edge similarity calculation module 432 and the texture similarity calculation module 433 are to extract indicators for evaluating the loss of details of the reconstructed image relative to the image to be processed from the reconstructed image, and the function of the total similarity calculation and discrimination module 434 is to summarize the three indicators and feedback the amount of noise added to the image in the image generation process.

[0144] The specific process of adjusting diffusion parameters based on the adaptive adjustment module to generate a reconstructed image is introduced below. Fig.14 A schematic flow chart of a method for adjusting diffusion parameters to generate a reconstructed image is shown, and the method can be executed by an electronic device, and specifically by an adaptive adjustment module in an image processing system of the electronic device. Fig.14 As shown, the method for adjusting diffusion parameters may include:

[0145] 1401: Extract depth information.

[0146] It can be understood that the depth information can be extracted from the reconstructed image and the image to be processed respectively.

[0147] 1402: Calculate the depth similarity.

[0148] It can be understood that the depth similarity between the reconstructed image and the image to be processed can be calculated based on the depth information of the reconstructed image and the depth information of the image to be processed. The depth similarity can be denoted as P depth .

[0149] In some optional instances, depth estimation can determine the depth relationship between each pixel and other pixels. Each pixel can be assigned a depth value through depth estimation, and after normalization to 0-255 values, it can be determined as a grayscale value.

[0150] In some optional examples, monocular depth estimation can be used to calculate the depth similarity between the reconstructed image and the image to be processed. In addition, midas, leres, leres++, zoe and other methods can also be used to calculate the depth similarity between the reconstructed image and the image to be processed.

[0151] 1403: Extract line information.

[0152] It can be understood that the line information can be extracted from the reconstructed image and the image to be processed respectively.

[0153] In some optional examples, the edge similarity between the reconstructed image and the image to be processed can be calculated by using the edge extraction method of the Canny operator. In addition, the edge information of the reconstructed image and the image to be processed can be extracted by using the edge extraction method of the Sobel operator.

[0154] 1404: Calculate edge similarity.

[0155] It can be understood that the edge similarity between the reconstructed image and the image to be processed can be calculated based on the line information of the reconstructed image and the line information of the image to be processed. The depth similarity can be recorded as P line .

[0156] In some optional instances, edge information can be used as supplementary information of the image. For example, text on clothes and glasses on people's eyes can be judged by edge similarity to determine whether they are similar.

[0157] 1405: Extract texture information.

[0158] It can be understood that texture information can be extracted from the reconstructed image and the image to be processed respectively.

[0159] In some optional examples, the texture information of the reconstructed image and the image to be processed can be calculated by the local binary pattern (LBP) detection algorithm under the LAB channel, that is, the color and material of the image can be compared. The conventional LBP algorithm is mainly used to detect the edge information of the grayscale image. For color images, the image can be converted from the RGB channel to the LAB channel, and the color information can be unified and converted into a single channel. In addition, the LBP algorithm has the characteristics of brightness invariance and insensitivity to illumination, and is very suitable for calculating whether the texture of the reconstructed image and the image to be processed is similar.

[0160] 1406: Calculate texture similarity.

[0161] It can be understood that the texture similarity between the reconstructed image and the image to be processed can be calculated based on the line information of the reconstructed image and the texture information of the image to be processed. The depth similarity can be denoted as P texture .

[0162] 1407: Calculate the total similarity.

[0163] In some optional examples, after obtaining the depth, line, and texture information of the reconstructed image and the image to be processed, they can be converted into comparable indicators.

[0164] 1408: Whether there is an image that meets the similarity requirement.

[0165] In some optional instances, for a batch of reconstructed images generated through diffusion model processing and the guidance and specification of img2img and control net, if there are images in the batch that meet the similarity requirements, it is possible to go to step 1410, that is, output and enter the calculation of the next angle, otherwise, go to step 1409, that is, adjust the diffusion parameters according to the adjustment formula and regenerate.

[0166] In some optional instances, it can be determined whether there are images in a batch of reconstructed images that meet the similarity requirements based on the depth, edge and texture information of the reconstructed image and the image to be processed. For example, the change threshold is set to 10%, and the depth values ​​of the pixels at the same position in the depth map corresponding to the reconstructed image and the depth map corresponding to the image to be processed are normalized to between 0 and 255. For example, the depth value of the pixel in the image to be processed is 100, and the depth value of the pixel at the same position in the reconstructed image is between 90 and 110. Both meet the requirement of not changing more than 10%. At this time, a pixel with a depth value of 98 will be judged as unchanged. By counting how many pixels in the reconstructed image and the image to be processed have not changed, a ratio value of 1-100% can be obtained. When the ratio value is greater than 80%, it can be considered that the reconstructed image meets the similarity requirements.

[0167] 1409: The diffusion parameters are adjusted and regenerated according to the adjustment formula.

[0168] In some optional instances, a larger diffusion parameter means more noise is added to the image, and the generated reconstructed image retains less information of the illumination reference map, while retaining more information of the image to be processed.

[0169] In some optional examples, the adjustment formula may take the form shown in Formula 2:

[0170]

[0171] Among them, P denoising It can be expressed as the diffusion parameter, P denoising The value range of P is (0,1). depth It can represent the depth similarity, P line It can represent the edge similarity, P texture Can represent texture similarity, P depth , P line , P texture The value range of is (10,100), α can represent the weight of depth similarity, β can represent the weight of edge similarity, γ can represent the weight of texture similarity, and 0<α, β, γ<1, and α+β+γ=1.

[0172] Fig.15 FIG. 1 shows a schematic diagram of the relationship between the diffusion parameter and the similarity, that is, the relationship between the diffusion parameter and the similarity when α=1, β=γ=0. Fig.15 As shown, when the depth similarity between the reconstructed image and the image to be processed is high, the diffusion parameter is small, that is, the more information of the illumination reference map is retained, the less information of the image to be processed is retained.

[0173] 1410: Output and enter the calculation of the next angle.

[0174] It can be understood that a frame of illumination reference map can be re-acquired from the omnidirectional illumination reference map as a new round of illumination reference map, wherein the illumination direction of the re-acquired illumination reference map and the illumination reference map acquired in the previous round can be different.

[0175] The following is an overall introduction to the image processing methods mentioned above using a specific example. Fig.16 A schematic diagram of another image processing method is shown in FIG. Fig.16 As shown, the image processing method may include:

[0176] 1601: Generate a lighting reference map.

[0177] It can be understood that a lighting reference map can be generated based on the drawing area and drawing graphics drawn by the user on the image to be processed. For example, a frame of lighting reference map is generated based on a circle drawn on the face area of ​​a portrait image with a weaker lighting intensity selected by the user, and a frame of lighting reference map is generated based on a triangle drawn on a hill image with a stronger lighting intensity selected by the user. A lighting reference map library can also be provided for the user to select the lighting reference map they want. A lighting reference map library can also be provided for the user to modify to obtain the lighting reference map they want.

[0178] In some optional examples, an omnidirectional illumination reference map may be generated, or a unidirectional illumination reference map may be generated, or part or all of the areas in the omnidirectional illumination reference map may be used as the illumination reference map, or part or all of the areas in the unidirectional illumination reference map may be used as the illumination reference map. Fig.17 As shown, the partial area shown in the dotted box in the one-way lighting reference map can be selected as the lighting reference map.

[0179] 1602: Calculate the illumination reference image result at a specific angle.

[0180] It can be understood that a frame of illumination reference map at a specific angle can be selected from the omnidirectional illumination reference map.

[0181] 1603: Get multiple calculation results.

[0182] It can be understood that the illumination direction and / or illumination intensity of an illumination reference image at a specific angle selected from the omnidirectional illumination reference image can be added to the image to be processed without the illumination direction and / or illumination intensity to obtain a batch of reconstructed images.

[0183] 1604: Calculate structure, edge, and texture similarity.

[0184] In some optional instances, after generating a reconstructed image, features such as depth, edge, and texture of the reconstructed image can be extracted, as well as features such as depth, edge, and texture of the image to be processed can be extracted, and the similarity between the reconstructed image and the image to be processed can be calculated based on the features of the reconstructed image and the features of the image to be processed.

[0185] 1605: Whether the overall similarity requirement is met.

[0186] It can be understood that, for a batch of reconstructed images, if there is an image that meets the overall similarity requirement, the process goes to step 1607 , otherwise, the process goes to step 1606 .

[0187] In some optional instances, it can be determined whether there are images in a batch of reconstructed images that meet the similarity requirements based on the depth, edge and texture information of the reconstructed image and the image to be processed. For example, the change threshold is set to 10%, and the depth values ​​of the pixels at the same position in the depth map corresponding to the reconstructed image and the depth map corresponding to the image to be processed are normalized to between 0 and 255. For example, the depth value of the pixel in the image to be processed is 100, and the depth value of the pixel at the same position in the reconstructed image is between 90 and 110. Both meet the requirement of not changing more than 10%. At this time, a pixel with a depth value of 98 will be judged as unchanged. By counting how many pixels in the reconstructed image and the image to be processed have not changed, a ratio value of 1-100% can be obtained. When the ratio value is greater than 80%, it can be considered that the reconstructed image meets the similarity requirements.

[0188] 1606: Adjust diffusion parameters according to the adaptive formula.

[0189] It can be understood that the diffusion parameters can be adjusted based on the above-mentioned formula (2), which will not be described in detail here.

[0190] 1607: Perform next angle calculation.

[0191] It can be understood that a frame of illumination reference map can be re-acquired from the omnidirectional illumination reference map as a new round of illumination reference map, wherein the illumination direction of the re-acquired illumination reference map and the illumination reference map acquired in the previous round can be different.

[0192] 1608: Whether rendering in all directions is completed.

[0193] It can be understood that a frame of illumination reference map can be re-acquired from the omnidirectional illumination reference map as a new round of illumination reference map until all illumination reference maps in the omnidirectional illumination reference map are selected, that is, rendering in all directions is completed, and the process ends. If there is an illumination reference map in the omnidirectional illumination reference map that has not been selected, go to step 1602.

[0194] The following is an introduction to the hardware structure of electronic equipment. Fig.18As shown, the electronic device 1800 may include a processor 1810, an external memory interface 1820, an internal memory 1821, a universal serial bus (USB) interface 1830, a charging management module 1840, a power management module 1841, a battery 1842, an antenna 1, an antenna 2, a mobile communication module 1850, a wireless communication module 1860, an audio module 1870, a speaker 1870A, a receiver 1870B, a microphone 1870C, an earphone interface 1870D, a sensor module 1880, a button 1890, a motor 1891, an indicator 1892, a camera 1893, a display screen 1894, and a subscriber identification module (SIM) card interface 1895, etc. The sensor module 1880 may include a pressure sensor 1880A, a gyroscope sensor 1880B, an air pressure sensor 1880C, a magnetic sensor 1880D, an acceleration sensor 1880E, a distance sensor 1880F, a proximity light sensor 1880G, a fingerprint sensor 1880H, a temperature sensor 1880J, a touch sensor 1880K, an ambient light sensor 1880L, a bone conduction sensor 1880M, and the like.

[0195] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device 1800 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0196] A memory may also be provided in the processor 1810 for storing instructions and data. In some embodiments, the memory in the processor 1810 is a cache memory. The memory may store instructions or data that the processor 1810 has just used or circulated. If the processor 1810 needs to use the instruction or data again, it may be directly called from the memory. Repeated access is avoided, the waiting time of the processor 1810 is reduced, and the efficiency of the system is improved. For example, the processor 1810 may execute the image processing method mentioned in the embodiment of the present application.

[0197] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0198] A memory may also be provided in the processor 1810 for storing instructions and data. In some embodiments, the memory in the processor 1810 is a cache memory. The memory may store instructions or data that the processor 1810 has just used or circulated. If the processor 1810 needs to use the instruction or data again, it may be directly called from the memory. Repeated access is avoided, the waiting time of the processor 1810 is reduced, and the efficiency of the system is improved. For example, the memory may store the image processing method mentioned in the embodiment of the present application.

[0199] The electronic device 1800 implements the display function through a GPU, a display screen 1894, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 1894 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 1810 may include one or more GPUs that execute program instructions to generate or change display information.

[0200] The display screen 1894 is used to display images, videos, etc. The display screen 1894 includes a display panel.

[0201] The electronic device 1800 can implement audio functions such as music playing and recording through the audio module 1870, the speaker 1870A, the receiver 1870B, the microphone 1870C, the headphone jack 1870D, and the application processor.

[0202] The pressure sensor 1880A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 1880A can be set on the display screen 1894. There are many types of pressure sensors 1880A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 1880A, the capacitance between the electrodes changes. The electronic device 1800 determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 1894, the electronic device 1800 detects the touch operation intensity according to the pressure sensor 1880A. The electronic device 1800 can also calculate the touch position according to the detection signal of the pressure sensor 1880A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to a first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0203] The touch sensor 1880K is also called a "touch device". The touch sensor 1880K can be set on the display screen 1894, and the touch sensor 1880K and the display screen 1894 form a touch screen, also called a "touch screen". The touch sensor 1880K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 1894. In other embodiments, the touch sensor 1880K can also be set on the surface of the electronic device 1800, which is different from the position of the display screen 1894.

[0204] The key 1890 includes a power key, a volume key, etc. The key 1890 may be a mechanical key or a touch key. The electronic device 1800 may receive key input and generate key signal input related to user settings and function control of the electronic device 1800.

[0205] The various embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.

[0206] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit, or a microprocessor.

[0207] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0208] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including, but not limited to, floppy disks, optical disks, optical disks, read-only memories (CD-ROMs), magneto-optical disks, read-only memories (ROMs), random access memories (RAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memory, or a tangible machine-readable memory for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in electrical, optical, acoustic, or other forms of propagation signals. Therefore, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a machine (e.g., computer) readable form.

[0209] The above describes the hardware structure that the electronic device may have. It is understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0210] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0211] The various embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.

[0212] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit, or a microprocessor.

[0213] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0214] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including, but not limited to, floppy disks, optical disks, optical disks, read-only memories (CD-ROMs), magneto-optical disks, read-only memories (ROMs), random access memories (RAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memory, or a tangible machine-readable memory for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in electrical, optical, acoustic, or other forms of propagation signals. Therefore, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a machine (e.g., computer) readable form.

[0215] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0216] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0217] Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the present application.

Claims

1. An image processing method, applied to an electronic device, characterized in that: include: Calculating a first image matrix of a first image to be processed, wherein data in the first image matrix represents image information of each pixel in the first image; Using a first reference image matrix, adjusting data in the first image matrix to obtain a second image matrix, wherein the first reference image matrix corresponds to an illumination reference image having a first illumination parameter value; A second image is obtained based on a second image matrix, wherein the illumination parameter value of the second image is different from the illumination parameter value of the first image.

2. The method according to claim 1, characterized in that: The lighting parameter value includes at least one of lighting direction and lighting intensity.

3. The method according to claim 1, characterized in that The image information includes at least one of color, texture, and brightness.

4. The method according to claim 3, characterized in that The first data element in the first image matrix includes first data, second data and third data, wherein: The first data represents the color value of the pixel corresponding to the first data element, the second data represents the texture value of the pixel, and the third data represents the brightness value of the pixel.

5. The method according to claim 1, characterized in that The second data element in the first reference image matrix includes fourth data, and the fourth data represents the brightness value of the pixel corresponding to the second data element.

6. The method according to claim 1, characterized in that The step of calculating a first image matrix of a first image to be processed comprises: The first image is input into a diffusion model, and encoding processing is performed on the first image based on the diffusion model to obtain a first image matrix of the first image.

7. The method according to claim 6, characterized in that The step of using a first reference image matrix and adjusting data in the first image matrix to obtain a second image matrix includes: Based on the diffusion parameters of the diffusion model and the data in the first reference image matrix, the data in the first image matrix is ​​adjusted to obtain a second image matrix, wherein the diffusion parameters represent the weight of the data in the first reference matrix participating in the adjustment in the process of adjusting the data in the first image matrix to the second image matrix.

8. The method according to claim 7, characterized in that The step of adjusting the data in the first image matrix to obtain a second image matrix based on the diffusion parameters of the diffusion model and the data in the first reference image matrix includes: multiplying the diffusion parameter by the first reference image matrix to obtain a third image matrix; The third image matrix is ​​added to the first image matrix to obtain the second image matrix.

9. The method according to any one of claims 6 to 8, characterized in that: The method further comprises: extracting a first feature of the first image, and extracting a second feature of the second image; performing a similarity comparison between the first feature and the second feature; In response to the similarity between the first feature and the second feature being less than or equal to a similarity threshold, a diffusion coefficient of the diffusion model is adjusted.

10. The method according to claim 9, characterized in that The first feature and the second feature include at least one of depth, edge, and texture.

11. The method according to claim 9, characterized in that The diffusion coefficient of the diffusion model is adjusted by the following formula: Among them, P denoising It can be expressed as the diffusion parameter, P denoising The value range of P is (0,1). depth It can represent the depth similarity, P line It can represent the edge similarity, P texture Can represent texture similarity, P depth , P line , P texture The value range of is (10,100), α can represent the weight of depth similarity, β can represent the weight of edge similarity, γ can represent the weight of texture similarity, and 0<α, β, γ<1, and α+β+γ=1.

12. An electronic device, characterized in that: include: A memory, used to store instructions executed by one or more processors of the electronic device, and a processor, which is one of the one or more processors of the electronic device, used to execute the image processing method described in any one of claims 1-11.

13. A readable storage medium, characterized in that: The readable medium stores instructions, and when the instructions are executed on an electronic device, the electronic device executes the image processing method according to any one of claims 1 to 11.

Images