An image processing method, apparatus, storage medium, and electronic device

Abstract

Embodiments of the present invention disclose an image processing method, apparatus, storage medium, and electronic device. The method includes: obtaining an image to be processed, inputting the image to be processed into a shadow prediction model to obtain a shadow image of the image to be processed; inputting the image to be processed into an albedo prediction model to obtain an initial albedo distribution of the image to be processed, and inputting the image to be processed into an albedo edge prediction model to obtain an albedo edge image of the image to be processed, and fusing the initial albedo distribution and the albedo edge image to obtain an albedo image of the image to be processed. By separately designing a prediction model for each intrinsic component, the characteristics of each component are separately encoded, thereby avoiding interference between components when using a single prediction model for prediction and improving the quality of intrinsic decomposition; in addition, each intrinsic component corresponds to a prediction model, which can be applied to more complex scenarios and improve the quality of intrinsic decomposition under complex lighting and texture conditions.

Description

Technical Field

[0001] Embodiments of the present invention relate to image processing technologies, and in particular, to an image processing method, apparatus, storage medium, and electronic device. Background Art

[0002] In the field of image processing, the physical components of an image are crucial for computer vision and graphics applications. The method of extracting image processing is called intrinsic image decomposition.

[0003] Figure 1 As a schematic flow diagram of existing intrinsic image decomposition methods, there are two existing intrinsic image decomposition schemes: 1. Use a single network to simultaneously predict the albedo image and the shadow image; 2. Only use a single network to predict the albedo image from the input image, and then calculate the shadow image by dividing the albedo by the albedo image.

[0004] In the above two schemes, in the first scheme, since the albedo image and the shadow image have different image characteristics, using the same network to simultaneously predict the two components will cause mutual interference and affect performance; in the second scheme, the modeling between the input image, the albedo image, and the shadow image is only applicable to scenes with only diffuse reflection and cannot be applied to daily scenes with complex lighting. Summary of the Invention

[0005] The present invention provides an image processing method, apparatus, storage medium, and electronic device to solve the problems in the prior art that mutual interference is formed when two intrinsic components are simultaneously predicted by the same prediction network and it cannot be applied to scenes with complex lighting, and to improve the prediction performance of each intrinsic component.

[0006] According to one aspect of the present invention, there is provided an image processing method, characterized by comprising:

[0007] Obtain an image to be processed, input the image to be processed into a shadow prediction model, and obtain the shadow image of the image to be processed;

[0008] Input the image to be processed into an albedo prediction model to obtain the initial albedo distribution of the image to be processed, and input the image to be processed into an albedo edge prediction model to obtain the albedo edge image of the image to be processed. Based on the initial albedo distribution and the albedo edge image, fuse to obtain the albedo image of the image to be processed.

[0009] Optionally, the fusing the initial albedo distribution and the albedo edge image to obtain the albedo image of the image to be processed includes:

[0010] Input the initial albedo distribution and the albedo edge image into an image fusion model to obtain the albedo image output by the image fusion model.

[0011] Among them, the shadow prediction model, the albedo prediction model, the albedo edge prediction model, and the image fusion model are respectively fully convolutional neural network models.

[0012] Optionally, there are at least two images to be processed;

[0013] The method further includes:

[0014] Adjust the albedo images of each image to be processed so that the albedo images of each image to be processed match;

[0015] Based on the shadow images and the adjusted albedo images of each image to be processed, fuse to obtain each updated image;

[0016] Merge the updated images to obtain a merged image.

[0017] Further, the adjusting the albedo images of each image to be processed includes:

[0018] Based on the initial albedo distribution of the reference image in the image to be processed, adjust the initial albedo distributions of other images to be processed to update the albedo images of other images to be processed;

[0019] Or,

[0020] Based on the initial albedo distributions in each image to be processed, determine a fused albedo distribution, and update the albedo images of each image to be processed based on the fused albedo distribution and the albedo edge images of each image to be processed.

[0021] Optionally, there are at least two images to be processed, and each image to be processed is obtained at the same acquisition angle for the same object;

[0022] The method further includes:

[0023] Perform a fusion process on the albedo images of each image to be processed to obtain a target albedo image;

[0024] Based on the target albedo image and the shadow image of any one of the images to be processed, perform a fusion to obtain enhanced images of each image to be processed.

[0025] Optionally, adjust the material of one or more objects in the shadow image.

[0026] According to another aspect of the present invention, there is provided an image processing device, characterized by including:

[0027] A shadow image decomposition module, configured to obtain an image to be processed, input the image to be processed into a shadow prediction model, and obtain a shadow image of the image to be processed;

[0028] An albedo image decomposition module, configured to input the image to be processed into an albedo prediction model to obtain an initial albedo distribution of the image to be processed, and input the image to be processed into an albedo edge prediction model to obtain an albedo edge image of the image to be processed, and fuse the initial albedo distribution and the albedo edge image to obtain an albedo image of the image to be processed.

[0029] According to another aspect of the present invention, there is provided an electronic device, including:

[0030] At least one processor; and

[0031] A memory communicatively connected to the at least one processor; wherein,

[0032] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the image processing method according to any embodiment of the present invention.

[0033] According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement the image processing method according to any embodiment of the present invention when executed.

[0034] Embodiments of the present invention disclose an image processing method, apparatus, storage medium, and electronic device. The method includes: obtaining an image to be processed, inputting the image to be processed into a shadow prediction model to obtain a shadow image of the image to be processed; inputting the image to be processed into an albedo prediction model to obtain an initial albedo distribution of the image to be processed, and inputting the image to be processed into an albedo edge prediction model to obtain an albedo edge image of the image to be processed, and fusing the initial albedo distribution and the albedo edge image to obtain an albedo image of the image to be processed. By separately designing a prediction model for each intrinsic component, the characteristics of each component are separately encoded, thereby avoiding interference between components when using a single prediction model for prediction and improving the quality of intrinsic decomposition; in addition, each intrinsic component corresponds to a prediction model, which can be applied to more complex scenarios and improve the quality of intrinsic image decomposition in complex lighting and texture situations.

[0035] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0037] Figure 1 It is a flowchart of an existing intrinsic image decomposition method;

[0038] Figure 2 It is a flowchart of an image processing method provided in Embodiment 1 of the present invention;

[0039] Figure 3 It is a flowchart of an image processing method provided in Embodiment 2 of the present invention;

[0040] Figure 4 It is a flowchart of an image processing method provided in Embodiment 3 of the present invention;

[0041] Figure 5 It is a flowchart of an image processing method provided in Embodiment 4 of the present invention;

[0042] Figure 6 It is a schematic structural diagram of an image processing apparatus provided in Embodiment 5 of the present invention;

[0043] Figure 7 It is a schematic structural diagram of an electronic device provided in Embodiment 6 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0044] In order to enable those skilled in the art to better understand the solutions of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0045] It should be noted that the terms "first", "second", etc. in the description, claims and above-mentioned drawings of the present invention are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that such data used can be interchanged under appropriate circumstances, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0046] Embodiment 1

[0047] Figure 2 FIG. is a flowchart of an image processing method provided for Embodiment 1 of the present invention. This embodiment is applicable to the case of intrinsic image decomposition. The image is decomposed into intrinsic components through a prediction network, and the intrinsic components can be a shadow image, an albedo image, an albedo edge image, etc. This method can be executed by the image processing device provided by the embodiments of the present invention. The image processing device can be implemented by software and / or hardware, and the image processing device can be configured on an electronic computing device. Specifically, the steps are as follows:

[0048] S110. Obtain an image to be processed, and input the image to be processed into a shadow prediction model to obtain a shadow image of the image to be processed.

[0049] S120. Input the image to be processed into an albedo prediction model to obtain an initial albedo distribution of the image to be processed, and input the image to be processed into an albedo edge prediction model to obtain an albedo edge image of the image to be processed. Based on the initial albedo distribution and the albedo edge image, an albedo image of the image to be processed is fused.

[0050] Among them, the image to be processed includes but is not limited to an image obtained by shooting, an image downloaded from the network, an image formed by extracting a video frame from an original video stream, etc.

[0051] In some embodiments, according to the processing requirements of the image, the brightness, light, material, etc. of the image are adjusted. However, since the image contains a large amount of information, when any adjustment is made to the image, the whole image is adjusted, resulting in interference with other information in the image. Exemplarily, adjusting the material of the image may affect the brightness of the image. By performing intrinsic decomposition on the image, any decomposed image of the image can be processed without interfering with other decomposed images, improving the accuracy of image processing and the quality of the processed image.

[0052] In this embodiment, the image is subjected to intrinsic decomposition to obtain the shadow image and the albedo image of the image to be processed. Among them, the shadow image is used to reflect the shadow effect in the scene of the image to be processed, including the effects caused by geometric shapes, occlusion, and mutual reflection; the albedo image is a decomposed image used to reflect the albedo information in the image to be processed, and the albedo is the degree of light reflection of the object material itself without depending on the observation point and specific lighting conditions. In order to improve the image quality of the shadow image and the albedo image, the shadow image and the albedo image of the image to be processed are obtained through independent decomposition methods respectively, and the decomposition processes of the shadow image and the albedo image do not interfere with each other, providing the image quality of the shadow image and the albedo image.

[0053] For the decomposition of the shadow image, it is achieved through the set shadow prediction model. This shadow prediction model has the function of extracting the shadow image of the image. For the input image to be processed, it outputs the shadow image of the image to be processed, solving the problem of mutual interference between components in the process of simultaneously predicting the shadow image and the albedo image by the same prediction network in the prior art, improving the quality of the shadow image, and enabling image decomposition to adapt to a more complex lighting environment. The process of the shadow prediction model predicting the shadow image can be exemplary: given the image to be processed I, the shadow prediction model D S aims to decompose the shadow image S from it, and the formula is as follows:

[0054] A = D s (I, Θ s )

[0055] where Θ S represents the shadow map prediction expert sub-network D S , that is, the convolutional weights of the shadow prediction model. D S is a convolutional neural network. It is also composed of 20 convolutional layers with a convolutional kernel size of 3×3. In order to increase the receptive field of the network, at the 3rd layer, the spatial resolution of the features is reduced to 1 / 4 of the original image, and at the 19th layer, it is restored to the original spatial resolution through a deconvolution layer with a stride of 2. Among them, the number of output channels of the 1st to 19th layers is set to 64. Different from the previous method of using grayscale images to represent shadow images, in order to depict the shadow images generated when the lighting conditions are not white light, we use D SThe number of output channels of the last layer is set to 3, that is, the shadow image is represented as a 3-channel color image. Among them, the convolutional neural network refers to a feedforward neural network that contains convolutional calculations and has a deep structure, has the ability of feature learning, and can also be said to be a deep learning model or a multi-layer perceptron similar to an artificial neural network, and is often used to analyze visual images. For the decomposition of the albedo image, it is realized by setting an albedo prediction model and an albedo edge prediction model. The albedo prediction model has the function of predicting the initial albedo distribution of the image, and the albedo edge prediction model has the function of predicting the albedo edge image of the image. For the input image to be processed, it is respectively input into the albedo prediction model and the albedo edge prediction model. The albedo prediction model outputs the initial albedo distribution of the image to be processed, and the albedo edge prediction model outputs the albedo edge image of the image to be processed. The albedo image of the image to be processed is obtained by fusing the initial albedo distribution and the albedo edge image. The shadow prediction model, the albedo prediction model, and the albedo edge prediction model are independent of each other and do not interfere with each other, solving the problem of mutual interference between components in the process of simultaneously predicting the albedo image and the shadow image in the prior art, and improving the quality of the albedo image; further, the albedo edge image includes the edges generated by the texture edge changes in the albedo image. By fusing the initial albedo distribution and the albedo edge image, the edges in the obtained albedo image are smoothed, improving the quality of the albedo image and enabling the image decomposition to adapt to a more complex lighting environment.

[0056] Based on the albedo prediction model, predict the initial albedo distribution of the image to be processed. Exemplarily, given the image to be processed I, the albedo prediction model D A aims to decompose and obtain the albedo distribution A from it, and the formula is as follows:

[0057] A = D A (I, Θ A )

[0058] where Θ A represents the albedo prediction expert sub-network D A , that is, the convolutional weights of the albedo prediction model. D A is a convolutional neural network. It consists of 20 convolutional layers with a convolutional kernel size of 3×3. The design of its first layer to the 19th layer is the same as that of D S , and the number of output channels of the last layer is set to 3, corresponding to the R, G, and B channels of the albedo image respectively.

[0059] Based on the albedo edge prediction model, predict the albedo edge image. Exemplarily, given the image to be processed I, the albedo edge prediction model D AE aims to decompose and obtain the albedo edge guidance map E A, the formula is as follows:

[0060] E A = D AE (I, Θ AE )

[0061] where Θ AE represents the albedo prediction expert sub-network D AE , that is, the convolution weights of the albedo edge prediction model. D AE is a convolutional neural network. It consists of 20 convolutional layers with a convolutional kernel size of 3×3. The design of its first 19 layers is the same as that of D S , and the number of output channels of the last layer is set to 1 because the edge of the image is represented by a grayscale image.

[0062] Based on the above embodiments, the shadow prediction model, the albedo prediction model, and the albedo edge prediction model are respectively full convolutional neural network models.

[0063] The technical solution of this embodiment separately designs a prediction model for each intrinsic component, so that the characteristics of each component are separately encoded, thereby avoiding interference between components when using a single prediction model for prediction and improving the quality of intrinsic decomposition; in addition, each intrinsic component corresponds to a prediction model, which can be applied to more complex scenarios and improve the quality of intrinsic image decomposition under complex lighting and texture conditions.

[0064] Embodiment 2

[0065] Figure 3 is a schematic flowchart of an image processing method provided by the second embodiment of the present invention. The embodiments of the present invention can be combined with each of the above optional solutions. In the embodiments of the present invention, optionally, the step of fusing the initial albedo distribution and the albedo edge image to obtain the albedo image of the image to be processed includes: inputting the initial albedo distribution and the albedo edge image into an image fusion model to obtain the albedo image output by the image fusion model.

[0066] As Figure 3 shown, the method of the embodiment of the present invention specifically includes the following steps:

[0067] S210. Obtain an image to be processed, and input the image to be processed into a shadow prediction model to obtain a shadow image of the image to be processed.

[0068] S220. Input the image to be processed into an albedo prediction model to obtain the initial albedo distribution of the image to be processed, and input the image to be processed into an albedo edge prediction model to obtain an albedo edge image of the image to be processed.

[0069] S230. Input the initial albedo distribution and the albedo edge image into an image fusion model to obtain the albedo image output by the image fusion model.

[0070] In this embodiment, by setting up an image fusion model, which has the function of optimizing the albedo image, can process the image to be processed, and output an optimized high-quality albedo image, the problem that the quality of the albedo image obtained only by predicting with the albedo prediction network is poor is solved, the albedo image is optimized, and the quality of the albedo image is improved.

[0071] The image fusion model is used to fuse the initial albedo distribution and the albedo edge image to generate a high-quality albedo image. Exemplarily, since the real albedo image is usually represented as an image with limited color intensity and blocky smoothness, and the direct prediction by the deep neural network usually presents poor results and usually requires post-processing using a filtering algorithm. To improve the quality of the albedo image, an image fusion model R following the eigen-decomposition network is set up. This image fusion model is implemented by a convolutional neural network, which can preserve the important features at the texture edges in the input image, while eliminating some artifacts and noises predicted by the network in the smooth regions, and improving the image quality of the obtained albedo image.

[0072] In the stage of fusing the initial albedo distribution and the albedo edge image, the input is set as the initial albedo distribution A predicted by D A and the albedo edge E predicted by D AE . After being processed by the image fusion model R, a high-quality albedo image A A is output. The formula is expressed as: R

[0073] A R = R(A, E A ) .

[0074] Since the adopted deep learning-based image fusion model can learn more complex image operations, a higher-quality albedo image can be obtained.

[0075] Among them, the image fusion model is a fully convolutional neural network model.

[0076] Working principle of the image processing method: The method includes an initial albedo prediction method, an albedo edge image prediction method, a shadow image prediction method, and an albedo image optimization method. After obtaining the image to be processed, the image to be processed is input into an albedo prediction model, an albedo edge prediction model, and a shadow prediction model respectively to predict the initial albedo distribution, the shadow image, and the albedo edge image. Then, the albedo distribution and the albedo edge image are input into an image fusion model to fuse and obtain a high-quality albedo image.

[0077] The technical solution of this embodiment enables the initial albedo distribution and the albedo edge image to be fused by using an image fusion model to obtain a high-quality albedo image, providing more possibilities for subsequent image editing and enhancement applications.

[0078] Based on the above embodiment, for the decomposed images of the image to be processed, the obtained shadow image and / or albedo image can be adjusted, and the processed image is obtained by combining the adjusted shadow image and / or albedo image. During the adjustment of any decomposed image, other decomposed images are not interfered, improving the accuracy of the adjustment object and the quality of the combined image.

[0079] In some embodiments, the image material in the shadow image can be adjusted. Exemplarily, the material of one or more objects in the shadow image is adjusted, and the shadow image will change due to the change of the material of the adjusted object, thereby affecting the entire shadow image. For example, the material of one or more objects in the shadow image can be adjusted. The objects in the shadow image can include, but are not limited to, objects such as floors, walls, and tables. The material of any object in the shadow image can be adjusted according to the adjustment requirements. For example, the material of the table is replaced with a metal material, etc.

[0080] By combining the albedo image and the adjusted shadow image, an image that does not affect the albedo distribution and meets the material adjustment requirements is obtained.

[0081] In some embodiments, the albedo distribution in the albedo image can be adjusted. Exemplarily, by adjusting the albedo distribution in the albedo image, the light information in the image to be processed can be realized. For example, the light in the morning can be adjusted to the light at dusk. During this process, the image content in the shadow image is not affected. By combining the shadow image and the adjusted albedo image, the processed image is obtained.

[0082] In some embodiments, the albedo image and the shadow image can be adjusted separately. During the adjustment process, the adjustments of the albedo image and the shadow image do not interfere with each other. The adjusted albedo image and the adjusted shadow image are combined to obtain the processed image.

[0083] Among them, the combination of the shadow image and the albedo image can be achieved through a pre-set image combination model, or can be obtained by fusing the pixel values of the corresponding pixels of the shadow image and the albedo image. For example, through weighted processing with a pre-set fusion ratio, and there is no limitation to this.

[0084] Embodiment 3

[0085] The embodiments of the present invention can be combined with each of the optional solutions in the above embodiments. Figure 4 It is a schematic flowchart of an image processing method provided by Embodiment 3 of the present invention. In the embodiments of the present invention, optionally, there are at least two images to be processed; the method of the embodiments of the present invention specifically includes the following steps:

[0086] S310. Obtain the image to be processed, and input the image to be processed into the shadow prediction model to obtain the shadow image of the image to be processed.

[0087] S320. Input the image to be processed into the albedo prediction model to obtain the initial albedo distribution of the image to be processed, and input the image to be processed into the albedo edge prediction model to obtain the albedo edge image of the image to be processed. Based on the initial albedo distribution and the albedo edge image, fuse to obtain the albedo image of the image to be processed.

[0088] S330. Adjust the albedo images of each image to be processed so that the albedo images of each image to be processed match.

[0089] S340. Based on the shadow images and the adjusted albedo images of each image to be processed, fuse to obtain each updated image.

[0090] S350. Perform image merging on each updated image to obtain a merged image.

[0091] In this embodiment, the image to be processed can be at least two images to be merged. Exemplarily, the images to be merged can be images including different merging objects. Among them, the merging objects can be people, animals, etc.; Exemplarily, the images to be merged can be background images and foreground images. Due to the differences in the scenes, lighting, shooting methods, etc. of each image, the images to be merged respectively correspond to different albedo distributions, resulting in the problem that the merged image is unnatural and rigid when directly merging the images to be merged. In this embodiment, by adjusting the albedo images of each image to be merged, the albedo distributions of each image are made to match, so that the lighting of the images to be merged matches, and the quality of the merged image is improved.

[0092] Optionally, based on the above embodiments, for adjusting the albedo images of each image to be processed, the method of the embodiments of the present invention specifically includes the following steps: adjusting the initial albedo distribution of other images to be processed based on the initial albedo distribution of the reference image in the images to be processed, so as to update the albedo images of other images to be processed.

[0093] Determine a reference image from the images to be processed, obtain the initial albedo distribution of the reference image through the image processing method provided in any of the above embodiments, and adjust the initial albedo distribution of other images to be processed based on the initial albedo distribution, so as to update the albedo images of other images to be processed. Among them, the reference image may be any image in the images to be processed, or may be the background image in the images to be merged. Exemplarily, the images to be merged include a background image and multiple foreground images including foreground objects. For example, the foreground objects may be people or animals, etc. The background image may be used as the reference image to adjust the albedo images of the foreground images.

[0094] In this embodiment, adjusting the initial albedo distribution of other images to be processed based on the initial albedo distribution of the reference image in the images to be processed may be using the initial albedo distribution of the reference image as the initial albedo distribution of other images to be processed; it may also be extracting the distribution trend of the initial albedo distribution of the reference image and adjusting the initial albedo distribution of other images to be processed based on the distribution trend of the initial albedo distribution of the reference image, so that the distribution trend of the initial albedo distribution of other images to be processed meets the distribution trend of the initial albedo distribution of the reference image.

[0095] Fuse the adjusted albedo distribution of each image to be processed with the corresponding albedo edge image to obtain the adjusted albedo image of each image to be processed.

[0096] Optionally, based on the above embodiments, for adjusting the albedo images of each image to be processed, the method of the embodiments of the present invention specifically includes the following steps: determining a fused albedo distribution based on the initial albedo distribution in each image to be processed, and updating the albedo images of each image to be processed based on the fused albedo distribution and the albedo edge images of each image to be processed.

[0097] The initial albedo distribution of each image to be processed is obtained respectively. Based on the initial albedo distribution, the fused albedo distribution is determined. Based on the fused albedo distribution and the albedo edge images of each image to be processed, the albedo images of each image to be processed are updated. Correspondingly, the albedo images of each image to be processed have a matching albedo distribution. Among them, the fused albedo distribution can be determined by performing weighted processing on the albedo distributions of each image to be processed with a preset fusion ratio. The preset fusion ratio can be determined in advance and is not limited herein. This image adjustment method first determines the fused albedo distribution based on the initial albedo distributions of all images to be processed, and then adjusts and updates all albedo images based on the fused albedo distribution. The use of the fused albedo distribution can provide a better basis for subsequent image fusion, making the fused image more realistic and of higher quality.

[0098] Based on the technical solution of this embodiment and on the basis of the above embodiment, the present application also provides an image merging solution, which merges multiple images (different scenes, lighting, shooting methods, etc.) into the image required by the user, meets the user's needs, and provides convenience for the user.

[0099] After determining the albedo images that match each image to be processed, based on the shadow image and the adjusted albedo image of the same image to be processed, each updated image is fused, and the updated images are merged to obtain a merged image.

[0100] Based on the technical solution of this embodiment, by adjusting the albedo images of multiple images to be merged, albedo images that match are obtained, and updated images of the images to be processed are obtained. The updated images with matching albedo distributions are merged to obtain a natural merging process, improving the quality of the merged image.

[0101] On the basis of the above embodiment, for any image to be processed to be merged, before fusing the shadow image and the adjusted albedo image, it further includes adjusting the material of one or more objects in the shadow image of any image to be processed. Correspondingly, the adjusted shadow image and the adjusted albedo image are fused to obtain the updated image of the image to be processed. The updated images of each image to be processed meet the matching of the albedo distribution and achieve differential material adjustment to obtain a merged image that meets the merging requirements. This solution is applicable to the situation of adjusting the material of one or more objects in an image during the image merging operation, such as adjusting the clothing material and color of a certain person in the image. The change of the material causes the change of the shadow image, and the corresponding albedo image changes accordingly. Adjust each of the albedo images to make them match; based on each adjusted albedo image, image merging is performed to obtain a merged image.

[0102] Based on the above image merging solution, this solution is an image merging solution that adds the adjustment of the shadow image material, which can further meet the user's requirements for the merged image.

[0103] Embodiment 4

[0104] Figure 5 It is a schematic flowchart of an image processing method provided in Embodiment 4 of the present invention. Based on the above embodiments, there are at least two images to be processed, and each image to be processed is obtained by acquiring the same object at the same acquisition angle. The method of the embodiment of the present invention specifically includes the following steps:

[0105] S410. Obtain the image to be processed, and input the image to be processed into the shadow prediction model to obtain the shadow image of the image to be processed.

[0106] S420. Input the image to be processed into the albedo prediction model to obtain the initial albedo distribution of the image to be processed, and input the image to be processed into the albedo edge prediction model to obtain the albedo edge image of the image to be processed. Based on the initial albedo distribution and the albedo edge image, fuse to obtain the albedo image of the image to be processed.

[0107] S430. Perform fusion processing on the albedo images of each image to be processed to obtain the target albedo image.

[0108] S440. Based on the target albedo image and the shadow image of any image to be processed, perform fusion to obtain the enhanced image of each image to be processed.

[0109] In this embodiment, by processing the images to be processed of the same object under different illuminations, the enhanced image of the image to be processed is obtained. It should be noted that the image contents of multiple images to be processed are the same, and correspondingly, the shadow images of each image to be processed are the same. By fusing the albedo images of each image to be processed, the target albedo image with enhanced illumination is obtained. Among them, the determination method of the target albedo image can be to perform fusion processing on the albedo images of each image to be processed with a preset fusion ratio, for example, perform weighted processing on the pixel values of the corresponding pixel points of each albedo image with a preset fusion ratio to obtain the target albedo image.

[0110] This image enhancement method obtains the target albedo image by fusing multiple albedo images. The target albedo image is fused with the shadow image of any image to be processed to obtain the enhanced image, making the image clearer and more complete. This method can efficiently obtain a batch of high-quality enhanced images, and also provides a good basis for the application of images in other fields.

[0111] Based on the technical solution of the above embodiment, the present application further provides an image enhancement solution. Through the above embodiment, an enhanced image of the image can be obtained, providing a better basis for subsequent image editing and applications.

[0112] Based on the above embodiment, before fusing the shadow image and the target albedo image of any image to be processed, it may further include adjusting the material of one or more objects in the shadow image. This solution is applicable to the situation of adjusting the material of one or more objects in the image during image enhancement operations. For example, when obtaining an equal-intensity image, it is necessary to adjust the material of a certain object in the image. The change in the material causes a change in the shadow image, and the corresponding albedo image changes accordingly. The target albedo image is obtained by fusing the changed albedo image, and the enhanced image after adjusting the material is obtained by fusing the changed shadow image and the target albedo image.

[0113] Based on the above image enhancement solution, this solution adds an image enhancement solution after adjusting the material in the image, which can further meet the user's needs for enhanced images.

[0114] Embodiment Five

[0115] Figure 6 FIG. 13 is a schematic structural diagram of an image processing apparatus provided in Embodiment Five of the present invention. The image processing apparatus can execute the image processing method provided in any embodiment of the present invention and has corresponding functional modules and beneficial effects for executing the method.

[0116] See Figure 6 , the image processing apparatus includes:

[0117] A shadow image decomposition module 710, configured to obtain an image to be processed, input the image to be processed into a shadow prediction model, and obtain a shadow image of the image to be processed;

[0118] An albedo image decomposition module 720, configured to input the image to be processed into an albedo prediction model to obtain an initial albedo distribution of the image to be processed, and input the image to be processed into an albedo edge prediction model to obtain an albedo edge image of the image to be processed, and fuse the initial albedo distribution and the albedo edge image to obtain an albedo image of the image to be processed.

[0119] Optionally, the albedo image decomposition module 720 includes:

[0120] An initial albedo distribution decomposition unit 721, configured to input the image to be processed into an albedo prediction model to obtain an initial albedo distribution of the image to be processed;

[0121] An albedo edge image decomposition unit 722 is configured to input the image to be processed into an albedo edge prediction model to obtain an albedo edge image of the image to be processed;

[0122] An image fusion unit 723 is configured to input the initial albedo distribution and the albedo edge image into an image fusion model to obtain an albedo image output by the image fusion model.

[0123] Optionally, the shadow prediction model, the albedo prediction model, the albedo edge prediction model, and the image fusion model are respectively full convolutional neural network models.

[0124] Optionally, there are at least two images to be processed;

[0125] The device further includes:

[0126] An albedo image adjustment module configured to adjust the albedo images of the images to be processed so that the albedo images of the images to be processed match;

[0127] An albedo image fusion module is configured to fuse and obtain respective updated images based on the shadow images and the adjusted albedo images of the images to be processed;

[0128] An image merging module is configured to merge the respective updated images to obtain a merged image.

[0129] Optionally, the albedo image adjustment module includes:

[0130] A first adjustment unit configured to adjust the initial albedo distributions of other images to be processed based on the initial albedo distribution of a reference image in the images to be processed, so as to update the albedo images of the other images to be processed;

[0131] Or,

[0132] A second adjustment unit configured to determine a fused albedo distribution based on the initial albedo distributions in the images to be processed, and update the albedo images of the images to be processed based on the fused albedo distribution and the albedo edge images of the images to be processed.

[0133] Optionally, there are at least two images to be processed, and each of the images to be processed is obtained at the same acquisition angle for the same object.

[0134] The device further includes:

[0135] A target albedo image fusion module configured to perform a fusion process on the albedo images of the images to be processed to obtain a target albedo image;

[0136] An image enhancement module, configured to fuse the target albedo image and the shadow image of any image to be processed, so as to obtain the enhanced image of each image to be processed.

[0137] Optionally, the apparatus further includes:

[0138] A shadow image adjustment module, configured to adjust the material of one or more objects in the shadow image. The image processing apparatus provided by the embodiments of the present invention can execute the image processing method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. For the technical details not described in detail, reference may be made to the image processing method provided by any embodiment of the present invention.

[0139] Embodiment Six

[0140] Figure 7 FIG. shows a schematic structural diagram of an electronic device 10 that can be used to implement the embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, a personal digital processor, a cellular phone, a smart phone, a wearable device (such as a helmet, glasses, a watch, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0141] As Figure 7 shown, the electronic device 10 includes at least one processor 11, and a memory communicatively connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program executable by the at least one processor, and the processor 11 can execute various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0142] Multiple components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disc, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0143] The processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as an image processing method.

[0144] In some embodiments, the image processing method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the image processing method described above can be executed. Alternatively, in other embodiments, the processor 11 can be configured to execute the image processing method by any other suitable means (e.g., by means of firmware).

[0145] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, the programmable processor can be a dedicated or general-purpose programmable processor, can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0146] A computer program for implementing the method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The computer programs can be executed entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine, or entirely on the remote machine or server.

[0147] In the context of the present invention, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0148] In order to provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, speech input, or tactile input).

[0149] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), blockchain network, and the Internet.

[0150] A computing system can include a client and a server. The client and the server are generally far from each other and typically interact through a communication network. The client - server relationship is created by computer programs running on respective computers and having a client - server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system and solves the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services.

[0151] It should be understood that various forms of the processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and this is not limited herein.

[0152] The above - described specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub - combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

[0153] Note that the above is only a preferred embodiment of the present invention and the applied technical principles. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, re - adjustments, and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments only. Without departing from the concept of the present invention, it can also include more other equivalent embodiments, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. An image processing method, characterized in that, Including: Obtain an image to be processed, input the image to be processed into a shadow prediction model, and obtain a shadow image of the image to be processed; The image to be processed is at least two images to be merged, and the images to be merged are images including different merging objects; the images to be merged include a foreground image and a background image; Input the image to be processed into an albedo prediction model to obtain an initial albedo distribution of the image to be processed, and input the image to be processed into an albedo edge prediction model to obtain an albedo edge image of the image to be processed, and fuse the initial albedo distribution and the albedo edge image to obtain an albedo image of the image to be processed; Based on the initial albedo distribution of the background image or the fused albedo distribution of each image to be processed, adjust the albedo images of each image to be processed so that the albedo images of each image to be processed match; The fused albedo distribution is obtained by performing weighted processing on the albedo distributions of each image to be processed with a preset fusion ratio; Adjust the material of one or more objects in the shadow image; Based on the adjusted shadow images and adjusted albedo images of each image to be processed, fuse to obtain updated images corresponding to each image to be merged; Perform image merging on the updated images corresponding to each image to be merged to obtain a merged image.

2. The method according to claim 1, wherein The fusing the initial albedo distribution and the albedo edge image to obtain the albedo image of the image to be processed includes: Input the initial albedo distribution and the albedo edge image into an image fusion model to obtain the albedo image output by the image fusion model.

3. The method according to claim 1 or 2, characterized in that, The shadow prediction model, the albedo prediction model, the albedo edge prediction model, and the image fusion model are respectively fully convolutional neural network models.

4. The method according to claim 1, characterized in that, The adjusting the albedo images of each image to be processed includes: Based on the initial albedo distribution of a reference image in the image to be processed, adjust the initial albedo distributions of other images to be processed to update the albedo images of other images to be processed; Or, Determine a fused albedo distribution based on the initial albedo distributions in each image to be processed, and update the albedo images of each image to be processed based on the fused albedo distribution and the albedo edge images of each image to be processed.

5. The method according to claim 1, characterized in that, The images to be processed are at least two, and each image to be processed is obtained at the same acquisition angle for the same object; The method further includes: Perform fusion processing on the albedo images of each image to be processed to obtain a target albedo image; Based on the target albedo image and the shadow image of any one of the images to be processed, perform fusion to obtain enhanced images of each image to be processed.

6. An image processing apparatus, characterized in that, Including: A shadow image decomposition module, configured to obtain an image to be processed, input the image to be processed into a shadow prediction model, and obtain a shadow image of the image to be processed; The image to be processed is at least two images to be merged, and the images to be merged include a foreground image and a background image; An albedo image decomposition module, configured to input the image to be processed into an albedo prediction model to obtain an initial albedo distribution of the image to be processed, and input the image to be processed into an albedo edge prediction model to obtain an albedo edge image of the image to be processed, and fuse the initial albedo distribution and the albedo edge image to obtain an albedo image of the image to be processed; An albedo image adjustment module, configured to adjust the albedo images of the images to be processed based on the initial albedo distribution of the background image or the fused albedo distribution of the images to be processed, so that the albedo images of the images to be processed match each other; A shadow image adjustment module, configured to adjust the materials of one or more objects in the shadow image; An albedo image fusion module, configured to fuse the adjusted shadow images and the adjusted albedo images of the images to be processed to obtain respective updated images; An image merging module, configured to merge the respective updated images to obtain a merged image.

7. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor, so that the at least one processor can execute the image processing method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the image processing method according to any one of claims 1-5 when executed by a processor.

Images