Image shadow removal method and device based on diffusion model

Abstract

The invention relates to the technical field of image processing, and discloses an image shadow removal method and device based on a diffusion model, and the method comprises the steps: obtaining a to-be-processed shadow removal data set which comprises a plurality of groups of to-be-processed shadow removal data; and based on a preset global guide sampling strategy, processing the to-be-processed shadow-removed data through a preset diffusion model, and generating an image after shadow removal. According to the method provided by the invention, through the preset diffusion model, the high-quality shadow-free image which has no obvious boundary and is consistent in illumination can be generated under any image resolution.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and device for removing image shadows based on a diffusion model. Background Art

[0002] In recent years, with the increasing number of cars, parking difficulties and disorderly parking have become one of the main problems that need to be solved in the development of urban transportation in my country. Traffic management departments in various places are also actively promoting the solution to the problem of difficult and disorderly parking on the roadside. In order to effectively alleviate this problem, traffic management departments in various places have tried and implemented various intelligent parking management technologies. Among them, the use of high-position video cameras for intelligent parking management is currently a mainstream solution. By installing high-position video cameras on the roadside for data collection, and then using visual algorithms for license plate detection and license plate recognition, intelligent parking charges and management are realized. However, in actual applications, shadows often appear in the captured video images. Shadows are a phenomenon that occurs when optical images are captured when light is blocked. The presence of shadows complicates image processing and many subsequent visual tasks, such as target detection, semantic segmentation, target tracking, license plate recognition, etc., affecting the accuracy of visual algorithm application.

[0003] The purpose of image shadow removal is to enhance the brightness of the shadow area of ​​the image and make the illumination distribution between the shadow area and the non-shadow area consistent. This task has been studied in depth and for a long time in the past years. The traditional method is based on manual features and physical lighting models to remove shadows. In recent years, with the development of deep learning technology, shadow removal methods have gradually improved, especially the introduction of the transformer architecture, which achieves shadow removal by introducing global information. However, in practical applications, this method has been found to have more residual shadows in the generated results. Recently, the diffusion model has attracted widespread attention in the field of image restoration due to its powerful modeling ability. The shadow removal task is essentially one of the image restoration tasks, but the diffusion model usually uses processed image blocks as input and lacks global information of the image. Therefore, there may be problems of inconsistent lighting or obvious shadow boundaries in the generated de-shadowed image. Summary of the invention

[0004] In view of this, the present invention provides a method and device for removing image shadows based on a diffusion model to solve the problem that there may be inconsistent lighting or obvious shadow boundaries in the generated de-shadowed image.

[0005] In a first aspect, the present invention provides an image shadow removal method based on a diffusion model, the method comprising:

[0006] Acquire a shadow removal data set to be processed, wherein the shadow removal data set to be processed includes a plurality of groups of shadow removal data to be processed;

[0007] Based on the preset global guided sampling strategy, the shadow removal data to be processed is processed through the preset diffusion model to generate an image after shadow removal.

[0008] In an optional implementation, each set of shadow removal data to be processed includes: a shadow-free image x0∈R HxWx3 , images with shadows And the mask image x of the shadow area m ∈R HxWx3 , where H represents the height of the image and W represents the width of the image.

[0009] In an optional implementation, the preset diffusion model is a dual-branch UNet network, wherein the dual-branch UNet network includes: a first branch network and a second branch network, the first network is used to estimate diffusion noise of the image, and the second branch network is used to extract global feature information of the image;

[0010] The first branch network input data is:

[0011]

[0012] in, Represents the first branch network input data, and Crop() represents the position information L i Extract the R*R image area block, where i represents the i-th image block;

[0013] The input data of the second branch network is:

[0014]

[0015] x' m = Downsample(x m )

[0016] Among them, x G Represents the second branch network input data, and Downsample() represents the downsampling of the image.

[0017] In an optional implementation, adjacent image blocks are combined based on a preset global guided sampling strategy to generate an image after shadow removal.

[0018] In an optional implementation, the preset global guided sampling strategy includes:

[0019] The contribution weight of each image block to the global image is calculated by the following formula:

[0020]

[0021] Among them, Sbright represents the score weight based on brightness, S mask represents the score weight based on the shadow mask, t represents each denoising time step, and T represents all time steps; where,

[0022]

[0023] L gray represents the grayscale image of the image patch restored by the local branch, G gray Represents the grayscale image of the global image restored by the global branch, eps is a preset value;

[0024] The score weight based on the shadow mask is calculated by the following formula:

[0025]

[0026] R sf =P sf / P total , T sf Indicates the preset ratio threshold, P sf Represents the number of pixels in the non-shadow area, P total Indicates the total number of pixels.

[0027] In an optional implementation, the preset diffusion model is calculated by the following formula:

[0028] L total =L diff +αL global

[0029]

[0030] L total represents the loss of the image after shadow removal, L diff represents the diffusion loss of the first branch network, L global represents the global loss of the second branch network, α is the weight coefficient, ∈ t represents the feature representation of the predicted output in the tth stage with gradual addition of noise, x r Indicates global output.

[0031] In an optional implementation, when the loss function of the image after shadow removal meets a preset condition, adjacent image blocks are combined to generate the image after shadow removal.

[0032] The image shadow removal method and device based on the diffusion model provided by the present invention obtain a shadow removal data set to be processed, wherein the shadow removal data set to be processed includes multiple groups of shadow removal data to be processed, and based on a preset global guided sampling strategy, the shadow removal data to be processed is processed by a preset diffusion model to generate an image after the shadow is removed. The method provided by the present invention can generate a high-quality shadow-free image without obvious boundaries and consistent lighting at any image resolution through a preset diffusion model.

[0033] In a second aspect, the present invention provides an image shadow removal device based on a diffusion model, the device comprising:

[0034] A data collection and preprocessing module is used to collect the data set to be processed, preprocess the data set to be processed, and store the preprocessing results in a preset time series medical knowledge base;

[0035] A graph community construction module, used to construct a graph community of a preset medical knowledge base according to a preset time-series medical knowledge base;

[0036] Hidden Markov model application module, used to perform time series inference at different time points on the graph community of the preset medical knowledge base based on the hidden Markov model;

[0037] An emission probability matrix construction module is used to construct an emission probability matrix by creating a preset retrieval system and setting an emission probability model;

[0038] The article writing module is used to select article fragments according to the emission probability matrix, write and modify the article according to the preset model, and perform manual review and revision to complete the writing of the medical article.

[0039] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the image shadow removal method based on the diffusion model of the above-mentioned first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0040] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the image shadow removal method based on a diffusion model according to the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 is a flow chart of a method for removing image shadows based on a diffusion model according to an embodiment of the present invention;

[0043] Figure 2 is a structural block diagram of an image shadow removal device based on a diffusion model according to an embodiment of the present invention;

[0044] Figure 3 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0046] According to an embodiment of the present invention, an embodiment of an image shadow removal method based on a diffusion model is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0047] In this embodiment, a method for removing image shadows based on a diffusion model is provided, which can be used in the above-mentioned mobile terminals, such as mobile phones, tablet computers, etc. Figure 1 FIG. 4 is a flow chart of a method for removing image shadows based on a diffusion model according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0048] Step S101 : obtaining a shadow removal data set to be processed, wherein the shadow removal data set to be processed includes a plurality of groups of shadow removal data to be processed.

[0049] In the embodiment of the present invention, each group of shadow removal data to be processed includes: a shadow-free image x0∈R HxWx3 , images with shadows And the mask image x of the shadow aream ∈R HxWx3 , where H represents the height of the image and W represents the width of the image.

[0050] Specifically, each set of shadow removal data to be processed is a high-level video scene, mainly including different locations such as urban roads, intersections, and roadside parking areas, covering the video image data of vehicles, pedestrians and other targets in the above different scenes in different weather and different time periods. Each set of shadow removal data to be processed includes three images: a shadow-free image, a shadowed image, and a mask image of the shadow area. The mask image of the shadow area is annotated by using a polygon annotation tool.

[0051] Specifically, due to the difficulty in acquiring paired images with and without shadows, open source shadow removal datasets can also be used, including but not limited to ISTD datasets, SRD datasets, etc. This is only an example and is not intended to be used as a reference. In actual applications, the corresponding acquisition method should be selected according to the actual situation.

[0052] Step S102 : Based on a preset global guided sampling strategy, the shadow removal data to be processed is processed by a preset diffusion model to generate a shadow-removed image.

[0053] In an embodiment of the present invention, the preset diffusion model is a dual-branch UNet network, wherein the dual-branch UNet network includes: a first branch network and a second branch network, the first network is used to estimate the diffusion noise of the image, and the second branch network is used to extract global feature information of the image.

[0054] The first branch network input data is:

[0055]

[0056]

[0057] in, The first branch network input data represents the input of the local branch. Local represents the local area obtained according to the location information Li. The input includes two parts, namely the local area of ​​the shadow image and the local area of ​​the mask of the shadow image. Crop() represents the local area of ​​the shadow image based on the location information L. i Extract the R*R image area block, where i represents the i-th image block.

[0058] The input data of the second branch network is:

[0059]

[0060] x' m = Downsample(x m )

[0061] Among them, x G Input data for the second branch network, representing the input of the global branch, which is the global feature map after downsampling the original image; the input includes two parts, namely the downsampled shadow image and the mask map of the downsampled shadow image, and Downsample() represents the downsampling of the image.

[0062] In the embodiment of the present invention, the local area refers to cutting the original image into multiple image blocks, each image block is used as the input of the local branch, and the global input is scaling (downsampling) the original image.

[0063] In the embodiment of the present invention, the first branch network is a local branch, and the input is It is used to estimate the diffuse noise of the image block; the second branch network is the global branch, and the input is Used to extract global feature information.

[0064] Specifically, the two branch network structures in the dual-branch UNet network are the same, and both consist of three parts: a downsampling module, a residual module, and an upsampling module. Each module consists of multiple convolutional layers, pooling layers, and activation function layers.

[0065] Specifically, the first branch network, namely the diffusion noise estimation branch, includes two parts: one is the forward diffusion process, in which the model begins to transform the pixels of the input image block and continuously adds Gaussian noise until the image block completely becomes a pure noise state; the other is the reverse diffusion process, which continuously removes noise until the original image block is restored, and finally obtains multiple image blocks after the shadow is removed. For the second branch network, the global features of the shadow area are obtained by learning the global features of the shadow image and its corresponding shadow image mask.

[0066] In the embodiment of the present invention, in the process of image restoration based on image blocks, each image block includes two weights, namely, a score weight based on brightness and a score weight based on shadow mask, which are defined as:

[0067]

[0068] S bright L represents the brightness-based score weight, which is used to constrain the brightness of the image patch restored by the local branch to be similar to the brightness of the complete image restored by the global branch. gray represents the grayscale image of the image block restored by the local branch; G gray A grayscale image representing the global image restored by the global branch; eps is a preset value, which is selected according to the actual situation and can be set to 0.01.

[0069] Specifically, the score weight based on the shadow mask pays more attention to the non-shadow area, which is specifically expressed as:

[0070]

[0071] Among them, R sf =P sf / P total , T sf It indicates the ratio threshold, which can be set to 0.5 according to the actual situation. sf Represents the number of pixels in the non-shadow area, P total Indicates the total number of pixels.

[0072] Specifically, when the non-shadow area is dominant in the image block, the feature map of the local branch contains both the shadow area and the non-shadow area, which provides effective information for eliminating the shadow through the non-shadow information; and when the shadow area is dominant in the image block, the brightness of the predicted restored image may be affected. Therefore, by utilizing the above-mentioned mask-based score weight calculation, more attention is paid to the non-shadow area.

[0073] In the embodiment of the present invention, the contribution weight of each image block to the global image is calculated as follows:

[0074]

[0075] Among them, t represents each denoising time step, and T represents all time steps.

[0076] In the embodiment of the present invention, the preset diffusion model includes: the diffusion loss of the first branch network and the global loss of the second branch network, and the preset diffusion model is calculated by the following formula:

[0077] L total =L diff +αL golbal

[0078]

[0079] L total represents the loss of the image after shadow removal, L diff represents the diffusion loss of the first branch network, L global represents the global loss of the second branch network, α is the weight coefficient, ∈ t represents the feature representation of the predicted output in the tth stage with gradual addition of noise, x r Indicates global output.

[0080] In the embodiment of the present invention, when the loss function of the image after shadow removal meets the preset condition, the adjacent image blocks are combined to generate the image after shadow removal. This is only an example and is not intended to be used as a reference. In actual applications, corresponding conditions are selected according to actual conditions, for example, when the loss value of the image after shadow removal is greater than the preset value.

[0081] The image shadow removal method based on the diffusion model provided in this embodiment utilizes the diffusion model and combines the global information of the image to perform the image shadow removal task. By constructing a globally guided diffusion model, a high-quality shadow-free image without obvious boundaries and consistent lighting can be generated at any image resolution. In the model, a dual-branch UNet network structure is adopted, wherein branch one, i.e., the local branch, is used to estimate the diffusion noise of the image block; branch two, i.e., the global branch, is used to extract global feature information. Finally, using a sampling strategy based on global guidance, adjacent image blocks are combined to obtain an image after shadow removal.

[0082] In this embodiment, a device for removing image shadows based on a diffusion model is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0083] This embodiment provides an image shadow removal device based on a diffusion model. Figure 2 As shown, including:

[0084] A data acquisition module 201 is used to acquire a shadow removal data set to be processed, wherein the shadow removal data set to be processed includes a plurality of groups of shadow removal data to be processed;

[0085] The image shadow removal module 202 processes the shadow removal data to be processed based on a preset global guided sampling strategy through a preset diffusion model to generate an image after the shadow is removed.

[0086] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0087] The image shadow removal device based on the diffusion model in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0088] The embodiment of the present invention also provides a computer device having the above Figure 3 The image shadow removal device based on the diffusion model is shown.

[0089] See also Figure 3 , Figure 3 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 3 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to an interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 3 A processor 10 is taken as an example.

[0090] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0091] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0092] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage devices. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0093] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0094] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0095] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0096] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for removing image shadows based on a diffusion model, characterized in that: The method comprises: Acquire a shadow removal data set to be processed, wherein the shadow removal data set to be processed includes a plurality of groups of shadow removal data to be processed; Based on the preset global guided sampling strategy, the shadow removal data to be processed is processed through the preset diffusion model to generate an image after shadow removal.

2. The method according to claim 1, characterized in that: Each set of shadow removal data to be processed includes: shadow-free image x0∈R HxWx3 , images with shadows And the mask image x of the shadow area m ∈R HxWx3 , where H represents the height of the image and W represents the width of the image.

3. The method according to claim 2, characterized in that The preset diffusion model is a dual-branch UNet network, wherein the dual-branch UNet network includes: a first branch network and a second branch network, the first network is used to estimate the diffusion noise of the image, and the second branch network is used to extract the global feature information of the image; The first branch network input data is: in, Represents the first branch network input data, and Crop() represents the position information L i Extract the R*R image area block, where i represents the i-th image block; The input data of the second branch network is: x′ m =Downsample(x m ) Among them, x G Represents the second branch network input data, and Downsample() represents the downsampling of the image.

4. The method according to claim 3, characterized in that Based on the preset global guided sampling strategy, adjacent image blocks are combined to generate an image after shadow removal.

5. The method according to claim 4, characterized in that The preset global guided sampling strategies include: The contribution weight of each image block to the global image is calculated by the following formula: Among them, S bright represents the score weight based on brightness, S mask represents the score weight based on the shadow mask, t represents each denoising time step, and T represents all time steps; where, L gray represents the grayscale image of the image patch restored by the local branch, G gray Represents the grayscale image of the global image restored by the global branch, eps is a preset value; The score weight based on the shadow mask is calculated by the following formula: R sf =P sf / P total , T sf Indicates the preset ratio threshold, P sf Represents the number of pixels in the non-shadow area, P total Indicates the total number of pixels.

6. The method according to claim 5, characterized in that The default diffusion model is calculated using the following formula: L total =L diff +αL global L total represents the loss of the image after shadow removal, L diff represents the diffusion loss of the first branch network, L golbal represents the global loss of the second branch network, α is the weight coefficient, ∈ t represents the feature representation of the predicted output in the tth stage with gradual addition of noise, x r Indicates global output.

7. The method according to claim 6, characterized in that When the loss function of the image after shadow removal meets the preset conditions, the adjacent image blocks are combined to generate the image after shadow removal.

8. An image shadow removal device based on a diffusion model, characterized in that: The device comprises: A data acquisition module, used for acquiring a shadow removal data set to be processed, wherein the shadow removal data set to be processed includes a plurality of groups of shadow removal data to be processed; Image shadow removal module, based on the preset global guided sampling strategy, processes the shadow removal data to be processed through the preset diffusion model to generate the image after shadow removal.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the image shadow removal method based on the diffusion model according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the image shadow removal method based on a diffusion model according to any one of claims 1 to 7.

Images