Light and shadow optimization method and device based on multi-modal feature fusion, equipment and medium

Abstract

The invention provides a multi-modal feature fusion-based light and shadow optimization method, apparatus and device, and a medium. The method comprises the steps of converting an original commodity graph, a Poisson fusion result graph and a masking graph into an HSV color space; performing weighted suppression on an H channel and an S channel of the Poisson fusion result graph; replacing the H channel and the S channel of the pixel of the mask area in the Poisson fusion result image with the H channel and the S channel of the pixel of the mask area in the original commodity image; according to the V channel of the original commodity image in the mask area, adjusting the V channel of the Poisson fusion result image in the mask area by using a Sigmoid-like function; according to the method, adaptive color correction is realized, the problems of color cast and light and shadow imbalance in a complex material scene are effectively solved, and the synthesized image presents a light and shadow transition effect conforming to a physical law.

Description

Technical Field

[0001] The present invention relates to the technical field of light image optimization, and in particular to a light image optimization method, device, equipment and medium based on multimodal feature fusion. Background Art

[0002] Among existing product image background replacement technologies, the Poisson fusion algorithm has two key technical flaws when achieving light and shadow fusion between foreground and background:

[0003] First, the algorithm does not fully consider the differences in material optical properties during gradient domain operations, resulting in color distortion in the processed image.

[0004] Second, due to the significant differences in surface reflectivity of objects made of different materials, the existing methods use a unified illumination compensation strategy, which will lead to light and shadow inconsistencies in the synthesized images. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a light and shadow optimization method, device, equipment and medium based on multimodal feature fusion to achieve adaptive color correction, effectively solve the problems of color deviation and light and shadow imbalance in complex material scenes, and make the synthetic image present a light and shadow transition effect that conforms to physical laws.

[0006] In a first aspect, the present invention provides a light and shadow optimization method based on multimodal feature fusion, comprising the following steps:

[0007] Step 1: Convert the original product image, Poisson fusion result image, and mask image to the HSV color space, and obtain the HSV value of each pixel of the Poisson fusion result image, which is H1, S1, and V1, and the HSV value of each pixel of the mask image, which is H2, S2, and V2; extract the HSV value of the area that is completely consistent with the mask image from the original product image, and record it as H0, S0, and V0. The mask image is the image of the original product image with the background removed;

[0008] Step 2: Perform weighted suppression on the H channel and S channel of the Poisson fusion result image to obtain H3 and S3;

[0009] Step 3: Replace the H channel and S channel of the pixels in the masked area of ​​the Poisson fusion result image with the H channel and S channel of the pixels in the masked area of ​​the original product image; that is, the H channel and S channel of the pixels in the non-masked area of ​​the Poisson fusion result image are H3 and S3 respectively, and the H channel and S channel of the pixels in the masked area of ​​the Poisson fusion result image are H0 and S0 respectively;

[0010] Step 4: According to the V channel of the original product image in the mask area, use the Sigmoid function to adjust the V channel of the Poisson fusion result image in the mask area to obtain V new ;

[0011] Step 5. The HSV values ​​of the non-masked area in the Poisson fusion result graph are H3, S3, and V1; the HSV values ​​of the masked area in the Poisson fusion result graph are H0, S0, and V new , converted back to RGB space to obtain the optimized fused image.

[0012] In a second aspect, the present invention provides a light and shadow optimization device based on multimodal feature fusion, comprising:

[0013] Convert the original product image, Poisson fusion result image, and mask image to HSV color space. The HSV values ​​of each pixel in the Poisson fusion result image are H1, S1, and V1, and the HSV values ​​of each pixel in the mask image are H2, S2, and V2. Extract the HSV values ​​of the area that is completely consistent with the mask image from the original product image, and record them as H0, S0, and V0. The mask image is the image of the original product image with the background removed.

[0014] The weighted suppression module performs weighted suppression on the H channel and S channel of the Poisson fusion result image to obtain H3 and S3;

[0015] The correction module replaces the H and S channels of the pixels in the masked area of ​​the Poisson fusion result image with the H and S channels of the pixels in the masked area of ​​the original product image. That is, the H and S channels of the pixels in the non-masked area of ​​the Poisson fusion result image are H3 and S3 respectively, and the H and S channels of the pixels in the masked area of ​​the Poisson fusion result image are H0 and S0 respectively.

[0016] The light compensation module uses a Sigmoid-like function to adjust the V channel of the Poisson fusion result image in the mask area according to the V channel of the original product image in the mask area to obtain V new ;

[0017] Optimize the restoration module. The HSV values ​​of the non-masked area in the Poisson fusion result graph are H3, S3, and V1 respectively; the HSV values ​​of the masked area in the Poisson fusion result graph are H0, S0, and V new , converted back to RGB space to obtain the optimized fused image.

[0018] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the program.

[0019] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the method described in the first aspect when the program is executed by a processor.

[0020] One or more technical solutions provided by the present invention have at least the following technical effects or advantages:

[0021] 1. Staged processing: first reduce noise and then restore to avoid the noise residual problem caused by direct replacement.

[0022] 2. Physical consistency Light compensation based on material reflectivity ensures that the synthesis results conform to real optical laws (such as dark objects absorb light and bright objects reflect light).

[0023] 3. Smooth transition: S-curve adjustment combined with edge feathering eliminates hard edges and brightness jumps, enhancing visual realism.

[0024] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0026] Figure 1 This is a flowchart of the method in Example 1 of the present invention;

[0027] Figure 2 This is a schematic diagram of the structure of the device in Example 2 of the present invention. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of this application have the following general ideas:

[0029] In the HSV channel, H refers to hue, S refers to saturation, and V refers to brightness.

[0030] Poisson fusion is an image synthesis technology based on partial differential equations, which achieves visual fusion by maintaining the gradient field characteristics of the source images.

[0031] A masked image refers to a product image with the background removed, leaving only the main body of the product.

[0032] Color Restoration: When product images are placed against different backgrounds, the resulting Poisson fusion image may be affected by the background, resulting in color deviation. Therefore, fusion against different backgrounds may cause color distortion. To correct this color shift, the H and S values ​​of the corresponding areas in the source image are replaced with the H and S channels of the masked area of ​​the Poisson fusion result image.

[0033] Light compensation is the process of adjusting for differences in brightness caused by varying reflectivity of objects. Different materials have different reflectivities, meaning they exhibit different brightness levels under the same lighting conditions. For example, dark objects absorb more light, while bright objects reflect more. By comparing the brightness differences between the Poisson fusion image and the source image, the brightness of the fusion result (V channel) can be dynamically adjusted to make the output image more consistent with the optical properties of the actual object. A method based on an S-curve (similar to a Sigmoid function) is used here to perform brightness adjustment. The S-curve smoothly changes brightness values, avoiding the visual discontinuity caused by hard cuts.

[0034] Example 1

[0035] like Figure 1 As shown, this embodiment provides a light and shadow optimization method based on multimodal feature fusion, including the following steps:

[0036] Step 1: Convert the original product image, Poisson fusion result image, and mask image to HSV color space, and obtain the HSV value of each pixel of the Poisson fusion result image, which is H1, S1, and V1, and the HSV value of each pixel of the mask image, which is H2, S2, and V2; extract the HSV value that is completely consistent with the area in the mask image from the original product image, and record it as H0, S0, and V0. The mask image is the image of the original product image after removing the background; the Poisson fusion result image is the image after fusion with the background; the mask image is the binary image of the marked product area;

[0037] Step 2: Perform weighted suppression on the H channel and S channel of the Poisson fusion result image to obtain H3 and S3;

[0038] Step 3: Replace the H channel and S channel of the pixels in the masked area of ​​the Poisson fusion result image with the H channel and S channel of the pixels in the masked area of ​​the original product image; that is, the H channel and S channel of the pixels in the non-masked area of ​​the Poisson fusion result image are H3 and S3 respectively, and the H channel and S channel of the pixels in the masked area of ​​the Poisson fusion result image are H0 and S0 respectively;

[0039] Step 4: According to the V channel of the original product image in the mask area, use the Sigmoid function to adjust the V channel of the Poisson fusion result image in the mask area to obtain V new ;

[0040] Step 5. The HSV values ​​of the non-masked area in the Poisson fusion result graph are H3, S3, and V1; the HSV values ​​of the masked area in the Poisson fusion result graph are H0, S0, and V new , converted back to RGB space to obtain the optimized fused image.

[0041] In this embodiment, preferably, step 2 is specifically as follows: weightedly suppressing the H channel and S channel of the Poisson fusion result image to obtain H3 and S3; H3 = H2 × 0.95, slightly reducing the hue difference to prevent abrupt color blocks; S3 = S2 × 0.95, suppressing saturation to avoid over-saturated noise.

[0042] In this embodiment, preferably, the V new The calculation formula is:

[0043]

[0044] Among them, V0: source image V channel; V1: Poisson fusion result image V channel; K: controls the adjustment strength, currently set to 0.005.

[0045] In this embodiment, preferably, the step 5 is specifically as follows: the HSV values ​​of the non-masked area in the Poisson fusion result image are H3, S3 and V1 respectively; the HSV values ​​of the masked area in the Poisson fusion result image are H0, S0 and V new , convert it back to the intermediate image in RGB space, and use Alpha blending to achieve a gradient transition of the set pixels on the edge of the mask area in the intermediate image to obtain the optimized fused image.

[0046] Based on the same inventive concept, this application also provides a device corresponding to the method in Example 1, see Example 2 for details.

[0047] Example 2

[0048] like Figure 2 As shown, in this embodiment, a light and shadow optimization device based on multimodal feature fusion is provided, including:

[0049] Convert the original product image, Poisson fusion result image, and mask image to HSV color space. The HSV values ​​of each pixel in the Poisson fusion result image are H1, S1, and V1, and the HSV values ​​of each pixel in the mask image are H2, S2, and V2. Extract the HSV values ​​of the area that is completely consistent with the mask image from the original product image, and record them as H0, S0, and V0. The mask image is the image of the original product image with the background removed.

[0050] The weighted suppression module performs weighted suppression on the H channel and S channel of the Poisson fusion result image to obtain H3 and S3;

[0051] The correction module replaces the H and S channels of the pixels in the masked area of ​​the Poisson fusion result image with the H and S channels of the pixels in the masked area of ​​the original product image. That is, the H and S channels of the pixels in the non-masked area of ​​the Poisson fusion result image are H3 and S3 respectively, and the H and S channels of the pixels in the masked area of ​​the Poisson fusion result image are H0 and S0 respectively.

[0052] The light compensation module uses a Sigmoid-like function to adjust the V channel of the Poisson fusion result image in the mask area according to the V channel of the original product image in the mask area to obtain V new ;

[0053] Optimize the restoration module. The HSV values ​​of the non-masked area in the Poisson fusion result graph are H3, S3, and V1 respectively; the HSV values ​​of the masked area in the Poisson fusion result graph are H0, S0, and V new , converted back to RGB space to obtain the optimized fused image.

[0054] In this embodiment, preferably, the weighted suppression module specifically performs weighted suppression on the H channel and the S channel of the Poisson fusion result image to obtain H3 and S3; H3=H2×0.95, S3=S2×0.95.

[0055] In this embodiment, preferably, the V new The calculation formula is:

[0056]

[0057] Among them, V0: source image V channel; V1: Poisson fusion result image V channel; K: controls the adjustment strength, currently set to 0.005.

[0058] In this embodiment, preferably, the optimization restoration module is specifically: the HSV values ​​of the non-masked area in the Poisson fusion result image are H3, S3 and V1 respectively; the HSV values ​​of the masked area in the Poisson fusion result image are H0, S0 and V new , convert it back to the intermediate image in RGB space, and use Alpha blending to achieve a gradient transition of the set pixels on the edge of the mask area in the intermediate image to obtain the optimized fused image.

[0059] Since the device described in the second embodiment of the present invention is used to implement the method of the first embodiment of the present invention, those skilled in the art will be able to understand the specific structure and variations of the device based on the method described in the first embodiment of the present invention, and therefore will not be described in detail here. All devices used in the method of the first embodiment of the present invention fall within the scope of protection of the present invention.

[0060] Based on the same inventive concept, this application provides an electronic device embodiment corresponding to the first embodiment, see the third embodiment for details.

[0061] Example 3

[0062] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any implementation method in the first embodiment can be implemented.

[0063] Since the electronic device described in this embodiment is the device used to implement the method in Example 1 of this application, based on the method described in Example 1 of this application, those skilled in the art will be able to understand the specific implementation of the electronic device of this embodiment and its various variations. Therefore, how the electronic device implements the method in the embodiment of this application will not be described in detail here. As long as the device used by those skilled in the art to implement the method in the embodiment of this application falls within the scope of protection to be provided by this application.

[0064] Based on the same inventive concept, this application provides a storage medium corresponding to Example 1, see Example 4 for details.

[0065] Example 4

[0066] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, any implementation method in the first embodiment can be implemented.

[0067] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0068] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0069] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0070] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0071] Although the specific embodiments of the present invention are described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A light and shadow optimization method based on multimodal feature fusion, characterized by: The steps include: Step 1: Convert the original product image, Poisson fusion result image, and mask image to the HSV color space, and obtain the HSV value of each pixel of the Poisson fusion result image, which is H1, S1, and V1, and the HSV value of each pixel of the mask image, which is H2, S2, and V2; extract the HSV value of the area that is completely consistent with the mask image from the original product image, and record it as H0, S0, and V0. The mask image is the image of the original product image with the background removed; Step 2: Perform weighted suppression on the H channel and S channel of the Poisson fusion result image to obtain H3 and S3; Step 3: Replace the H channel and S channel of the pixels in the masked area of ​​the Poisson fusion result image with the H channel and S channel of the pixels in the masked area of ​​the original product image; that is, the H channel and S channel of the pixels in the non-masked area of ​​the Poisson fusion result image are H3 and S3 respectively, and the H channel and S channel of the pixels in the masked area of ​​the Poisson fusion result image are H0 and S0 respectively; Step 4: According to the V channel of the original product image in the mask area, use the Sigmoid function to adjust the V channel of the Poisson fusion result image in the mask area to obtain V new ; Step 5. The HSV values ​​of the non-masked area in the Poisson fusion result graph are H3, S3, and V1; the HSV values ​​of the masked area in the Poisson fusion result graph are H0, S0, and V new , converted back to RGB space to obtain the optimized fused image.

2. The light and shadow optimization method based on multimodal feature fusion according to claim 1, characterized in that: The step 2 is specifically as follows: performing weighted suppression on the H channel and the S channel of the Poisson fusion result image to obtain H3 and S3; H3=H2×0.95, S3=S2×0.

95.

3. The light and shadow optimization method based on multimodal feature fusion according to claim 1, characterized in that: The V new The calculation formula is: Among them, V0: source image V channel; V1: Poisson fusion result image V channel; K: controls the adjustment strength, currently set to 0.

005.

4. The light and shadow optimization method based on multimodal feature fusion according to claim 1, characterized in that: The step 5 is specifically as follows: the HSV values ​​of the non-masked area in the Poisson fusion result image are H3, S3 and V1 respectively; the HSV values ​​of the masked area in the Poisson fusion result image are H0, S0 and V new , convert it back to the intermediate image in RGB space, and use Alpha blending to achieve a gradient transition of the set pixels on the edge of the mask area in the intermediate image to obtain the optimized fused image.

5. A light and shadow optimization device based on multimodal feature fusion, characterized by: include: Convert the original product image, Poisson fusion result image, and mask image to HSV color space. The HSV values ​​of each pixel in the Poisson fusion result image are H1, S1, and V1, and the HSV values ​​of each pixel in the mask image are H2, S2, and V2. Extract the HSV values ​​of the area that is completely consistent with the mask image from the original product image, and record them as H0, S0, and V0. The mask image is the image of the original product image with the background removed. The weighted suppression module performs weighted suppression on the H channel and S channel of the Poisson fusion result image to obtain H3 and S3; The correction module replaces the H and S channels of the pixels in the masked area of ​​the Poisson fusion result image with the H and S channels of the pixels in the masked area of ​​the original product image. That is, the H and S channels of the pixels in the non-masked area of ​​the Poisson fusion result image are H3 and S3 respectively, and the H and S channels of the pixels in the masked area of ​​the Poisson fusion result image are H0 and S0 respectively. The light compensation module uses a Sigmoid-like function to adjust the V channel of the Poisson fusion result image in the mask area according to the V channel of the original product image in the mask area to obtain V new ; Optimize the restoration module. The HSV values ​​of the non-masked area in the Poisson fusion result graph are H3, S3, and V1 respectively; the HSV values ​​of the masked area in the Poisson fusion result graph are H0, S0, and V new , converted back to RGB space to obtain the optimized fused image.

6. The light and shadow optimization device based on multimodal feature fusion according to claim 5, characterized in that: The weighted suppression module specifically performs weighted suppression on the H channel and the S channel of the Poisson fusion result image to obtain H3 and S3; H3=H2×0.95, S3=S2×0.

95.

7. The light and shadow optimization device based on multimodal feature fusion according to claim 5, characterized in that: The V new The calculation formula is: Among them, V0: source image V channel; V1: Poisson fusion result image V channel; K: controls the adjustment strength, currently set to 0.

005.

8. The light and shadow optimization device based on multimodal feature fusion according to claim 5, characterized in that: The optimization restoration module is specifically as follows: the HSV values ​​of the non-masked area in the Poisson fusion result image are H3, S3 and V1 respectively; the HSV values ​​of the masked area in the Poisson fusion result image are H0, S0 and V new , convert it back to the intermediate image in RGB space, and use Alpha blending to achieve a gradient transition of the set pixels on the edge of the mask area in the intermediate image to obtain the optimized fused image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images