An image processing method and apparatus
By determining the sky region mask map and brightness distribution map, and combining image processing algorithms and linear blending techniques, the problem of low fusion between sky materials and non-sky regions in the image was solved, achieving better edge blending effects and user experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGZHOU GUANGZHUIYUAN INFORMATION TECH CO LTD
- Filing Date
- 2022-12-29
- Publication Date
- 2026-05-12
AI Technical Summary
In existing technologies, the fusion between sky materials and non-sky areas in the image is low, resulting in problems such as jagged edges and high edge roughness.
By determining the sky region mask map and the original image to be fused based on the image to be processed and the sky material image, image fusion is performed using the brightness distribution map of the sky material and the mask map. Image processing algorithms, including semantic segmentation, guided filtering, Gaussian blur, etc., are combined with linear blending and color transfer techniques to achieve accurate fusion of the sky material and non-sky regions.
It improves the blending effect between sky elements and non-sky areas in the image, reduces jagged edges, and enhances the user experience.
Smart Images
Figure CN115880197B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer technology, and specifically relates to an image processing method and apparatus. Background Technology
[0002] Images are an important tool for people to record beautiful moments. When users take pictures, if the sky is not well represented, the sky area in the image will be replaced. Usually, the sky area is directly identified by an algorithm, and then the new sky material replaces the original sky area. However, since sky recognition cannot achieve pixel-level precision, when the edges of the sky in the image are irregular, the boundary between the sky area and the non-sky area in the replaced image will appear very harsh and unharmonious. Current technology generally performs a simple feathering process between the original image and the sky material, but simple feathering still cannot solve the problem of edge connection and blending between the sky material and the non-sky area in the image.
[0003] Therefore, how to blend sky material with non-sky areas in an image and reduce jagged edges and roughness between sky material and non-sky areas is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention
[0004] The purpose of this invention is to solve the technical problems in the prior art, such as low fusion between sky materials and non-sky areas in images, resulting in jagged edges and high edge roughness.
[0005] To achieve the above-mentioned technical objectives, in one aspect, the present invention provides an image processing method, the method comprising:
[0006] Based on the image to be processed and the sky material image, a sky region mask image and a light and shadow fusion original image are determined. The sky region mask image is used to distinguish the sky region and non-sky region in the image to be processed.
[0007] Based on the aforementioned sky image, a brightness distribution map of the sky image is determined, and based on this brightness distribution map, the original image to be blended, and the sky region masking map, the original image to be blended is determined.
[0008] The original image to be fused, the sky image, and the sky region masking image are fused to obtain the fused image.
[0009] Furthermore, the step of fusing the original image to be fused, the sky image, and the sky region masking image to obtain the fused image is characterized by specifically including:
[0010] Determine the G channel value in the RGB three-channel image of the sky region masking map;
[0011] Determine the edge blending values of the original image to be blended and the sky image;
[0012] The G channel value in the RGB three channels of the sky region masking image is scaled and updated based on the edge blending value;
[0013] The edge blending value is multiplied by the updated G channel value to obtain the blending parameter;
[0014] Based on the mixing parameters, the original image to be fused and the sky image are linearly blended.
[0015] Furthermore, determining the sky region masking map based on the image to be processed and the sky source image specifically includes:
[0016] The first mask image is segmented from the image to be processed using a preset semantic segmentation model;
[0017] The first mask image and the image to be processed are subjected to guided filtering to obtain the second mask image;
[0018] The first masking image is Gaussian blurred and then multiplied with the second masking image to obtain the sky region masking image.
[0019] Furthermore, determining the original image to be fused based on the image to be processed and the sky image specifically includes:
[0020] Obtain the average color of the sky image, as well as the grayscale value and alpha channel value of the image to be processed;
[0021] The first image is obtained by performing an inverse premultiplication operation on the image to be processed;
[0022] The smoothing step function value of the first image is determined based on the gray value;
[0023] A first temporary image is obtained based on the smoothing step function value, average color, and alpha channel value.
[0024] The average color is multiplied by a preset color migration value to obtain a second temporary image;
[0025] The image to be processed is mixed with the first temporary image using a first blending mode, and then mixed with the second temporary image using a second blending mode. The result is multiplied by the transparency channel value to obtain the original image to be fused.
[0026] Furthermore, determining the brightness distribution map of the sky image based on the sky image specifically includes:
[0027] Convert the sky image from the RGB color space to the LAB color space;
[0028] Gaussian blurring of the L channel in the LAB color space yields the brightness distribution map of the sky image.
[0029] Furthermore, determining the original image to be blended based on the brightness distribution map of the sky material, the original image to be blended, and the sky region masking map specifically includes:
[0030] Determine the G channel value in the RGB three-channel image of the sky region mask and the G channel value in the brightness distribution image of the sky material;
[0031] If the preset color migration value is not less than 0, a third temporary image is constructed, and the transparency channel of the third temporary image is scaled. Then, the scaled third temporary image is blended with the original image to be blended using the first blending mode to obtain the original image to be blended. In this case, the RGB channels of the third temporary image are all pure black, the transparency channel value is the inverse of the G channel value of the brightness distribution map of the sky material, and the scaling ratio is the product of the G channel value of the sky area mask map, the preset light and shadow blending degree value, and the transparency channel value of the third temporary image.
[0032] If the preset color migration value is less than 0, a fourth temporary image is constructed and scaled. Then, the scaled fourth temporary image is blended with the original image to be blended using the first blending mode to obtain the original image to be blended. The RGB channels of the fourth temporary image are all pure white, the transparency channel value is the G channel value of the brightness distribution map of the sky material, and the scaling ratio is the product of the G channel value of the sky area mask map, the preset light and shadow blending degree value, and the transparency channel value of the fourth temporary image.
[0033] On the other hand, the present invention also provides an image processing apparatus, comprising:
[0034] The first determining module is used to determine a sky region masking image and a light and shadow fusion original image based on the image to be processed and the sky material image. The sky region masking image is an image used to distinguish the sky region and non-sky region in the image to be processed.
[0035] The second determining module is used to determine the brightness distribution map of the sky material based on the sky material map, and to determine the original image to be fused based on the brightness distribution map of the sky material, the original image to be fused, and the sky region masking map;
[0036] The fusion module is used to fuse the original image to be fused, the sky image, and the sky area masking image to obtain the fused image.
[0037] This invention provides an image processing method and apparatus. Compared with existing technologies, this method first determines a sky region mask map and a light and shadow fusion original image based on the image to be processed and a sky material image. The sky region mask map is used to distinguish between sky regions and non-sky regions in the image to be processed. Then, a sky material brightness distribution map is determined based on the sky material image, and a fusion original image is determined based on the sky material brightness distribution map, the fusion original image, and the sky region mask map. Finally, the fusion original image, the sky material image, and the sky region mask map are fused to obtain a fused image. This achieves better fusion between sky materials and non-sky regions in the image, improving the user experience. Attached Figure Description
[0038] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0039] Figure 1 The diagram shown is a schematic flowchart of the image processing method provided in the embodiments of this specification;
[0040] Figure 2 The diagram shown is a structural schematic of the image processing apparatus provided in the embodiments of this specification;
[0041] Figure 3 The diagram shown is a hardware structure block diagram of the image processing server provided in the embodiments of this specification. Detailed Implementation
[0042] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0043] like Figure 1The diagram illustrates the image processing method provided in the embodiments of this specification. Although this specification provides the method operation steps or device structure shown in the embodiments or figures below, based on conventional methods or without creative effort, the method or device may include more or fewer operation steps or module units after partial merging. In steps or structures where there is no necessary causal relationship logically, the execution order of these steps or the module structure of the device are not limited to the execution order or module structure shown in the embodiments or figures of this specification. When the method or module structure is applied in actual devices, servers, or terminal products, it can be executed sequentially or in parallel according to the method or module structure shown in the embodiments or figures (e.g., in a parallel processor or multi-threaded processing environment, or even in a distributed processing or server cluster implementation environment).
[0044] The image processing methods provided in this embodiment can be applied to terminal devices such as clients and servers, for example... Figure 1 As shown, the method specifically includes the following steps:
[0045] Step S101: Based on the image to be processed and the sky material image, determine the sky region mask image and the original image to be fused with light and shadow. The sky region mask image is an image used to distinguish the sky region and non-sky region in the image to be processed.
[0046] In this embodiment of the application, determining the sky region masking map based on the image to be processed and the sky material image specifically includes:
[0047] The first mask image is segmented from the image to be processed using a preset semantic segmentation model;
[0048] The first mask image and the image to be processed are subjected to guided filtering to obtain the second mask image;
[0049] The first masking image is Gaussian blurred and then multiplied with the second masking image to obtain the sky region masking image.
[0050] Specifically, the sky region masking image is a single-channel grayscale image with a value range of 0-1. The larger the value, the higher the probability that the pixel is sky. The preset semantic segmentation model can be flexibly set by those skilled in the art according to the actual situation.
[0051] Specifically, regarding the generation of the sky region mask map, a machine learning model is used to segment the sky region of the image to be processed to obtain the sky region mask map. Then, guided filtering, Gaussian filtering and other image processing algorithms are used for further fine-tuning, so that the confidence of the sky and non-sky edges of the sky region mask map used later is more accurate, refined and without jagged edges.
[0052] In this embodiment of the application, determining the original image to be fused based on the image to be processed and the sky image specifically includes:
[0053] Obtain the average color of the sky image, as well as the grayscale value and alpha channel value of the image to be processed;
[0054] The first image is obtained by performing an inverse premultiplication operation on the image to be processed;
[0055] The smoothing step function value of the first image is determined based on the gray value;
[0056] A first temporary image is obtained based on the smoothing step function value, average color, and alpha channel value.
[0057] The average color is multiplied by a preset color migration value to obtain a second temporary image;
[0058] The image to be processed is mixed with the first temporary image in the first blending mode, which is the normal mode in the common blending modes, and then mixed with the second temporary image in the second blending mode, which is the hardlight mode in the common blending modes. Finally, the image is multiplied by the transparency channel value to obtain the original image to be blended.
[0059] Specifically, the inverse premultiplication operation can avoid edge distortion problems such as black and white edges, and also avoid the appearance of white edges in the image, making the image color more balanced. The first temporary image is obtained based on the smoothing step function value, average color, grayscale value of the image to be processed, and alpha channel value, and is determined by the following formula:
[0060] First temporary image = Average color × (0.25 + 0.25 × smooth step function value) × (1.0 - alpha channel value)
[0061] The method of determining the smoothing step function value based on the grayscale value of the image to be processed can be flexibly set by those skilled in the art according to the actual situation, which does not affect the scope of protection of this application.
[0062] The average color of the sky image is statistically calculated and then transferred to the image to be processed. This ensures that the overall tone and atmosphere of the image to be processed are consistent with the tone and atmosphere of the sky image, thus making the final result conform to the characteristics of the tone and atmosphere of the sky image.
[0063] In specific application scenarios, the algorithm flow for determining the original image to be blended with light and shadow is as follows:
[0064] vec4srcColor; Original image
[0065] vec4avgColor; Average Color
[0066] float srcAlpha=srcColor.a;
[0067] vec4unmulSrcColor=vec4(srcColor.rgb / (srcAlpha+step(srcAlpha,0.0)),1.0);
[0068] float prgs=smoothstep(-0.1,0.1,0.5-dot(unmulSrcColor.rgb,vec3(0.30,0.59,0.11)));
[0069] unmulSrcColor=vec4(clamp(unmulSrcColor.rgb*2.7,0.0,1.0),1.0);
[0070] vec4color1=avgColor*(0.25+0.5*prgs)*(1.0-srcAlpha);
[0071] vec4color2=avgColor*hueOpacity;
[0072] vec4result=blendHardlight(blendNormal(unmulSrcColor,color1),color2)*srcAlpha;
[0073] Among them, srcColor refers to the original image input by the user, avgColor refers to the average color obtained by statistical calculation of the sky image, result refers to the original image to be blended, blendNormal is the normal mode among commonly used blending modes, blendHardlight is the hardlight mode among commonly used blending modes, and hueOpacity refers to the degree of color migration, with a value range of 0 to 1.
[0074] Step S102: Determine the brightness distribution map of the sky material based on the sky material map, and determine the original image to be blended based on the brightness distribution map of the sky material, the original image to be blended, and the sky area masking map;
[0075] In this embodiment of the application, determining the brightness distribution map of the sky image based on the sky image specifically includes:
[0076] Convert the sky image from the RGB color space to the LAB color space;
[0077] Gaussian blurring of the L channel in the LAB color space yields the brightness distribution map of the sky image.
[0078] Specifically, by converting the sky image from RGB space to LAB color space and applying Gaussian blur to its L channel to obtain the brightness distribution map of the sky image, and then using image processing operations including blending modes, the light and shadow effects of the sky image are integrated into the image to be processed, so that the final result conforms to the light and shadow distribution characteristics of the sky image.
[0079] In this embodiment of the application, determining the original image to be blended based on the brightness distribution map of the sky material, the original image to be blended, and the sky region masking map specifically includes:
[0080] Determine the G channel value in the RGB three-channel image of the sky region mask and the G channel value in the brightness distribution image of the sky material;
[0081] If the preset color migration value is not less than 0, a third temporary image is constructed, and the transparency channel of the third temporary image is scaled. Then, the scaled third temporary image is blended with the original image to be blended using the first blending mode to obtain the original image to be blended. In this case, the RGB channels of the third temporary image are all pure black, the transparency channel value is the inverse of the G channel value of the brightness distribution map of the sky material, and the scaling ratio is the product of the G channel value of the sky area mask map, the preset light and shadow blending degree value, and the transparency channel value of the third temporary image.
[0082] If the preset color migration value is less than 0, a fourth temporary image is constructed and scaled. Then, the scaled fourth temporary image is blended with the original image to be blended using the first blending mode to obtain the original image to be blended. The RGB channels of the fourth temporary image are all pure white, the transparency channel value is the G channel value of the brightness distribution map of the sky material, and the scaling ratio is the product of the G channel value of the sky area mask map, the preset light and shadow blending degree value, and the transparency channel value of the fourth temporary image.
[0083] Specifically, the preset color migration value and preset light and shadow blending degree value mentioned above are all preset and can be flexibly adjusted by those skilled in the art according to actual needs.
[0084] In specific application scenarios, the algorithm flow for determining the original image to be merged can be as follows:
[0085] vec4result = vec4(1.);
[0086] vec4skyLumColor;
[0087] vec4srcColor;
[0088] vec4skyFineMask;
[0089] if (intensity >= 0.) {
[0090] float alpha=1.-skyLumColor.g;
[0091] vec4ttt=vec4(0.,0.,0.,alpha);
[0092] ttt.a*=abs(intensity)*skyFineMask;
[0093] result=blendNormal(ttt,srcColor);
[0094] }else{
[0095] float alpha=skyLumColor.g;
[0096] vec4ttt=vec4(1.,1.,1.,alpha);
[0097] ttt.a*=abs(intensity)*skyFineMask;
[0098] result=blendNormal(ttt,srcColor);
[0099] }
[0100] Among them, srcColor refers to the original image to be blended, skyLumColor refers to the brightness distribution map of the sky material, skyFineMask refers to the sky area masking optimization map, result refers to the original image to be blended, blendNormal is the normal mode among commonly used blending modes, and intensity refers to the degree of color migration, with a value range of 0 to 1.
[0101] Step S103: The original image to be fused, the sky image, and the sky area masking image are fused to obtain the fused image.
[0102] In this embodiment of the application, the step of fusing the original image to be fused, the sky image, and the sky region masking image to obtain the fused image is characterized by specifically including:
[0103] Determine the G channel value in the RGB three-channel image of the sky region mask;
[0104] Determine the edge blending values of the original image to be blended and the sky image;
[0105] The G channel value in the RGB three channels of the sky region masking image is scaled and updated based on the edge blending value;
[0106] The edge blending value is multiplied by the updated G channel value to obtain the blending parameter;
[0107] Based on the mixing parameters, the original image to be fused and the sky image are linearly blended.
[0108] Specifically, by scaling the confidence level of the sky region with the edge blending degree input by the user using a custom numerical scale, and then linearly blending the sky image with the image to be processed that has undergone tone transfer and light and shadow blending based on the scaled confidence level of the sky region and the blending degree input by the user, the final result is obtained with unified tone and light and shadow and no jagged edges on the sky.
[0109] In specific application scenarios, the above fusion algorithm flow can be described as follows:
[0110] vec4srcColor;
[0111] vec4skyColor;
[0112] vec4skyFineMask;
[0113] float r = skyFineMask.g;
[0114] if (mixIntensity >= 0.) {
[0115] r=pow(srcColor.g,6.)*min(1.,(skyFineMask+mixIntensity));
[0116] }else{
[0117] r=min(1.,pow(srcColor.g,mix(6.,1.,abs(mixIntensity)))*skyFineMask+skyFineMask*abs(mixIntensity));
[0118] }
[0119] vec4result=mix(srcColor,skyColor,r*opacityIntensity)
[0120] In this context, srcColor refers to the original image to be blended, skyColor refers to the sky image, skyFineMask refers to the sky area masking optimization image, result refers to the final result, mixIntensity refers to the degree of edge blending, with a value ranging from 0 to 1, and opacityIntensity refers to the degree of image blending, with a value ranging from 0 to 1.
[0121] Based on the image processing method described above, one or more embodiments of this specification also provide an image processing platform or terminal. This platform or terminal may include devices, software, modules, plug-ins, servers, clients, etc., using the methods described in the embodiments of this specification, combined with necessary hardware implementation devices. Based on the same innovative concept, the systems in one or more embodiments provided in this specification are as described in the following embodiments. Since the implementation schemes and methods for solving the system problem are similar, the specific system implementations in the embodiments of this specification can refer to the implementations of the foregoing methods. Repeated descriptions will not be repeated. The terms "unit" or "module" used below can refer to a combination of software and / or hardware that achieves a predetermined function. Although the systems described in the following embodiments are preferably implemented in software, hardware implementations, and a combination of software and hardware, are also possible and contemplated.
[0122] Specifically, Figure 2 This is a schematic diagram of the module structure of one embodiment of the image processing apparatus provided in this specification, as shown below. Figure 2 As shown, the image processing apparatus provided in this specification includes:
[0123] The first determining module 201 is used to determine a sky region masking image and a light and shadow fusion original image based on the image to be processed and the sky material image. The sky region masking image is an image used to distinguish the sky region and non-sky region in the image to be processed.
[0124] The second determining module 202 is used to determine the brightness distribution map of the sky material based on the sky material map, and to determine the original image to be fused based on the brightness distribution map of the sky material, the original image to be fused, and the sky region masking map;
[0125] The fusion module 203 is used to fuse the original image to be fused, the sky image, and the sky area masking image to obtain the fused image.
[0126] It should be noted that the system described above may include other implementation methods based on the description of the corresponding method embodiments. The specific implementation methods can be referred to the description of the corresponding method embodiments above, and will not be elaborated here.
[0127] This application also provides an electronic device, including:
[0128] processor;
[0129] Memory used to store the processor's executable instructions;
[0130] The processor is configured to perform the methods provided in the embodiments described above.
[0131] The electronic device provided in this application embodiment stores executable instructions of the processor in a memory. When the processor executes the executable instructions, it can first determine a sky region mask map and a light and shadow fusion original image based on the image to be processed and the sky material map. The sky region mask map is used to distinguish the sky region and non-sky region in the image to be processed. Then, it determines a sky material brightness distribution map based on the sky material map, and determines a material fusion original image based on the sky material brightness distribution map, the light and shadow fusion original image, and the sky region mask map. Finally, it fuses the material fusion original image, the sky material map, and the sky region mask map to obtain a fused image, thereby achieving better fusion between the sky material and the non-sky region in the image and improving the user experience.
[0132] The methods and embodiments provided in this specification can be executed on mobile terminals, computer terminals, servers, or similar computing devices. Taking execution on a server as an example... Figure 3 This is a hardware structure block diagram of an image processing server according to one embodiment of this specification. The computer terminal can be the image processing server or image processing device described in the above embodiments. It may include one or more (only one is shown in the figure) processors 100 (processors 100 may include, but are not limited to, processing devices such as microprocessors MCUs or programmable logic devices FPGAs), non-volatile memory 200 for storing data, and a transmission module 300 for communication functions.
[0133] The non-volatile memory 200 can be used to store software programs and modules of application software, such as program instructions / modules corresponding to the image processing method in the embodiments of this specification. The processor 100 executes various functional applications and resource data updates by running the software programs and modules stored in the non-volatile memory 200. The non-volatile memory 200 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the non-volatile memory 200 may further include memory remotely located relative to the processor 100, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0134] The transmission module 300 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the computer terminal. In one example, the transmission module 300 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission module 300 may be a radio frequency (RF) module, used for wireless communication with the Internet.
[0135] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.
[0136] The methods or apparatus described in the embodiments provided in this specification can implement business logic through a computer program and record it on a storage medium. The storage medium can be read and executed by a computer to achieve the effects of the solutions described in the embodiments of this specification, such as:
[0137] Based on the image to be processed and the sky material image, a sky region mask image and a light and shadow fusion original image are determined. The sky region mask image is used to distinguish the sky region and non-sky region in the image to be processed.
[0138] Based on the aforementioned sky image, a brightness distribution map of the sky image is determined, and based on this brightness distribution map, the original image to be blended, and the sky region masking map, the original image to be blended is determined.
[0139] The original image to be fused, the sky image, and the sky region masking image are fused to obtain the fused image.
[0140] The storage medium can include physical devices for storing information, typically digitizing the information and then storing it using electrical, magnetic, or optical methods. The storage medium can include: devices that store information using electrical energy, such as various types of memory, like RAM and ROM; devices that store information using magnetic energy, such as hard disks, floppy disks, magnetic tapes, magnetic core memory, bubble memory, and USB flash drives; and devices that store information using optical methods, such as CDs or DVDs. Of course, there are other readable storage media, such as quantum memories and graphene memories.
[0141] The embodiments in this specification are not limited to conforming to industry communication standards, standard computer resource data update and data storage rules, or the situations described in one or more embodiments of this specification. Slightly modified implementations based on certain industry standards or custom methods or embodiments can also achieve the same, equivalent, or similar, or predictable, implementation effects as described above. Embodiments that utilize these modified or modified methods for data acquisition, storage, judgment, and processing still fall within the scope of optional implementations of the embodiments in this specification.
[0142] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, ASICs, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.
[0143] The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or plug-ins may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.
[0144] These computer program instructions can also be loaded onto a computer or other programmable resource data updating device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0145] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, system embodiments are basically similar to method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. In the description of this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.
[0146] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.
Claims
1. An image processing method, characterized in that, The method includes: Based on the image to be processed and the sky image, a sky region masking image and a source image for light and shadow fusion are determined. The sky region masking image is used to distinguish between sky regions and non-sky regions in the image to be processed. Specifically, determining the source image for light and shadow fusion based on the image to be processed and the sky image includes: obtaining the average color of the sky image, and the grayscale value and alpha channel value of the image to be processed; performing an inverse premultiplication operation on the image to be processed to obtain a first image; determining the smoothing step function value of the first image based on the grayscale value; obtaining a first temporary image based on the smoothing step function value, average color, and alpha channel value; multiplying the average color by a preset color migration value to obtain a second temporary image; mixing the image to be processed with the first temporary image using a first blending mode, then mixing it with the second temporary image using a second blending mode, and finally multiplying the result by the alpha channel value to obtain the source image for light and shadow fusion. A sky brightness distribution map is determined based on the sky image, and a fusion original image is determined based on the sky brightness distribution map, the original image to be fused, and the sky region masking map. Specifically, determining the fusion original image based on the sky brightness distribution map, the original image to be fused, and the sky region masking map includes: determining the G channel value in the RGB three channels of the sky region masking map and the G channel value in the sky brightness distribution map; if the preset color shift value is not less than 0, a third temporary image is constructed, and the transparency channel of the third temporary image is scaled. Then, the scaled third temporary image is blended with the original image to be fused using a first blending mode to obtain the fusion original image. The RGB three channels of the third temporary image... All channels are pure black, the transparency channel value is the inverse of the G channel value of the brightness distribution map of the sky material, and the scaling ratio is the product of the G channel value of the sky area mask map, the preset light and shadow blending degree value, and the transparency channel value of the third temporary image; if the preset color migration value is less than 0, a fourth temporary image is constructed and scaled, and then the scaled fourth temporary image is blended with the original image to be blended through the first blending mode to obtain the original image to be blended. In the fourth temporary image, all three RGB channels are pure white, the transparency channel value is the G channel value of the brightness distribution map of the sky material, and the scaling ratio is the product of the G channel value of the sky area mask map, the preset light and shadow blending degree value, and the transparency channel value of the fourth temporary image; The process of fusing the original image to be fused, the sky image, and the sky region masking image to obtain a fused image specifically includes: determining the G channel value in the RGB three channels of the sky region masking image; determining the edge blending value between the original image to be fused and the sky image; scaling and updating the G channel value in the RGB three channels of the sky region masking image based on the edge blending value; multiplying the edge blending value by the updated G channel value as a blending parameter; and linearly blending the original image to be fused and the sky image based on the blending parameter.
2. The image processing method as described in claim 1, characterized in that, The process of determining the sky region masking map based on the image to be processed and the sky source image specifically includes: The first mask image is segmented from the image to be processed using a preset semantic segmentation model; The first mask image and the image to be processed are subjected to guided filtering to obtain the second mask image; The first masking image is Gaussian blurred and then multiplied with the second masking image to obtain the sky region masking image.
3. The image processing method as described in claim 1, characterized in that, The step of determining the brightness distribution map of the sky image based on the sky image specifically includes: Convert the sky image from the RGB color space to the LAB color space; Gaussian blurring of the L channel in the LAB color space yields the brightness distribution map of the sky image.
4. An image processing apparatus, characterized in that, The device includes: The first determining module is used to determine a sky region mask image and a light and shadow fusion original image based on the image to be processed and a sky material image. The sky region mask image is used to distinguish between sky regions and non-sky regions in the image to be processed. Specifically, it is used to obtain the average color of the sky material image, as well as the grayscale value and alpha channel value of the image to be processed; perform an inverse premultiplication operation on the image to be processed to obtain a first image; determine the smoothing step function value of the first image based on the grayscale value; obtain a first temporary image based on the smoothing step function value, average color, and alpha channel value; multiply the average color by a preset color migration value to obtain a second temporary image; mix the image to be processed with the first temporary image using a first blending mode, then mix it with the second temporary image using a second blending mode, and finally multiply it by the alpha channel value to obtain the light and shadow fusion original image. The second determining module is used to determine the brightness distribution map of the sky image based on the sky image, and to determine the original image to be blended based on the brightness distribution map, the original image to be blended, and the sky region masking map. Specifically, it is used to determine the G channel value in the RGB three channels of the sky region masking map and the G channel value in the sky image brightness distribution map. If the preset color migration value is not less than 0, a third temporary image is constructed, and the transparency channel of the third temporary image is scaled. Then, the scaled third temporary image is blended with the original image to be blended through the first blending mode to obtain the original image to be blended. In this case, the RGB three channels of the third temporary image are all pure black, and the transparency channel value is the sky image brightness distribution map. The inverted G channel value of the material brightness distribution map is scaled by the product of the G channel value of the sky area mask map, the preset light and shadow blending degree value, and the transparency channel value of the third temporary image. If the preset color migration value is less than 0, a fourth temporary image is constructed and scaled. The scaled fourth temporary image is then blended with the original image to be blended using the first blending mode to obtain the original image to be blended. In this original image, all three RGB channels of the fourth temporary image are pure white, the transparency channel value is the G channel value of the sky material brightness distribution map, and the scaling ratio is the product of the G channel value of the sky area mask map, the preset light and shadow blending degree value, and the transparency channel value of the fourth temporary image. The fusion module is used to fuse the original image to be fused, the sky image, and the sky region mask image to obtain a fused image. Specifically, it is used to determine the G channel value in the RGB three channels of the sky region mask image; determine the edge blending value between the original image to be fused and the sky image; scale and update the G channel value in the RGB three channels of the sky region mask image based on the edge blending value; multiply the edge blending value and the updated G channel value as a blending parameter; and perform linear blending of the original image to be fused and the sky image based on the blending parameter.