Image processing method, apparatus, device, storage medium and program product

Abstract

The application provides an image processing method and device, electronic equipment, computer readable storage medium and computer program product. The method comprises: performing opacity estimation processing on a to-be-processed image to obtain the opacity of the to-be-processed image; performing an erosion operation on the opacity of the to-be-processed image to obtain a foreground region and an uncertain region of the to-be-processed image; performing update processing on the opacity of the uncertain region based on the channel observation chroma of the uncertain region and the channel reference chroma of a reference pixel to obtain new opacity of the uncertain region; and determining a target foreground chroma map of the to-be-processed image based on the new opacity of the uncertain region, the opacity of the foreground region and the channel reference chroma of the reference pixel. Through the application, the synthesis effect of the to-be-processed image can be improved.

Description

Technical Field

[0001] The present application relates to image processing technology, and in particular to an image processing method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] An image can be considered to consist of two parts, the foreground and the background. Image matting is the process of extracting the foreground object of interest from an image while filtering out the background.

[0003] With the development of image processing technology, image cutout technology is applied to film shooting and post-production scenes, so as to achieve diversity and flexibility of film effects. However, with the improvement of production requirements, the background of the image is no longer limited to clean pure colors. There may be uneven lighting and the background itself may have texture. Therefore, the image cutout technology in related technologies cannot obtain a clean foreground image, thereby achieving a higher synthesis effect. Summary of the Invention

[0004] The embodiments of the present application provide an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the synthesis effect of the image to be processed.

[0005] The technical solutions of the embodiments of this application are implemented as follows:

[0006] Each embodiment of the present application provides an image processing method, including:

[0007] Performing opacity estimation processing on the image to be processed to obtain the opacity of the image to be processed;

[0008] Performing an erosion operation on the opacity of the image to be processed to obtain a foreground area and an uncertain area of ​​the image to be processed;

[0009] Based on the channel observation chromaticity of the uncertain area and the channel reference chromaticity of the reference pixel, the opacity of the uncertain area is updated to obtain a new opacity of the uncertain area;

[0010] Determining a target foreground color map of the image to be processed based on the new opacity of the uncertain area, the opacity of the foreground area, and the channel reference chromaticity of the reference pixel;

[0011] A target foreground color map of the image to be processed is determined based on the new opacity of the uncertain area, the opacity of the foreground area, and the channel reference chromaticity of the reference pixel.

[0012] Each embodiment of the present application provides an image processing device, including:

[0013] An opacity module is used to perform opacity estimation processing on the image to be processed to obtain the opacity of the image to be processed;

[0014] an erosion module, configured to perform an erosion operation on the opacity of the image to be processed to obtain a foreground area and an uncertain area of ​​the image to be processed;

[0015] an optimization module, configured to update the opacity of the uncertain area based on the channel observation chromaticity of the uncertain area and the channel reference chromaticity of the reference pixel, to obtain a new opacity of the uncertain area;

[0016] a foreground module, configured to determine a target foreground color map of the image to be processed based on the new opacity of the uncertain area, the opacity of the foreground area, and the channel reference chromaticity of the reference pixel;

[0017] The synthesis module is used to determine the target foreground color map of the image to be processed based on the new opacity of the uncertain area, the opacity of the foreground area and the channel reference chromaticity of the reference pixel.

[0018] In the above scheme, the opacity module is also used to: perform the following processing for each pixel of the image to be processed: obtain the channel observation chromaticity of the pixel, and determine the color channel difference of the pixel based on the channel observation chromaticity of the pixel; determine the opacity of the pixel based on the color channel difference of the pixel.

[0019] In the above scheme, the channel observation chromaticity includes the green channel observation chromaticity, the red channel observation chromaticity and the blue channel observation chromaticity, and the opacity module is further used to: obtain the larger observation chromaticity of the red channel observation chromaticity and the blue channel observation chromaticity; perform difference processing on the green channel observation chromaticity and the larger observation chromaticity to obtain the color channel difference value of the pixel.

[0020] In the above scheme, the opacity module is also used to: obtain a first cutoff value and a second cutoff value; based on the first cutoff value and the second cutoff value, perform interception mapping processing on the color channel difference of the pixel to obtain a interception mapping result; based on the interception mapping result, determine the opacity of the pixel.

[0021] In the above scheme, the opacity module is also used to: when the color channel difference of the pixel is less than the second truncation value, determine the value 1 as the truncated mapping result of the pixel; when the color channel difference of the pixel is greater than the first truncation value, determine the value 0 as the truncated mapping result of the pixel, wherein the second truncation value is less than the first truncation value; when the color channel difference of the pixel is not less than the second truncation value and not greater than the first truncation value, perform the following processing: use the reciprocal of the first difference between the second truncation value and the first truncation value as the slope, and use the product of the reciprocal of the slope and the first truncation value as the vertical channel intercept; multiply the slope by the color channel difference of the pixel, and add the multiplication result to the vertical channel intercept to obtain the truncated mapping result of the pixel.

[0022] In the above solution, the opacity module is further used to: perform any one of the following processes: determining the intercepted mapping result of the pixel as the opacity of the pixel; smoothing the intercepted mapping result to obtain the opacity of the pixel.

[0023] In the above solution, the opacity module is further used to: obtain the cube value and the square value of the intercepted mapping result; and obtain the opacity that is negatively correlated with the cube value and positively correlated with the square value.

[0024] In the above scheme, the corrosion module is also used to: obtain the corrosion operation matrix; perform the following processing on each pixel of the image to be processed: perform an AND operation on the opacity of the pixel through the corrosion operation matrix to obtain the corrosion pixel value of the pixel, and determine the second difference between the standard pixel value and the corrosion pixel value; pixels whose second difference is the standard pixel value constitute the uncertain area, and pixels whose difference is zero constitute the foreground area.

[0025] In the above scheme, the operation matrix includes N*N operation elements, N is an integer greater than 1, and the value of each operation element is the standard pixel value. The erosion module is further used to: when the opacity of the pixel is greater than the opacity threshold, update the opacity of the pixel to the standard pixel value; when the opacity of the pixel is not greater than the opacity threshold, update the opacity of the pixel to zero; obtain at least one associated pixel corresponding to the pixel; obtain the operation element corresponding to the pixel and the operation element corresponding to each associated pixel in the operation matrix; perform an AND process on the updated transparency of the pixel and the value of the corresponding operation element to obtain a first AND process result; perform the following processing on each associated pixel: perform an AND process on the updated transparency of the associated pixel and the value of the corresponding operation element to obtain a second AND process result; when there is a zero value in the first AND process result and at least one second AND process result, determine the eroded pixel value of the pixel to be zero; when there is no zero value in the first AND process result and at least one second AND process result, determine the eroded pixel value of the pixel to be the standard pixel value.

[0026] In the above scheme, the optimization module is also used to: determine the chromaticity distance between each pixel in the uncertain area and the reference pixel based on the channel reference chromaticity and the channel observed chromaticity; based on the chromaticity distance, update the opacity of the uncertain area to obtain a new opacity of the uncertain area.

[0027] In the above scheme, the optimization module is also used to: before determining the chromaticity distance between each pixel in the uncertain area and the reference pixel based on the channel reference chromaticity and the channel observed chromaticity, obtain the green channel reference chromaticity, the red channel reference chromaticity and the blue channel reference chromaticity as the channel reference chromaticity of the reference pixel; obtain the green channel observed chromaticity, the red channel observed chromaticity and the blue channel observed chromaticity as the channel observed chromaticity of each pixel in the uncertain area.

[0028] In the above scheme, the optimization module is also used to: perform spatial conversion processing on the channel reference chromaticity to obtain target reference chromaticity; perform the following processing on each pixel in the uncertain area: perform spatial conversion processing on the channel observation chromaticity to obtain target observation chromaticity; obtain the target channel corresponding to the chromaticity distance; obtain the target observation chromaticity corresponding to the target channel from the target observation chromaticity, and obtain the target reference chromaticity corresponding to the target channel from the target reference chromaticity; determine the chromaticity distance between the pixel and the reference pixel based on the target observation chromaticity corresponding to the target channel and the target reference chromaticity corresponding to the target channel.

[0029] In the above scheme, the optimization module is also used to: smooth the chromaticity distance between each pixel in the uncertain area and the reference pixel to obtain the shadow reference opacity of each pixel in the uncertain area; perform the following processing on each pixel in the uncertain area: when the shadow reference opacity of the pixel is less than the opacity of the pixel, determine the shadow reference opacity of the pixel as the new opacity of the pixel; when the shadow reference opacity of the pixel is not less than the opacity of the pixel, determine the opacity of the pixel as the new opacity of the pixel.

[0030] In the above scheme, the foreground module is also used to: perform the following processing on each pixel in the image to be processed: when the pixel is in the foreground area, obtain the transparency of the pixel based on the opacity of the pixel; when the pixel is in the uncertain area, obtain the transparency of the pixel based on the new opacity of the pixel; obtain the transparency of the pixel and multiply it with the channel reference chromaticity of the reference pixel, and obtain the third difference between the channel observed chromaticity of the pixel and the multiplication result; when the pixel is in the foreground area, determine the ratio of the third difference to the opacity of the pixel as the foreground color chromaticity of the pixel; when the pixel is in the uncertain area, determine the ratio of the third difference to the new opacity of the pixel as the foreground color chromaticity of the pixel; and compose the foreground color chromaticity of multiple pixels in the image to be processed into a target foreground color chromaticity map of the image to be processed.

[0031] In the above scheme, the foreground color saturation of the pixel includes red foreground color saturation, green foreground color saturation and blue foreground color saturation. After determining the target foreground color saturation map of the image to be processed, the foreground module is further used to: perform the following processing for each pixel in the image to be processed: obtain the average value of the red foreground color saturation of the pixel and the blue foreground color saturation of the pixel; when the green foreground color saturation of the pixel is greater than the average value, update the foreground color saturation of the pixel to the average value; when the green foreground color saturation of the pixel is not greater than the average value, keep the foreground color saturation of the pixel unchanged.

[0032] Each embodiment of the present application provides an electronic device, including:

[0033] a memory for storing executable instructions;

[0034] The processor is used to implement the image processing method provided in each embodiment of the present application when executing the executable instructions stored in the memory.

[0035] Each embodiment of the present application provides a computer-readable storage medium storing executable instructions for implementing the image processing method provided in each embodiment of the present application when executed by a processor.

[0036] Each embodiment of the present application has the following beneficial effects:

[0037] By performing an erosion operation on the opacity of the image to be processed, the foreground area and the uncertain area of ​​the image to be processed are obtained, which can effectively improve the accuracy of the foreground edge and the uncertain area. Based on the channel observation chromaticity of the uncertain area and the channel reference chromaticity of the reference pixel, the opacity of the uncertain area is updated to obtain a new opacity of the uncertain area. Since the opacity of the uncertain area is updated, the transparency of the shadow in the uncertain area can be optimized. Based on the optimized and updated transparency, the current target scene map is inferred and combined with the target background chromaticity map to obtain a target image with the original content in the uncertain area accurately removed. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a schematic structural diagram of the image processing system provided in each embodiment of the present application;

[0039] Figure 2 is a schematic structural diagram of an electronic device provided in each embodiment of the present application;

[0040] Figures 3A-3C is a flowchart of the image processing method provided in each embodiment of the present application;

[0041] Figure 4 is a logical diagram of the image processing method provided in each embodiment of the present application;

[0042] Figures 5A-5B is a schematic diagram of a mapping function of the image processing method provided in each embodiment of the present application;

[0043] Figures 6A-6B is a schematic diagram of green elimination of the image processing method provided in each embodiment of the present application;

[0044] Figure 7A Schematic diagram of the effects of the image processing methods provided in various embodiments of the present application;

[0045] Figure 7B It is a schematic diagram of the effect of the image processing method provided by the related art;

[0046] Figure 7C Schematic diagram of the effects of the image processing methods provided in various embodiments of the present application;

[0047] Figure 7D It is a schematic diagram of the effect of the image processing method provided by the related art;

[0048] Figure 7E Schematic diagram of the effects of the image processing methods provided in various embodiments of the present application;

[0049] Figure 8A Schematic diagram of the effects of the image processing methods provided in various embodiments of the present application;

[0050] Figure 8B It is a schematic diagram of the effect of the image processing method provided by the related art;

[0051] Figure 8C Schematic diagram of the effects of the image processing methods provided in various embodiments of the present application;

[0052] Figure 8D It is a schematic diagram of the effect of the image processing method provided by the related art;

[0053] Figure 8E It is a schematic diagram of the effects of the image processing method provided in each embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0055] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0056] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the various embodiments of this application only and are not intended to limit this application.

[0058] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0059] Green screen cutout: Green screen cutout technology was first used in film production. When the actors cannot reach the shooting site or the background required by the plot cannot be made, the actors shoot alone in the studio with a green screen, and then the portrait is synthesized with the outdoor scene or special effects scene using the cutout method.

[0060] Mask: A mask is a single-channel grayscale image. It has the same dimensions as the original color channel image and is used to represent the position of the character in the original image. The area formed by pixels with a value of 1 (corresponding to a pixel value of 255) represents the foreground, the area formed by pixels with a value of 0 (corresponding to a pixel value of 0) represents the background, and the remaining pixels represent the semi-transparent parts of the character, often seen at the edges of the character, such as hair or handheld transparent objects. The fineness of the mask reflects the quality of the cutout. If hairlines are clearly visible in the mask, the cutout quality is high. Conversely, a rough edge texture indicates a low cutout quality.

[0061] Foreground color map: The foreground color map is a color channel image of the same size as the original color channel image, used to represent the color components of the pixels belonging to the foreground. Generally speaking, the foreground color saturation of the area composed of pixels with a corresponding value of 1 in the mask image is the same as the chromaticity of the corresponding pixels in the original color channel image. The area composed of pixels with a corresponding value of 0 in the mask image does not belong to the foreground and therefore has no foreground color saturation, or has a random foreground color saturation (which has no practical meaning). The foreground color saturation of the area composed of the remaining pixels is usually different from the chromaticity of the corresponding pixels in the original color channel image, and its value is closer to the foreground color saturation of the surrounding pixels with a corresponding value of 1.

[0062] In related technologies, the front and back backgrounds can be separated based on color differences. This can be done by comparing the color differences of different channels. Taking a green screen background as an example, the larger the green value is compared to red and blue, the greater the probability that the pixel belongs to the background. Conversely, the smaller the difference between green and red and blue, the greater the probability that the pixel belongs to the foreground. By intercepting with two thresholds, the opacity of semi-transparent pixels can be calculated. The method of separating the front and back backgrounds based on color differences works better in practice for solid color backgrounds, and because the calculations for each pixel are independent of each other, parallel calculations can be achieved, resulting in a faster clipping speed. However, the method of separating the front and back backgrounds based on color differences will fail when the green screen color is uneven and has shadows, and there will be green overflow in the final synthesized target image.

[0063] Based on the above technical problems, the embodiments of the present application provide an image processing method, device, electronic device, computer-readable storage medium and computer program product, which can use reference pixels as a reference on the basis of transparency division area, remove shadows and light textures, and improve the synthesis effect of the image to be processed. The following describes the exemplary applications of the electronic devices provided by the embodiments of the present application. The electronic devices provided by the embodiments of the present application can be implemented as a terminal or as a server.

[0064] The image processing method provided in the embodiments of the present application can be implemented by various electronic devices, for example, it can be implemented by a terminal or a server alone, or it can be implemented by a terminal and a server in collaboration.

[0065] See also Figure 1 , Figure 1 This is a schematic diagram of the architecture of the image processing system provided in each embodiment of the present application. To support a photo application, the terminal 400 is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0066] In some embodiments, in response to a user's shooting operation, the terminal 400 receives an image to be processed that is shot by the user in a green screen scene. In response to a user's designation operation for a reference pixel and a setting operation for a target background chromaticity map, the terminal 400 receives the reference pixel and the target background chromaticity map. The terminal 400 sends the image to be processed, the reference pixel, and the target background chromaticity map to the server 200. The server 200 performs opacity estimation processing on the image to be processed to obtain the opacity of the image to be processed; performs an erosion operation on the opacity of the image to be processed to obtain a foreground area and an uncertain area of ​​the image to be processed; updates the opacity of the uncertain area based on the channel observed chromaticity of the uncertain area and the channel reference chromaticity of the reference pixel to obtain a new opacity of the uncertain area; determines a target foreground chromaticity map of the image to be processed based on the new opacity of the uncertain area, the opacity of the foreground area, and the channel reference chromaticity of the reference pixel; synthesizes a target image corresponding to the image to be processed based on the target foreground chromaticity map of the image to be processed, the target background chromaticity map of the image to be processed, the new opacity of the uncertain area, and the opacity of the foreground area, and the server 200 sends the target image to the terminal 400 for display.

[0067] In other embodiments, when the image processing method provided in the embodiments of the present application is implemented by a terminal alone, in the various application scenarios described above, the terminal can determine the target image corresponding to the image to be processed or the target image of the video frame in the corresponding video, and directly present the target image.

[0068] As an example, the server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal 400 can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart TV, smart car terminal and smart watch, etc., but is not limited to these. The terminal 400 and the server 200 can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0069] In some embodiments, the terminal or server can implement the image processing method provided in each embodiment of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a live broadcast APP or a photo APP; it can also be a small program, that is, a program that can be run only by downloading it into a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form.

[0070] Next, the structure of the electronic device for implementing the image processing method provided by the embodiment of the present application is described. As mentioned above, the electronic device provided by the embodiment of the present application can be Figure 1 The server 200 or terminal 400 in FIG. Figure 2 , Figure 2 is a schematic diagram of the structure of the electronic device provided in each embodiment of the present application, Figure 2 The terminal 400 shown includes: at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in the terminal 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 440 is not shown in FIG. Figure 2 Various buses are labeled as bus system 440 .

[0071] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0072] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0073] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0074] The memory 450 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in various embodiments of the present application is intended to include any suitable type of memory.

[0075] In some embodiments, the memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0076] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0077] A network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include Bluetooth, Wi-Fi, and Universal Serial Bus (USB).

[0078] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0079] The input processing module 454 is configured to detect one or more user inputs or interactions from one of the one or more input devices 432 and to translate the detected inputs or interactions.

[0080] In some embodiments, the image processing apparatus provided in each embodiment of the present application may be implemented in software. Figure 2 An image processing device 455 stored in memory 450 is shown. This device can be software in the form of a program or plug-in, and includes the following software modules: an opacity module 4551, an erosion module 4552, an optimization module 4553, a foreground module 4554, and a synthesis module 4555. These modules are logical and can be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.

[0081] The image processing method provided in each embodiment of the present application will be described in conjunction with exemplary applications and implementations of the terminal provided in each embodiment of the present application.

[0082] See also Figure 3A , Figure 3A This is a flow chart of the image processing method provided by each embodiment of the present application, which will be combined with Figure 3A Steps 101 to 105 are shown for explanation.

[0083] In step 101, opacity estimation processing is performed on the image to be processed to obtain the opacity of the image to be processed.

[0084] As an example, the image to be processed can be a photographed image or an image extracted from a gallery. The image to be processed can be shot against a green screen or other background, or extracted from a gallery. Although not shot against a green screen, the image to be processed still has a foreground area and an uncertain area. Opacity refers to the degree of opacity of a pixel, with 100% indicating complete opacity and 0% indicating complete transparency. Pixels with an opacity of 0 constitute the foreground area, areas with an opacity of 1 constitute the background area, and pixels with an opacity between 0 and 1 constitute the edge of the foreground area or the background area.

[0085] In some embodiments, the opacity estimation process performed on the image to be processed in step 101 to obtain the opacity of the image to be processed can be implemented by the following technical solution: performing the following process for each pixel in the image to be processed: obtaining the channel-observed chromaticity of the pixel, and determining the color channel difference of the pixel based on the channel-observed chromaticity of the pixel; and determining the opacity of the pixel based on the color channel difference of the pixel. By performing the above process in parallel for each pixel, the processing speed can be increased, and the opacity estimation accuracy can be improved by using the color channel difference to estimate the opacity.

[0086] In some embodiments, the channel observed chromaticity includes green channel observed chromaticity, red channel observed chromaticity, and blue channel observed chromaticity. Determining the color channel difference of a pixel based on the channel observed chromaticity of the pixel can be achieved by the following technical solution: obtaining the larger observed chromaticity of the red channel observed chromaticity and the blue channel observed chromaticity; performing a difference processing on the green channel observed chromaticity and the larger observed chromaticity to obtain the color channel difference of the pixel. Because various image processing application scenarios require separating human figures from the image to be processed, and the color of human skin, hair, and clothing typically has the greatest color difference from green, performing a difference processing on the green channel observed chromaticity and the larger observed chromaticity to obtain the color channel difference can improve the accuracy of opacity estimation when estimating opacity based on the obtained color channel difference.

[0087] As an example, the color channel difference D(p) between the green channel and the red and blue channels of each pixel p in the image to be processed is calculated using formula (1):

[0088] D(p)=C g (p)-max(C r (p),C b (p)) (1);

[0089] As an example, the color channel difference D(p) of pixel p is substituted into formula (2) to obtain the opacity of the pixel, see formula (2):

[0090] α(p)=g(D(p)) (2);

[0091] As an example, p is a pixel, D(p) is the color channel difference D(p), g is the mapping function, α(p) is the initial opacity α(p) of pixel P, α(p)∈[0,1].

[0092] In some embodiments, users can also personalize separation requirements. For example, if the forest needs to be separated from the sky, the larger observed chromaticity of the blue channel and the green channel can be obtained; the red channel observed chromaticity and the larger observed chromaticity are differenced to obtain the color channel difference of the pixel, that is, based on the foreground content that needs to be separated, the channel observed chromaticity with the largest color difference from the foreground content is differenced with the larger observed chromaticity to obtain the color channel difference. When estimating the opacity based on the obtained color channel difference, the accuracy of the opacity estimation can be improved.

[0093] In some embodiments, determining the opacity of a pixel based on its color channel difference can be achieved by the following technical solution: obtaining a first cutoff value and a second cutoff value; performing a clipping mapping process on the color channel difference of the pixel based on the first cutoff value and the second cutoff value to obtain a clipping mapping result; and determining the opacity of the pixel based on the clipping mapping result. The clipping mapping process limits the clipping mapping result to a fixed range, meeting the data range requirements for opacity during image processing.

[0094] As an example, the opacity of a pixel is negatively correlated with the color channel difference of the pixel as a whole. For example, the larger the color channel difference, the higher the probability that the pixel belongs to the foreground area, and the lower its opacity. The smaller the color channel difference, the lower the probability that the pixel belongs to the foreground area, and the higher its opacity. Since there are upper and lower boundaries of opacity, that is, opacity α(p)∈[0,1], there is a first truncation value and a second truncation value. Based on the first truncation value and the second truncation value, a truncation mapping is performed to obtain a truncation mapping result, and the opacity of the pixel is determined based on the truncation mapping result.

[0095] In some embodiments, the above-mentioned interception mapping processing of the color channel difference of the pixel based on the first cutoff value and the second cutoff value to obtain the interception mapping result can be achieved by the following technical solution: when the color channel difference of the pixel is less than the second cutoff value, the value 1 is determined as the interception mapping result of the pixel; when the color channel difference of the pixel is greater than the first cutoff value, the value 0 is determined as the interception mapping result of the pixel, wherein the second cutoff value is less than the first cutoff value; when the color channel difference of the pixel is not less than the second cutoff value and not greater than the first cutoff value, the following processing is performed: the reciprocal of the first difference between the second cutoff value and the first cutoff value is used as the slope, and the product of the inverse of the slope and the first cutoff value is used as the vertical channel intercept; the slope is multiplied by the color channel difference of the pixel, and the multiplication result is added to the vertical channel intercept to obtain the interception mapping result of the pixel. The interception mapping process is linear, thereby reducing the amount of calculation in the interception process, and the linear interception mapping method helps to ensure the stability and uniformity of opacity, thereby improving the reference value of opacity.

[0096] For example, see Figure 5A , Figure 5A is a schematic diagram of the mapping function of the image processing method provided in each embodiment of the present application. The mapping function g can be a linear hierarchical function, see formula (3):

[0097]

[0098] Among them, t2 and t1 are set thresholds, that is, t1 is the first truncation value, t2 is the second truncation value, clamp() represents the truncation function, when the color channel difference D of pixel p is greater than t1 (second truncation value), the initial opacity α(p) is 0, when the color difference of pixel p is less than t2 (first truncation value), the initial opacity α(p) is 1, when the color difference of pixel p is not less than t2 and not greater than t1, the initial opacity α(p) is a value between 0 and 1, that is, the reciprocal of the first difference between the second truncation value and the first truncation value As the slope, multiply the inverse of the slope by the first cutoff value As the vertical channel intercept; multiply the slope by the color channel difference of the pixel, and add the multiplication result to the vertical channel intercept to obtain the intercept mapping result of the pixel.

[0099] In some embodiments, the above-mentioned determination of the opacity of the pixel based on the intercepted mapping result can be achieved through the following technical solutions: performing any one of the following processes: determining the intercepted mapping result of the pixel as the opacity of the pixel; smoothing the intercepted mapping result to obtain the opacity of the pixel.

[0100] In some embodiments, smoothing the intercepted mapping results to obtain pixel opacity can be achieved by obtaining the following technical solution: obtaining the cubed and squared values ​​of the intercepted mapping results; and obtaining an opacity that is negatively correlated with the cubed value and positively correlated with the squared value. The introduction of square and cube calculations can smooth the transparency. Since transparency in real scenes cannot be absolutely stable and uniform, using a smoothed curve to describe opacity can improve the accuracy of subsequent foreground area separation.

[0101] For example, see Figure 5B , Figure 5B is a schematic diagram of the mapping function of the image processing method provided in each embodiment of the present application, Figure 5B The curve shown in the figure is actually Figure 5A The mapping function shown in is improved, and a smoother result can be obtained by using the SmoothStep function, see formula (4) and formula (5):

[0102] SmoothStep(x,t1,t2)=g(x,t1,t2)*g(x,t1,t2)*(3-2*g(x,t1,t2)) (4);

[0103] α(p)=SmoothStep(D(p),t1,t2) (5);

[0104] Among them, t2 and t1 are set thresholds, that is, t1 is the first truncation value, t2 is the second truncation value, clamp() represents the truncation function, SmoothStep represents the smoothing function, and g(x, t1, t2) is the intercepted mapping result, that is, the color channel difference D is substituted into the mapping function shown in formula (3), and after calculating g(x, t1, t2), g(x, t1, t2) is substituted into formula (4) to calculate SmoothStep(D, t1, t2) as the initial opacity.

[0105] In step 102, an erosion operation is performed on the opacity of the image to be processed to obtain a foreground area and an uncertain area of ​​the image to be processed.

[0106] See also Figure 3B , Figure 3B is a flow chart of the image processing method provided by each embodiment of the present application. In step 102, the opacity of the image to be processed is eroded to obtain the foreground area and the uncertain area of ​​the image to be processed. Figure 3B Steps 1021 to 1023 are implemented as shown.

[0107] In step 1021, an erosion operation matrix is ​​obtained.

[0108] As an example, a corrosion operation matrix is ​​required in the corrosion operation. The corrosion operation matrix is ​​usually 3×3 in size. The corrosion operation matrix is ​​based on set operations. The corrosion operation matrix is ​​used to scan each pixel in the image, so that each pixel can be judged in turn whether it belongs to the foreground area.

[0109] In step 1022, the following processing is performed for each pixel of the image to be processed: performing an AND operation on the opacity of the pixel through the erosion operation matrix to obtain the eroded pixel value of the pixel, and determining a second difference between the standard pixel value and the eroded pixel value.

[0110] In some embodiments, the operation matrix includes N*N operation elements, where N is an integer greater than 1, and the value of each operation element is a standard pixel value. In step 1022, the opacity of the pixel is ANDed by the erosion operation matrix to obtain the eroded pixel value of the pixel, which can be achieved by the following technical solution: when the opacity of the pixel is greater than the opacity threshold, the opacity of the pixel is updated to the standard pixel value; when the opacity of the pixel is not greater than the opacity threshold, the opacity of the pixel is updated to zero; at least one associated pixel of the corresponding pixel is obtained; the operation element of the corresponding pixel in the operation matrix and the operation element corresponding to each associated pixel are obtained; the updated transparency of the pixel is ANDed with the value of the corresponding operation element to obtain a first ANDed processing result; the following processing is performed on each associated pixel: the updated transparency of the associated pixel is ANDed with the value of the corresponding operation element to obtain a second ANDed processing result; when there is a zero value in the first ANDed processing result and at least one second ANDed processing result, the zero value is determined as the eroded pixel value of the pixel; when there is no zero value in the first ANDed processing result and at least one second ANDed processing result, the standard pixel value is determined as the eroded pixel value of the pixel.

[0111] As an example, the erosion operation matrix is ​​used to scan each pixel in the image to be processed, and the value of each operation element is the standard pixel value 255. Each element in the operand matrix is ​​ANDed with the covered pixel, where the AND result between 255 and 0 is 0, and the AND result between 255 and 255 is 255. If the AND operation results of all elements in the operand matrix are 255, the eroded pixel value of the pixel in the image to be processed is 255, otherwise the eroded pixel value of the pixel in the image to be processed is 0. For example, when the opacity of pixel p is 200, which is greater than the opacity threshold of 120, the opacity of pixel p is updated to the standard pixel value 255. When the opacity of pixel p is 12, which is not greater than the opacity threshold of 120, the opacity of pixel p is updated to 0, and at least one associated pixel of the corresponding pixel is obtained. The associated pixel is a pixel adjacent to pixel p. When scanning, the central element of the center of the corrosion operation matrix is ​​usually used as the element corresponding to pixel p. The element position relationship between other elements in the corrosion operation matrix and the central element is the same as the pixel position relationship between pixel p and the associated pixel. For example, the adjacent element above the central element is the same as the adjacent element above pixel P. Corresponding to adjacent pixels, the updated transparency 255 of the pixel is processed with the value 255 of the corresponding operation element to obtain a first AND processing result 255, and the following processing is performed on each associated pixel: the updated transparency 255 of the associated pixel is processed with the value 255 of the corresponding operation element to obtain a second AND processing result 255; when there is a zero value in the first AND processing result and at least one second AND processing result, the eroded pixel value of the pixel is determined to be a zero value, for example, there is a second AND processing result of 0, when there is no zero value in the first AND processing result and at least one second AND processing result, the eroded pixel value of the pixel is determined to be a standard pixel value 255.

[0112] In step 1023, pixels whose second difference value is the standard pixel value are formed into an uncertain area, and pixels whose difference value is zero are formed into a foreground area.

[0113] As an example, an erosion operation (erode) is performed on the initial opacity to exclude the edge of the character from the foreground area, and then the foreground area is inverted to obtain the uncertain area, which includes the edge of the character and the green screen background, see formula (6):

[0114] U = 255-erode(α) (6);

[0115] Among them, U is the grayscale value of the pixel of the image to be processed (the second difference), α is the initial opacity of the pixel of the image to be processed, erode(α) is the eroded pixel value of the pixel, when U of a pixel is 255, the pixel belongs to the uncertain area, when U of a pixel is 0, the pixel belongs to the foreground area.

[0116] The role of the erosion operation is to eliminate the boundary points of the foreground area, thereby shrinking the foreground area. The erosion operation will eliminate those small and meaningless objects and shrink the boundary inward. The uncertain area obtained based on the erosion operation can include the edge part of the foreground area, thereby improving the reliability of foreground area separation.

[0117] In step 103 , the opacity of the uncertain area is updated based on the channel observation chromaticity of the uncertain area and the channel reference chromaticity of the reference pixel to obtain a new opacity of the uncertain area.

[0118] As an example, the reference pixel can be a real pixel in the image to be processed, or a virtual pixel with only relevant parameter values. By introducing the reference pixel, the large area of ​​strong shadows included in the uncertain area can be effectively removed. For example, if a light green QR code is printed on a green screen background, the light green can also be considered as a strong shadow.

[0119] See also Figure 3C , Figure 3C is a flow chart of the image processing method provided by each embodiment of the present application. In step 103, the opacity of the uncertain area is updated based on the channel observation chromaticity of the uncertain area and the channel reference chromaticity of the reference pixel. The new opacity of the uncertain area can be obtained by Figure 3C Steps 1031 to 1032 are implemented as shown.

[0120] In step 1031 , based on the channel reference chromaticity and the channel observed chromaticity, the chromaticity distance between each pixel in the uncertainty area and the reference pixel is determined.

[0121] In some embodiments, before determining the chromaticity distance between each pixel in the uncertain area and the reference pixel based on the channel reference chromaticity and the channel observed chromaticity, the green channel reference chromaticity, the red channel reference chromaticity, and the blue channel reference chromaticity are obtained as the channel reference chromaticity of the reference pixel; the green channel observed chromaticity, the red channel observed chromaticity, and the blue channel observed chromaticity are obtained as the channel observed chromaticity of each pixel in the uncertain area.

[0122] As an example, the chromaticity in image processing is usually the chromaticity of the RGB space. For example, the channel reference chromaticity includes the green channel reference chromaticity, the red channel reference chromaticity, and the blue channel reference chromaticity, that is, the reference pixel is described by the green channel reference chromaticity, the red channel reference chromaticity, and the blue channel reference chromaticity, and the channel observation chromaticity includes the green channel observation chromaticity, the red channel observation chromaticity, and the blue channel observation chromaticity.

[0123] In some embodiments, in step 1031, the chromaticity distance between each pixel in the uncertain area and the reference pixel is determined based on the channel reference chromaticity and the channel observed chromaticity, which can be achieved by the following technical solutions: spatial conversion processing is performed on the channel reference chromaticity to obtain the target reference chromaticity; the following processing is performed for each pixel in the uncertain area: spatial conversion processing is performed on the channel observed chromaticity to obtain the target observed chromaticity; the target channel corresponding to the chromaticity distance is obtained; the target observed chromaticity corresponding to the target channel is obtained from the target observed chromaticity, and the target reference chromaticity corresponding to the target channel is obtained from the target reference chromaticity; based on the target observed chromaticity corresponding to the target channel and the target reference chromaticity corresponding to the target channel, the chromaticity distance between the pixel and the reference pixel is determined. There are different space options for chromaticity distance calculation, such as RGB space, YUV space, HSV space, etc. Therefore, the target reference chromaticity and the target observed chromaticity can be obtained through spatial conversion processing, so that the calculation of the chromaticity distance is more objective and accurate.

[0124] As an example, the color space distance between all pixels in the uncertain area and the reference pixel is normalized to the range of 0 to 1. When calculating the color space distance, it can be calculated in the UV plane of the YUV space. The channel reference chromaticity and the channel observation chromaticity can be converted from the RGB space to the YUV space by formula (7):

[0125]

[0126] Among them, R is the red chromaticity value of the RGB space (red channel reference chromaticity or red channel observed chromaticity), G is the green chromaticity value of the RGB space (green channel reference chromaticity or green channel observed chromaticity), B is the blue chromaticity value of the RGB space (blue channel reference chromaticity or blue channel observed chromaticity), Y is the Y channel chromaticity value of the YUV space (Y channel reference chromaticity or Y channel observed chromaticity), U is the U channel chromaticity value of the YUV space (U channel reference chromaticity or U channel observed chromaticity), and V is the V channel chromaticity value of the YUV space (V channel reference chromaticity or V channel observed chromaticity).

[0127] After completing the spatial conversion, the target channel corresponding to the chromaticity distance is obtained. The target channel is pre-configured for the chromaticity distance. For example, the target channel is configured as the U channel and the V channel. The target observation chromaticity corresponding to the target channel is obtained from the target observation chromaticity, and the target reference chromaticity corresponding to the target channel is obtained from the target reference chromaticity. The UV space distance between each pixel and the reference pixel is calculated using formula (8):

[0128] KeyD(p)=(C u (p)-Key u (p)) 2+(C v (p)-Key v (p)) 2 (8);

[0129] Among them, KeyD(p) is the spatial distance between pixel p and the reference pixel in the UV plane, C u (p) is the observed chromaticity of the U channel of pixel p, C v (p) is the V channel observed chromaticity of pixel p, Key u (p) is the U channel reference chromaticity, Key v (p) is the V channel reference chromaticity.

[0130] In step 1032 , the opacity of the uncertain region is updated based on the chromaticity distance to obtain a new opacity of the uncertain region.

[0131] In some embodiments, in step 1032, the opacity of the uncertain region is updated based on the chromatic distance to obtain a new opacity of the uncertain region. This can be achieved by the following technical solution: based on the chromatic distance between each pixel in the uncertain region and the reference pixel, the shadow reference opacity of each pixel in the uncertain region is determined; and for each pixel in the uncertain region, the following processing is performed: when the shadow reference opacity of the pixel is less than the opacity of the pixel, the shadow reference opacity of the pixel is determined as the new opacity of the pixel; when the shadow reference opacity of the pixel is not less than the opacity of the pixel, the opacity of the pixel is determined as the new opacity of the pixel. By updating the opacity, large areas of strong shadows contained in the uncertain region can be effectively removed.

[0132] As an example, the chromaticity distance between each pixel in the uncertain area and the reference pixel is intercepted and mapped to obtain the intercepted mapping result, and the intercepted mapping result is smoothed to obtain the shadow reference opacity of each pixel in the uncertain area. The SmoothStep function can be used to perform interception mapping and smoothing in sequence to obtain the opacity of the reference pixel, see formula (9):

[0133] Keyα(p)=SmoothStep(KeyD(p),t1,t2) (9);

[0134] Among them, Keyα(p) is the opacity of the reference pixel, and SmoothStep(KeyD(p),t1,t2) can be seen in Formula (4) and Formula (5).

[0135] As an example, the following processing is performed for each pixel in the uncertainty area: when the shadow reference opacity of the pixel is less than the opacity of the pixel, the shadow reference opacity of the pixel is determined as the new opacity of the pixel; when the shadow reference opacity of the pixel is not less than the opacity of the pixel, the opacity of the pixel is determined as the new opacity of the pixel.

[0136] The opacity of the uncertain region is optimized according to the following formula (10) to obtain the new opacity of the uncertain region:

[0137] α(p)=min(Keyα(p),α(p)), if U(p)=255 (10);

[0138] Among them, Key α(p) is the opacity of the reference pixel, U(p) is the grayscale value of the pixel of the image to be processed (the second difference), and α(p) is the initial opacity of the pixel p of the image to be processed.

[0139] In step 104 , a target foreground color map of the image to be processed is determined based on the new opacity of the uncertain area, the opacity of the foreground area, and the channel reference chromaticity of the reference pixel.

[0140] In some embodiments, determining the target foreground color saturation map of the image to be processed in step 104 based on the new opacity of the uncertain area, the opacity of the foreground area, and the channel reference chromaticity of the reference pixel can be achieved by the following technical solution: performing the following processing for each pixel in the image to be processed: when the pixel is in the foreground area, obtaining the transparency of the pixel based on the opacity of the pixel; when the pixel is in the uncertain area, obtaining the transparency of the pixel based on the new opacity of the pixel; obtaining the multiplication result of the transparency of the pixel and the channel reference chromaticity of the reference pixel, and obtaining a third difference between the channel observed chromaticity of the pixel and the multiplication result; when the pixel is in the foreground area, determining the foreground color saturation of the pixel by the ratio of the third difference to the opacity of the pixel; when the pixel is in the uncertain area, determining the foreground color saturation of the pixel by the ratio of the third difference to the new opacity of the pixel. By inferring the foreground color saturation based on the updated transparency, shadows in the uncertain area can be effectively eliminated.

[0141] As an example, when a pixel is in the foreground area, the transparency of the pixel is obtained based on the opacity α(p) of the pixel, that is, 1-α(p) is calculated; when the pixel is in the uncertain area, the transparency of the pixel is obtained based on the new opacity α(p) of the pixel, 1-α(p); the multiplication result of the transparency of the pixel and the channel reference chromaticity of the reference pixel (1-α(p))*Key(p) is obtained, and the third difference C between the channel observation chromaticity of the pixel and the multiplication result is obtained. r(p)-(1-α(p))*Key(p) When the pixel is in the foreground area, the ratio of the third difference to the opacity of the pixel (C r (p)-(1-α(p))*Key(p)) / α(p) is determined as the foreground color of the pixel; when the pixel is in the uncertain area, the ratio of the third difference to the new opacity of the pixel (C r (p)-(1-α(p))*Key(p)) / α(p) is determined as the foreground color saturation of the pixel. Specifically, the foreground color saturation of all pixels in the three channels can be approximately inferred according to formula (11):

[0142]

[0143] Among them, C r (p) is the observed chromaticity of the red channel, C g (p) is the observed chromaticity of the green channel, C b (p) is the observed chromaticity of the blue channel, F r (p) is the red foreground color intensity, F g (p) is the green foreground color intensity, F b (p) is the blue foreground color, Key r (p) is the red channel chromaticity of the reference pixel, Key g (p) is the green channel chromaticity of the reference pixel, Key b (p) is the blue channel chromaticity of the reference pixel, and α(p) is the transparency of pixel p.

[0144] In some embodiments, the foreground color saturation of a pixel includes red, green, and blue foreground color saturation. After determining the target foreground color saturation map for the image to be processed, the following processing is performed for each pixel in the image to be processed: the average of the pixel's red foreground color saturation and the pixel's blue foreground color saturation is obtained; when the pixel's green foreground color saturation is greater than the average, the pixel's foreground color saturation is updated to the average; when the pixel's green foreground color saturation is not greater than the average, the pixel's foreground color saturation remains unchanged. By comparing the green foreground color saturation with the average of the red and blue foreground color saturations, the phenomenon of greening the foreground color saturation can be avoided, thereby improving the accuracy of the foreground color saturation.

[0145] After inverting the foreground color saturation, the foreground color saturation may appear green. Figure 6A , Figure 6A is a schematic diagram of green elimination of the image processing method provided in each embodiment of the present application, Figure 6A There is a greenish color at 601A in the image. The green color in the foreground color can be removed by the following formula (12):

[0146]

[0147] Among them, F r (p) is the red foreground color intensity, F g (p) is the green foreground color intensity, F b (p) is the blue foreground color saturation, that is, F is calculated by formula (12) g (p) is the green foreground color saturation to update, thereby removing the green in the foreground color, see Figure 6B , Figure 6B is a schematic diagram of green elimination of the image processing method provided in each embodiment of the present application, Figure 6B There is no green overflow at 601B.

[0148] In some embodiments, the user can also personalize the separation requirements. For example, if the forest needs to be separated from the sky, the red foreground color saturation of the pixel and the average value of the green foreground color saturation of the pixel are obtained; when the blue foreground color saturation of the pixel is greater than the average value, the foreground color saturation of the pixel is updated to the average value; when the blue foreground color saturation of the pixel is not greater than the average value, the foreground color saturation of the pixel is kept unchanged. That is, based on the content of the uncertain area to be separated, the overflow color that needs to be eliminated is determined. If the forest needs to be separated from the sky, blue overflow needs to be avoided.

[0149] In step 105 , a target image corresponding to the image to be processed is synthesized based on the target foreground chromaticity map of the image to be processed, the target background chromaticity map of the image to be processed, the new opacity of the uncertain area, and the opacity of the foreground area.

[0150] As an example, the image synthesis model of the image processing method provided in each embodiment of the present application can be referred to formula (13):

[0151] C(p)=α(p)*F(p)+(1-α(p))*B(p) (13);

[0152] Among them, p represents any pixel in the image to be processed, C is the observed pixel color, which includes RGB values, α is the opacity (the new opacity of the pixel in the uncertain area, or the opacity of the pixel in the foreground area), F is the target foreground color, which includes RGB values, and B is the target background chromaticity map (which can be set arbitrarily according to user needs), which includes RGB values.

[0153] Expanding formula (13) according to the three channels of RGB, we get formula (14):

[0154]

[0155] Among them, C r (p) is the observed chromaticity of the red channel, C g(p) is the observed chromaticity of the green channel, C b (p) is the observed chromaticity of the blue channel, F r (p) is the red foreground color intensity, F g (p) is the green foreground color intensity, F b (p) is the blue foreground color, B r (p) is the red background chromaticity, B g (p) is the green background chromaticity, B b (p) is the blue background chromaticity.

[0156] The following describes an exemplary application of each embodiment of the present application in a practical application scenario.

[0157] In some embodiments, the image processing methods provided in various embodiments of the present application can be applied to a virtual live broadcast scene, where a host broadcasts live in front of a green screen. In response to a user's designated operation on a reference pixel and a setting operation on a target background chromaticity map, the terminal 400 receives the reference pixel and the target background chromaticity map. In response to the user's live broadcast operation, the terminal 400 receives a live image stream (multiple images to be processed) shot by the user in the green screen scene. The terminal 400 sends the images to be processed, the reference pixel, and the target background chromaticity map to the server 200. The server 200 synthesizes a target image corresponding to the multiple images to be processed, and then the server 200 sends the target image stream corresponding to the live image stream to the terminal 400 for display. The target image shows the host's live broadcast in the virtual scene. The virtual scene is determined based on the target background chromaticity map and can be a forest, an ocean, a studio, etc., thereby realizing virtual live broadcast through the image processing methods provided in various embodiments of the present application. When the green screen includes a QR code, replacing the green screen with a virtual scene can also realize camera movement in a three-dimensional virtual scene, bringing an immersive three-dimensional virtual experience to users watching the live broadcast.

[0158] In some embodiments, see Figure 4 , Figure 4 This is an algorithm diagram of the image processing method provided in each embodiment of the present application. In step 401, the initial opacity of the image to be processed is generated. In step 402, the uncertain area is generated. In step 403, the initial opacity of the uncertain area is optimized. In step 404, the foreground color saturation is determined and the green overflow of the foreground color saturation is eliminated. In step 405, the target image is synthesized based on the foreground color saturation and opacity.

[0159] The image synthesis model of the image processing method provided in each embodiment of the present application can be referred to formula (15). The image processing method provided in each embodiment of the present application actually needs to estimate the opacity α and foreground color F.

[0160] C(p)=α(p)*F(p)+(1-α(p))*B(p) (15);

[0161] Among them, p represents any pixel in the image to be processed, C is the observed pixel color, which includes RGB values, α is the opacity (the new opacity of the pixel in the uncertain area, or the opacity of the pixel in the foreground area), F is the target foreground color, which includes RGB values, and B is the target background chromaticity map (which can be set arbitrarily according to user needs), which includes RGB values.

[0162] Expanding formula (1) according to the three channels of RGB, we get formula (16):

[0163]

[0164] Among them, C r (p) is the observed chromaticity of the red channel, C g (p) is the observed chromaticity of the green channel, C b (p) is the observed chromaticity of the blue channel, F r (p) is the red foreground color intensity, F g (p) is the green foreground color intensity, F b (p) is the blue foreground color, B r (p) is the red background chromaticity, B g (p) is the green background chromaticity, B b (p) is the blue background chromaticity.

[0165] In some embodiments, generating the initial opacity in step 401 can be achieved by the following technical solution: calculating the color channel difference D between the green channel and the red and blue channels of each pixel in the image to be processed by formula (17):

[0166] D(p)=C g (p)-max(C r (p),C b (p)) (17);

[0167] Substitute the color difference map D into the mapping function g to calculate the initial opacity α(p) of the pixel P, α(p)∈[0,1], see formula (18):

[0168] α(p)=g(D(p)) (18);

[0169] See also Figure 5A , the mapping function g can be a linear hierarchical function, see formula (19):

[0170]

[0171] Where t2 and t1 are set thresholds, and clamp represents the truncation function. When the color channel difference D of pixel p is greater than t1, the initial opacity α(p) is 0. When the color difference of pixel p is less than t2, the initial opacity α(p) is 1. When the color difference of pixel p is not less than t2 and not greater than t1, the initial opacity α(p) is a value between 0 and 1.

[0172] See also Figure 5B ,right Figure 5A The mapping function shown in is improved, and a smoother result can be obtained by using the SmoothStep function, see formula (20) and formula (21):

[0173] SmoothStep(x,t1,t2)=g(x,t1,t2)*g(x,t1,t2)*(3-2*g(x,t1,t2)) (20);

[0174] α(p)=SmoothStep(D(p),t1,t2) (21);

[0175] Among them, t2 and t1 are set thresholds, clamp represents the truncation function, and SmoothStep represents the smoothing function. That is, the color channel difference D is substituted into the mapping function shown in formula (19) to calculate g(x, t1, t2). Then, g(x, t1, t2) is substituted into formula (20) to calculate SmoothStep(D, t1, t2) as the initial opacity.

[0176] In some embodiments, the foreground color is determined by the initial opacity, and most of the foreground can be correctly cut out. However, due to the presence of light and shadow in the green screen, the light and shadow in the green screen cannot be completely removed, and the edge of the character will also be inaccurate. Therefore, it is necessary to obtain an uncertain region to further optimize the transparency of the uncertain region. The generation of the uncertain region in step 402 can be achieved by the following technical solution: first, an erosion operation (erode) is performed on the initial opacity map to exclude the edge of the character from the foreground region, and then the foreground region is inverted to obtain the uncertain region. The uncertain region includes the edge of the character and the green screen background, see formula (22):

[0177] U = 255-erode(α) (22);

[0178] Among them, U is the grayscale value of the pixel of the image to be processed, α is the initial opacity of the pixel of the image to be processed, when U of a pixel is 255, the pixel belongs to the uncertain area, when U of a pixel is 0, the pixel belongs to the foreground area.

[0179] In some embodiments, the uncertain region may include a large area of ​​strong shadows. When the green screen background includes a light green QR code, the light green can also be considered as a strong shadow. In order to effectively remove the strong shadow, it is necessary to use a manually selected reference pixel as a reference. The optimization of the initial opacity of the uncertain region in step 403 can be achieved by the following technical solution: calculate the color space distance between all pixels in the image to be processed and the reference pixel, and normalize the color space distance to the range of 0 to 1. When calculating the color space distance, the calculation can be selected in the UV plane of the YUV space. The chromaticity can be converted from the RGB space to the YUV space by formula (23):

[0180]

[0181] Among them, R is the red chromaticity value of the RGB space (red channel reference chromaticity or red channel observed chromaticity), G is the green chromaticity value of the RGB space (green channel reference chromaticity or green channel observed chromaticity), B is the blue chromaticity value of the RGB space (blue channel reference chromaticity or blue channel observed chromaticity), Y is the Y channel chromaticity value of the YUV space (Y channel reference chromaticity or Y channel observed chromaticity), U is the U channel chromaticity value of the YUV space (U channel reference chromaticity or U channel observed chromaticity), and V is the V channel chromaticity value of the YUV space (V channel reference chromaticity or V channel observed chromaticity).

[0182] Continue to calculate the UV space distance between each pixel and the reference pixel using formula (24):

[0183] KeyD(p)=(C u (p)-Key u (p)) 2 +(C v (p)-Key v (p)) 2 (twenty four);

[0184] Among them, KeyD(p) is the spatial distance between pixel p and the reference pixel in the UV plane, C u (p) is the observed chromaticity of the U channel of pixel p, C v (p) is the V channel observed chromaticity of pixel p, Key u (p) is the U channel reference chromaticity, Key v (p) is the V channel reference chromaticity.

[0185] Continue to use the SmoothStep function for mapping to obtain the opacity of the reference pixel, see formula (25):

[0186] Keyα(p)=SmoothStep(KeyD(p),t1,t2) (25);

[0187] Among them, Keyα(p) is the opacity of the reference pixel, and SmoothStep(KeyD(p),t1,t2) can refer to Formula (19) and Formula (20).

[0188] Finally, the opacity of the uncertain region is optimized according to the following formula (26) to obtain the new opacity of the uncertain region:

[0189]

[0190] Among them, Key α(p) is the opacity of the reference pixel, U(p) is the grayscale value of pixel p of the image to be processed, and α(p) is the initial opacity of pixel p of the image to be processed.

[0191] The above formula can completely remove the shadows on the green screen and retain the natural edges of the characters.

[0192] In some embodiments, determining the foreground color saturation in step 404 and eliminating the green overflow of the foreground color saturation can be achieved by the following technical solution: Based on the opacity calculated above, the foreground color saturation of all pixels can be approximately inferred according to formula (27):

[0193]

[0194] Among them, C r (p) is the observed chromaticity of the red channel, C g (p) is the observed chromaticity of the green channel, C b (p) is the observed chromaticity of the blue channel, F r (p) is the red foreground color intensity, F g (p) is the green foreground color intensity, F b (p) is the blue foreground color, Key r (p) is the red channel chromaticity of the reference pixel, Key g (p) is the green channel chromaticity of the reference pixel, Key b (p) is the blue channel chromaticity of the reference pixel, and α(p) is the transparency of pixel p.

[0195] After inverting the foreground color saturation, the character's foreground color may appear green. Figure 6A , Figure 6A There is a greenish color at 601A in the image. The green color in the foreground color can be removed by the following formula (28):

[0196]

[0197] Among them, F r (p) is the red foreground color intensity, F g(p) is the green foreground color intensity, F b (p) is the blue foreground color saturation, that is, F is calculated by formula (28) g (p) is the green foreground color saturation to update, thereby removing the green in the foreground color, see Figure 6B , Figure 6B There is no green overflow at 601B.

[0198] See also Figure 7A and Figure 8A , Figure 7A and Figure 8A is a schematic diagram of the effects of the image processing methods provided in various embodiments of the present application, Figure 7A and Figure 8A The green screen scene used in the virtual live broadcast is shown. A light green QR code is printed on the green screen to help the camera identify and locate the scene, thereby realizing virtual camera movement and reconstructing the three-dimensional structure of the virtual scene. Figure 7B and Figure 8B , Figure 7B and Figure 8B It is a schematic diagram of the effect of the image processing method provided in the related art. Figure 7B and Figure 8B The transparency map obtained by the cutout method in the related art is shown. It can be seen that the shadows in the uncertain area are not eliminated. Figure 7C and Figure 8C , Figure 7C and Figure 8C is a schematic diagram of the effects of the image processing methods provided in various embodiments of the present application, Figure 7C and Figure 8C The transparency map obtained by the image processing method provided by each embodiment of the present application is shown. It can be seen that the shadows in the uncertain area have been eliminated. Figure 7D and Figure 8D , Figure 7D and Figure 8D It is a schematic diagram of the effect of the image processing method provided in the related art. Figure 7D and Figure 8D The target image synthesized by the cutout method in the related art is shown. It can be seen that the shadows in the uncertain area are not eliminated. Figure 7E and Figure 8E , Figure 7E and Figure 8E is a schematic diagram of the effects of the image processing methods provided in various embodiments of the present application, Figure 7E and Figure 8E The target image synthesized using the image processing method provided in each embodiment of the present application is shown. It can be seen that the shadows in the uncertain area have been eliminated and the entire green curtain wall can be correctly removed.

[0199] It is understandable that in the various embodiments of the present application, when user information, user images and other related data are involved, when the various embodiments of the present application are applied to specific products or technologies, user permission or consent must be obtained, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0200] The following continues to describe the exemplary structure of the image processing device 455 provided in each embodiment of the present application as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the image processing device 455 of the memory 450 may include: an opacity module 4551, which is used to perform opacity estimation processing on the image to be processed to obtain the opacity of the image to be processed; an erosion module 4552, which is used to perform an erosion operation on the opacity of the image to be processed to obtain a foreground area and an uncertain area of ​​the image to be processed; an optimization module 4553, which is used to update the opacity of the uncertain area based on the channel observed chromaticity of the uncertain area and the channel reference chromaticity of the reference pixel to obtain a new opacity of the uncertain area; a foreground module 4554, which is used to determine the target foreground color map of the image to be processed based on the new opacity of the uncertain area, the opacity of the foreground area and the channel reference chromaticity of the reference pixel; and a synthesis module 4555, which is used to determine the target foreground color map of the image to be processed based on the new opacity of the uncertain area, the opacity of the foreground area and the channel reference chromaticity of the reference pixel.

[0201] In some embodiments, the opacity module 4551 is also used to perform the following processing for each pixel of the image to be processed: obtain the channel observation chromaticity of the pixel, and determine the color channel difference of the pixel based on the channel observation chromaticity of the pixel; determine the opacity of the pixel based on the color channel difference of the pixel.

[0202] In some embodiments, the channel observed chromaticity includes the green channel observed chromaticity, the red channel observed chromaticity and the blue channel observed chromaticity, and the opacity module 4551 is further used to: obtain the larger observed chromaticity between the red channel observed chromaticity and the blue channel observed chromaticity; perform difference processing on the green channel observed chromaticity and the larger observed chromaticity to obtain the color channel difference of the pixel.

[0203] In some embodiments, the opacity module 4551 is also used to: obtain a first cutoff value and a second cutoff value; based on the first cutoff value and the second cutoff value, perform interception mapping processing on the color channel difference of the pixel to obtain a interception mapping result; based on the interception mapping result, determine the opacity of the pixel.

[0204] In some embodiments, the opacity module 4551 is also used to: when the color channel difference of the pixel is less than the second cutoff value, determine the value 1 as the truncated mapping result of the pixel; when the color channel difference of the pixel is greater than the first cutoff value, determine the value 0 as the truncated mapping result of the pixel, wherein the second cutoff value is less than the first cutoff value; when the color channel difference of the pixel is not less than the second cutoff value and not greater than the first cutoff value, perform the following processing: use the inverse of the first difference between the second cutoff value and the first truncation value as the slope, and use the product of the inverse of the slope and the first cutoff value as the vertical channel intercept; multiply the slope by the color channel difference of the pixel, and add the multiplication result to the vertical channel intercept to obtain the truncated mapping result of the pixel.

[0205] In some embodiments, the opacity module 4551 is further used to: perform any one of the following processes: determining the intercepted mapping result of the pixel as the opacity of the pixel; smoothing the intercepted mapping result to obtain the opacity of the pixel.

[0206] In some embodiments, the opacity module 4551 is further used to: obtain the cube value and the square value of the intercepted mapping result; and obtain the opacity that is negatively correlated with the cube value and positively correlated with the square value.

[0207] In some embodiments, the corrosion module 4552 is also used to: obtain the corrosion operation matrix; perform the following processing on each pixel of the image to be processed: perform an AND operation on the opacity of the pixel through the corrosion operation matrix to obtain the corrosion pixel value of the pixel, and determine the second difference between the standard pixel value and the corrosion pixel value; pixels whose second difference is the standard pixel value are formed into an uncertain area, and pixels whose difference is zero are formed into a foreground area.

[0208] In some embodiments, the operation matrix includes N*N operation elements, N is an integer greater than 1, and the value of each operation element is a standard pixel value. The corrosion module 4552 is also used to: when the opacity of the pixel is greater than the opacity threshold, update the opacity of the pixel to the standard pixel value; when the opacity of the pixel is not greater than the opacity threshold, update the opacity of the pixel to zero; obtain at least one associated pixel of the corresponding pixel; obtain the operation element of the corresponding pixel in the operation matrix and the operation element corresponding to each associated pixel; perform AND processing on the updated transparency of the pixel and the value of the corresponding operation element to obtain a first AND processing result; perform the following processing on each associated pixel: perform AND processing on the updated transparency of the associated pixel and the value of the corresponding operation element to obtain a second AND processing result; when there is a zero value in the first AND processing result and at least one second AND processing result, determine the eroded pixel value of the pixel to be a zero value; when there is no zero value in the first AND processing result and at least one second AND processing result, determine the eroded pixel value of the pixel to be a standard pixel value.

[0209] In some embodiments, the optimization module 4553 is also used to: determine the chromaticity distance between each pixel in the uncertain area and the reference pixel based on the channel reference chromaticity and the channel observed chromaticity; based on the chromaticity distance, update the opacity of the uncertain area to obtain a new opacity of the uncertain area.

[0210] In some embodiments, the optimization module 4553 is also used to: obtain the green channel reference chromaticity, the red channel reference chromaticity, and the blue channel reference chromaticity as the channel reference chromaticity of the reference pixel before determining the chromaticity distance between each pixel in the uncertain area and the reference pixel based on the channel reference chromaticity and the channel observed chromaticity; obtain the green channel observed chromaticity, the red channel observed chromaticity, and the blue channel observed chromaticity as the channel observed chromaticity of each pixel in the uncertain area.

[0211] In some embodiments, the optimization module 4553 is further used to: perform spatial conversion processing on the channel reference chromaticity to obtain the target reference chromaticity; perform the following processing on each pixel in the uncertain area: perform spatial conversion processing on the channel observation chromaticity to obtain the target observation chromaticity; obtain the target channel corresponding to the chromaticity distance; obtain the target observation chromaticity corresponding to the target channel from the target observation chromaticity, and obtain the target reference chromaticity corresponding to the target channel from the target reference chromaticity; determine the chromaticity distance between the pixel and the reference pixel based on the target observation chromaticity corresponding to the target channel and the target reference chromaticity corresponding to the target channel.

[0212] In some embodiments, the optimization module 4553 is also used to: smooth the chromaticity distance between each pixel in the uncertain area and the reference pixel to obtain the shadow reference opacity of each pixel in the uncertain area; perform the following processing on each pixel in the uncertain area: when the shadow reference opacity of the pixel is less than the opacity of the pixel, determine the shadow reference opacity of the pixel as the new opacity of the pixel; when the shadow reference opacity of the pixel is not less than the opacity of the pixel, determine the opacity of the pixel as the new opacity of the pixel.

[0213] In some embodiments, the foreground module 4554 is also used to: perform the following processing on each pixel in the image to be processed: when the pixel is in the foreground area, obtaining the transparency of the pixel based on the opacity of the pixel; when the pixel is in the uncertain area, obtaining the transparency of the pixel based on the new opacity of the pixel; multiplying the obtained transparency of the pixel with the channel reference chromaticity of the reference pixel, and obtaining the third difference between the channel observed chromaticity of the pixel and the multiplication result; when the pixel is in the foreground area, determining the ratio of the third difference to the opacity of the pixel as the foreground color chromaticity of the pixel; when the pixel is in the uncertain area, determining the ratio of the third difference to the new opacity of the pixel as the foreground color chromaticity of the pixel, and forming the foreground color chromaticity of multiple pixels in the image to be processed into a target foreground color chromaticity map of the image to be processed.

[0214] In some embodiments, the foreground color saturation of a pixel includes red foreground color saturation, green foreground color saturation, and blue foreground color saturation. After determining the target foreground color saturation map of the image to be processed, the foreground module 4554 is further used to: perform the following processing for each pixel in the image to be processed: obtain the red foreground color saturation of the pixel and the average value of the blue foreground color saturation of the pixel; when the green foreground color saturation of the pixel is greater than the average value, update the foreground color saturation of the pixel to the average value; when the green foreground color saturation of the pixel is not greater than the average value, keep the foreground color saturation of the pixel unchanged.

[0215] Each embodiment of the present application provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image processing method described in each embodiment of the present application.

[0216] Each embodiment of the present application provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the image processing method provided by each embodiment of the present application, for example, Figures 3A-3C The image processing method is shown.

[0217] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or various devices including one or any combination of the above memories.

[0218] In some embodiments, executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0219] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0220] As an example, executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0221] In summary, through the embodiments of the present application, by performing an erosion operation on the opacity of the image to be processed, the foreground area and the uncertain area of ​​the image to be processed are obtained, which can effectively improve the accuracy of the foreground edge and the uncertain area. Based on the channel observation chromaticity of the uncertain area and the channel reference chromaticity of the reference pixel, the opacity of the uncertain area is updated to obtain a new opacity of the uncertain area. Since the opacity of the uncertain area is updated, the transparency of the shadow in the uncertain area can be optimized. Based on the optimized and updated transparency, the foreground target scene map is inferred, and combined with the target background chromaticity map for merging, the target image with the original content in the uncertain area accurately removed can be obtained.

[0222] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Performing opacity estimation processing on the image to be processed to obtain the opacity of the image to be processed; Performing an erosion operation on the opacity of the image to be processed to obtain a foreground area and an uncertain area of ​​the image to be processed; Based on the channel observation chromaticity of the uncertain area and the channel reference chromaticity of the reference pixel, the opacity of the uncertain area is updated to obtain a new opacity of the uncertain area; Determining a target foreground color map of the image to be processed based on the new opacity of the uncertain area, the opacity of the foreground area, and the channel reference chromaticity of the reference pixel; A target image corresponding to the image to be processed is synthesized based on the target foreground chromaticity map of the image to be processed, the target background chromaticity map of the image to be processed, the new opacity of the uncertain area, and the opacity of the foreground area.

2. The method according to claim 1, characterized in that The performing opacity estimation processing on the image to be processed to obtain the opacity of the image to be processed includes: The following processing is performed for each pixel of the image to be processed: Obtaining the channel-observed chromaticity of the pixel, and determining the color channel difference of the pixel based on the channel-observed chromaticity of the pixel; The opacity of the pixel is determined based on the color channel difference value of the pixel.

3. The method according to claim 2, characterized in that The channel observation chromaticity includes a green channel observation chromaticity, a red channel observation chromaticity, and a blue channel observation chromaticity, and determining the color channel difference of the pixel based on the channel observation chromaticity of the pixel includes: Obtaining the larger observed chromaticity of the red channel and the blue channel observed chromaticity; A difference process is performed on the green channel observed chromaticity and the larger observed chromaticity to obtain a color channel difference value of the pixel.

4. The method according to claim 2, characterized in that The determining the opacity of the pixel based on the color channel difference of the pixel includes: Obtaining a first cutoff value and a second cutoff value; Based on the first truncation value and the second truncation value, performing a truncation mapping process on the color channel difference of the pixel to obtain a truncation mapping result; Based on the intercepted mapping result, the opacity of the pixel is determined.

5. The method according to claim 4, characterized in that The performing interception mapping processing on the color channel difference of the pixel based on the first truncation value and the second truncation value to obtain an interception mapping result includes: When the color channel difference value of the pixel is less than the second cutoff value, determining a value of 1 as the cutoff mapping result of the pixel; When the color channel difference value of the pixel is greater than the first truncation value, determining a value of 0 as a truncation mapping result of the pixel, wherein the second truncation value is less than the first truncation value; When the color channel difference of the pixel is not less than the second cutoff value and not greater than the first cutoff value, perform the following processing: The reciprocal of the first difference between the second cutoff value and the first cutoff value is used as the slope, and the product of the reciprocal of the slope and the first cutoff value is used as the longitudinal channel intercept; The slope is multiplied by the color channel difference of the pixel, and the multiplication result is added to the vertical channel intercept to obtain the intercepted mapping result of the pixel.

6. The method according to claim 4, characterized in that The determining the opacity of the pixel based on the intercepted mapping result includes: Perform any of the following: determining the intercepted mapping result of the pixel as the opacity of the pixel; The intercepted mapping result is smoothed to obtain the opacity of the pixel.

7. The method according to claim 6, characterized in that The smoothing process on the intercepted mapping result to obtain the opacity of the pixel includes: Obtaining the cube value and the square value of the intercepted mapping result; Obtain an opacity that is negatively correlated with the cube value and positively correlated with the square value.

8. The method according to claim 1, characterized in that The performing an erosion operation on the opacity of the image to be processed to obtain a foreground area and an uncertain area of ​​the image to be processed includes: Get the corrosion operation matrix; The following processing is performed for each pixel of the image to be processed: Performing an AND operation on the opacity of the pixel using the corrosion operation matrix to obtain an eroded pixel value of the pixel, and determining a second difference between the standard pixel value and the eroded pixel value; The pixels whose second difference value is the standard pixel value constitute the uncertain area, and the pixels whose difference value is zero constitute the foreground area.

9. The method according to claim 8, characterized in that The operation matrix includes N*N operation elements, where N is an integer greater than 1, and the value of each operation element is the standard pixel value. The opacity of the pixel is subjected to an AND operation by the erosion operation matrix to obtain the eroded pixel value of the pixel, including: When the opacity of the pixel is greater than the opacity threshold, updating the opacity of the pixel to the standard pixel value; When the opacity of the pixel is not greater than the opacity threshold, updating the opacity of the pixel to zero; Obtain at least one associated pixel corresponding to the pixel, and obtain an operation element corresponding to the pixel and an operation element corresponding to each of the associated pixels in the operation matrix; Performing AND processing on the updated transparency of the pixel and the value of the corresponding operation element to obtain a first AND processing result; For each of the associated pixels, the following processing is performed: performing AND processing on the updated transparency of the associated pixel and the value of the corresponding operation element to obtain a second AND processing result; When there is a zero value in the first AND processing result and at least one second AND processing result, determining the zero value as the eroded pixel value of the pixel; When the zero value does not exist in the first AND processing result and at least one second AND processing result, the standard pixel value is determined as the eroded pixel value of the pixel.

10. The method according to claim 1, characterized in that The updating process of the opacity of the uncertain area based on the channel observation chromaticity of the uncertain area and the channel reference chromaticity of the reference pixel to obtain a new opacity of the uncertain area includes: Determining a chromaticity distance between each pixel in the uncertain area and the reference pixel based on the channel reference chromaticity and the channel observed chromaticity; Based on the chromaticity distance, the opacity of the uncertain area is updated to obtain a new opacity of the uncertain area.

11. The method according to claim 10, characterized in that Before determining the chromaticity distance between each pixel in the uncertain area and the reference pixel based on the channel reference chromaticity and the channel observed chromaticity, the method further includes: Obtain green channel reference chromaticity, red channel reference chromaticity, and blue channel reference chromaticity as channel reference chromaticities of the reference pixel; The green channel observation chromaticity, the red channel observation chromaticity, and the blue channel observation chromaticity are obtained as the channel observation chromaticity of each pixel in the uncertain area.

12. The method according to claim 10, characterized in that The determining, based on the channel reference chromaticity and the channel observed chromaticity, a chromaticity distance between each pixel in the uncertain area and the reference pixel, includes: Performing spatial conversion processing on the channel reference chromaticity to obtain target reference chromaticity; The following processing is performed for each pixel in the uncertainty area: Performing spatial conversion processing on the channel observation chromaticity to obtain the target observation chromaticity; Obtaining a target channel corresponding to the chromaticity distance; Acquire a target observation chromaticity corresponding to the target channel from the target observation chromaticity, and acquire a target reference chromaticity corresponding to the target channel from the target reference chromaticity; A chromaticity distance between the pixel and the reference pixel is determined based on a target observed chromaticity corresponding to the target channel and a target reference chromaticity corresponding to the target channel.

13. The method according to claim 10, characterized in that The updating of the opacity of the uncertain region based on the chromaticity distance to obtain a new opacity of the uncertain region includes: determining a reference opacity of each pixel in the uncertain area based on a chromaticity distance between each pixel in the uncertain area and the reference pixel; The following processing is performed for each pixel in the uncertainty area: When the reference opacity of the pixel is less than the opacity of the pixel, determining the reference opacity of the pixel as a new opacity of the pixel; When the reference opacity of the pixel is not less than the opacity of the pixel, the opacity of the pixel is determined as the new opacity of the pixel.

14. The method according to claim 1, wherein The determining, based on the new opacity of the uncertain area, the opacity of the foreground area, and the channel reference chromaticity of the reference pixel, of a target foreground color map of the image to be processed includes: The following processing is performed for each pixel in the image to be processed: When the pixel is in the foreground area, obtaining the transparency of the pixel based on the opacity of the pixel; When the pixel is in the uncertainty area, obtaining the transparency of the pixel based on the new opacity of the pixel; Obtaining a multiplication result of the transparency of the pixel and the channel reference chromaticity of the reference pixel, and obtaining a third difference between the channel observation chromaticity of the pixel and the multiplication result; When the pixel is in the foreground area, determining a ratio of the third difference value to the opacity of the pixel as the foreground color saturation of the pixel; When the pixel is in the uncertain area, determining a ratio of the third difference value to the new opacity of the pixel as the foreground color saturation of the pixel; The foreground color intensities of a plurality of pixels in the image to be processed are combined into a target foreground color intensities map of the image to be processed.

15. The method according to claim 14, characterized in that The foreground color saturation of the pixel includes red foreground color saturation, green foreground color saturation, and blue foreground color saturation. After determining the target foreground color saturation map of the image to be processed, the method further includes: The following processing is performed for each pixel in the image to be processed: Obtaining an average value of the red foreground color saturation of the pixel and the blue foreground color saturation of the pixel; When the green foreground color saturation of the pixel is greater than the average value, updating the foreground color saturation of the pixel to the average value; When the green foreground color saturation of the pixel is not greater than the average value, the foreground color saturation of the pixel is kept unchanged.

16. An image processing device, characterized in that: The device comprises: An opacity module is used to perform opacity estimation processing on the image to be processed to obtain the opacity of the image to be processed; an erosion module, configured to perform an erosion operation on the opacity of the image to be processed to obtain a foreground area and an uncertain area of ​​the image to be processed; an optimization module, configured to update the opacity of the uncertain area based on the channel observation chromaticity of the uncertain area and the channel reference chromaticity of the reference pixel, to obtain a new opacity of the uncertain area; a foreground module, configured to determine a target foreground color map of the image to be processed based on the new opacity of the uncertain area, the opacity of the foreground area, and the channel reference chromaticity of the reference pixel; The synthesis module is used to determine the target foreground color map of the image to be processed based on the new opacity of the uncertain area, the opacity of the foreground area and the channel reference chromaticity of the reference pixel.

17. An electronic device, characterized in that: The electronic device comprises: a memory for storing executable instructions; A processor, configured to implement the image processing method according to any one of claims 1 to 15 when executing the executable instructions stored in the memory.

18. A computer-readable storage medium storing executable instructions, characterized in that: When the executable instructions are executed by a processor, the image processing method according to any one of claims 1 to 15 is implemented.

19. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the image processing method according to any one of claims 1 to 15 is implemented.

Images