Image synthesis method, device and equipment

Abstract

The invention discloses an image synthesis method, device and equipment, and the method comprises the steps: carrying out the image matting of an input image, obtaining a main channel image of the input image, and enabling the input image to comprise a foreground image and a pure-color background image; obtaining a comprehensive channel image of the input image according to each color channel value corresponding to the foreground image and the pure color background image and the main body channel image; preprocessing the comprehensive channel image, and determining a target image matting image according to a target channel image obtained through preprocessing and the foreground image; and synthesizing a background image to be synthesized, the target matting image and the target channel image to obtain a target synthesized image. According to the invention, a more natural image matting image can be accurately and quickly obtained with lower cost, so that a high-quality target composite image can be obtained, and the availability of the composite image is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image synthesis method, device and equipment. Background Art

[0002] In the field of image processing, accurate and rapid image extraction is crucial for image synthesis. This process is called keying, which involves extracting a foreground image of interest from an image and separating it from the background. Keying is widely used in game development, film and television special effects, virtual reality, video conferencing, and image and video editing.

[0003] Traditional matting methods (such as color key matting or chroma matting) require manual intervention or manual marking of matting areas, and the efficiency and cost-effectiveness of matting are low. In addition, traditional matting methods ignore some detailed information, and the edges of the matting results are blurred and distorted, resulting in poor matting effects. Although the traditional edge feathering method can make the edges of the image appear softer and more continuous, it requires additional edge processing steps after matting, which wastes a lot of time and cost. In addition, the traditional edge feathering method may cause the edges of the matting results to be too blurred, losing some details and clarity, affecting the quality and realism of the image. Therefore, the existing technology has the problem of not being able to obtain a more natural matting image at a lower cost, accurately and quickly, and thus cannot obtain a high-quality target composite image.

[0004] To address the above issues, no effective solutions have been proposed so far. Summary of the Invention

[0005] The embodiments of this specification provide an image synthesis method, apparatus, and device to solve the problem that the existing technology cannot obtain a more natural matte image at a low cost, accurately, and quickly, and thus cannot obtain a high-quality target synthesized image.

[0006] In a first aspect, an embodiment of this specification provides an image synthesis method, the method comprising:

[0007] Performing a matting process on an input image to obtain a main channel image of the input image, wherein the input image includes a foreground image and a solid color background image;

[0008] Obtaining a comprehensive channel image of the input image according to the color channel values corresponding to the foreground image and the pure color background image, and the main channel image;

[0009] Preprocessing the integrated channel image to determine a target matting image based on the target channel image and the foreground image obtained by the preprocessing;

[0010] The background image to be synthesized, the target matted image and the target channel image are synthesized to obtain a target synthesized image.

[0011] In some embodiments, obtaining the integrated channel image of the input image according to the color channel values corresponding to the foreground image and the pure color background image, and the main channel image, includes:

[0012] Obtaining a first channel image of the input image according to the color channel values corresponding to the foreground image and the solid color background image;

[0013] performing background noise removal processing on the first channel image to obtain a second channel image;

[0014] The main channel image and the second channel image are summed to obtain a comprehensive channel image of the input image.

[0015] In some embodiments, the color channel values corresponding to the foreground image include: a foreground red channel value, a foreground green channel value, and a foreground blue channel value; the color channel values corresponding to the solid color background image include: a background red channel value, a background green channel value, and a background blue channel value;

[0016] Accordingly, obtaining the first channel image of the input image according to the color channel values corresponding to the foreground image and the pure color background image includes:

[0017] According to the foreground red channel value, the foreground green channel value, the foreground blue channel value, the background red channel value, the background green channel value, and the background blue channel value, the first channel image of the input image is obtained according to the following formula:

[0018]

[0019] Among them, a1 is the first channel image; F r 、F g 、F b are the foreground red channel value, foreground green channel value, and foreground blue channel value respectively; B r 、B g 、B b They are the background red channel value, background green channel value, and background blue channel value respectively.

[0020] In some embodiments, the preprocessing of the integrated channel image includes:

[0021] The comprehensive channel image is subjected to an anti-overflow process, and then the comprehensive channel image after the anti-overflow process is subjected to a filtering process to obtain a pre-processed target channel image.

[0022] In some embodiments, determining the target matte image based on the target channel image obtained by preprocessing and the foreground image includes:

[0023] Fusing the target channel image and the foreground image to obtain an initial matted image;

[0024] The initial matte image is subjected to a de-spill process to obtain a matte image after de-spill, and the matte image after de-spill is used as the target matte image.

[0025] In some embodiments, performing de-spill processing on the initial matte image includes:

[0026] Determine the matte red channel value, matte green channel value, and matte blue channel value corresponding to each pixel point in the initial matte image;

[0027] Calculate the average value of the red and blue channels corresponding to each pixel point based on the keying red channel value and keying blue channel value corresponding to each pixel point;

[0028] Determine whether the average value of the red and blue channels is less than the green channel value of the matte;

[0029] If yes, take the average value of the red and blue channels as the new green channel value;

[0030] An image composed of the matte red channel value, the new matte green channel value, and the matte blue channel value corresponding to each pixel point is used as the matte image after de-spill.

[0031] In some embodiments, the synthesizing process of the background image to be synthesized, the target matte image, and the target channel image includes:

[0032] A linear interpolation operation is performed on the background image to be synthesized, the target matte image and the target channel image to obtain a target synthesized image.

[0033] In a second aspect, embodiments of this specification further provide an image synthesis device, the device comprising:

[0034] A subject channel image determination module is used to perform matting processing on an input image to obtain a subject channel image of the input image, wherein the input image includes a foreground image and a solid color background image;

[0035] a comprehensive channel image determination module, configured to obtain a comprehensive channel image of the input image based on the color channel values corresponding to the foreground image and the pure color background image, and the subject channel image;

[0036] a target matting image determination module, configured to pre-process the integrated channel image to determine a target matting image based on the target channel image obtained by the pre-processing and the foreground image;

[0037] The synthesis module is used to synthesize the background image to be synthesized, the target matte image and the target channel image to obtain a target synthesized image.

[0038] In a third aspect, an embodiment of this specification further provides an image synthesis device, comprising a memory, a processor, and a computer program / instruction stored in the memory, wherein the processor executes the computer program / instruction to implement the steps of the above-mentioned image synthesis method.

[0039] In a fourth aspect, the embodiments of this specification further provide a computer-readable storage medium having a computer program / instruction stored thereon, which implements the steps of the above-mentioned image synthesis method when executed by a processor.

[0040] In a fifth aspect, the embodiments of this specification also provide a computer program product, which implements the steps of the above-mentioned image synthesis method when the computer program / instructions are executed by a processor.

[0041] Embodiments of this specification provide an image synthesis method, apparatus, and device. First, an input image is matted to obtain a subject channel image of the input image, the input image comprising a foreground image and a solid-color background image. Then, a composite channel image of the input image is obtained based on the color channel values corresponding to the foreground image and the solid-color background image, as well as the subject channel image. The composite channel image is then preprocessed to determine a target matted image based on the target channel image obtained by the preprocessing and the foreground image. Finally, the background image to be synthesized, the target matted image, and the target channel image are synthesized to obtain a target composite image. In this embodiment, a first channel image is calculated based on the color channel values corresponding to the foreground image and the solid-color background image. The first channel image has a smooth transition without discontinuities and more effectively preserves image details. The composite channel image is then determined by combining the subject channel image. This allows the resulting composite channel image to preserve the main content while ensuring a natural transition without loss of edge details, resulting in a more perfect matting effect and reducing the complexity and processing cost of pre-matting followed by edge processing. By preprocessing the integrated channel image, a smoothed target channel image can be obtained, and thus an accurate matting image can be obtained based on the smoothed target channel image, thereby obtaining a high-quality target composite image, effectively improving the usability of the composite image. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0043] Figure 1 This is a flowchart of an image synthesis method provided in an embodiment of this specification;

[0044] Figure 2 is a schematic diagram of the image synthesis process provided by the embodiments of this specification;

[0045] Figure 3 This is a comparison chart of image synthesis effects provided by the embodiments of this specification;

[0046] Figure 4 This is a schematic diagram of the structure of an image synthesis device provided in an embodiment of this specification;

[0047] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0048] To help those skilled in the art better understand the technical solutions in this specification, the following will provide a clear and complete description of the technical solutions in the embodiments of this specification, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. All other embodiments derived by those skilled in the art based on the embodiments in this specification without creative effort shall fall within the scope of protection of this specification.

[0049] In the field of image processing, accurate and rapid image extraction is crucial for image synthesis. This process is called keying, which involves extracting a foreground image of interest from an image and separating it from the background. Keying is widely used in game development, film and television special effects, virtual reality, video conferencing, and image and video editing.

[0050] Traditional keying methods (such as chroma keying or chroma keying) require manual intervention or manual marking of keying areas, resulting in low efficiency and cost-effectiveness. Furthermore, these methods often overlook some fine details, resulting in blurred and distorted edges and poor keying results. While traditional edge feathering methods can make image edges appear softer and more continuous, they require additional post-keying edge processing, which wastes significant time and money. Furthermore, traditional edge feathering methods can cause excessively blurred edges in the keyed image, losing detail and clarity, and compromising image quality and realism. Chroma keying or chroma keying methods first calculate the similarity between the color of pixels in the processed image and the key color (the user-selected background color), then convert this similarity into an image transparency, which is then used to complete the keying. Edge feathering is often used to reduce the sharpness of image edges, creating a softer and more natural transition. During edge feathering, the edges of an image gradually blur, mitigating the sudden change between the edge and the background, resulting in a softer and more continuous image.

[0051] Therefore, some limitations and shortcomings of the existing methods of using chroma keying or chroma keying combined with edge feathering are:

[0052] 1. Limitations of matte matting effects: Chroma key matting or chroma matting combined with edge feathering often have difficulty handling matte matting tasks in complex scenes, which can easily lead to unsatisfactory matte matting effects. For example, when the main object and the background are similar in color, inaccurate matte matting or loss of details may occur.

[0053] 2. The fuzziness of edge feathering: Edge feathering may cause the edges of the matte result to be too blurred, resulting in the loss of some details and clarity, affecting the quality and realism of the image.

[0054] 3. Processing complexity: Combining chroma keying and edge feathering usually requires multiple steps and parameter adjustments, which increases the complexity and time cost of processing. At the same time, it requires more manual intervention and adjustments, which reduces efficiency and automation. The cost of manual intervention is relatively high.

[0055] In summary, the existing technology has the problem of being unable to obtain a more natural matting image at a low cost, accurately and quickly, and thus being unable to obtain a high-quality target composite image.

[0056] In response to the above-mentioned problems existing in the existing methods and the specific reasons for the above-mentioned problems, this application considers introducing an image synthesis method, device and equipment that can produce a more accurate and natural matting effect at a lower cost (that is, the matting image can be obtained accurately and quickly at a lower cost), thereby obtaining a high-quality target composite image and effectively improving the usability of the composite image.

[0057] Based on the above ideas, this specification proposes an image synthesis method. First, the input image is subjected to matting processing to obtain the main channel image of the input image, wherein the input image includes a foreground image and a solid-color background image. Then, based on the color channel values corresponding to the foreground image and the solid-color background image, as well as the main channel image, a comprehensive channel image of the input image is obtained. Then, the comprehensive channel image is preprocessed to determine the target matting image based on the target channel image obtained by the preprocessing and the foreground image. Finally, the background image to be synthesized, the target matting image, and the target channel image are synthesized to obtain a target synthesized image.

[0058] Figure 1 It is a flow chart of an image synthesis method provided by an embodiment of this specification. Although this specification provides the method operation steps or device structure shown in the following embodiments or drawings, the method or device may include more or fewer operation steps or module units after partial merger based on routine or no creative labor. In the steps or structures where there is no necessary causal relationship logically, the execution order of these steps or the module structure of the device is not limited to the execution order or module structure shown in the embodiments or drawings of this specification. When the method or module structure is applied to an actual device, server or terminal product, it can be executed sequentially or in parallel according to the method or module structure shown in the embodiment or drawings (for example, a parallel processor or multi-threaded processing environment, or even a distributed processing, server cluster implementation environment). For specific implementation, please refer to Figure 1 As shown, the method may include the following contents.

[0059] S101: performing matting processing on an input image to obtain a main channel image of the input image, wherein the input image includes a foreground image and a pure color background image.

[0060] In some embodiments, the input image can be the image to be processed, and can include a foreground image and a solid-color background image (e.g., a green (blue) screen). The foreground image and solid-color background image (e.g., a green (blue) screen) can be obtained by camera capture or video stream capture, etc., and this specification does not specifically limit this. The above-mentioned matting process can be performed using traditional chroma key matting, that is, the similarity between the color of the pixels in the above-mentioned input image and the key color (the background screen color selected by the user) can be calculated, and then the similarity is converted into the image transparency. The matting is then completed based on the transparency, and finally the main channel image a0 of the input image is obtained. Chroma key matting can calculate a transparent a0 of the main body with a high similarity to the background. This method can effectively preserve the main content, but it may lead to problems such as unnatural transitions in transparency information or loss of details. Therefore, after obtaining the above-mentioned main channel image, it is necessary to adopt a partitioning method and perform special processing on edge details. This can simultaneously focus on and carefully process the differences between the main body and the edges, making the matting result more realistic and clear, thereby effectively improving the quality and usability of the synthesized image. The method of performing special processing on edge details will be explained later and will not be repeated in this specification.

[0061] S102: Obtaining a comprehensive channel image of the input image according to the color channel values corresponding to the foreground image and the pure color background image, and the main channel image.

[0062] In some embodiments, the above S102 obtains the integrated channel image of the input image based on the color channel values corresponding to the foreground image and the pure color background image, and the main channel image. In specific implementation, it may include:

[0063] S21: Obtaining a first channel image of the input image according to the color channel values corresponding to the foreground image and the pure color background image;

[0064] S22: performing background noise removal processing on the first channel image to obtain a second channel image;

[0065] S23: performing summation processing on the main channel image and the second channel image to obtain a comprehensive channel image of the input image.

[0066] In some embodiments, the color channel values corresponding to the foreground image in S102 may include: a foreground red channel value, a foreground green channel value, and a foreground blue channel value, and the color channel values corresponding to the solid color background image may include: a background red channel value, a background green channel value, and a background blue channel value;

[0067] Accordingly, the step S21 above of obtaining the first channel image of the input image according to the color channel values corresponding to the foreground image and the solid color background image may include:

[0068] S211: Obtain a first channel image of the input image according to the following formula based on the foreground red channel value, the foreground green channel value, the foreground blue channel value, the background red channel value, the background green channel value, and the background blue channel value:

[0069]

[0070] Among them, a1 is the first channel image; F r 、F g 、F b are the foreground red channel value, foreground green channel value, and foreground blue channel value respectively; B r 、B g 、B b They are the background red channel value, background green channel value, and background blue channel value respectively.

[0071] In some embodiments, each color channel in the RGB channel of the foreground image and the solid color background image of the input image can be obtained (the RGB channel is a channel that stores image color information, and each channel in the RGB channel represents the brightness information of red, green, and blue respectively), such as: obtaining the foreground red channel value F corresponding to the foreground image r , foreground green channel value F g and the foreground blue channel value F b , get the background red channel value B corresponding to the pure background image r , background green channel value B g and background blue channel value B b , and then calculate the first channel image a1 of the input image according to the following formula (1) (the first channel image a1 can be a semi-transparent mask image; mask means blocking or covering, which can block part of the image content and display the image content of a specific area, equivalent to a window):

[0072]

[0073] The principle of the above formula (1) is to calculate a1 by the channel value difference between the color channel values in the foreground image and the color channel values in the background image of the input image. This can make the obtained a1 transition smooth and without discontinuity, and at the same time can more effectively preserve image details. It should be noted that if the above formula (1) is used alone, the subject in the foreground image will usually appear semi-transparent.

[0074] In some embodiments, the background noise elimination process in S22 can be performed by a power operation, that is, a power operation can be performed on the first channel image. The power operation can be a power(number, power) function, which returns the power of a given number, where number is the base and power is the exponent. The specific calculation formula is as follows:

[0075] a2=power(a1,1.8)(2)

[0076] Where a2 is the second channel image; a1 is the first channel image, which serves as the base; 1.8 is the exponent; and power is the power operation function.

[0077] In some embodiments, the subject channel image and the second channel image are summed (or fused) in S23 to obtain a comprehensive channel image (or third channel image). This can be done according to the following formula:

[0078] a3=a0+a2(3)

[0079] Among them, a3 is the integrated channel image (or called the third channel image); a2 is the second channel image; and a0 is the main channel image.

[0080] By summing the main channel image and the second pass image after background noise removal, the matting results obtained by the two different matting methods can be fused, so that the resulting composite channel image can retain the main content while also allowing the edge details to transition naturally without causing detail loss, making the matting effect more perfect. The above-mentioned special processing of edge details can be used to obtain the first channel image a1 through the above formula (1), and then perform background noise removal on a1 through the above formula (2), so that the obtained second channel image a2 can transition smoothly. By fusing a0 and a2, the problem of the unnatural transition of the main channel image a0 can be compensated.

[0081] S103: Preprocessing the integrated channel image to determine a target matting image based on the target channel image obtained by the preprocessing and the foreground image.

[0082] In some embodiments, the pre-processing of the integrated channel image in S103 may include:

[0083] S31: performing anti-overflow processing on the comprehensive channel image, and then performing filtering processing on the comprehensive channel image after the anti-overflow processing to obtain a pre-processed target channel image.

[0084] In some embodiments, the above-mentioned pre-processing may include but is not limited to: anti-overflow processing, filtering processing (or mean filtering processing).

[0085] The above-mentioned overflow prevention process can be performed using the following formula (the comprehensive channel image after the above-mentioned overflow prevention process can be used as the fourth channel image):

[0086] a4=Clamp(a3,0,1)(4)

[0087] Among them, a4 is the fourth channel image; a3 is the integrated channel image (or called the third channel image); the Clamp function is a function used to limit the value within a specified range, which can be called an anti-overflow processing function; 0 and 1 represent the limit range.

[0088] By performing anti-overflow processing on the integrated channel image a3, a3 can be limited to between 0 and 1 to prevent value overflow.

[0089] The filtering process of the comprehensive channel image after the anti-overflow process can be performed according to the following formula:

[0090]

[0091] Among them, a5 is the target channel image; a4 is the fourth channel image (i.e., the comprehensive channel image after anti-overflow processing); the pixel points on the x and y images; Used to calculate the sum of the pixel values of all pixels in a 3x3 window.

[0092] Specifically, a 3x3 convolution kernel can be taken, with the current pixel as the center, and the surrounding 8 pixels including the top, bottom, left, right, and diagonal pixels. The 9 pixel values are added together and then divided by 9 to get the mean value, which is then used to replace the center pixel value. Among them, the 3x3 convolution kernel will slide on the image, and the above formula (5) can be applied to each position to calculate the new pixel value, thereby obtaining the smoothed target channel image. By filtering the comprehensive channel image after anti-overflow processing, the mutation and noise on the Alpha channel after the fusion of a0 and a2 can be effectively reduced, making the transition smoother.

[0093] In some embodiments, determining the target matting image based on the target channel image obtained by preprocessing and the foreground image in S103 may include:

[0094] S32: Fusing the target channel image and the foreground image to obtain an initial matted image;

[0095] S33: performing de-spill processing on the initial matte image to obtain a matte image after de-spill, and using the matte image after de-spill as the target matte image.

[0096] In some embodiments, the fusion of the target channel image and the foreground image in S32 may be performed according to the following formula:

[0097] I k =I f *a5(6)

[0098] Among them, I k is the initial matte image; I f is the foreground image; a5 is the target channel image.

[0099] The above fusion is to multiply the foreground image and the target channel image to obtain the initial matting result (i.e., the initial matting image). There are some overflow colors in the initial matting image, and further de-overflow processing is required to obtain the matting image after de-overflowing.

[0100] In some embodiments, the de-spill processing of the initial matted image in S33 may include:

[0101] S331: Determine the matting red channel value, matting green channel value, and matting blue channel value corresponding to each pixel point in the initial matting image;

[0102] S332: Calculate the average value of the red and blue channels corresponding to each pixel point based on the keying red channel value and the keying blue channel value corresponding to each pixel point;

[0103] S333: Determine whether the average value of the red and blue channels is less than the green channel value of the matte;

[0104] S334: If yes, use the average value of the red and blue channels as a new green channel value for keying;

[0105] S335: An image composed of the matted red channel value, the new matted green channel value, and the matted blue channel value corresponding to each pixel point is used as the matted image after the overflow is removed.

[0106] In some embodiments, the initial matte image I may be determined k The keying red channel value R, keying green channel value G and keying blue channel value B in the RGB channels corresponding to each pixel in the image. Then, for each pixel, the corresponding red and blue channel average values are calculated, and the calculation formula is as follows:

[0107]

[0108] Among them, avg is the average value of the red and blue channels; B is the keying blue channel value; R is the keying red channel value.

[0109] Then determine whether the average value avg of the red and blue channels is less than the keying green channel value G. If it is less than the keying green channel value G, the average value avg of the red and blue channels is assigned to the keying green channel to obtain the new keying green channel value G. new :

[0110]

[0111] The keying red channel value R corresponding to each pixel point and the new keying green channel value G new The image composed of the blue channel value B and the keying image is used as the keying image after removing the overflow color, that is, the target keying image I d If the average value of the red and blue channels avg is greater than the keyed green channel value G, the keyed green channel value remains unchanged.

[0112] S104: Perform synthesis processing on the background image to be synthesized, the target matted image and the target channel image to obtain a target synthesized image.

[0113] In some embodiments, the synthesis process of the background image to be synthesized, the target matted image, and the target channel image in S104 may include:

[0114] A linear interpolation operation is performed on the background image to be synthesized, the target matte image and the target channel image to obtain a target synthesized image.

[0115] In some embodiments, the background image I to be synthesized may be obtained. b , the acquisition method is the same as the foreground image, etc. According to the following formula, the background image I b , the above-mentioned target matting image I d , the target channel image a5 is subjected to linear interpolation operation to obtain the target composite image I C :

[0116] I C =I d (1-a5)+I b ·a5(9)

[0117] The target matting image I in the above formula (9) d It is obtained based on the target channel image, which is also a fusion of two different matting methods (i.e., the fusion of a0 obtained by chroma key matting and a2 obtained by special processing of edge details). The target channel image a5 determined in this way can handle the difference between the subject and the edge more carefully, which can not only retain the details of the subject object, but also make its edge and background transition naturally. In addition, after the fusion of a0 and a2, anti-overflow processing, filtering processing, and de-overflow processing are performed, so that the target matting image Id It is more realistic and clear, improves the quality and stability of the matte effect, and reduces the complexity and cost of processing. Ultimately, a high-quality target composite image can be obtained at a lower cost, accurately and quickly. C .

[0118] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. For details, please refer to the description of the aforementioned related processing embodiments, and no further description is given here.

[0119] The above describes the present invention. However, it is worth noting that this specific embodiment is only intended to better illustrate the present application and to describe specific embodiments of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0120] The above method is described below in conjunction with a specific embodiment. However, it should be noted that this specific embodiment is only for better illustrating the present application and does not constitute an improper limitation to the present application.

[0121] In a specific implementation, first, the input image is subjected to matting processing to obtain the main channel image a0 of the input image. Then, the first channel image a1 is calculated based on the color channel values corresponding to the foreground image and the pure background image in the input image, and the first channel image a1 is subjected to a power operation to eliminate background noise to obtain the second channel image a2. The main channel image a0 and the second channel image a2 are then summed (or fused) to obtain a comprehensive channel image a3. Then, the comprehensive channel image a3 is subjected to anti-overflow processing to obtain a fourth channel image a4, and the fourth channel image a4 is filtered to obtain a target channel image a5. Then, the target channel image a5 and the foreground image are fused to obtain the initial matting image I k Then the initial matte image I k Perform de-coloring to obtain the target matte image I d Finally, obtain the background image I to be synthesized b , according to the background image to be synthesized I b , target matting image I d Perform linear interpolation operation on the target channel image a5 to obtain the target composite image I C .

[0122] The above method has the following technical effects:

[0123] 1. No need for manual area marking: This invention eliminates the need for manual intervention or marking of matting areas, significantly improving the efficiency and cost-effectiveness of matting. Compared to traditional matting methods that require manual marking, this method saves a lot of manpower and time costs, speeds up the matting process, and improves production efficiency.

[0124] 2. Superior matting effects: This invention provides superior matting effects compared to traditional chroma keying, preserving more details. This means no additional post-production corrections or manual refinement are required, saving costs and resources, and improving matting quality and efficiency.

[0125] 3. Efficient edge processing: The edge processing of this invention is superior to traditional edge feathering methods, reducing additional blur and distortion. No additional edge processing steps are required, saving time and cost, and achieving clearer and more accurate keying results.

[0126] 4. Low performance overhead and real-time stream input support: The present invention has low performance overhead and can run efficiently under real-time stream input.

[0127] In a specific implementation scenario, see Figure 2 As shown, Figure 2 This is a schematic diagram of the image synthesis process provided in the embodiments of this specification. Specifically, the image synthesis process may include the following steps:

[0128] Step 1: Get the input image, which includes a foreground image and a solid-color background image (which can be used as a calibration image in step 2). Perform matting on the input image to obtain the main channel image a0.

[0129] Step 2: Take the foreground red channel value, foreground green channel value and foreground blue channel value in the RGB channel of the foreground image, take the background red channel value, background green channel value and background blue channel value in the RGB channel of the pure color background image, and calculate the first channel image a1 according to the above formula (1)

[0130] Step 3: Perform background noise removal on the first channel image a1 in step 2, that is, perform a power operation on a1 using the above formula (2) to obtain the second channel image a2.

[0131] Step 4: Use the above formula (3) to perform summation processing (or fusion processing or addition operation) on a0 in step 1 and a2 in step 3 to obtain a comprehensive channel image (or called the third channel image) a3.

[0132] Step 5: Preprocess a3 in step 4 (i.e., use the above formula (4) to perform anti-overflow processing on a3 to obtain the fourth channel image a4; use the above formula (5) to filter a4 to obtain the target channel image a5).

[0133] Step 6: Use the above formula (6) to fuse a5 in step 5 and the foreground image in step 1 to obtain the initial matting image I k (or the initial matte result), there are overflow parts in the initial matte image, which requires further de-overflow processing.

[0134] Step 7: Use the above formula (7) and formula (8) to calculate I in step 6. k Perform de-coloring to obtain the target matte image I d .

[0135] Step 8: Use the above formula (9) to combine the background image I to be synthesized b , the target matting image I in step 7 d , the target channel image a5 in step 5, and the target composite image I C .

[0136] See Figure 3 As shown, Figure 3 This is a comparison chart of the image synthesis effects provided in the embodiments of this specification. Figure 3 From left to right are the original image, the composite image obtained by chroma keying, and the target composite image of the present invention. Figure 3 As can be seen from the black circle in the figure, the present invention has a better matting effect and retains more details, such as the hands are smoother, the transition is natural, and the hair is finer.

[0137] Although this specification provides examples such as the following examples or the accompanying Figure 4 The method operation steps or device structure shown, but based on routine or no creative labor, the method or device may include more or fewer operation steps or module units after partial merger. In the steps or structures that do not logically have a necessary causal relationship, the execution order of these steps or the module structure of the device is not limited to the execution order or module structure shown in the embodiments or drawings of this specification. When the method or module structure described is applied to an actual device, server or terminal product, it can be executed sequentially or in parallel according to the method or module structure shown in the embodiments or drawings (for example, a parallel processor or multi-threaded processing environment, or even a distributed processing, server cluster implementation environment). Based on the above-mentioned image synthesis method, the embodiment of this specification also proposes an embodiment of an image synthesis device. As Figure 4 As shown, the device may specifically include the following modules:

[0138] The subject channel image determination module 401 may be used to perform matting processing on an input image to obtain a subject channel image of the input image, wherein the input image includes a foreground image and a solid color background image;

[0139] The comprehensive channel image determination module 402 may be configured to obtain a comprehensive channel image of the input image based on the color channel values corresponding to the foreground image and the pure color background image, and the subject channel image;

[0140] The target matting image determination module 403 may be used to pre-process the integrated channel image to determine the target matting image based on the target channel image and the foreground image obtained by the pre-processing;

[0141] The synthesis module 404 may be used to synthesize the background image to be synthesized, the target matted image and the target channel image to obtain a target synthesized image.

[0142] In some embodiments, the above-mentioned comprehensive channel image determination module 402 can be specifically used to obtain the first channel image of the input image based on the color channel values corresponding to the foreground image and the pure color background image; perform background noise elimination processing on the first channel image to obtain the second channel image; and perform summation processing on the main channel image and the second channel image to obtain the comprehensive channel image of the input image.

[0143] In some embodiments, the color channel values corresponding to the foreground image in the above-mentioned comprehensive channel image determination module 402 may include: a foreground red channel value, a foreground green channel value, and a foreground blue channel value, and the color channel values corresponding to the solid color background image may include: a background red channel value, a background green channel value, and a background blue channel value; accordingly, the above-mentioned comprehensive channel image determination module 402 may also be specifically used to obtain the first channel image of the input image according to the foreground red channel value, the foreground green channel value, the foreground blue channel value, the background red channel value, the background green channel value, and the background blue channel value according to the following formula:

[0144]

[0145] Among them, a1 is the first channel image; F r 、F g 、F b are the foreground red channel value, foreground green channel value, and foreground blue channel value respectively; B r 、B g 、B b They are the background red channel value, background green channel value, and background blue channel value respectively.

[0146] In some embodiments, the target matting image determination module 403 may be specifically configured to perform anti-overflow processing on the integrated channel image, and then perform filtering processing on the anti-overflow processed integrated channel image to obtain a pre-processed target channel image.

[0147] In some embodiments, the target matting image determination module 403 can also be used to fuse the target channel image and the foreground image to obtain an initial matting image; perform de-spill processing on the initial matting image to obtain a matting image after de-spilling, and use the matting image after de-spilling as the target matting image.

[0148] In some embodiments, the target matting image determination module 403 can also be specifically used to determine the matting red channel value, matting green channel value and matting blue channel value corresponding to each pixel point in the initial matting image; calculate the red and blue channel average value corresponding to each pixel point based on the matting red channel value and matting blue channel value corresponding to each pixel point; determine whether the red and blue channel average value is less than the matting green channel value; if so, use the red and blue channel average value as the new matting green channel value; use the image composed of the matting red channel value, the new matting green channel value and the matting blue channel value corresponding to each pixel point as the matting image after removing the overflow color.

[0149] In some embodiments, the synthesis module 404 may be specifically configured to perform a linear interpolation operation on the background image to be synthesized, the target matte image, and the target channel image to obtain a target synthesized image.

[0150] As can be seen from the above, an image synthesis device provided based on the embodiments of this specification can obtain a more natural matte image at a lower cost, accurately and quickly, thereby obtaining a high-quality target synthesized image, effectively improving the usability of the synthesized image.

[0151] An embodiment of this specification also provides an electronic device based on the above-mentioned image synthesis method, including a processor and a memory for storing processor-executable programs / instructions. When the processor is specifically implemented, it can perform the following steps according to the program / instructions: perform matting processing on the input image to obtain a main channel image of the input image, and the input image includes a foreground image and a solid background image; obtain a comprehensive channel image of the input image according to the color channel values corresponding to the foreground image and the solid background image, and the main channel image; pre-process the comprehensive channel image to determine a target matting image based on the target channel image obtained by pre-processing and the foreground image; and perform synthesis processing on the background image to be synthesized, the target matting image and the target channel image to obtain a target synthesized image.

[0152] In order to complete the above instructions more accurately, refer to Figure 5 As shown, an embodiment of this specification also provides another specific electronic device, wherein the electronic device includes a network communication port 501, a processor 502 and a memory 503, and the above structures are connected through internal cables so that each structure can perform specific data interaction.

[0153] The network communication port 501 can be used to perform matting processing on an input image to obtain a main channel image of the input image, wherein the input image includes a foreground image and a solid color background image;

[0154] The processor 502 can be specifically used to obtain a comprehensive channel image of the input image based on the color channel values corresponding to the foreground image and the pure color background image, as well as the main channel image; preprocess the comprehensive channel image to determine a target matte image based on the target channel image obtained by preprocessing and the foreground image; and synthesize the background image to be synthesized, the target matte image and the target channel image to obtain a target synthesized image.

[0155] The memory 503 may be specifically used to store corresponding instruction programs.

[0156] In this embodiment, the network communication port 501 can be a virtual port that is bound to different communication protocols, thereby being capable of sending or receiving different data. For example, the network communication port can be a port responsible for web data communication, a port responsible for FTP data communication, or a port responsible for email data communication. Furthermore, the network communication port can also be a physical communication interface or communication chip. For example, it can be a wireless mobile network communication chip, such as GSM, CDMA, etc.; it can also be a Wi-Fi chip; or it can be a Bluetooth chip.

[0157] In this embodiment, the processor 502 may be implemented in any suitable manner. For example, the processor may take the form of a microprocessor or a processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, an embedded microcontroller, etc. This specification is not intended to limit this.

[0158] In this embodiment, the memory 503 may include multiple levels. In a digital system, anything that can store binary data can be a memory. In an integrated circuit, a circuit with a storage function that has no physical form is also called a memory, such as RAM, FIFO, etc. In a system, a storage device with a physical form is also called a memory, such as a memory stick, TF card, etc.

[0159] An embodiment of the present specification also provides a computer storage medium based on the above-mentioned image synthesis method, wherein the computer storage medium stores a computer program / instruction, which, when executed, implements: performing matting processing on an input image to obtain a main channel image of the input image, wherein the input image includes a foreground image and a solid background image; obtaining a comprehensive channel image of the input image based on the color channel values corresponding to the foreground image and the solid background image, and the main channel image; preprocessing the comprehensive channel image to determine a target matting image based on the target channel image obtained by preprocessing and the foreground image; and synthesizing the background image to be synthesized, the target matting image, and the target channel image to obtain a target synthesized image.

[0160] In this embodiment, the storage medium includes, but is not limited to, random access memory (RAM), read-only memory (ROM), cache, hard disk drive (HDD), or memory card. The memory can be used to store computer program instructions. The network communication unit can be an interface configured in accordance with the standards specified by the communication protocol for network connection communication.

[0161] In this embodiment, the functions and effects specifically implemented by the program instructions stored in the computer storage medium can be explained in comparison with other implementations and will not be repeated here.

[0162] An embodiment of this specification also provides a computer program product based on the above-mentioned image synthesis method, including a non-transitory computer-readable storage medium storing a computer program / instruction, wherein the computer program / instruction is operable to cause a computer to execute the following steps: performing matting processing on an input image to obtain a main channel image of the input image, wherein the input image includes a foreground image and a solid background image; obtaining a comprehensive channel image of the input image based on the color channel values corresponding to the foreground image and the solid background image, and the main channel image; preprocessing the comprehensive channel image to determine a target matting image based on the target channel image obtained by preprocessing and the foreground image; and synthesizing the background image to be synthesized, the target matting image, and the target channel image to obtain a target synthesized image.

[0163] Although this specification provides the method operation steps as described in the embodiments or flow charts, more or fewer operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the device or client product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The term "comprise", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such process, method, product or device. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or device including the elements. Words such as first and second are used to represent names and do not represent any particular order.

[0164] Those skilled in the art will also appreciate that, in addition to implementing the controller in pure computer-readable program code, it is entirely possible to implement the same functionality by logically programming the method steps in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, and the like. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered structures within the hardware component. Alternatively, the devices for implementing various functions can be considered both software modules implementing the method and structures within the hardware component.

[0165] This specification may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, classes, and the like that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0166] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that this specification can be implemented by means of software plus the necessary general hardware platform. Based on this understanding, the technical solution of this specification can essentially be embodied in the form of a software product. This computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of this specification.

[0167] The various embodiments in this specification are described in a progressive manner. References to the common or similar parts of the various embodiments are sufficient. Each embodiment focuses on the differences from the other embodiments. This specification can be used in a variety of general-purpose or specialized computer system environments or configurations. For example, personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable electronic devices, network PCs, minicomputers, mainframe computers, and distributed computing environments that include any of the above systems or devices.

[0168] Although the present specification has been described with reference to the embodiments, persons skilled in the art will appreciate that there are many variations to the present specification without departing from the spirit of the present specification, and it is intended that the appended claims encompass such variations without departing from the spirit of the present specification.

Claims

1. An image synthesis method, characterized in that: include: Performing a matting process on an input image to obtain a main channel image of the input image, wherein the input image includes a foreground image and a solid color background image; Obtaining a comprehensive channel image of the input image according to the color channel values corresponding to the foreground image and the pure color background image, and the main channel image; Preprocessing the integrated channel image to determine a target matting image based on the target channel image and the foreground image obtained by the preprocessing; The background image to be synthesized, the target matted image and the target channel image are synthesized to obtain a target synthesized image.

2. The method according to claim 1, characterized in that The step of obtaining a comprehensive channel image of the input image according to the color channel values corresponding to the foreground image and the pure color background image, and the main channel image, comprises: Obtaining a first channel image of the input image according to the color channel values corresponding to the foreground image and the solid color background image; performing background noise removal processing on the first channel image to obtain a second channel image; The main channel image and the second channel image are summed to obtain a comprehensive channel image of the input image.

3. The method according to claim 2, characterized in that The color channel values corresponding to the foreground image include: a foreground red channel value, a foreground green channel value, and a foreground blue channel value; the color channel values corresponding to the solid color background image include: a background red channel value, a background green channel value, and a background blue channel value; Accordingly, obtaining the first channel image of the input image according to the color channel values corresponding to the foreground image and the pure color background image includes: According to the foreground red channel value, the foreground green channel value, the foreground blue channel value, the background red channel value, the background green channel value, and the background blue channel value, the first channel image of the input image is obtained according to the following formula: Among them, a1 is the first channel image; F r 、F g 、F b are the foreground red channel value, foreground green channel value, and foreground blue channel value respectively; B r 、B g 、B b They are the background red channel value, background green channel value, and background blue channel value respectively.

4. The method according to claim 1, wherein The preprocessing of the integrated channel image comprises: The comprehensive channel image is subjected to an anti-overflow process, and then the comprehensive channel image after the anti-overflow process is subjected to a filtering process to obtain a pre-processed target channel image.

5. The method according to claim 1, wherein Determining a target matting image based on the target channel image and the foreground image obtained by preprocessing includes: Fusing the target channel image and the foreground image to obtain an initial matted image; The initial matte image is subjected to a de-spill process to obtain a matte image after de-spill, and the matte image after de-spill is used as the target matte image.

6. The method according to claim 5, characterized in that The performing de-spill processing on the initial matte image includes: Determine the matte red channel value, matte green channel value, and matte blue channel value corresponding to each pixel point in the initial matte image; Calculate the average value of the red and blue channels corresponding to each pixel point based on the keying red channel value and keying blue channel value corresponding to each pixel point; Determine whether the average value of the red and blue channels is less than the green channel value of the matte; If yes, take the average value of the red and blue channels as the new green channel value; An image composed of the matte red channel value, the new matte green channel value, and the matte blue channel value corresponding to each pixel point is used as the matte image after de-spill.

7. The method according to claim 1, characterized in that The synthesizing process of the background image to be synthesized, the target matted image and the target channel image comprises: A linear interpolation operation is performed on the background image to be synthesized, the target matte image and the target channel image to obtain a target synthesized image.

8. An image synthesis device, characterized in that: include: A subject channel image determination module is used to perform matting processing on an input image to obtain a subject channel image of the input image, wherein the input image includes a foreground image and a solid color background image; a comprehensive channel image determination module, configured to obtain a comprehensive channel image of the input image based on the color channel values corresponding to the foreground image and the pure color background image, and the subject channel image; a target matting image determination module, configured to pre-process the integrated channel image to determine a target matting image based on the target channel image obtained by the pre-processing and the foreground image; The synthesis module is used to synthesize the background image to be synthesized, the target matte image and the target channel image to obtain a target synthesized image.

9. An image synthesis device comprising a memory, a processor, and a computer program / instruction stored in the memory, characterized in that: The processor executes the computer program / instructions to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images