Image synthesis method and device based on chroma keying and video live broadcast system

Abstract

The application relates to an image synthesis method and device based on chroma keying and a video live broadcast system, the method comprising the following steps: extracting and enhancing foreground details of a foreground image to obtain an enhanced detail image of the foreground image; obtaining a semi-transparent channel image obtained by pre-keying the foreground image; synthesizing a background image to be synthesized with the enhanced detail image to obtain an enhanced background image; and synthesizing the foreground image, the semi-transparent channel image and the enhanced background image to obtain a target synthesis image; in the foreground image and background image synthesis process, the details in the foreground image are enhanced, details objects in a part of the foreground image which cannot be processed into an effective foreground image by keying technology are restored in the final synthesis image, the synthesis picture is more realistic, and the display effect of the synthesis image is greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image synthesis method and device based on chroma keying and a video live streaming system. BACKGROUND

[0002] In the image processing technology, the chroma keying (commonly known as green (blue) screen keying) technology has been widely applied in image editing, video live streaming, film and television creation and other fields. The basic principle of the technology is to first calculate the similarity of the color of the pixels in the image to be processed and the key color (the color of the background screen selected by the user), then convert the similarity into the transparency of the image, and then complete the keying according to the transparency.

[0003] Generally, the objects in the image to be processed that are expected to appear as foreground usually have characteristics such as large difference from the color of the curtain, blockiness or relatively thick line shape, so that they are not easily removed by the keying algorithm and are retained in the final imaged image. In actual applications, the image to be processed often contains a person, and the person himself and the objects carried by the person usually contain small objects such as hair, earphone wire and glasses frame. These small objects will contain a considerable degree of background color in imaging due to green reflection, motion blur (with the movement of the person) and out-of-focus blur. In addition, the shadow area of the person in the image due to the on-site lighting and the foreground also has similar performance. These small objects and shadows help to improve the detail richness of the synthesized image. If the synthesized image cannot be restored, the synthesized image lacks realism, and in conventional technologies, these important details are very easy to be converted into semi-transparent or removed, which seriously affects the display effect of the synthesized image. SUMMARY

[0004] Therefore, it is necessary to provide an image synthesis method and device based on chroma keying and a video live streaming system to improve the image synthesis effect in view of at least one of the above technical defects.

[0005] An image synthesis method based on chroma keying, comprising:

[0006] extracting and enhancing the foreground details of the foreground image to obtain an enhanced detail map of the foreground image;

[0007] obtaining a semi-transparent channel image obtained by pre-keying the foreground image;

[0008] synthesizing the background image to be synthesized with the enhanced detail map to obtain an enhanced background image;

[0009] synthesizing the foreground image, the semi-transparent channel image and the enhanced background image to obtain a target synthesized image.

[0010] In one embodiment, the foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, comprising:

[0011] The foreground image is preprocessed.

[0012] The preprocessed foreground image is subjected to background region centering processing to obtain a foreground detail map.

[0013] The foreground detail map is subjected to nonlinear transformation to obtain an enhanced detail map.

[0014] In one embodiment, the foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, comprising:

[0015] The foreground image is converted into a grayscale image, and the grayscale image is subjected to normalization processing.

[0016] In one embodiment, the preprocessed foreground image is subjected to background region centering processing to obtain a foreground detail map, comprising:

[0017] The numerical distribution of the preprocessed foreground image on the background region is obtained.

[0018] The representative numerical value of the background region is calculated according to the numerical distribution.

[0019] The foreground detail map in which the background region takes the value of "1" is calculated according to the representative numerical value.

[0020] In one embodiment, the foreground detail map is subjected to nonlinear transformation to obtain an enhanced detail map, comprising:

[0021] The detail information of the foreground detail map is subjected to transformation processing by using a nonlinear function to obtain an enhanced detail map; wherein the detail information of the foreground detail map is used to describe the difference between the numerical value of the background region and the numerical value of the non-background region.

[0022] In one embodiment, the nonlinear function is an S-shaped curve function; wherein the variable value of the S-shaped curve function satisfies a monotonically increasing range of [0, 2], and the function value is 0 when the variable is 0, and the function value is 1 when the variable is 1.

[0023] In one embodiment, the background image to be synthesized is synthesized with the enhanced detail map to obtain an enhanced background image, comprising:

[0024] The enhanced detail map is synthesized into the background image by a multiplication synthesis method to obtain an enhanced background image.

[0025] In one embodiment, the foreground image, the semi-transparent channel image, and the enhanced background image are synthesized to obtain a target synthesis image, comprising:

[0026] The foreground image, the semi-transparent channel image and the enhanced background are synthesized by using an alpha synthesis mode to obtain a target synthesized image with enhanced details.

[0027] In an embodiment, the foreground image is a video image of a host in front of a background curtain of a set color.

[0028] Before the foreground details of the foreground image are extracted and enhanced, the method further comprises:

[0029] A live connection is established with each host terminal participating in the live streaming.

[0030] Video image data uploaded by at least two host terminals is received, wherein the video image data includes a foreground image of a host collected by a host terminal and a semi-transparent channel image obtained by pre-matting the foreground image.

[0031] In an embodiment, the image synthesis method based on chroma matting further comprises:

[0032] The video image data uploaded by other host terminals is pushed to any host terminal, so that the host terminal performs mixed picture synthesis based on the foreground image of the host collected by the host terminal and the semi-transparent channel image obtained by pre-matting the foreground image, the foreground image of other hosts and the semi-transparent channel image obtained by pre-matting the foreground image, and a background image.

[0033] In an embodiment, after the foreground image, the semi-transparent channel image and the enhanced background image are synthesized to obtain a target synthesized image, the method further comprises:

[0034] A live video image and an interactive special effect generated by an interactive host are obtained based on the target synthesized image.

[0035] The live video image and the interactive special effect are encoded into a live video stream and pushed to a viewer terminal.

[0036] An image synthesis device based on chroma matting comprises:

[0037] A detail enhancement processing module is configured to extract and enhance foreground details of a foreground image to obtain an enhanced detail image of the foreground image.

[0038] A transparent channel acquisition module is configured to acquire a semi-transparent channel image obtained by pre-matting the foreground image.

[0039] An enhanced background generation module is configured to synthesize a background image to be synthesized with the enhanced detail image to obtain an enhanced background image.

[0040] An image synthesis module is configured to synthesize the foreground image, the semi-transparent channel image and the enhanced background image to obtain a target synthesized image.

[0041] A video live streaming system includes at least one anchor client and a live streaming server. The anchor client is configured to acquire the foreground image and the corresponding semi-transparent channel image and upload them to the live streaming server.

[0042] The live streaming server synthesizes the foreground image and the background image using the chroma keying-based image synthesis method.

[0043] An electronic device includes:

[0044] One or more processors;

[0045] A memory;

[0046] One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the chroma keying-based image synthesis method.

[0047] A computer-readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to execute the chroma keying-based image synthesis method.

[0048] The chroma keying-based image synthesis method, device and video live streaming system extract and enhance the foreground details of the foreground image to obtain an enhanced detail image, synthesize the background image to be synthesized and the enhanced detail image to obtain an enhanced background image, and finally synthesize the foreground image, the semi-transparent channel image and the enhanced background image to obtain a target synthesized image. In the process of synthesizing the foreground image and the background image, the details in the foreground image are enhanced, so that the details of the objects that cannot be processed into effective foreground image parts by the matting technology can be restored in the synthesized image, thereby making the synthesized image more realistic and greatly improving the display effect of the synthesized image. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 is a synthesized image effect diagram of a conventional technology of an example;

[0050] Figure 2 is a synthesized image effect diagram of another example of the conventional technology;

[0051] Figure 3 is a flowchart of the chroma keying-based image synthesis method of an embodiment.

[0052] Figure 4 is an example of a synthetic image effect diagram based on the technical solutions of the present application;

[0053] Figure 5 is an example of a color key-based image synthesis processing flow diagram;

[0054] Figure 6 is an example of a network live streaming system topology diagram;

[0055] Figure 7 is an example of a color key-based image synthesis device structure diagram of an embodiment;

[0056] Figure 8 is an example of a video live streaming system structure diagram;

[0057] Figure 9 is a block diagram of an electronic device. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0059] In the embodiments of the present application, the words "first", "second", etc. are used to distinguish the same or similar items with basically the same function, "at least one" means one or more, and "multiple" means two or more, for example, multiple objects refer to two or more objects. The words "include" or "contain" and similar words mean that the information appearing before "include" or "contain" covers the information listed after "include" or "contain" and its equivalents, and does not exclude other information. In the embodiments of the present application, "and / or" means that there can be three relationships, and the character " / " generally means that the objects before and after are in an "or" relationship.

[0060] In the color key-based image synthesis technology, the green (blue) screen keying technology is generally used, and then the foreground image of the picture is synthesized with the foreground image generated semi-transparent channel image and the background image to obtain the final synthesis image effect.

[0061] For example, the foreground image F color ∈R H×W×C , the semi-transparent channel image M f ∈R H×W , the background image B∈R H×W×C , and the synthesis image I direct =M f *Fcolor +(1-M f )*B,I direct ∈R H×W×C Among them, F color R represents the foreground image, and M represents the standard matrix. f B represents the semi-transparent channel image, I represents the background image, and I represents the background image. direct This indicates direct image synthesis, where W and H represent the width and height of the image, respectively, C represents the number of channels in the input image (typically 3), and M... f The value is between 0 and 1. The higher the value, the lower the similarity between the corresponding pixel in the input foreground image and the key color, and the lower the transparency.

[0062] refer to Figure 1 As shown, Figure 1 This is an example of a composite image produced using a common technique, such as... Figure 1 In the example shown by the arrow in the image, after the foreground image is keyed out and directly composited into the background image, the fuzz in the original image cannot be effectively displayed. (Reference) Figure 2 , Figure 2 Here is another example of a composite image produced using conventional techniques, such as... Figure 2 In this case, after the foreground image is chroma keyed and directly composited into the background image, as shown by the arrow in the figure, the shadow of the person in the original image cannot be effectively displayed.

[0063] Accordingly, in order to restore the detailed information in the foreground image, this application provides an image synthesis method based on chroma keying, such as... Figure 3 As shown, Figure 3 This is a flowchart of an image compositing method based on chroma keying, including the following steps:

[0064] S10, perform foreground detail extraction and enhancement processing on the foreground image to obtain an enhanced detail map of the foreground image.

[0065] In this step, by analyzing the foreground image F... color The processing thus obtains the foreground image F. color Enhanced detail image F with various details detail ; where the foreground image F color ∈R H×W×C Semi-transparent channel M f ∈R H×W Among them, F detail This indicates an image with enhanced details.

[0066] In one embodiment, the scheme for obtaining an enhanced detail map of the foreground image in step S10 may include the following steps:

[0067] S110, preprocess the foreground image.

[0068] As an example, in the foreground image F color During preprocessing, the foreground image F is first... color Convert to grayscale image F gray ∈R H×W Then, the grayscale image is normalized to obtain F′. gray ∈R H×W Specifically, let's assume F color It is an RGB image, and its grayscale representation is F. gray =F color ×v,v=[0.299,0.587,0.114] T Normalized foreground image F′ gray =[F gray -min(F gray )] / [min(F gray )-max(F gray )], where min(F gray ) and max(F gray ) represent grayscale images F gray The minimum and maximum values ​​of pixels; where F gray Indicated by F color The converted grayscale image; v = [0.299, 0.587, 0.114] T This represents the standard operation process for converting an RGB image to a grayscale image, specifically: grayscale channel = 0.299 * red + 0.587 * green + 0.114 * blue.

[0069] The technical solution of the above embodiments can convert the foreground image to grayscale mode for processing through grayscale and normalization processing, thereby reducing the amount of image data to be processed in the background area and improving processing efficiency.

[0070] S120, perform background region centering processing on the preprocessed foreground image to obtain a foreground detail image.

[0071] In this step, by performing background region centering processing on the foreground image, the values ​​of the non-foreground regions in the foreground image are brought close to a set value (usually "1"). The detail information of the foreground detail map can be used to describe the difference between the values ​​of the background region and the non-background region, which facilitates the enhancement of the background part, so that the detail information of the background region can be enhanced when synthesizing the background image.

[0072] As an example, when performing background region centering processing on a foreground image, the following may be included:

[0073] (1) Obtain the numerical distribution of the preprocessed foreground image in the background region;

[0074] (2) calculating a representative value of the background region according to the numerical distribution;

[0075] (3) calculating a foreground detail map with the background region centered at "1" according to the representative value.

[0076] Specifically, the normalized foreground image F′ gray is analyzed to obtain the numerical distribution of the background region (M f <τ) and a representative value f′ BG of the background region is calculated. const F gray = F′ BG + (1-f′ const ), F H×W ∈R BG ; wherein τ represents an opaque threshold value, which can be assigned values such as 0.01, 0.1, 0.2, etc., and f′

[0077] The technical solution of the above embodiment, through the background region centered processing of the background region, the value of the non-foreground region in the foreground image is close to 1, i.e. the value of the background region is centered close to 1, which weakens the influence of the non-foreground region in the foreground image when the foreground image detail information is multiplied with the background image to synthesize the background picture, and ensures that the non-foreground region in the synthesized foreground image remains unchanged, thereby ensuring the synthesis effect.

[0078] S130, performing nonlinear transformation on the foreground detail map to obtain an enhanced detail map.

[0079] In this step, through nonlinear transformation processing, the foreground detail map is nonlinearly transformed according to a specific function relationship, the features of the foreground detail map are transformed into a new image feature, and linear classification is performed using the new image feature, so that the difference between the background region and the non-background region of the foreground detail map is greater, the image difference between the two regions is presented, and an enhanced detail map is obtained.

[0080] As an embodiment, when performing nonlinear transformation on the foreground detail map, a nonlinear function can be used to transform the detail information of the foreground detail map to obtain an enhanced detail map; wherein the detail information of the foreground detail map is within a set distance between the background region value and the non-background region value; preferably, the nonlinear function can be any S-shaped curve function; wherein the variable value of the S-shaped curve function satisfies a monotonically increasing range of [0, 2], and the value is 0 when the variable is 0, and the value is 1 when the variable is 1.

[0081] Specifically, the detail information of the foreground image is nonlinearly transformed Fdetail ∈R H×W At this time, any S-shaped curve y = func(x) can be adopted, func represents function function, such as Sigmoid, Sin, Cos and its variants, etc., and the S-shaped curve function meets the following characteristics:

[0082] (a) When 0≤x≤2, it is monotonically increasing;

[0083] (b) func(0) = 0, func(1) = 1;

[0084] For the selected function type, the specific calculation process expression is as follows: F detail = func(F const ), F detail represents the foreground image after nonlinear transformation.

[0085] The technical solution of the above embodiment can enlarge the distance between the background area value (the value is basically "1") and the non-background area value of the foreground detail image F const , so as to better enlarge the subtle picture difference.

[0086] S20, obtaining the semi-transparent channel image obtained by pre-masking the foreground image.

[0087] In this step, the semi-transparent channel image M color ∈R H×W×C can be obtained through green (blue) screen masking technology, and the pre-masking process can be completed remotely or locally. f ∈R H×W

[0088] S30, synthesizing the background image to be synthesized with the enhanced detail image to obtain an enhanced background image.

[0089] As an embodiment, when synthesizing the background image to be synthesized with the enhanced detail image, the multiplication synthesis method can be used for synthesis; specifically, for the enhanced background image B'∈R H×W×C , the foreground detail image F detail is synthesized into the background image B through multiplication synthesis; wherein the calculation formula can be as follows:

[0090] B' = Clip(F detail ×B, b min , b max ),

[0091] In the formula, Clip is a clipping function, which sets the value less than b min to b min ​greater than b max the value of b max B' represents the enhanced background image.

[0092] It should be noted that b min and b max are generally the upper and lower limits of the pixel value of the image, b min is generally 0, and b max is generally determined according to the data type, such as 1 for a floating point number and 255 for an 8-bit unsigned integer.

[0093] S40, the foreground image, the semi-transparent channel image and the enhanced background image are synthesized to obtain a target synthesized image.

[0094] As an embodiment, when the foreground image, the semi-transparent channel image and the enhanced background image are synthesized, a standard alpha synthesis method can be used for synthesis, that is, the transparency and semi-transparency value of a semi-transparent channel image with an alpha channel is operated with the pixel value of the corresponding position of the foreground image and the enhanced background image.

[0095] Specifically, the input foreground image F color is first synthesized with the enhanced background image B' to obtain a final synthesized image I, and the synthesis formula can be as follows:

[0096] I = M f * F color + (1-M f )*B'

[0097] In the formula, B' represents the enhanced background image, I represents the synthesized image, M f represents the semi-transparent channel image.

[0098] As the technical solution of the above embodiment, the image synthesis method based on chroma keying of the present application is suitable for application scenarios such as green (blue) screen keying live broadcast in live broadcast services, and reference Figure 4 is shown, Figure 4 is an example of a synthesized image effect diagram based on the technical solution of the present application, which is a scheme for synthesizing related images in the examples of Figure 1 and 2 , and it can be seen from Figure 4 that when the green (blue) screen keying is performed and the synthesized image is synthesized with the background image, the technical solution of the present application can avoid the details of the foreground image from being removed, such as Figure 4 which can restore the hair details and shadow details of the host, wherein the left image corresponds to the example of Figure 1 , and the right image corresponds to Figure 2As an example, through the technical solution of the present application, most of the detail information of the foreground image can be restored, so that the synthesized picture is more realistic, and the display effect of the synthesized image is greatly improved.

[0099] In order to make the technical solution of the above-mentioned embodiment clearer, the following will be combined with the drawings to illustrate the image synthesis process based on chroma matting, and reference will be made to Figure 5 as shown, Figure 5 is an example of an image synthesis processing flow diagram based on chroma matting. As shown in the figure, when the background image needs to be synthesized with the foreground image, the foreground image is first enhanced for detail processing, including foreground preprocessing, background region centering, and nonlinear transformation processing; the enhanced detail image obtained after processing is multiplied with the background image to obtain an enhanced background image, and then the foreground image, the semi-transparent channel image obtained by pre-matting, and the enhanced background image are synthesized by alpha synthesis to obtain a synthesized image; as shown in the figure, the detail information in the foreground image is enhanced in the processing process, so that the synthesized image can restore the detail information of the foreground image, and the synthesized image is more realistic. This technical solution is particularly suitable for the synthesis stage of the foreground image and the background image, and can restore (enhance) details such as hair, earphone wire, and shadow in the final synthesized image.

[0100] The image synthesis method based on chroma matting of the above-mentioned embodiment can be applied to various scenes requiring chroma matting and image synthesis. As an example, the image synthesis method based on chroma matting provided by the present application is particularly suitable for a network live streaming system. By performing portrait image matting processing based on chroma matting on each host participating in live streaming in the live streaming process, and then synthesizing with the background image, a mixed picture synthesis process of multiple live streaming hosts in the same background can be formed.

[0101] Therefore, the following will provide an embodiment of the present application applied to a mixed picture synthesis in a live streaming scene, and reference will be made to Figure 6 as shown, Figure 6 is a topology diagram of a network live streaming system. Based on the diagram, the network live streaming system includes a host end (i.e., a host client), a live streaming server, and a viewer end. Multiple host ends (host end 1 and host end 2 shown in the figure, corresponding to host 1 and host 2) establish a live streaming connection with the live streaming server, establish live streaming through the live streaming server, and use the background image to perform matting on the foreground image uploaded by each host end to obtain the portrait image of each host, and perform live streaming interaction on the background image. The live streaming interaction of the live streaming hosts generates a live streaming video stream and pushes it to the viewer end.

[0102] In one embodiment, the foreground image can be a video image taken by the anchor end of the co-host anchor in front of a background curtain of a set color; generally a green screen background, green screen cutout is performed; accordingly, the technical solution of the present embodiment, in the early stage of co-host live broadcast, first establishes a live broadcast connection between the live broadcast server and each anchor end participating in the co-host live broadcast; specifically, the anchor end 1 and the anchor end 2 co-host through the live broadcast server, and select a virtual background material as the background image of the co-host (the type of background image includes static pictures, dynamic videos, etc.), the anchor can select the background image according to the demand in this process, the anchor can select the favorite background image on the interface panel, and can perform real-time preview on the client interface, the right image in the figure is a single-color background real-time image, and the left image is a synthesized preview view replacing the background image, the right side is a function area, and the anchor can use the functions provided by the panel.

[0103] After the co-host live broadcast is established, the live broadcast server starts to receive video image data uploaded by at least two anchor ends, such as the video image data of the anchor end 1 and the anchor end 2; wherein the video image data can include the foreground image of the anchor collected by the anchor end and the semi-transparent channel image obtained by pre-cutting the foreground image; specifically, the anchor end can collect the video image of the anchor, extract the behavior data (such as arm movement, gesture, whole contour of the body, etc.) to obtain the semi-transparent channel image by pre-cutting the foreground image; then on the live broadcast server, the image synthesis method based on chroma cutout in the foregoing embodiment can be used to perform foreground detail extraction and enhancement processing on the foreground image, then the background image to be synthesized and the enhanced detail image are synthesized to obtain an enhanced background image, and finally the foreground image, the semi-transparent channel image and the enhanced background image are synthesized to obtain a target synthesis image, such as mixing and synthesizing the portrait images of anchor 1 and anchor 2 of the anchor end 1 and the anchor end 2 on the background image to form a virtual same stage scene, thereby realizing the virtual same stage function.

[0104] Further, after the target synthesis image is synthesized, the co-host live video image generated by the co-host interaction and the interactive special effect can also be obtained based on the target synthesis image; the co-host live video image and the interactive special effect are encoded into a co-host live video stream and pushed to the audience end; for example, after the virtual same stage scene is mixed and synthesized, the joint virtual gift special effect generated by the co-host interaction in the virtual same stage scene is placed on the virtual same stage scene for rendering, and a co-host live video stream is generated and pushed to the CDN network, and distributed to each audience end for viewing.

[0105] In one embodiment, the image synthesis method based on chroma keying of the embodiments of the present application, the live server can also send other anchor end uploaded video image data to any anchor end, so that each anchor end based on the collected foreground image of the anchor and the semi-transparent channel image obtained by the pre-keying thereof, and the foreground image of other anchors and the semi-transparent channel image obtained by the pre-keying thereof, and the background image are mixed and synthesized.

[0106] For example, when the anchor end 1 initiates the mic connection to the anchor end 2, the URL of the selected background image is added to the video image data; after the live server receives the video data, it is forwarded to the anchor end 2, and the anchor end 2 can perform the mixed picture synthesis process locally, thereby realizing the effect of quickly generating a preview mixed picture image locally.

[0107] The embodiments of the image synthesis device based on chroma keying are described below.

[0108] Reference Figure 7 As shown in the figure, Figure 7 is a structural schematic diagram of the image synthesis device based on chroma keying of one embodiment, which comprises:

[0109] The detail enhancement processing module 10 is configured to extract and enhance the foreground details of the foreground image to obtain an enhanced detail image of the foreground image.

[0110] The transparent channel acquisition module 20 is configured to acquire a semi-transparent channel image obtained by pre-keying the foreground image.

[0111] The enhanced background generation module 30 is configured to synthesize the background image to be synthesized with the enhanced detail image to obtain an enhanced background image.

[0112] The image synthesis module 40 is configured to synthesize the foreground image, the semi-transparent channel image, and the enhanced background image to obtain a target synthesis image.

[0113] The image synthesis device based on chroma keying of the embodiments of the present application can perform the image synthesis method based on chroma keying provided by the embodiments of the present application, and the implementation principles are similar. The actions performed by each module in the image synthesis device based on chroma keying in each embodiment of the present application are corresponding to the steps in the image synthesis method based on chroma keying in each embodiment of the present application. For the detailed function description of each module of the image synthesis device based on chroma keying, please refer to the description of the corresponding image synthesis method based on chroma keying shown in the foregoing, which will not be described here.

[0114] The embodiments of the video live system are described below.

[0115] The video live broadcast system of the present application comprises at least one anchor client and a live broadcast server; the anchor client is used to acquire a foreground image and a corresponding semi-transparent channel image and upload them to the live broadcast server; the live broadcast server uses the image synthesis method based on chroma keying of the above-mentioned embodiments to synthesize the keying image of the foreground image and a background image.

[0116] In order to more clearly embody the effect of the image synthesis technical solution based on chroma keying of the present application, the following describes an example in a video live broadcast system. As described in the above embodiments, referring to Figure 8 , Figure 8 is a structural schematic diagram of an example of a video live broadcast system, comprising an anchor client and a live broadcast server, and the following describes an example in which anchor client 1 and anchor client 2 are taken as examples in a live broadcast in which microphones are connected.

[0117] In the above example, the anchor client can acquire a foreground image of an anchor through a camera, output a semi-transparent channel image by pre-keying the green screen background of the foreground image, and then compress and send the semi-transparent channel image to the live broadcast server using an encoding algorithm such as H264 or HEVC. The live broadcast server acquires a background image, obtains an enhanced background image using the above-mentioned image synthesis method based on chroma keying, and then synthesizes the foreground image and the semi-transparent channel image and the enhanced background image using an alpha synthesis method to obtain a synthesized image, thereby synthesizing anchor 1 and anchor 2 of anchor client 1 and anchor client 2 onto one background image, so as to realize virtual co-spectacle mixed drawing. Specifically, as shown in Figure 8 , the anchor end can be composed of a mobile phone and a PC, or composed of a camera and a portable computer, and of course, the devices can be selected and combined according to the needs in actual applications. Each anchor end is connected to the live broadcast server through a network, the live broadcast server performs a live broadcast in which microphones are connected, and then pushes the live broadcast video stream to each viewer end for viewing, thereby realizing the virtual co-spectacle mixed drawing function of the live broadcast in which microphones are connected. The viewer end can be a PDA, a tablet computer, a PC, or a portable computer, and the like.

[0118] As an example, the anchor end can comprise a broadcast tool and a client, wherein the broadcast tool integrates a virtual camera and has various functions such as beautifying and keying, and the client is a software client based on voice and video live broadcast. The live broadcast can provide various types (entertainment / friendship / battle / gaming / education, etc.) of live broadcast templates, and in the present example, the live broadcast in which microphones are connected can be realized through an entertainment template for show live broadcast.

[0119] For the anchor end:

[0120] (1) The starting tool is responsible for camera acquisition, and performs beautification, skin smoothing, slimming, etc. on the video image of the host, then performs pre-matting processing on the foreground image based on the chroma keying technology, extracts behavior data (such as arm movement, gestures, the whole contour of the body, etc.) to obtain a semi-transparent channel image; the client uploads the foreground image, semi-transparent channel image and image related information (AI key point information transmitted using SEI information, key point information such as face, gesture and head, starting effect information, play gift information, and other information, etc.) to the live server.

[0121] (2) The host end initiates a mic connection, for example, the host end 1 and the host end 2 perform mic through the live server, and select a background image (static picture, dynamic video, etc.). The background image is sent to the live server in the form of a URL in SEI information.

[0122] (3) The video image of the host is beautified and virtual effect processing function is performed; for example, skin smoothing, face slimming, face changing, wearing sunglasses, etc.

[0123] For the live server:

[0124] (1) Forward the video stream information of the host end, for example, forward the video stream information of the host end 1 to the host end 2, so that the host end 2, so that the host end 1 can perform mix picture synthesis locally.

[0125] (2) Push the video stream generated by the mic live to the audience end, for example, mix the portrait image and the background image in the foreground image uploaded by the host end 1 and the host end 2, encode the mixed image and push it to the CDN distribution network, and send it to each audience end.

[0126] (3) Re-render the virtual effect content on the host end; for example, the virtual gift effects displayed by the host end 1 and the host end 2 during the mic process are calculated according to the AI key point information to obtain the rendering position of the virtual effect content, and then re-rendered on the live server; in addition, the joint virtual gift effect generated by the mic interaction is synchronously rendered, and a mic video stream is pushed to the audience end.

[0127] For the audience end:

[0128] (1) Receive the mic video stream pushed by the live server, and play it on the audience end interface.

[0129] (2) Perform mic during the mic live process, first establish a connection with the live server, download the video image data of the host end 1 and the host end 2, preview locally, and upload the audio stream of the mic user to the live server, which is added to the mic video stream and pushed to the audience end.

[0130] Embodiments of an electronic device and a computer-readable storage medium are described below.

[0131] The present application provides a technical solution of an electronic device to realize the functions related to the chroma-based image matting method.

[0132] In one embodiment, the present application provides an electronic device, which comprises:

[0133] one or more processors;

[0134] a memory;

[0135] one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured for the chroma-based image matting method of any embodiment.

[0136] As Figure 9 shown, Figure 9 is a block diagram of an example electronic device. The electronic device can be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, and the like. Referring to Figure 9 , the apparatus 900 can include one or more of the following components: a processing component 902, a memory 904, a power supply component 906, a multimedia component 908, an audio component 909, an input / output (I / O) interface 912, a sensor component 914, and a communication component 916.

[0137] The processing component 902 usually controls the overall operations of the apparatus 900, such as operations associated with displaying, making phone calls, data communications, camera operations, and recording operations.

[0138] The memory 904 is configured to store various types of data to support the operations of the apparatus 900. As static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0139] The power supply component 906 provides power for the various components of the apparatus 900.

[0140] The multimedia component 909 includes a screen that provides an output interface between the apparatus 900 and the user. In some embodiments, the screen can include a liquid crystal display (LCD) and a touch panel (TP). In some embodiments, the multimedia component 908 includes a front camera and / or a rear camera.

[0141] The audio component 909 is configured to output and / or input audio signals.

[0142] The I / O interface 912 provides an interface between the processing component 902 and peripheral interface modules, which can be a keyboard, a click wheel, a button, and the like. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0143] The sensor component 914 includes one or more sensors for providing status assessments of various aspects of the device 900. The sensor component 914 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact.

[0144] The communication component 916 is configured to facilitate wired or wireless communication between the device 900 and other devices. The device 900 can access a wireless network based on a communication standard, such as WiFi, an operator network (such as 2G, 3G, 4G, or 5G), or a combination thereof.

[0145] The present application provides a computer readable storage medium to realize the functions of the image synthesis method based on chroma matting. The computer readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the image synthesis method based on chroma matting.

[0146] In the exemplary embodiments, the computer readable storage medium can be a non-transitory computer readable storage medium including instructions, such as a memory including instructions, for example, the non-transitory computer readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0147] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be noted that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A method for image compositing based on chroma keying, characterized by, The method comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps:

2. The chroma-key based image compositing method of claim 1, wherein, The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps:

3. The chroma-key based image compositing method of claim 2, wherein, The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps:

4. The chroma-key based image compositing method of claim 3, wherein, The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps:

5. The chroma-key based image compositing method of claim 1, wherein, The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps:

6. The chroma-key based image compositing method of claim 1, wherein, The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps:

7. The chroma-key based image compositing method according to any of claims 1-6, wherein, The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps:

8. The chroma-key based image compositing method of claim 7, wherein, The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps:

9. The chroma-key based image compositing method of claim 8, wherein, The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps:

10. An image compositing apparatus based on chroma keying, characterized by comprising: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map of the foreground image, which comprises the following steps: The foreground image is preprocessed, and the foreground details of the foreground image are extracted and enhanced to obtain an enhanced detail map The detail enhancement processing module is configured to extract and enhance foreground details of the foreground image to obtain an enhanced detail map of the foreground image, including: preprocessing the foreground image; obtaining a numerical distribution of the preprocessed foreground image on the background region; calculating a representative value of the background region according to the numerical distribution; calculating a foreground detail map with the representative value of the background region being "1" according to the representative value; performing a non-linear transformation on the foreground detail map to obtain an enhanced detail map; The transparent channel acquisition module is configured to obtain a semi-transparent channel image obtained by pre-masking the foreground image; The enhanced background generation module is configured to synthesize the foreground image and the enhanced detail map to obtain an enhanced background image; The image synthesis module is configured to synthesize the foreground image, the semi-transparent channel image, and the enhanced background image to obtain a target synthesized image.

11. A video live streaming system, characterized by, The electronic device comprises: one or more processors; a memory; one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the chroma-based image synthesis method according to any one of claims 1-9.

12. An electronic device, comprising: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to execute the chroma-based image synthesis method according to any one of claims 1-9. ​ ​ ​ 13. A computer-readable storage medium, characterized in that, ​

Images