Image matting model training method and device, image matting method and device, equipment and medium

Abstract

The invention discloses an image matting model training method, an image matting method and device, equipment and a medium. The image matting model training method comprises the following steps: acquiring a data set for training; constructing a background probability model formula; marking each pixel point of all the image frames by using the sum obtained by the background probability model formula to obtain a probability thermodynamic diagram of whether the pixel is the background, and obtaining a binary portrait foreground label according to the thermodynamic diagram; using the chrominance component of the data set for training and the binarized portrait foreground label to train a color threshold model of the multivariate nonlinear regression model; brightness and chrominance components of image blocks are randomly acquired in a bbox range, and a nonlinear regression model is trained to represent a calculation model for fitting an edge part of a foreground portrait into an expected fuzzy weight under blue or green backgrounds under different conditions. According to the method, the foreground imaging effect can be optimized during superposition operation, and the computing power of an image matting algorithm is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and more specifically, to a matting model training method, an image matting method, a device, equipment and a medium. Background Art

[0002] Micro-class is a new teaching mode, which uses short videos as the main carrier, usually about 5-10 minutes long, and the longest should not exceed 20 minutes. In the production of micro-class, some backgrounds are usually added according to the needs of the course, such as adding PPT teaching plans, or adding pictures related to the teaching topic. Therefore, when making micro-classes, blue screen or green screen keying technology is often used to separate teachers or specific objects from the original shooting background and synthesize them into a new background.

[0003] Traditional micro-class teaching video recording applications based on blue screen or green screen keying generally have high requirements for shooting equipment, venue, lighting, etc. For example, the lighting needs to be uniform and the pixel number of the shooting equipment must be high enough. Among them, some traditional matting applications are generally based on pixel data in the red, green, blue (RGB) color space, have low adaptability to scenes, and are easily affected by the color difference of the background curtain, lighting shooting and other conditions, which further increases the requirements for the recording scene layout; and for matting applications with certain adaptability, they generally need to be run on a platform with a certain computing power, such as a server with parallel computing functions or a mobile terminal with good performance, such as a mobile phone and other devices. The common point of these solutions is that they cannot form a good match with professional video recording equipment. For example, professional recording equipment is equipped with a server with parallel computing functions, and the cost of this configuration is often high; and the mobile terminal with good performance has a certain gap between its own shooting equipment and professional video recording equipment; in addition, the general micro-class recording scene also requires a second video stream for displaying PPT and other teaching plans as the background of the final composite video. The solution of using mobile terminals for matting often requires secondary processing in the later stage to finally output a composite video stream with the teacher's explanation foreground and the teaching plan background, which increases the time and manpower required for micro-class recording applications. Summary of the invention

[0004] In order to solve the problems in the prior art that the application of matting recording scenes has high requirements for scene layout, or the computing power of the computing equipment, and the high cost required for post-synthesis, the present invention provides a matting model training method, image matting method, device, equipment and medium. The technical solution adopted by the present invention is as follows.

[0005] In a first aspect, the present invention provides a matting model training method, comprising:

[0006] Acquire a training data set, wherein the training data set is composed of image blocks obtained by extracting and segmenting original video data under a preset rule to reduce the proportion of redundant data; the original video data is a video with a blue or green background and containing two situations: no lecturer or human portrait exists and the presence of a lecturer or human portrait exists, and the original video data is original data in a YCrCb color space;

[0007] Construct blue and green background probability model formula P B (X) and P G (X), where X = (v Cb ,v Cr ) is the CrCb component value of each pixel in the image, P is the probability density function satisfying the parameter θ, and the parameter θ is obtained by training the training data set;

[0008] Use the sum obtained by the background probability model formula to label each pixel of all image frames, obtain all image frames used for training and their corresponding labels, that is, the probability heat map of whether the pixel is the background, and obtain the binary portrait foreground mask label based on the heat map;

[0009] Use the chromaticity components in the YCrCb data of the training dataset and the binary portrait foreground mask label to train the multivariate nonlinear regression model color threshold model F in the YCrCb color space BG , the color threshold model F BG It is used to fit the probability heat map of each pixel in the image frame as the background under different backgrounds;

[0010] Using the brightness and chromaticity components of the YCrCb data of the image block randomly obtained within the bbox range, a nonlinear regression model is trained in the YCrCb color space to characterize the edge of the foreground portrait under the blue or green background in different situations. The calculation model F is fitted to the desired blur weight. BLUR , wherein the bbox range is the maximum bounding box of the portrait mask of the image frame containing the portrait in the training data set.

[0011] In one embodiment, it further includes:

[0012] Use the image frames in the original video data to sample image blocks and finally construct a training data set;

[0013] The process of sampling image blocks using image frames in the original video data to finally construct a training data set includes:

[0014] Randomly extract N frames of images from the original video data and convert these images into RGB color space;

[0015] Use the foreground segmentation algorithm to detect and extract N frames of images in the RGB color space to classify images containing portraits and images not containing portraits, and record the portrait mask of the images containing portraits;

[0016] For blue or green background image frames that do not contain portraits, randomly extract a certain number of image blocks of random sizes, and record the positions and length and width data of these image blocks;

[0017] For image frames containing portraits, a certain number of image blocks of random sizes are randomly extracted from the image part outside the mask, and the position, length and width data of these image blocks are recorded. For the image part within the mask, the maximum bounding box bbox of the mask is calculated, and a certain number of image blocks of fixed sizes are randomly extracted within the range of the bbox, and the position, length and width data of these image blocks are recorded.

[0018] In one embodiment, the original video data includes imaging pictures under a variety of lighting conditions, and the lighting conditions include: lighting under different lighting methods, lighting without lighting methods, lighting interfered by external ambient light, and / or lighting under different screen brightness when the screen is used as the background.

[0019] In one embodiment, the multivariate nonlinear regression model color threshold model F is trained in the YCrCb color space using the chromaticity component in the YCrCb data of the training data set and the binary portrait foreground mask label. BG The process includes:

[0020] Constructing a multivariate nonlinear regression model Color threshold model F BG =sigmoid(M(P(X))) or F BG =softmax(M(P(X))), where P represents the blue and green background probability model formula P B (X) and P G (X), X=(v Cb ,v Cr ) is the CrCb component value of each pixel in the image frame, M is the nonlinear regression model parameter to be trained, sigmoid and softmax are nonlinear functions with normalization ability;

[0021] The chromaticity component in the YCrCb data of the training dataset and the binary portrait foreground mask label are used as training data to train the multivariate nonlinear regression model color threshold model F BG .

[0022] In one embodiment, the brightness and chromaticity components in the YCrCb data of the image block are randomly obtained within the bbox range, and a nonlinear regression model is trained in the YCrCb color space to characterize the edge of the foreground portrait under the blue or green background in different situations. The calculation model F is fitted to the desired blur weight. BLUR The process includes:

[0023] Construct the nonlinear function with parameter φ and fit the target function F TARGET , and construct the calculation model F of the fuzzy weight of the nonlinear regression model BLUR =sigmoid(M CrCb (X CrCb )+M(X YCrCb )) is used to train the fitting F TARGET , where X CrCb is the CrCb component value of each pixel in the image frame (v Cb ,v Cr ), X YCrCb is the YCrCb component value of each pixel in the image frame (v Y ,v Cb ,v Cr ), M CrCb is the nonlinear regression model parameter to be trained for the CrCb component of each pixel in the image frame of the YCrCb color space, M YCrCb is the correlation model parameter to be trained between the Y component and the CrCb component of each pixel in the image frame of the YCrCb color space;

[0024] The image frame data within the obtained bbox range is used as training data to train the calculation model F of the fuzzy weight of the nonlinear regression model BLUR .

[0025] In a second aspect, the present invention provides an image matting method, comprising:

[0026] Obtain a video stream shot in a non-portrait recording environment, and randomly extract N frames from the video stream in its original YCrCb color space, wherein the video stream is a video stream using a blue or green background;

[0027] The extracted N frames of images are passed through the background probability model formula and the color threshold model F BG Calculate the probability p of each pixel in the current image frame being the background, take all pixels whose p is greater than or equal to the parameter α, and obtain the maximum and minimum CrCb component values ​​of these pixels as the threshold interval for distinguishing the foreground and background in the current recording scene, where the parameter α is the preset threshold parameter for the matting degree, and the background probability model formula is the color of the matting background in the blue background probability model formula PB (X) and green background probability model formula P G (X);

[0028] Acquire the original YCrCb color space video stream of the captured portrait video stream, and segment the foreground and background of the video stream based on the foreground and background distinguishing threshold interval to obtain a rough mask of the portrait foreground;

[0029] According to the rough mask of the portrait foreground, the calculation model F of the fuzzy weight is used BLUR Calculate each pixel in the area covered by the mask to obtain the fuzzy weight of each pixel covered by the mask in the image superposition;

[0030] Cut out the foreground of the portrait in the frame of the current video stream according to the blur weight and superimpose it with the background video stream for superimposition;

[0031] Among them, the blue background probability model formula P B (X), green background probability model formula P G (X), color threshold model F BG And the calculation model of fuzzy weight F BLUR All are obtained through the matting model training method described in any one of claims 1-5.

[0032] In one implementation, the process of cutting out the portrait foreground in the frame of the current video stream according to the blur weight and superimposing it with the background video stream for superimposition includes:

[0033] Perform color space transformation on the foreground area covered by the foreground mask fuzzy weight and calculate the grayscale value v of each pixel in this area. g , and convert the grayscale color value back to the corresponding YCrCb component v GYCrCb , according to the formula: v DYCrCb =f blur *v SYCrCb +(1-f blur )*v GYCrCb Calculate the YCrCb component v of the foreground pixel used for superposition DYCrCb , get the foreground pixel value, where f blur is the fuzzy weight, v SYCrCb YCrCb components of the original scene pixel;

[0034] The background video stream for superimposition is calculated according to the fuzzy weight, the area to be superimposed is set to 0, and the foreground pixel value is filled in according to the corresponding position to complete the matting and superimposition operations.

[0035] In one implementation, the process of segmenting the foreground and background of the video stream based on the foreground-background distinction threshold interval to obtain a rough mask of the portrait foreground includes:

[0036] When the CrCb value of each pixel in the image frame falls within the foreground-background distinction threshold interval, it is judged as the background, and the corresponding mask pixel is set to 0. When the CrCb value of each pixel in the image frame falls outside the foreground-background distinction threshold interval, it is judged as the foreground, and the corresponding mask pixel is set to 1, and finally a rough mask of the portrait foreground is obtained.

[0037] In a third aspect, the present invention provides a matting model training device, comprising:

[0038] The training acquisition module is used to acquire a training data set, wherein the training data set is composed of image blocks obtained by extracting and segmenting the original video data under a preset rule to reduce the proportion of redundant data; the original video data is a video with a blue or green background and containing two situations: no lecturer or portrait exists and the presence of a lecturer or portrait exists, and the original video data is original data in the YCrCb color space;

[0039] Sample generation module, used to construct the blue and green background probability model formula P B (X) and P G (X), where X = (v Cb ,v Cr ) is the CrCb component value of each pixel in the image, P is the probability density function that satisfies the parameter θ, and the parameter θ is obtained by training the training data set; B (X) and P G (X) obtain and label each pixel of all image frames to obtain all image frames used for training and their corresponding labels, that is, the probability heat map of whether the pixel is background, and obtain the binary portrait foreground mask label according to the heat map;

[0040] The model training module is used to train the multivariate nonlinear regression model color threshold model F in the YCrCb color space using the chromaticity components in the YCrCb data of the training dataset and the binary portrait foreground mask label. BG , the color threshold model F BGIt is used to fit the probability heat map of each pixel in the image frame as the background under different backgrounds; the brightness and chromaticity components in the YCrCb data of the image block randomly obtained within the bbox range are used to train a nonlinear regression model in the YCrCb color space to characterize the edge of the foreground portrait under the blue or green background under different situations. Fit the calculation model F to the expected blur weight BLUR , wherein the bbox range is the maximum bounding box of the portrait mask of the image frame containing the portrait in the training data set.

[0041] In a fourth aspect, the present invention provides an image matting device, comprising:

[0042] A sample acquisition module is used to acquire a video stream shot in a non-person recording environment and randomly extract N frames therefrom in its original YCrCb color space, wherein the video stream is a video stream using a blue or green background;

[0043] The template extraction module is used to extract the N frames of images through the background probability model formula and the color threshold model F BG Calculate the probability p of each pixel in the current image frame being the background, take all pixels whose p is greater than or equal to the parameter α, and obtain the maximum and minimum CrCb component values ​​of these pixels as the threshold interval for distinguishing the foreground and background in the current recording scene, where the parameter α is the preset threshold parameter for the matting degree, and the background probability model formula is the color of the matting background in the blue background probability model formula P B (X) and green background probability model formula P G (X);

[0044] The processing module is used to obtain the original YCrCb color space video stream of the portrait video stream, and segment the foreground and background of the video stream based on the foreground and background distinguishing threshold interval to obtain a rough mask of the portrait foreground; according to the rough mask of the portrait foreground, use the fuzzy weight calculation model F BLUR Calculate each pixel in the area covered by the mask to obtain the fuzzy weight of each pixel covered by the mask in the image superposition;

[0045] A superposition module, used for cutting out the foreground of the portrait in the frame of the current video stream according to the fuzzy weight and superimposing it with the background video stream for superposition;

[0046] Among them, the blue background probability model formula P B (X), green background probability model formula P G (X), color threshold model F BG And the calculation model of fuzzy weight F BLURAll of them are obtained through the matting model training device described in claim 9.

[0047] In a fifth aspect, the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-mentioned methods when executing the program.

[0048] In a sixth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the method of any of the above-mentioned embodiments is implemented.

[0049] In the present invention, in view of the complicated arrangement of micro-class shooting scenes and the weak adaptability in traditional matting applications, a method for adaptively determining the foreground background threshold by collecting recorded videos of blue or green backgrounds under a variety of different scenes, using a designed sample collection rule for training data collection, and using the training data to construct a series of background probability models, which makes up for the lack of adaptability in the traditional matting scheme to a certain extent, so that the thresholds of each color component required for matting can be more robust in different recording environments, reducing the requirements for the arrangement of the recording scene, and the setting of the matting application is also simplified. In the simplest case, the user only needs to set two parameters (blue or green background, matting blur intensity). In addition, the training of the model is based on the original data of the YCrCb color space obtained by the general recording device, and does not need to be converted into the RGB color space, and the foreground background color threshold model proposed in step S30 only needs to be calculated before starting to record the video, which reduces the real-time requirements for determining the foreground background threshold algorithm, and can design a more complex and more robust foreground background threshold algorithm model, and at the same time, after the threshold is determined, it is no longer necessary to use the model, and the amount of calculation during the subsequent formal recording is further reduced, so that the requirements of the matting function on the computing device are further reduced.

[0050] In addition, the proposed method of using the trained foreground edge blur weight calculation model to further refine the foreground obtained in the matting application can not only optimize the imaging effect of the foreground when performing overlay operations, but also the calculation range of the model is limited to the foreground background threshold set previously in the video frame. The calculation range and the demand for computing power are also greatly reduced compared with the matting algorithms with a certain degree of adaptivity. Because of the real-time requirement, the common matting algorithms with a certain degree of adaptivity need to operate on every pixel point of each frame in the video stream to obtain the corresponding pixel transparency and other parameters, which has high requirements on the computing power of the equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a flow chart of an implementation method of the first embodiment of the present invention.

[0052] Figure 2 It is a flow chart of an implementation method of the second embodiment of the present invention.

[0053] Figure 3 It is a schematic diagram of the overall structure of an implementation method of Example 3 of the present invention.

[0054] Figure 4 It is a schematic diagram of the overall structure of another implementation method of Example 3 of the present invention. DETAILED DESCRIPTION

[0055] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0056] It should be noted that the terms "first\second\..." involved in the embodiments of the present invention are only used to distinguish similar objects, and do not represent a specific order for the objects. It is understandable that the specific order or sequence of "first\second\..." can be interchanged where permitted. It should be understood that the objects distinguished by "first\second\..." can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein.

[0057] Embodiment 1

[0058] See also Figure 1 , Figure 1 1 is a flow chart of a matting model training method provided in Embodiment 1 of the present invention, the method comprising: step S110, step S120, step S130, step S140 and step S150. It should be noted that step S110, step S120, step S130, step S140 and step S150 are only reference numerals for clear explanation of the embodiment and the appendix. Figure 1 The corresponding relationship does not limit the order of the steps in this embodiment.

[0059] Step S110, obtaining a training data set, wherein the training data set is composed of image blocks obtained by extracting and segmenting the original video data under a preset rule to reduce the proportion of redundant data; the original video data is a video with a blue or green background and containing two situations: no lecturer or portrait exists and the presence of a lecturer or portrait exists, and the original video data is original data in a YCrCb color space;

[0060] Step S120, constructing a blue and green background probability model formula P B (X) and P G (X), where X = (v Cb ,v Cr ) is the CrCb component value of each pixel in the image, P is the probability density function satisfying the parameter θ, and the parameter θ is obtained by training the training data set;

[0061] Step S130, using the sum obtained by the background probability model formula to label each pixel of all image frames, obtain all image frames used for training and their corresponding labels, that is, a probability heat map of whether the pixel is background, and obtain a binary portrait foreground mask label according to the heat map;

[0062] Step S140, using the chromaticity components in the YCrCb data of the training data set and the binary portrait foreground mask label, train a multivariate nonlinear regression model color threshold model F in the YCrCb color space BG , the color threshold model F BG It is used to fit the probability heat map of each pixel in the image frame as the background under different backgrounds;

[0063] Step S150, using the brightness and chromaticity components of the YCrCb data of the image block randomly obtained within the bbox range, a nonlinear regression model is trained in the YCrCb color space to characterize the edge of the foreground portrait under the blue or green background in different situations to fit the calculation model F of the desired blur weight BLUR , wherein the bbox range is the maximum bounding box of the portrait mask of the image frame containing the portrait in the training data set.

[0064] In step S110, a training data set is obtained. A large number of micro-course videos under different circumstances are pre-recorded. The video data is usually original data in the YCrCb color space. Such original data usually contains at least the following features:

[0065] (1) The background of the video data used as the training data source should usually be blue or green;

[0066] (2) Materials of blue or green background include but are not limited to: blue or green curtains, blue or green walls, screens that display blue or green, etc.;

[0067] (3) The image should be captured under different lighting conditions, including but not limited to: lighting under different lighting methods, lighting without lighting, lighting interfered by external ambient light (such as sunlight), and / or lighting under different screen brightness when the screen is used as the background;

[0068] (4) Based on (1), (2), and (3), there is no lecturer or person in the picture;

[0069] (5) Based on (1)(2)(3), there is a lecturer or a portrait in the picture.

[0070] In the video stream set that meets the above conditions, there must be a large amount of redundant data, such as a large number of videos with blue or green backgrounds and no people in the foreground. Too much redundant data will inevitably reduce the general performance of the trained model. Therefore, it is necessary to extract and segment the original data under certain rules to reduce the proportion of redundant data, that is, it is necessary to sample the image frames extracted from the original data, and the sampled image blocks ultimately constitute the training data set.

[0071] In one implementation, step S100 is further included before step S110.

[0072] Step S100: Use image frames in the original video data to perform image block sampling to finally construct a training data set.

[0073] The process of step S100 includes: step S101, step S102, step S103 and step S104.

[0074] Step S101, randomly extracting N frames of images from the original video data, and converting these images into RGB color space;

[0075] Step S102, using a foreground segmentation algorithm, classifying images containing portraits and images not containing portraits from the N frames of images detected in the RGB color space, and recording portrait masks of the images containing portraits;

[0076] Step S103, for a blue or green background image frame that does not contain a portrait, randomly extract a certain number of image blocks of random sizes, and record the positions and length and width data of these image blocks;

[0077] Step S104, for the image frame containing the portrait, randomly extract a certain number of image blocks of random sizes from the image part outside the mask, record the position and length and width data of these image blocks, and for the image part within the mask, calculate the maximum bounding box bbox of the mask, randomly extract a certain number of image blocks of fixed sizes within the range of the bbox, and record the position and length and width data of these image blocks.

[0078] First, N frames of images are randomly extracted from the acquired video raw data, and these images are converted into red, green, blue (RGB) color space. Then, a foreground segmentation algorithm is used to detect which images of the extracted N frames of images contain portraits and which do not in the RGB color space, and the portrait mask containing the portrait image is recorded. Next, for the blue or green background image frames that do not contain portraits, a certain number of image blocks of random sizes are randomly extracted, and the positions and length and width data of these image blocks are recorded. Finally, for the image frames containing portraits, the operation described in step S103 is performed on the image part outside the mask. For the mask mask, the maximum bounding box (bbox) of the mask mask is calculated, and a certain number of fixed-size image blocks are randomly extracted within the range of the bbox, and the positions and length and width data of these image blocks are recorded.

[0079] After step S110, after obtaining the training data set, the color threshold model F for distinguishing the foreground and background in the YCrCb color space is trained. BG The calculation model F of the fuzzy weight of the foreground edge BLUR Among them, steps S120 to S140 are how to train the color threshold model F of the foreground and background. BG In the process, step S150 is how to use the color threshold model F of the foreground and background that has been obtained. BG To get the calculation model F of fuzzy weight BLUR process.

[0080] In step S120, construct the blue and green background probability model formula P B (X) and P G (X), where X = (v Cb ,v Cr ) is the CrCb component value of each pixel in the image, P is a probability density function satisfying a parameter θ, the parameter θ is obtained by training according to the training data set, and more specifically, the specific value of the parameter θ can be obtained by training the blue or green background image blocks obtained in steps S103 and S104.

[0081] In step S130, the sum obtained in step S120 is used to label each pixel of all image frames to obtain all image frames used for training and their corresponding labels, that is, a probability heatmap of whether the pixel is the background, and the pixels in the heatmap with a probability greater than or equal to 0.5 are labeled as 1, and the rest are labeled as 0, thereby obtaining a binary portrait foreground mask label.

[0082] In step S140, the chromaticity (Cr and Cb) components in the YCrCb data of the training data set obtained in step S110 and the binarized portrait foreground mask label are used to train a multivariate nonlinear regression model color threshold model F in the YCrCb color space. BG , the color threshold model F BG It is used to fit the probability heat map of each pixel in the image frame as the background under different backgrounds.

[0083] In one implementation, the specific process of step S140 includes: step S141 and step S142.

[0084] Step S141, constructing a multivariate nonlinear regression model color threshold model F BG =sigmoid(M(P(X))) or F BG =softmax(M(P(X))), where P represents the blue and green background probability model formula P B (X) and P G (X), X=(v Cb ,v Cr ) is the CrCb component value of each pixel in the image frame, M is the nonlinear regression model parameter to be trained, sigmoid and softmax are nonlinear functions with normalization ability;

[0085] Step S142: Use the chromaticity components in the YCrCb data of the training data set and the binary portrait foreground mask label as training data to train the multivariate nonlinear regression model color threshold model F BG .

[0086] In step S141, a nonlinear model F is constructed. BG =sigmoid(M(P(X))) or F BG =softmax(M(P(X))), where P represents the blue or green background model formula P obtained by training in step S130 B and P G , X=(v Cb ,v Cr ) is the CrCb component value of each pixel in the image frame, M is the nonlinear regression model parameter to be trained, sigmoid and softmax are nonlinear functions with normalization ability.

[0087] Step S141 uses the image block data obtained in step S110 and step S130 as training data for training the nonlinear regression model F BG .

[0088] In step S150, all data in the YCrCb data of the image block randomly obtained within the bbox range, that is, the chrominance (Cr and Cb) components of brightness (Y), are used to train a nonlinear regression model F in the YCrCb color space. BLUR , which is used to characterize the computational model of fitting the edge of the foreground portrait to the desired blur weight under the blue or green background in different situations. Specifically, the bbox range is the maximum bounding box of the portrait mask of the image frame containing the portrait in the training data set, which can be obtained by the method of the aforementioned step S104.

[0089] In one implementation, the process of step S150 includes: step S151 and step S152.

[0090] Step S151, constructing a nonlinear function with parameter φ as the expected fitting objective function F TARGET , and construct the calculation model F of the fuzzy weight of the nonlinear regression model BLUR =sigmoid(M CrCb (X CrCb )+M(X YCrCb )) is used to train the fitting F TARGET , where X CrCb is the CrCb component value of each pixel in the image frame (v Cb ,v Cr ), X YCrCb is the YCrCb component value of each pixel in the image frame (v Y ,v Cb ,v Cr ), M CrCb is the nonlinear regression model parameter to be trained for the CrCb component of each pixel in the image frame of the YCrCb color space, M YCrCb is the correlation model parameter to be trained between the Y component and the CrCb component of each pixel in the image frame of the YCrCb color space;

[0091] Step S152: using the acquired image frame data within the bbox range as training data to train the calculation model F of the fuzzy weight of the nonlinear regression model BLUR .

[0092] Step S151 constructs a nonlinear function with parameters φ and an expected fitting objective function F TARGET , and construct the calculation model F of the fuzzy weight of the nonlinear regression model BLUR =sigmoid(M CrCb (X CrCb )+M(X YCrCb )) is used to train the fitting F TARGET , where X CrCbis the CrCb component value of each pixel in the image frame (v Cb ,v Cr ), X YCrCb is the YCrCb component value of each pixel in the image frame (v Y ,v Cb ,v Cr ), M CrCb is the nonlinear regression model parameter to be trained for the CrCb component of each pixel in the image frame of the YCrCb color space, M YCrCb are the correlation model parameters to be trained between the Y component and the CrCb component of each pixel in the image frame of the YCrCb color space.

[0093] Step S152 uses the acquired image frame data within the bbox range as training data for training the nonlinear regression model F BLUR Specifically, the bbox range is the maximum bounding box of the portrait mask of the image frame containing the portrait in the training data set, which can be obtained by the method of the aforementioned step S104.

[0094] In this method, in view of the complex arrangement of micro-class shooting scenes and the weak adaptability in traditional matting applications, a method is proposed to collect recorded videos of blue or green backgrounds in a variety of different scenes, use designed sample collection rules to collect training data, and use the training data to build a series of background probability models to adaptively determine the foreground background threshold. This method makes up for the lack of adaptability in the traditional matting scheme to a certain extent, so that the thresholds of each color component required for matting can be more robust in different recording environments, reducing the requirements for the arrangement of the recording scene, and the settings of the matting application are also simplified. In the simplest case, the user only needs to set two parameters (blue or green background, matting blur intensity). In addition, the training of the model is based on the original data of the YCrCb color space obtained by the general recording device, and does not need to be converted into the RGB color space, and the foreground background color threshold model proposed in step S30 only needs to be calculated before starting to record the video, which reduces the real-time requirements for determining the foreground background threshold algorithm, and can design a more complex and more robust foreground background threshold algorithm model. At the same time, after the threshold is determined, the model is no longer needed, and the amount of calculation during the subsequent formal recording is further reduced, so that the requirements of the matting function on the computing device are further reduced.

[0095] Embodiment 2

[0096] See also Figure 2 , Figure 21 is a flow chart of an image matting method provided in Embodiment 1 of the present invention, the method comprising: step S210, step S220, step S230, step S240 and step S250. It should be noted that step S210, step S220, step S230, step S240 and step S250 are only reference numerals for the purpose of clearly explaining the embodiment and the appendix. Figure 2 The corresponding relationship does not limit the order of the steps in this embodiment.

[0097] Step S210, obtaining a video stream shot in a non-person recording environment, and randomly extracting N frames therefrom in its original YCrCb color space, wherein the video stream is a video stream using a blue or green background;

[0098] Step S220: The extracted N frames of images are processed by the background probability model formula and the color threshold model F BG Calculate the probability p of each pixel in the current image frame being the background, take all pixels whose p is greater than or equal to the parameter α, and obtain the maximum and minimum CrCb component values ​​of these pixels as the threshold interval for distinguishing the foreground and background in the current recording scene, where the parameter α is the preset threshold parameter for the matting degree, and the background probability model formula is the color of the matting background in the blue background probability model formula P B (X) and green background probability model formula P G (X);

[0099] Step S230, obtaining the original YCrCb color space video stream of the captured portrait video stream, and segmenting the foreground and background of the video stream based on the foreground and background distinguishing threshold interval to obtain a rough mask of the portrait foreground;

[0100] Step S240: Based on the rough mask of the portrait foreground, a fuzzy weight calculation model F is used. BLUR Calculate each pixel in the area covered by the mask to obtain the fuzzy weight of each pixel covered by the mask in the image superposition;

[0101] Step S250: extracting the foreground of the portrait in the frame of the current video stream according to the blur weight and superimposing it with the background video stream for superimposition.

[0102] Among them, the blue background probability model formula P B (X), green background probability model formula P G (X), color threshold model F BG And the calculation model of fuzzy weight F BLUR It is obtained through the training method of embodiment 1.

[0103] This embodiment is a method of how to use the method of embodiment one to perform image matting. Before step S210, a micro-course recording environment is arranged, and a blue or green background is used. The background can be a screen, curtain, wallpaper or other equipment or material that displays or is colored in blue or green, and normal shooting scene lighting is ensured. Then on this basis, the user sets the background color of the portrait matting (green or blue), as well as the threshold parameter α of the matting degree. Step S210 obtains the video stream shot in the non-portrait recording environment from the recording and broadcasting terminal, and randomly extracts N frames from it in its original YCrCb color space. Then, in step S220, the extracted N frames of image are passed through the blue or green background model formula P B and P G (selected according to the background color of the matte set by the user), and the color threshold model F BG The probability p of each pixel in the current image frame being the background is calculated, and all pixels with p greater than or equal to the parameter α are selected to obtain the maximum and minimum CrCb component values ​​of these pixels as the threshold interval for distinguishing the foreground and background in the current recording scene.

[0104] After step S220 obtains the threshold interval for distinguishing the foreground and the background, step S230 uses this threshold interval. Specifically, the terminal device implementing the matting, such as the recording and broadcasting terminal, obtains the video stream in the original YCrCb color space of the captured portrait video stream, and then uses the threshold interval obtained in step S220 to perform foreground and background segmentation, that is, specifically, if the CrCb value of each pixel in the image frame falls within the threshold interval set in step S220, it is determined to be the background, and the corresponding mask pixel is set to 0, otherwise it is considered to be the foreground, and the corresponding mask pixel is set to 1, and finally a rough mask of the portrait foreground is obtained.

[0105] In step S240, based on the rough mask of the portrait foreground obtained in step S230, the foreground edge blur weight calculation model F is used. BLUR The calculation is performed on each pixel in the area covered by the mask to obtain the blur weight of each pixel covered by the mask in the image superposition.

[0106] Finally, step S250 cuts out the portrait foreground in the current video frame and superimposes it with the background video stream according to the foreground mask blur weight obtained in step S240 and another background video stream for superimposition received by the recording and broadcasting terminal.

[0107] In one implementation, the process of step S250 includes: step S251 and step S252.

[0108] Step S251, perform color space transformation on the foreground area covered by the foreground mask fuzzy weight, and calculate the grayscale color value v of each pixel in this area. g, and convert the grayscale color value back to the corresponding YCrCb component v GYCrCb , according to the formula: v DYCrCb =f blur *v SYCrCb +(1-f blur )*v GYCrCb Calculate the YCrCb component v of the foreground pixel used for superposition DYCrCb , get the foreground pixel value, where f blur is the fuzzy weight, v SYCrCb YCrCb components of the original scene pixel;

[0109] Step S252, the background video stream for superimposition is calculated according to the fuzzy weight, the area to be superimposed is set to 0, and the foreground pixel value is filled in according to the corresponding position to complete the matting and superimposition operation.

[0110] In step S251, the foreground area covered by the foreground mask fuzzy weight obtained in step S240 is transformed into a color space, and the grayscale color value v of each pixel in this area is calculated. g , and convert the grayscale color value back to the corresponding YCrCb component v GYCrCb , according to the formula: v DYCrCb =f blur *v SYCrCb +(1-f blur )*v GYCrCb Calculate the YCrCb component v of the foreground pixel used for superposition DYCrCb Where f blur is the fuzzy weight of the current pixel obtained in step S240, v SYCrCb It is the YCrCb component of the original scene pixel.

[0111] In step S252, another background video stream for superimposition at the recording and broadcasting terminal is calculated according to the foreground mask blur weight obtained in step S240, the area to be superimposed is set to 0, and then the foreground pixel value calculated in step S251 is filled in according to the corresponding position to complete the matting and superimposition operation.

[0112] In this method, it is proposed to use the trained foreground edge blur weight calculation model to further refine the foreground obtained in the matting application, which can not only optimize the imaging effect of the foreground when performing the overlay operation, but also the calculation range of the model is limited to the foreground background threshold set in the video frame. The calculation range and the demand for computing power are also greatly reduced compared with the matting algorithm with a certain degree of adaptivity. Because under the requirement of real-time performance, the common matting algorithm with a certain degree of adaptivity needs to calculate every pixel point of every frame in the video stream to obtain the corresponding pixel transparency and other parameters, which has high requirements on the computing power of the device.

[0113] Embodiment 3

[0114] Corresponding to the matting model training method of embodiment 1, Figure 3 As shown, the present invention also provides a matting model training device, including: a training acquisition module 310, a sample generation module 320, and a model training module 330;

[0115] The training acquisition module 310 is used to acquire a training data set, wherein the training data set is composed of image blocks obtained by extracting and segmenting the original video data under a preset rule to reduce the proportion of redundant data; the original video data is a video with a blue or green background and containing two situations: no lecturer or human portrait exists and the presence of a lecturer or human portrait exists, and the original video data is original data in the YCrCb color space;

[0116] Sample generation module 320, used to construct blue and green background probability model formula P B (X) and P G (X), where X = (v Cb ,v Cr ) is the CrCb component value of each pixel in the image, P is the probability density function that satisfies the parameter θ, and the parameter θ is obtained by training the training data set; B (X) and P G (X) obtain and label each pixel of all image frames to obtain all image frames used for training and their corresponding labels, that is, the probability heat map of whether the pixel is background, and obtain the binary portrait foreground mask label according to the heat map;

[0117] The model training module 330 is used to train a multivariate nonlinear regression model color threshold model F in the YCrCb color space using the chrominance components in the YCrCb data of the training data set and the binary portrait foreground mask label. BG , the color threshold model F BGIt is used to fit the probability heat map of each pixel in the image frame as the background under different backgrounds; the brightness and chromaticity components in the YCrCb data of the image block randomly obtained within the bbox range are used to train a nonlinear regression model in the YCrCb color space to characterize the edge of the foreground portrait under the blue or green background under different situations. Fit the calculation model F to the expected blur weight BLUR , wherein the bbox range is the maximum bounding box of the portrait mask of the image frame containing the portrait in the training data set.

[0118] Corresponding to the image matting method of the first embodiment, as Figure 4 As shown, the present invention further provides an image matting device, including: a template acquisition module 410, a template extraction module 420, a processing module 430, and an overlay module 440.

[0119] The template acquisition module 410 is used to acquire a video stream shot in a non-person recording environment and randomly extract N frames therefrom in its original YCrCb color space, wherein the video stream is a video stream using a blue or green background;

[0120] The template extraction module 420 is used to extract the N frames of images through the background probability model formula and the color threshold model F BG Calculate the probability p of each pixel in the current image frame being the background, take all pixels whose p is greater than or equal to the parameter α, and obtain the maximum and minimum CrCb component values ​​of these pixels as the threshold interval for distinguishing the foreground and background in the current recording scene, where the parameter α is the preset threshold parameter for the matting degree, and the background probability model formula is the color of the matting background in the blue background probability model formula P B (X) and green background probability model formula P G (X);

[0121] The processing module 430 is used to obtain the original YCrCb color space video stream of the portrait video stream, and segment the foreground and background of the video stream based on the foreground and background distinguishing threshold interval to obtain a rough mask of the portrait foreground; according to the rough mask of the portrait foreground, use the fuzzy weight calculation model F BLUR Calculate each pixel in the area covered by the mask to obtain the fuzzy weight of each pixel covered by the mask in the image superposition;

[0122] A superposition module 440, configured to cut out the foreground of the portrait in the frame of the current video stream according to the blur weight and superimpose it with the background video stream for superposition;

[0123] Among them, the blue background probability model formula P B(X), green background probability model formula P G (X), color threshold model F BG And the calculation model of fuzzy weight F BLUR All of them are obtained through the matte model training device mentioned above.

[0124] In this device, in view of the complex arrangement of micro-class shooting scenes and the weak adaptability in traditional matting applications, a method is proposed to collect recorded videos of blue or green backgrounds in a variety of different scenes, use designed sample collection rules to collect training data, and use the training data to build a series of background probability models to adaptively determine the foreground background threshold. This method makes up for the lack of adaptability in the traditional matting scheme to a certain extent, so that the thresholds of each color component required for matting can be more robust in different recording environments, reducing the requirements for the arrangement of the recording scene, and the settings of the matting application are also simplified. In the simplest case, the user only needs to set two parameters (blue or green background, matting blur intensity). In addition, the training of the model is based on the original data of the YCrCb color space obtained by the general recording device, and does not need to be converted into the RGB color space, and the foreground background color threshold model proposed in step S30 only needs to be calculated before starting to record the video, which reduces the real-time requirements for determining the foreground background threshold algorithm, and can design a more complex and more robust foreground background threshold algorithm model. At the same time, after the threshold is determined, the model is no longer needed, and the amount of calculation during the subsequent formal recording is further reduced, so that the requirements of the matting function on the computing device are further reduced.

[0125] In addition, the proposed method of using the trained foreground edge blur weight calculation model to further refine the foreground obtained in the matting application can not only optimize the imaging effect of the foreground when performing overlay operations, but also the calculation range of the model is limited to the foreground background threshold set previously in the video frame. The calculation range and the demand for computing power are also greatly reduced compared with the matting algorithms with a certain degree of adaptivity. Because of the real-time requirement, the common matting algorithms with a certain degree of adaptivity need to operate on every pixel point of each frame in the video stream to obtain the corresponding pixel transparency and other parameters, which has high requirements on the computing power of the equipment.

[0126] Embodiment 4

[0127] An embodiment of the present invention further provides a storage medium on which computer instructions are stored. When the instructions are executed by a processor, the matting model training method and / or image matting method of any of the above embodiments are implemented.

[0128] Those skilled in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, random access memories (RAM), read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0129] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to execute all or part of the methods of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, RAM, ROM, magnetic disks or optical disks.

[0130] Corresponding to the above-mentioned computer storage medium, in one embodiment, a computer device is also provided, which includes a memory, an encoder, and a computer program stored in the memory and runnable on the encoder, wherein when the encoder executes the program, it implements any one of the matting model training methods and / or image matting methods in the above-mentioned embodiments.

[0131] The above computer device, in view of the complicated arrangement of micro-class shooting scenes and the weak adaptability in traditional matting applications, proposes a method for adaptively determining the foreground background threshold by collecting recorded videos of blue or green backgrounds in a variety of different scenes, using designed sample collection rules for training data collection, and using the training data to build a series of background probability models. This method makes up for the lack of adaptability in the traditional matting scheme to a certain extent, so that the thresholds of each color component required for matting can be more robust in different recording environments, reducing the requirements for the arrangement of recording scenes, and the settings of the matting application are also simplified. In the simplest case, the user only needs to set two parameters (blue or green background, matting blur intensity). In addition, the training of the model is based on the original data of the YCrCb color space obtained by the general recording device, and does not need to be converted into the RGB color space, and the foreground background color threshold model proposed in step S30 only needs to be calculated before starting to record the video, which reduces the real-time requirements for determining the foreground background threshold algorithm, and can design a more complex and more robust foreground background threshold algorithm model. At the same time, after the threshold is determined, the model is no longer needed to be used, which further reduces the amount of calculation during the subsequent formal recording, so that the requirements of the matting function on the computing device are further reduced.

[0132] In addition, the proposed method of using the trained foreground edge blur weight calculation model to further refine the foreground obtained in the matting application can not only optimize the imaging effect of the foreground when performing overlay operations, but also the calculation range of the model is limited to the foreground background threshold set previously in the video frame. The calculation range and the demand for computing power are also greatly reduced compared with the matting algorithms with a certain degree of adaptivity. Because of the real-time requirement, the common matting algorithms with a certain degree of adaptivity need to operate on every pixel point of each frame in the video stream to obtain the corresponding pixel transparency and other parameters, which has high requirements on the computing power of the equipment.

[0133] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0134] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A matting model training method, characterized in that: include: Acquire a training data set, wherein the training data set is composed of image blocks obtained by extracting and segmenting original video data under a preset rule to reduce the proportion of redundant data; the original video data is a video with a blue or green background and containing two situations: no lecturer or human portrait exists and the presence of a lecturer or human portrait exists, and the original video data is original data in a YCrCb color space; Construct blue and green background probability model formula P B (X) and P G (X), where X = (v Cb ,v Cr ) is the CrCb component value of each pixel in the image, P is the probability density function satisfying the parameter θ, and the parameter θ is obtained by training the training data set; Use the sum obtained by the background probability model formula to label each pixel of all image frames, obtain all image frames used for training and their corresponding labels, that is, the probability heat map of whether the pixel is the background, and obtain the binary portrait foreground mask label based on the heat map; Use the chromaticity components in the YCrCb data of the training dataset and the binary portrait foreground mask label to train the multivariate nonlinear regression model color threshold model F in the YCrCb color space BG , the color threshold model F BG It is used to fit the probability heat map of each pixel in the image frame as the background under different backgrounds; Using the brightness and chromaticity components of the YCrCb data of the image block randomly obtained within the bbox range, a nonlinear regression model is trained in the YCrCb color space to characterize the edge of the foreground portrait under the blue or green background in different situations. The calculation model F is fitted to the desired blur weight. BLUR , wherein the bbox range is the maximum bounding box of the portrait mask of the image frame containing the portrait in the training data set.

2. The matting model training method according to claim 1, characterized in that: Also includes: Use the image frames in the original video data to sample image blocks and finally construct a training data set; The process of sampling image blocks using image frames in the original video data to finally construct a training data set includes: Randomly extract N frames of images from the original video data and convert these images into RGB color space; Use the foreground segmentation algorithm to detect and extract N frames of images in the RGB color space to classify images containing portraits and images not containing portraits, and record the portrait mask of the images containing portraits; For blue or green background image frames that do not contain portraits, randomly extract a certain number of image blocks of random sizes, and record the positions and length and width data of these image blocks; For image frames containing portraits, a certain number of image blocks of random sizes are randomly extracted from the image part outside the mask, and the position, length and width data of these image blocks are recorded. For the image part within the mask, the maximum bounding box bbox of the mask is calculated, and a certain number of image blocks of fixed sizes are randomly extracted within the range of the bbox, and the position, length and width data of these image blocks are recorded.

3. The matting model training method according to claim 1, characterized in that: The original video data includes imaging images under a variety of lighting conditions, and the lighting conditions include: lighting under different lighting methods, lighting without lighting, lighting interfered by external ambient light, and / or lighting under different screen brightness when the screen is used as the background.

4. The matting model training method according to claim 1, characterized in that: The method uses the chromaticity component in the YCrCb data of the training data set and the binary portrait foreground mask label to train a multivariate nonlinear regression model color threshold model F in the YCrCb color space. BG The process includes: Constructing a multivariate nonlinear regression model Color threshold model F BG =sigmoid(M(P(X))) or F BG =softmax(M(P(X))), where P represents the blue and green background probability model formula P B (X) and P G (X), X=(v Cb ,v Cr ) is the CrCb component value of each pixel in the image frame, M is the nonlinear regression model parameter to be trained, sigmoid and softmax are nonlinear functions with normalization ability; The chromaticity component in the YCrCb data of the training dataset and the binary portrait foreground mask label are used as training data to train the multivariate nonlinear regression model color threshold model F BG .

5. The matting model training method according to claim 1, characterized in that: The method uses the brightness and chromaticity components of the YCrCb data of the image block randomly obtained within the bbox range, and trains a nonlinear regression model in the YCrCb color space to characterize the edge of the foreground portrait under the blue or green background in different situations to fit the calculation model F with the desired blur weight. BLUR The process includes: Construct the nonlinear function with parameter φ and fit the objective function F TARGET , and construct the calculation model F of the fuzzy weight of the nonlinear regression model BLUR =sigmoid(M CrCb (X CrCb )+M(X YCrCb )) is used to train and fit F TARGET , where X CrCb is the CrCb component value of each pixel in the image frame (v Cb ,v Cr ), X YCrCb is the YCrCb component value of each pixel in the image frame (v Y ,v Cb ,v Cr ), M CrCb is the nonlinear regression model parameter to be trained for the CrCb component of each pixel in the image frame of the YCrCb color space, M YCrCb is the correlation model parameter to be trained between the Y component and the CrCb component of each pixel in the image frame of the YCrCb color space; The image frame data within the obtained bbox range is used as training data to train the calculation model F of the fuzzy weight of the nonlinear regression model BLUR .

6. An image matting method, characterized in that: include: Obtain a video stream shot in a non-portrait recording environment, and randomly extract N frames from the video stream in its original YCrCb color space, wherein the video stream is a video stream using a blue or green background; The extracted N frames of images are passed through the background probability model formula and the color threshold model F BG Calculate the probability p of each pixel in the current image frame being the background, take all pixels whose p is greater than or equal to the parameter α, and obtain the maximum and minimum CrCb component values ​​of these pixels as the threshold interval for distinguishing the foreground and background in the current recording scene, where the parameter α is the preset threshold parameter for the matting degree, and the background probability model formula is the color of the matting background in the blue background probability model formula P B (X) and green background probability model formula P G (X); Acquire the original YCrCb color space video stream of the captured portrait video stream, and segment the foreground and background of the video stream based on the foreground and background distinguishing threshold interval to obtain a rough mask of the portrait foreground; According to the rough mask of the portrait foreground, the calculation model F of the fuzzy weight is used BLUR Calculate each pixel in the area covered by the mask to obtain the fuzzy weight of each pixel covered by the mask in the image superposition; Cut out the foreground of the portrait in the frame of the current video stream according to the blur weight and superimpose it with the background video stream for superimposition; Among them, the blue background probability model formula P B (X), green background probability model formula P G (X), color threshold model F BG And the calculation model of fuzzy weight F BLUR All are obtained through the matting model training method described in any one of claims 1-5.

7. The image matting method according to claim 6, characterized in that: The process of cutting out the portrait foreground in the frame of the current video stream according to the blur weight and superimposing it with the background video stream for superimposition includes: Perform color space transformation on the foreground area covered by the foreground mask fuzzy weight and calculate the grayscale value v of each pixel in this area. g , and convert the grayscale color value back to the corresponding YCrCb component v GYCrCb , according to the formula: v DYCrCb =f blur *v SYCrCb +(1-f blur )*v GYCrCb Calculate the YCrCb component v of the foreground pixel used for superposition DYCrCb , get the foreground pixel value, where f blur is the fuzzy weight, v SYCrCb YCrCb components of the original scene pixel; The background video stream for superimposition is calculated according to the fuzzy weight, the area to be superimposed is set to 0, and the foreground pixel value is filled in according to the corresponding position to complete the matting and superimposition operations.

8. The image matting method according to claim 6, characterized in that: The process of segmenting the foreground and background of the video stream based on the foreground and background distinguishing threshold interval to obtain a rough mask of the portrait foreground includes: When the CrCb value of each pixel in the image frame falls within the foreground-background distinction threshold interval, it is judged as the background, and the corresponding mask pixel is set to 0. When the CrCb value of each pixel in the image frame falls outside the foreground-background distinction threshold interval, it is judged as the foreground, and the corresponding mask pixel is set to 1, and finally a rough mask of the portrait foreground is obtained.

9. A matting model training device, characterized in that: include: The training acquisition module is used to acquire a training data set, wherein the training data set is composed of image blocks obtained by extracting and segmenting the original video data under a preset rule to reduce the proportion of redundant data; the original video data is a video with a blue or green background and containing two situations: no lecturer or portrait exists and the presence of a lecturer or portrait exists, and the original video data is original data in the YCrCb color space; Sample generation module, used to construct the blue and green background probability model formula P B (X) and P G (X), where X = (v Cb ,v Cr ) is the CrCb component value of each pixel in the image, P is the probability density function that satisfies the parameter θ, and the parameter θ is obtained by training the training data set; B (X) and P G (X) obtain and label each pixel of all image frames to obtain all image frames used for training and their corresponding labels, that is, the probability heat map of whether the pixel is background, and obtain the binary portrait foreground mask label according to the heat map; The model training module is used to train the multivariate nonlinear regression model color threshold model F in the YCrCb color space using the chromaticity components in the YCrCb data of the training dataset and the binary portrait foreground mask label. BG , the color threshold model F BG It is used to fit the probability heat map of each pixel in the image frame as the background under different backgrounds; the brightness and chromaticity components in the YCrCb data of the image block randomly obtained within the bbox range are used to train a nonlinear regression model in the YCrCb color space to characterize the edge of the foreground portrait under the blue or green background under different situations. Fit the calculation model F to the expected blur weight BLUR , wherein the bbox range is the maximum bounding box of the portrait mask of the image frame containing the portrait in the training data set.

10. An image matting device, characterized in that: include: A sample acquisition module is used to acquire a video stream shot in a non-person recording environment and randomly extract N frames therefrom in its original YCrCb color space, wherein the video stream is a video stream using a blue or green background; The template extraction module is used to extract the N frames of images through the background probability model formula and the color threshold model F BG Calculate the probability p of each pixel in the current image frame being the background, take all pixels whose p is greater than or equal to the parameter α, and obtain the maximum and minimum CrCb component values ​​of these pixels as the threshold interval for distinguishing the foreground and background in the current recording scene, where the parameter α is the preset threshold parameter for the matting degree, and the background probability model formula is the color of the matting background in the blue background probability model formula P B (X) and green background probability model formula P G (X); The processing module is used to obtain the original YCrCb color space video stream of the portrait video stream, and segment the foreground and background of the video stream based on the foreground and background distinguishing threshold interval to obtain a rough mask of the portrait foreground; according to the rough mask of the portrait foreground, use the fuzzy weight calculation model F BLUR Calculate each pixel in the area covered by the mask to obtain the fuzzy weight of each pixel covered by the mask in the image superposition; A superposition module, used for cutting out the foreground of the portrait in the frame of the current video stream according to the fuzzy weight and superimposing it with the background video stream for superposition; Among them, the blue background probability model formula P B (X), green background probability model formula P G (X), color threshold model F BG And the calculation model of fuzzy weight F BLUR All of them are obtained through the matting model training device described in claim 9.

11. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images