Video processing method and device

Through the video processing method, the feature extraction and binarized mask image generation of frame-by-frame video frame diagrams are used to use the segmentation model, and the synthetic image processing is carried out in combination with the background image, which solves the problem of professional recording scenes in the prior art portrait segmentation and background replacement in the prior art. , achieving efficient, lightweight and high-quality video processing effects.

CN113112508BActive Publication Date: 2024-12-06BEIJING DA MI TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110340843.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-03-30
Publication Date
2024-12-06
Estimated Expiration
2041-03-30

AI Technical Summary

Technical Problem

When performing portrait segmentation and background replacement in live broadcast or video conference, professional recording scenes and good uniform light are required, limiting the flexibility and quality of video processing.

Method used

By using video processing method, by obtaining the to-process video and background images, using segmentation models to perform feature extraction and binarized mask images for frame-by-frame video frame images, combined with background images for synthetic image processing, to realize portrait segmentation and background replacement. And not restricted by recording scenes.

Benefits of technology

It realizes efficient, lightweight, and high-quality portrait segmentation and background replacement of videos to be processed without being restricted by recording scenes, improving the flexibility and effect of video processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113112508B_ABST
    Figure CN113112508B_ABST
Patent Text Reader

Abstract

The embodiment of the present invention discloses a video processing method and device. After obtaining a video to be processed and a background image, the method obtains multiple video frame images from the video to be processed, and then inputs each video frame image into a segmentation model to obtain a binary mask image corresponding to each video frame image, and then generates a synthetic image corresponding to each video frame image based on the video frame image, the binary mask image and the background image, and finally regenerates the video based on the synthetic image. Through the video processing method, the video to be processed can be efficiently, light-weighted, and high-quality portrait segmentation and background replacement can be performed without being restricted by the recording scene.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of video technology, and in particular to a video processing method and device. Background Art

[0002] During a live broadcast or video conference, if the speaker does not want to reveal the actual background he / she is currently in, it is necessary to separate the speaker's image from the actual background and replace the actual background with a virtual background.

[0003] In the prior art, traditional OBS tools are usually used to separate the portrait and the background by color. However, this method has very strict requirements on the recording scene during video recording. For example, a pure color filter and good and uniform light are required to achieve better results. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a video processing method and device, so that portrait segmentation and background replacement can be performed on the processed video in an efficient, lightweight and high-quality manner without the need for professional recording scenes.

[0005] In a first aspect, an embodiment of the present invention provides a video processing method, the method comprising:

[0006] Get the video and background image to be processed;

[0007] Obtaining multiple video frame images from the video to be processed;

[0008] For each video frame image, input the video frame image into a segmentation model to obtain a binary mask image corresponding to the video frame image;

[0009] Generate a synthetic image corresponding to each video frame image according to the video frame image, the binary mask image and the background image;

[0010] A video is generated according to the composite image.

[0011] Furthermore, obtaining a plurality of video frame images from the video to be processed includes:

[0012] The video to be processed is extracted frame by frame to obtain multiple video frame images in the video to be processed.

[0013] Furthermore, the step of inputting the video frame image into a segmentation model to obtain a binary mask image corresponding to the video frame image includes:

[0014] Inputting the video frame graph into the encoder part of the segmentation model to obtain an encoding feature extraction graph;

[0015] The encoded feature extraction map is input into the decoder part of the segmentation model to obtain the binary mask map.

[0016] Furthermore, the step of inputting the video frame graph into an encoder part of the segmentation model to obtain an encoding feature extraction graph comprises:

[0017] Inputting the video frame graph into a first feature extraction module, a second feature extraction module, a third feature extraction module, a fourth feature extraction module and a fifth feature extraction module connected in series to obtain the coding feature extraction graph;

[0018] The second feature extraction module, the third feature extraction module, the fourth feature extraction module and the fifth feature extraction module respectively use the output of the previous feature extraction module as input.

[0019] Furthermore, the first feature extraction module, the second feature extraction module, the third feature extraction module, the fourth feature extraction module and the fifth feature extraction module all include a MobileNetV2 structure.

[0020] Furthermore, the MobileNetV2 structure includes a sequentially connected 3*3 depth-wise separable convolutional layer, a 1*1 convolutional layer, a 3*3 depth-wise separable convolutional layer with a stride of 2, and a 1*1 convolutional layer.

[0021] Further, the step of inputting the encoded feature extraction map into a decoder part of the segmentation model to obtain the binary mask map comprises:

[0022] Inputting the encoding feature extraction map into a first upsampling module, a second upsampling module, a third upsampling module, a fourth upsampling module and a fifth upsampling module connected in series to obtain a decoding feature restoration map;

[0023] Inputting the decoded feature restoration image into the convolution prediction layer to obtain the binary mask image;

[0024] The second up-sampling module, the third up-sampling module, the fourth up-sampling module and the fifth up-sampling module respectively use the output of the previous up-sampling module as input.

[0025] Further, the first upsampling module, the second upsampling module, the third upsampling module, the fourth upsampling module and the fifth upsampling module all include a feature restoration module and a transposed convolution layer;

[0026] Wherein, the feature restoration modules all adopt a residual structure.

[0027] Furthermore, the residual structure includes a skip connection convolutional layer and a sequentially connected 3*3 depth-separable convolutional layer, a 1*1 convolutional layer, a 3*3 depth-separable convolutional layer and a 1*1 convolutional layer;

[0028] The jump connection convolution layer is a 1*1 convolution layer, and takes the feature extraction graph jumped from the encoder part as input.

[0029] Furthermore, the step of inputting the decoded feature restoration map into a convolution prediction layer to obtain the binary mask map comprises:

[0030] The decoded feature restoration image is binarized to obtain the binarized mask image.

[0031] Furthermore, the segmentation model is obtained by training based on a joint loss function of binary cross entropy and focal-loss;

[0032] The joint loss function L is calculated according to the following formula:

[0033] L=L m +λ*L e

[0034]

[0035]

[0036] Among them, n represents the total number of pixels in an image, m represents the total number of pixels at the image boundary, yi represents the label of the i-th pixel in an image, xi represents the label of the i-th pixel at the image boundary, ρi is the predicted probability that the i-th pixel belongs to the segmentation target, γ is the adjustment factor, and λ is the loss weight parameter.

[0037] Furthermore, the image boundary label is an output obtained by processing the decoded feature restoration image using a canny edge detection algorithm.

[0038] Further, generating a synthetic image corresponding to each video frame image according to the video frame image, the binary mask image and the background image includes:

[0039] Normalizing the binary mask image to obtain a normalized binary mask image;

[0040] The video frame image, the normalized binary mask image and the background image are subjected to image operation processing according to the following formula to obtain the composite image I:

[0041] I=αF+(1-α)B

[0042] Among them, α is the normalized mask image, F is the video frame image, and B is the background image.

[0043] In a second aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method described in the first aspect.

[0044] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium for storing computer program instructions, wherein the computer program instructions implement the method described in the first aspect when executed by a processor.

[0045] After obtaining the video to be processed and the background image, the method of the embodiment of the present invention obtains multiple video frame images from the video to be processed, and then inputs each video frame image into the segmentation model respectively to obtain a binary mask image corresponding to each video frame image, and then generates a synthetic image corresponding to each video frame image based on the video frame image, the binary mask image and the background image, and finally regenerates the video based on the synthetic image. Through the video processing method, efficient, lightweight and high-quality portrait segmentation and background replacement can be performed on the video to be processed without being restricted by the recording scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0047] Figure 1 A schematic diagram of a flow chart of a video processing method according to an embodiment of the present invention;

[0048] Figure 2 Another schematic diagram of a video processing method according to an embodiment of the present invention;

[0049] Figure 3 A schematic diagram of another flow chart of a video processing method according to an embodiment of the present invention;

[0050] Figure 4 A schematic diagram of the composition of a segmentation model according to an embodiment of the present invention;

[0051] Figure 5 is a structural schematic diagram of a feature extraction module according to an embodiment of the present invention;

[0052] Figure 6 A schematic diagram of the composition of an upsampling module according to an embodiment of the present invention;

[0053] Figure 7 is a schematic flow chart of a video frame processing method according to an embodiment of the present invention;

[0054] Figure 8 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The present invention is described below based on embodiments, but the present invention is not limited to these embodiments. In the detailed description of the present invention below, some specific details are described in detail. It is possible for a person skilled in the art to fully understand the present invention without the description of these details. In order to avoid confusing the essence of the present invention, known methods, processes, flows, components and circuits are not described in detail.

[0056] In addition, persons of ordinary skill in the art will appreciate that the drawings provided herein are for illustration purposes and are not necessarily drawn to scale.

[0057] Unless the context clearly requires otherwise, words such as “include”, “including” and the like in the specification should be interpreted as including rather than being exclusive or exhaustive; that is, as meaning “including but not limited to”.

[0058] In the description of the present invention, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, "plurality" means two or more.

[0059] Figure 1 FIG. 1 is a flow chart of a video processing method according to an embodiment of the present invention. Figure 1 As shown, the video processing method of this embodiment includes the following steps:

[0060] In step S101, a video to be processed and a background image are obtained.

[0061] The video to be processed is a video that needs to be segmented and replaced with a background. The video to be processed can be a video stream generated in real time during a live broadcast or video conference, or a recorded video. This embodiment does not limit this. The background image is a virtual background image selected by the speaker. The speaker can select from the background images preset on the local end, or it can be uploaded by the speaker himself.

[0062] For example: After a teacher has finished recording a course video, he or she wants to replace the actual background in the course video with an uploaded picture related to the course video. At this time, the video to be processed is the recorded course video, and the background picture is the uploaded picture related to the course video selected by the teacher.

[0063] For example, a teacher is conducting a live teaching session. When explaining a certain knowledge point, he wants to replace the current actual background with a local image related to the knowledge point. At this time, the video to be processed is the data stream generated in real time during the recording process, and the background image is the local image related to the knowledge point selected by the teacher.

[0064] In step S102, a plurality of video frame images are obtained from the video to be processed.

[0065] Specifically, a video is composed of a plurality of still images, each still image is a frame of the video, and after the video to be processed is acquired, a plurality of video frame images can be obtained from the video to be processed.

[0066] For example, if the acquired video to be processed is a recorded course video, multiple course video frames can be obtained from the course video.

[0067] For another example: if the acquired video to be processed is a data stream generated in real time during the recording process, the data stream can be decoded first, and then a plurality of course video frames can be obtained from the decoded data stream.

[0068] In step S103, for each video frame image, the video frame image is input into a segmentation model to obtain a binary mask image corresponding to the video frame image.

[0069] Specifically, the obtained video frame images are respectively input into the segmentation model to obtain a binary mask image corresponding to each video frame image.

[0070] Figure 7 This is a flow chart of a video frame processing method according to an embodiment of the present invention, which is for the purpose of protecting the portrait of the person in the picture. Figure 7 The face of the person in the image is blurred. It should be understood that in actual applications, the portrait can be clear. Figure 7 As shown, the video frame image 71 is one of the multiple video frame images obtained. After the video frame image 71 is input into the segmentation model, a corresponding binary mask image 72 can be obtained.

[0071] Among them, the segmentation model is a model pre-trained by a neural network, which is used to segment the segmentation target from the actual background. The segmentation target is a portrait target in this embodiment. The segmentation model can obtain a binary mask image corresponding to each video frame image. The binary mask image is obtained after the segmentation model performs feature extraction and binarization processing on the video frame image. The grayscale value of the pixel point in the binary mask image is 0 or 255, which belongs to the portrait target and the background target in the video frame image respectively. Through the segmentation model, the portrait target and the background target can be extracted from the video frame image.

[0072] For example, the teacher portrait target in the course video frame is the segmentation target, and the rest is the background target. After the course video frame is input into the segmentation model, the segmentation model will perform feature extraction and binarization on the course video frame to obtain a binary mask corresponding to the course video frame. The binary mask consists of pixels with gray values ​​of 0 or 255, which belong to the teacher portrait target and the background target respectively.

[0073] In step S104, a synthetic image corresponding to each video frame image is generated according to the video frame image, the binary mask image and the background image.

[0074] Specifically, the video frame image, the binary mask image and the background image are synthesized to obtain a synthesized image corresponding to each video frame image, wherein the synthesized image is a synthesized image after portrait segmentation and background replacement are performed on the video frame image.

[0075] like Figure 7 As shown, based on the video frame image 71 , the binary mask image 72 and the background image 74 , a synthetic image 73 corresponding to the video frame image 71 is generated.

[0076] In another possible implementation, the image to be processed may be directly input into the segmentation model to obtain a binary mask image corresponding to the image to be processed, and then the image to be processed, the binary mask image and the background image are synthesized to obtain a synthesized image.

[0077] For example, after obtaining the binary mask image corresponding to the course video frame image, a composite image can be generated according to the uploaded image, the course video frame image and the binary mask image.

[0078] In step S105, a video is generated according to the composite image.

[0079] Specifically, after obtaining the composite images corresponding to each video frame image, the video can be regenerated according to the multiple composite images.

[0080] Optionally, the processing method may be to synthesize a plurality of synthetic images through a predetermined image processing program.

[0081] Optionally, if the video to be processed is video stream data generated in real time during a live broadcast, the synthesized video should also be encoded and then converted back into a video stream.

[0082] After obtaining the video to be processed and the background image, the embodiment of the present invention obtains multiple video frame images from the video to be processed, and then inputs each video frame image into the segmentation model respectively to obtain a binary mask image corresponding to each video frame image, and then generates a synthetic image corresponding to each video frame image according to the video frame image, the binary mask image and the background image, and finally regenerates the video according to the synthetic image. This embodiment performs portrait segmentation through the segmentation model, and can perform efficient, lightweight and high-quality portrait segmentation and background replacement on the video to be processed without being restricted by the recording scene.

[0083] Figure 2 FIG. 2 is another flow chart of a video processing method according to an embodiment of the present invention. Figure 2 As shown, the video processing method of this embodiment includes the following steps:

[0084] S201: Obtain the video to be processed and the background image.

[0085] S202: Extract the video to be processed frame by frame to obtain multiple video frame images in the video to be processed.

[0086] S203: Input the video frame image into the encoder part of the segmentation model to obtain an encoding feature extraction image.

[0087] S204: Input the encoded feature extraction image into the decoder part of the segmentation model to obtain the binary mask image.

[0088] S205: Generate a synthetic image corresponding to each video frame image according to the video frame image, the binary mask image and the background image.

[0089] S206: Generate a video based on the composite image

[0090] Steps S201 and S205-S206 can refer to the above Figure 1 The description shown is not repeated here.

[0091] For step S202, specifically, after obtaining the video to be processed, the video to be processed may be extracted frame by frame to obtain a plurality of video frame images in the video to be processed. Optionally, the video to be processed may be input into image processing software for frame by frame extraction.

[0092] For step S203, Figure 4 As shown, the segmentation model consists of an encoder part 41 and a decoder part 42, wherein the encoder part 41 is used to extract features of the input image by performing multiple downsampling operations on the input image. During the multiple downsampling operations, feature extraction of the input image from shallow to deep can be achieved.

[0093] Specifically, after the video frame image is input into the segmentation model, the video frame image will first be transmitted to the encoder part, and the encoder part will downsample the video frame image multiple times to extract features, thereby obtaining a coding feature extraction map, wherein the coding feature extraction map can reflect the depth features in the video frame image.

[0094] For example, after a 572*572*1 video frame image is input into the encoder part, the encoder part will perform deep feature extraction on the video frame image to obtain a 28*28*1024 encoding feature extraction image.

[0095] For step S204, Figure 4 As shown, the decoder part is used to restore the features of the coding feature extraction map. The feature restoration method is to upsample the coding feature extraction map multiple times. During the multiple upsampling operations, the features of the coding feature extraction map can be gradually restored.

[0096] Specifically, after obtaining the encoded feature extraction map, the segmentation model will transmit the encoded feature extraction map to the decoder part of the segmentation model. The decoder part will restore the features of the encoded feature extraction map and perform binarization processing to obtain a binary mask map.

[0097] For example, after a 28*28*1024 coded feature extraction image is input into the decoder part, the decoder part will perform feature restoration on the coded feature extraction image to obtain a 388*388*2 feature restoration image, and then perform binarization on the feature restoration image to obtain a binary mask image.

[0098] Alternatively, if Figure 4 As shown, in the process of restoring the features of the encoded feature extraction graph, the corresponding feature extraction graph of the decoder part can also be obtained to perform feature splicing.

[0099] After obtaining the video to be processed and the background image, the embodiment of the present invention obtains multiple video frame images from the video to be processed, and then inputs each video frame image into the segmentation model respectively to obtain a binary mask image corresponding to each video frame image, and then generates a synthetic image corresponding to each video frame image according to the video frame image, the binary mask image and the background image, and finally regenerates the video according to the synthetic image. This embodiment performs portrait segmentation through the segmentation model, and can perform efficient, lightweight and high-quality portrait segmentation and background replacement on the video to be processed without being restricted by the recording scene.

[0100] Figure 3 FIG. 2 is another flow chart of a video processing method according to an embodiment of the present invention. Figure 3 As shown, the video processing method of this embodiment includes the following steps:

[0101] S301: Obtain the video to be processed and the background image.

[0102] S302: Extract the video to be processed frame by frame to obtain multiple video frame images in the video to be processed.

[0103] S303: Input the video frame image into a first feature extraction module, a second feature extraction module, a third feature extraction module, a fourth feature extraction module and a fifth feature extraction module connected in series to obtain the encoding feature extraction image.

[0104] S304: Input the encoding feature extraction map into a first upsampling module, a second upsampling module, a third upsampling module, a fourth upsampling module and a fifth upsampling module connected in series to obtain a decoding feature restoration map.

[0105] S305: Input the decoded feature restoration image into a convolution prediction layer to obtain the binary mask image.

[0106] S306: performing normalization processing on the binary mask image to obtain a normalized binary mask image.

[0107] S307: Perform image operation processing on the video frame image, the normalized binary mask image and the background image to obtain the composite image.

[0108] S308: Generate a video according to the composite image.

[0109] Steps S301-S302 and S308 can be found in the above Figure 2 The description shown is not repeated here.

[0110] For step S303, Figure 4 As shown, the encoder part 41 of the segmentation model is composed of a first feature extraction module, a second feature extraction module, a third feature extraction module, a fourth feature extraction module and a fifth feature extraction module connected in series, and can perform feature extraction operations on the input image.

[0111] Specifically, after the video frame image is input into the segmentation model, the segmentation model will transmit the video frame image to the encoder part, and then input the first feature extraction module, the second feature extraction module, the third feature extraction module, the fourth feature extraction module and the fifth feature extraction module connected in series, perform multiple feature extractions, and obtain a coded feature extraction image. Among them, the second feature extraction module, the third feature extraction module, the fourth feature extraction module and the fifth feature extraction module respectively use the output of the previous feature extraction module as input. In this step, each time the video frame image passes through a feature extraction module, a downsampling operation is performed to extract features.

[0112] Optionally, the feature extraction modules in this embodiment all adopt the MobileNetV2 structure. Figure 5 FIG. 1 is a schematic diagram of the structure of a feature extraction module according to an embodiment of the present invention. Figure 5 As shown, the feature extraction module includes a 3*3 depthwise separable convolutional layer 51, a 1*1 convolutional layer 52, a 3*3 depthwise separable convolutional layer 53 with a stride of 2, and a 1*1 convolutional layer 54 which are sequentially connected.

[0113] The depthwise separable convolutional layer is a special convolutional layer that can perform convolution on each input channel separately during depthwise convolution to speed up the convolution.

[0114] Among them, the 3*3 depthwise separable convolution layer 51 is used to expand the number of channels of the input image, the 1*1 convolution layer 52, the 3*3 depthwise separable convolution layer 53 with a stride of 2, and the 1*1 convolution layer 54 can temporarily increase the number of channels of the input image, and then perform convolution and downsampling operations, and then reduce it back to the original number of channels.

[0115] For example, a 572*572*1 video frame is input into a 3*3 depthwise separable convolutional layer 51 to expand the number of channels and obtain a 570*570*64 feature map. Then, when the 570*570*64 feature map is input into a 1*1 convolutional layer 52, the number of channels of the feature map is temporarily increased to obtain a 570*570*128 feature map. Then, the 570*570*128 feature map is input into a 3*3 depthwise separable convolutional layer 53 with a stride of 2 to obtain a 284*284*128 feature map. Then, the 284*284*128 feature map is input into a 1*1 convolutional layer 54 to reduce the number of channels and obtain a 284*284*64 feature map.

[0116] Optionally, an activation layer may be set after the convolution layer, and the activation method may adopt a ReLu function.

[0117] For step S304, Figure 4 As shown, the decoder part 42 of the segmentation model is composed of a first upsampling module, a second upsampling module, a third upsampling module, a fourth upsampling module and a fifth upsampling module connected in series, wherein the upsampling module is used to restore the features of the input image, and the convolution prediction layer is used to perform binarization on the input image. In this step, the feature of the encoded feature extraction map is restored each time it passes through an upsampling module.

[0118] Specifically, after obtaining the encoding feature extraction map, the segmentation model transmits the encoding feature extraction map to the decoder part 42, and then inputs the first upsampling module, the second upsampling module, the third upsampling module, the fourth upsampling module and the fifth upsampling module connected in series to obtain a decoding feature restoration map. Among them, the second upsampling module, the third upsampling module, the fourth upsampling module and the fifth upsampling module respectively use the output of the previous upsampling module as input.

[0119] Optionally, the first upsampling module, the second upsampling module, the third upsampling module, the fourth upsampling module and the fifth upsampling module all include a feature restoration module and a transposed convolution layer. The feature restoration module is used to perform feature restoration on the input image, and the transposed convolution layer is used to perform an upsampling operation on the input image.

[0120] Optionally, the feature restoration modules in this embodiment all adopt a residual structure. Figure 6 FIG. 4 is a schematic diagram of the composition of an upsampling module according to an embodiment of the present invention. Figure 6 As shown, the upsampling module includes a transposed convolution layer 61 and a feature restoration module 62, wherein the feature restoration module 62 includes a skip connection convolution layer 625 and a sequentially connected 3*3 depth-separable convolution layer 621, a 1*1 convolution layer 622, a 3*3 depth-separable convolution layer 623 and a 1*1 convolution layer 624.

[0121] Among them, the 3*3 depthwise separable convolution layer 621 is used to reduce the number of channels of the input image, and the 1*1 convolution layer 622, the 3*3 depthwise separable convolution layer 623 and the 1*1 convolution layer 624 can temporarily reduce the number of channels of the input image, and then perform convolution, and then restore the original number of channels. The input of the skip connection convolution layer 625 is the feature extraction map passed by the encoder part.

[0122] For example, when a 28*28*1024 coded feature extraction map is input to the upsampling module, the upsampling module will transfer the 28*28*1024 coded feature extraction map to the 3*3 depth-separable convolution layer 621, reduce the number of channels to obtain a 26*26*512 feature map, and then input the 26*26*512 feature map into the 1*1 convolution layer 622 to temporarily reduce the number of channels to obtain a 26*26*256 feature map, and then input the 26*26*256 feature map into the 3*3 depth-separable convolution layer 623 for The row convolution obtains a 24*24*256 feature map, and then the 24*24*256 feature map is input into the 1*1 convolution layer 624 to restore the number of channels to obtain a 24*24*512 feature map. At this time, the feature extraction map 24*24*512 corresponding to the encoder part connected by the jump connection is convolved through the jump connection convolution layer 625 and superimposed with the obtained 24*24*512 feature map to obtain a 24*24*1024 feature map, which is then upsampled to obtain a 48*48*512 feature map.

[0123] Optionally, an activation layer may be set after the convolution layer, and the activation method may adopt a ReLu function.

[0124] Optionally, the superimposed feature maps are normalized before being superimposed.

[0125] For step S305, after obtaining the decoded feature restoration map, the decoded feature restoration map is input into the convolution prediction layer to obtain a binary mask map.

[0126] Optionally, the convolution prediction layer can traverse each pixel in the decoded feature restoration image through a Sigmoid function to obtain the confidence of each pixel, wherein the confidence range is limited between 0 and 1, and the confidence of each pixel is proportional to the probability that the pixel belongs to the segmentation target. The confidence of each pixel is judged, and the confidence of the pixel whose confidence is greater than a certain threshold (for example, 0.5) is reset to 1, and the confidence of the pixel whose confidence is less than 0.5 is reset to 0, and then the reset confidence of each pixel is multiplied by 255 and used as the pixel value of each pixel to obtain a binary mask image.

[0127] For step S306, specifically, after the binary mask image is obtained, the binary mask image is normalized.

[0128] For step S307, specifically, image operation processing is performed on the video frame image, the normalized binary mask image, and the background image according to the following formula to obtain the composite image I:

[0129] I=αF+(1-α)B

[0130] Among them, α is the normalized mask image, F is the video frame image, and B is the background image.

[0131] The segmentation model in this embodiment is obtained by training based on a joint loss function of binary cross entropy and focal-loss, wherein the binary cross entropy can calculate the mask loss, and the focal-loss can calculate the boundary loss.

[0132] Specifically, the joint loss function L is calculated according to the following formula:

[0133] L=L m +λ*L e

[0134]

[0135]

[0136] Among them, n represents the total number of pixels in an image, m represents the total number of pixels at the image boundary, yi represents the label of the i-th pixel in an image, xi represents the label of the i-th pixel at the image boundary, ρi is the predicted probability that the i-th pixel belongs to the segmentation target, γ is a predetermined adjustment factor, and λ is a predetermined loss weight parameter.

[0137] Optionally, the image boundary can be obtained by processing the decoded feature restoration map using the canny edge detection algorithm in the convolution prediction layer.

[0138] The embodiment of the present invention is Figure 2 On the basis of the method shown, the video frame image is processed by a feature extraction module including a MobileNetV2 structure, a feature restoration module including a residual structure and a transposed convolution layer, and a convolution prediction layer, thereby achieving efficient and lightweight portrait segmentation. At the same time, this embodiment also uses a joint function to train the segmentation model to improve the accuracy of portrait segmentation.

[0139] Figure 8 Schematic diagram of an electronic device according to an embodiment of the present invention. Figure 8As shown, the electronic device is a general data processing device, which includes a general computer hardware structure, which at least includes a processor 81 and a memory 82. The processor 81 and the memory 82 are connected via a bus 83. The memory 82 is suitable for storing instructions or programs executable by the processor 81. The processor 81 can be an independent microprocessor or a collection of one or more microprocessors. Thus, the processor 81 executes the instructions stored in the memory 82, thereby executing the method flow of the embodiment of the present invention as described above to realize the processing of data and the control of other devices. The bus 83 connects the above-mentioned multiple components together, and at the same time connects the above-mentioned components to the display controller 84 and the display device and the input / output (I / O) device 85. The input / output (I / O) device 85 can be a mouse, a keyboard, a modem, a network interface, a touch input device, a somatosensory input device, a printer, and other devices known in the art. Typically, the input / output device 85 is connected to the system via an input / output (I / O) controller 86.

[0140] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices (equipment) or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may adopt a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0141] The present application is described with reference to flowcharts of methods, apparatuses (devices) and computer program products according to embodiments of the present application. It should be understood that each process in the flowchart can be implemented by computer program instructions.

[0142] These computer program instructions may be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device that implements the process Figure 1 A function specified in a process or multiple processes.

[0143] These computer program instructions may also be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce the instructions for implementing the process Figure 1 A device that specifies functions in a process or multiple processes.

[0144] Another embodiment of the present invention relates to a non-volatile storage medium for storing a computer-readable program, wherein the computer-readable program is used for a computer to execute part or all of the above method embodiments.

[0145] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by specifying relevant hardware through a program, and the program is stored in a storage medium, including several instructions for a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0146] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A video processing method, characterized in that: The method comprises: obtaining a video to be processed and a background image; Obtaining multiple video frame images from the video to be processed; For each video frame image, input the video frame image into a segmentation model to obtain a binary mask image corresponding to the video frame image; Generate a synthetic image corresponding to each video frame image according to the video frame image, the binary mask image and the background image; generating a video according to the composite image; The step of inputting the video frame image into a segmentation model to obtain a binary mask image corresponding to the video frame image includes: Inputting the video frame image into the encoder part of the segmentation model to perform deep feature extraction to obtain a coding feature extraction image; Input the encoded feature extraction map into the decoder part of the segmentation model, obtain the corresponding feature extraction map of the decoder part for feature splicing, so as to perform feature restoration and binarization processing, and obtain the binarized mask map; The decoder part includes an upsampling module and a convolution prediction layer, the convolution prediction layer is used to binarize the input image, the upsampling module includes a transposed convolution layer and a feature restoration module; the feature restoration module includes a skip connection convolution layer and a sequentially connected 3*3 depth separable convolution layer, a 1*1 convolution layer, a 3*3 depth separable convolution layer and a 1*1 convolution layer; the 3*3 depth separable convolution layer is used to reduce the number of channels of the input image, the 1*1 convolution layer, the 3*3 depth separable convolution layer and the 1*1 convolution layer temporarily reduce the number of channels of the input image, and then perform convolution, and then restore the original number of channels, and the input of the skip connection convolution layer is the feature extraction map passed by the encoder part; The segmentation model is obtained by training based on the joint loss function L of binary cross entropy and focal-loss; wherein the joint loss function L is calculated according to the following formula: L=L m +λ*L e Among them, n represents the total number of pixels in an image, m represents the total number of pixels at the image boundary, yi represents the label of the i-th pixel in an image, xi represents the label of the i-th pixel at the image boundary, ρi is the predicted probability that the i-th pixel belongs to the segmentation target, γ is the adjustment factor, and λ is the loss weight parameter.

2. The method according to claim 1, characterized in that The step of obtaining a plurality of video frame images from the video to be processed comprises: The video to be processed is extracted frame by frame to obtain multiple video frame images in the video to be processed.

3. The method according to claim 1, characterized in that The step of inputting the video frame graph into the encoder part of the segmentation model to obtain the encoded feature extraction graph comprises: Inputting the video frame graph into a first feature extraction module, a second feature extraction module, a third feature extraction module, a fourth feature extraction module and a fifth feature extraction module connected in series to obtain the coding feature extraction graph; The second feature extraction module, the third feature extraction module, the fourth feature extraction module and the fifth feature extraction module respectively use the output of the previous feature extraction module as input.

4. The method according to claim 3, characterized in that The first feature extraction module, the second feature extraction module, the third feature extraction module, the fourth feature extraction module and the fifth feature extraction module all include a MobileNetV2 structure.

5. The method according to claim 4, characterized in that The MobileNetV2 structure includes a sequentially connected 3*3 depth-wise separable convolutional layer, a 1*1 convolutional layer, a 3*3 depth-wise separable convolutional layer with a stride of 2, and a 1*1 convolutional layer.

6. The method according to claim 4, characterized in that The step of inputting the encoded feature extraction map into the decoder part of the segmentation model to obtain the binary mask map comprises: Inputting the encoding feature extraction map into a first upsampling module, a second upsampling module, a third upsampling module, a fourth upsampling module and a fifth upsampling module connected in series to obtain a decoding feature restoration map; Inputting the decoded feature restoration image into the convolution prediction layer to obtain the binary mask image; The second up-sampling module, the third up-sampling module, the fourth up-sampling module and the fifth up-sampling module respectively use the output of the previous up-sampling module as input.

7. The method according to claim 6, characterized in that The first upsampling module, the second upsampling module, the third upsampling module, the fourth upsampling module and the fifth upsampling module all include a feature restoration module and a transposed convolution layer; Wherein, the feature restoration modules all adopt a residual structure.

8. The method according to claim 7, characterized in that The step of inputting the decoded feature restoration image into a convolution prediction layer to obtain the binary mask image comprises: The decoded feature restoration image is binarized to obtain the binarized mask image.

9. The method according to claim 8, characterized in that The image boundary label is the output obtained by processing the decoded feature restoration image using the canny edge detection algorithm.

10. The method according to claim 1, characterized in that The step of generating a synthetic image corresponding to each video frame image according to the video frame image, the binary mask image and the background image comprises: Normalizing the binary mask image to obtain a normalized binary mask image; The video frame image, the normalized binary mask image and the background image are processed by image operation according to the following formula to obtain the composite image I: I=αF+(1-α)B Among them, α is the normalized mask image, F is the video frame image, and B is the background image.

11. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1-10.

12. A computer-readable storage medium for storing computer program instructions, characterized in that: The computer program instructions implement the method according to any one of claims 1 to 10 when executed by a processor.