A method for image cutout

Abstract

The invention discloses an image matting method, which relates to the field of image processing technology. The method comprises: obtaining a video frame image; inputting the video frame image into a matting model, extracting RGB features and frequency features and fusing them to obtain a fused feature; performing local attention enhancement on the fused feature to obtain a fine semantic feature; splicing the fused feature with the fine semantic feature to obtain a semantic feature; processing the semantic feature to obtain a semantic map; extracting the detail features of the fused feature, and then compensating for the underlying texture information lost by the detail feature to obtain a detail map; fusing the semantic map and the detail map to obtain a transparent map Alpha matte; separating the foreground and background in the video frame image according to the transparent map Alpha matte to obtain a matted image. The invention effectively avoids the error accumulation problem through a single-image double-branch design, so that the matting result is closer to the real scene.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image matting method. Background Art

[0002] In recent years, with the rapid development of artificial intelligence technology and computer vision, the field of education is undergoing a transformation in the information age. The application of artificial intelligence technology in the field of education is becoming more and more extensive, covering aspects such as precision teaching, intelligent classroom assessment, personalized learning and educational games. In this context, it is an inevitable need to deeply integrate artificial intelligence technology with the field of educational applications. There is a long-standing problem to be solved in distance education, that is, how to enhance the interactivity and immersion of teaching. Cutting technology can cut out the target object from the original video or image, and embed the target object into various teaching scenes or virtual environments, which helps to enhance the interactivity and immersion of the course and improve students' learning interest and participation.

[0003] Existing matting methods divide the matting process into the semantic segmentation stage and the matting stage; in the semantic segmentation stage, deep learning technology is used to analyze the content of the input image, automatically distinguish and generate mask trimaps for the foreground, background and transition areas; in the matting stage, the alpha value of the transition area is further predicted and refined based on the mask trimap generated in the segmentation stage; the alpha value is an important indicator to measure the transparency of each pixel in the image, and its accurate prediction is directly related to the quality of the final matting result.

[0004] The defect of the above-mentioned prior art is that if the segmentation stage outputs an erroneous mask, the cutout stage cannot repair the erroneous mask, resulting in error accumulation, thereby making the cutout result inaccurate and incomplete. Summary of the invention

[0005] Based on this, it is necessary to provide an image cutout method for the above technical problems.

[0006] An embodiment of the present invention provides an image cutout method, comprising:

[0007] Acquire video frame images of the teacher and each student during the class process in the remote teaching environment;

[0008] A dual-branch structure including a dual-stream detail prediction branch and an attention semantic estimation branch is constructed, and a feature extractor HRNet48, a discrete cosine transform module, and a domain aggregation module are added before the input of the dual-branch structure; the output of the feature extractor HRNet48 and the discrete cosine transform module are connected to the input of the domain aggregation module, and a spatial pyramid pooling module SPPELAN is added between the output of the domain aggregation module and the attention semantic estimation branch; a cross-regional attention step-by-step refinement module is added to the output of the dual-branch structure, and a fusion module Fusion is added between the attention semantic estimation branch and the cross-regional attention step-by-step refinement module to construct a cutout model;

[0009] The video frame image is input into the matting model, and the RGB features of the video frame image are extracted by the feature extractor HRNet48, and the frequency features of the video frame image are extracted by the discrete cosine transform module; the RGB features and the frequency features are fused by the domain aggregation module to obtain the fused features; the fused features are locally enhanced by the spatial pyramid pooling module SPPELAN to obtain fine semantic features; the fused features are spliced ​​with the fine semantic features to obtain semantic features; the semantic features are processed by multiple convolutions and Softmax activation functions by the attention semantic estimation branch to obtain the semantic map of the video frame image; the detail features of the fused features are extracted by the dual-stream detail prediction branch, and the underlying texture information lost by the detail features is compensated by the cross-regional attention step-by-step refinement module to obtain the detail map of the video frame image; the semantic map and the detail map are fused by the fusion module Fusion to obtain the transparent map Alpha matte; the foreground and background in the video frame image are separated according to the transparent map Alpha matte to obtain the matted image.

[0010] Optionally, video data of the teacher-student distance learning classroom process is obtained, specifically including:

[0011] Get video data of the teacher and each student during the class in a remote teaching environment V ;

[0012] For video data V Perform multi-threaded extraction to obtain video frame images ,in Indicates the total number of video frame images.

[0013] Optionally, the RGB features of the video frame image are extracted by the feature extractor HRNet48, and the expression is:

[0014] ;

[0015] in, I is the video frame image, is the RGB feature, is a feature extractor;

[0016] The frequency characteristics of the video frame image are extracted through the discrete cosine transform module, including:

[0017] Divide the input video frame image into three color channels: R, G, and B;

[0018] The RGB features Divide into a set of 8×8 patches, and get ;

[0019] For each block of the color channel Perform discrete cosine transform to obtain the channel spectrum ;

[0020] By concatenating each channel spectrum , get the image spectrum ;

[0021] Image spectrum Perform flattening and reshape operations to obtain the frequency characteristics of the video frame image .

[0022] Optionally, the RGB features and the frequency features are fused through a domain aggregation module, which specifically includes:

[0023] Using binary basis filtering and convolutional modules Conv Adjust the frequency characteristics of the channel for cascading to obtain a frequency domain matrix ;

[0024] Feed the RGB features to another Sigmoid Convolutional module with activation function and two output channels Conv , get the RGB domain matrix ;

[0025] The RGB domain matrix and frequency domain matrix and the learning vector Multiply to get the RGB domain aggregation feature and frequency domain aggregation features , and its calculation formula is:

[0026] ;

[0027] ;

[0028] Add the RGB domain aggregation features and the frequency domain aggregation features to obtain the fusion features , the formula is:

[0029] ;

[0030] in, is the RGB feature, is the frequency characteristic, is the RGB domain aggregation feature, is the frequency domain aggregation feature.

[0031] Optionally, obtaining a semantic graph of the video frame image specifically includes:

[0032] The fusion features Input the spatial pyramid pooling module SPPELAN to obtain fine semantic features P ;

[0033] The fusion features and fine semantic features P Splice to get semantic features , the formula is;

[0034] ;

[0035] ;

[0036] in, is a fine semantic feature. is the convolution and Upsampling, To fusion features, For channel-level splicing, It is a convolutional neural network composed of 3×3 convolution, batch normalization BN and rectified linear unit ReLU. DAS It is a deformable attention mechanism;

[0037] The last layer output features Input Softmax activation function to get semantic graph , the formula is:

[0038] ;

[0039] in, is the Softmax activation function.

[0040] Optionally, the detail features of the fused features are extracted through the dual-stream detail prediction branch, which specifically includes:

[0041] The fusion features Input the high-resolution feature extraction stream in the dual-stream detail prediction branch to obtain high-resolution detail features , the formula is:

[0042] ;

[0043] ;

[0044] in, is the enhanced high-resolution feature of the previous layer, is the residual group structure, is a 1×1 convolutional layer, i is the number of treatments;

[0045] The original image Input the shallow feature extraction stream in the dual-stream detail prediction branch and execute 4 times ConvBNReLU And bilinear upsampling to obtain shallow detail features , the formula is:

[0046] ;

[0047] ;

[0048] in, It is a convolutional neural network composed of 3×3 convolution, batch normalization BN and rectified linear unit ReLU. is the original image, is bilinear upsampling;

[0049] High-resolution detail features and shallow detail features Perform channel-level splicing and fuse through the convolutional neural network ConvBNReLU to obtain detailed fusion features , the formula is:

[0050] ;

[0051] Semantic features and detail fusion features Splicing is performed, and convolution, normalization and sigmoid function are performed in sequence to obtain auxiliary feature maps , the formula is:

[0052] ;

[0053] Generate detail feature map , the formula is:

[0054] ;

[0055] in, For auxiliary flow, is the element-by-element product, T is the transpose, is the channel weighted kernel;

[0056] The detail feature map output by the last layer As detail feature D_fea.

[0057] Optionally, the underlying texture information lost by the detail features is compensated by a cross-region attention step-by-step refinement module, specifically including:

[0058] Identifying semantic graphs Areas where texture information is lost in the middle and bottom layers;

[0059] The K-nearest neighbor algorithm KNN is used to locate the closest area KNNs of each area where the underlying texture information is lost, and the KNNs are obtained from the lookup table Query the relative position deviation in

[0060] set up For Region Features, For Region i of neighbors, for each region i , the original image I and the semantic map processed by the cross-region attention step-by-step refinement module Stitching to get low-level texture features , the formula is:

[0061] ;

[0062] ;

[0063] in, is the original image, Gradually refine the semantic graph processed by the module for cross-region attention With the original image Perform channel-level splicing. is bilinear upsampling, It is a convolutional neural network composed of 3×3 convolution, batch normalization BN and rectified linear unit ReLU;

[0064] The output feature map D_fea of ​​the two-stream detail prediction branch and the low-level texture features Send to fusion module , get the detailed features , the formula is:

[0065] ;

[0066] in, The detail feature D_fea output by the dual-stream detail prediction branch;

[0067] Fusion Module The previous layer of detail features after processing and Fusion Module The processed detail features of the current layer Input the progressive refinement module to fuse the output of the previous layer and the current layer to obtain the final layer of features ;

[0068] The last layer of features As the final refined output feature, the Sigmoid activation function is input to obtain the final detail map , the formula is:

[0069] ;

[0070] in, is the Sigmoid activation function.

[0071] Optionally, the semantic map and the detail map are fused through a fusion module, which specifically includes:

[0072] By taking the maximum index value of the video frame image, the semantic map is converted from S_map Mapping to index map ;

[0073] Determine the position of the pixel in the video frame image by the value in the index map;

[0074] All pixels are filtered and separated to obtain two index maps, which represent the index of the foreground area and the index of the transition area respectively;

[0075] The index map of the transition area With details Perform a dot product operation to map the foreground index to the foreground image;

[0076] Add the foreground map and the dot-producted image to generate the final transparent image Alpha matte, the formula is:

[0077] ;

[0078] in, is the feature map with index 1, For detail pictures, For pure foreground image, is the set of pixels belonging to the transition region, is the set of pixels belonging to the foreground area.

[0079] Optionally, determining the position of a pixel in the video frame image by using the value in the index map specifically includes:

[0080] When index=0 in the index map, the pixel in the video frame image belongs to the foreground area;

[0081] When index=1 in the index map, the pixel in the video frame image belongs to the transition area.

[0082] Compared with the prior art, the image cutout method provided by the embodiment of the present invention has the following beneficial effects:

[0083] The present invention proposes a cutout model, which is based on a dual-branch structure consisting of a dual-stream detail prediction branch and an attention semantic estimation branch. The single-image dual-branch design effectively avoids the problem of error accumulation. A cross-regional attention step-by-step refinement module is added to the output of the dual-branch structure. The cross-regional attention step-by-step refinement module accurately captures and restores the underlying texture information lost due to detail features through step-by-step iteration and refinement processing. Whether it is subtle textures or complex backgrounds in the image, they can be accurately and naturally presented, making the cutout results closer to the real scene. This improvement not only improves the practicality of the cutout technology, but also opens up a broader space for innovative applications in image editing, synthesis and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 A method flow chart of an image cutout method provided in one embodiment;

[0085] Figure 2 A model structure diagram of an image cutout method provided in an embodiment;

[0086] Figure 3 A schematic diagram of feature fusion of an image matting method provided in an embodiment;

[0087] Figure 4 FIG. 1 is a schematic diagram of a distillation process of an image cutout method provided in an embodiment. Figure 4 (a) is a schematic diagram of the optimal teacher distillation process. Figure 4 (b) is a schematic diagram of the dense distillation process. Figure 4 (c) is a schematic diagram of the transfer distillation process. DETAILED DESCRIPTION

[0088] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0089] In one embodiment, a method for cutting out an image is provided, such as Figure 1 As shown, the method includes:

[0090] 1. Main assumptions

[0091] In a distance learning environment, each student is an individual. Video data of the teacher and each student is obtained during the classroom process in the distance learning environment. V . For video data V Perform multi-threaded extraction to obtain video frame images ,in Indicates the total number of video frame images.

[0092] Data is the basis of driving model analysis. In a classroom environment, it is usually necessary to read camera videos or perform inferences from video files to extract frames from the video to obtain data. However, for larger models, the inference speed often cannot keep up with the frame rate of the camera or video. If a single-threaded processing method is adopted, that is, detection is performed after reading a frame, the inference process will block the normal transmission of the video, resulting in a significant delay in the camera video. For applications with high real-time requirements, this delay is unacceptable. Therefore, the present invention adopts a multi-threaded method to place video reading and deep learning inference in two threads to achieve independent and real-time effects.

[0093] Next, the background of the target person in the video frame image is separated and cut out.

[0094] In order to ensure that the model not only has a strong perception of local tiny details but also can accurately distinguish the foreground boundary, the present invention uses RGB stream and frequency stream to extract features of the image respectively. The extracted RGB features and frequency features are used respectively. and express.

[0095] At the same time, in order to make the distance education interaction more realistic, a more refined background separation and cutout is required. Therefore, a dual-branch structure is used, which is a dual-stream detail prediction branch. The output features are , and the attention semantic estimation branch, the output feature is and semantic graph .

[0096] In addition, a cross-region attention step-by-step refinement module is designed to capture contextual information across adjacent regions and make up for the low-level texture information lost upstream to generate detailed maps. , the semantic map and the detail map are fused to generate a transparent map Alpha matte. Finally, the foreground of the teacher and each student is separated from the background according to the generated transparent map Alpha matte, and different scenes are set according to different distance education classes to achieve the effect of enriching remote interaction and increasing the fun of remote classes.

[0097] 2. Establishment of background separation and cutout model

[0098] 2.1 Model Structure

[0099] A dual-branch structure including a dual-stream detail prediction branch and an attention semantic estimation branch is constructed, and a feature extractor HRNet48, a discrete cosine transform module and a domain aggregation module are added before the input of the dual-branch structure; the output of the feature extractor HRNet48 and the discrete cosine transform module are connected to the input of the domain aggregation module, and a spatial pyramid pooling module SPPELAN is added between the output of the domain aggregation module and the attention semantic estimation branch; a cross-regional attention step-by-step refinement module is added to the output of the dual-branch structure, and a fusion module Fusion is added between the attention semantic estimation branch and the cross-regional attention step-by-step refinement module to construct a cutout model.

[0100] 2.2 Model workflow

[0101] like Figure 2 As shown, the video frame image is input into the cutout model, the RGB features of the video frame image are extracted by the feature extractor HRNet48, and the frequency features of the video frame image are extracted by the discrete cosine transform module.

[0102] The RGB features and frequency features are fused through the domain aggregation module to obtain fused features. The fused features are locally enhanced through the spatial pyramid pooling module SPPELAN to obtain fine semantic features.

[0103] The fusion features are spliced ​​with the fine semantic features to obtain the semantic features. The semantic features are processed by multiple convolutions and Softmax activation functions through the attention semantic estimation branch to obtain the semantic map of the video frame image. The detail features of the fusion features are extracted through the dual-stream detail prediction branch, and the underlying texture information lost by the detail features is compensated through the cross-region attention step-by-step refinement module to obtain the detail map of the video frame image.

[0104] The semantic map and the detail map are fused through the fusion module to obtain the transparent map Alpha matte.

[0105] The foreground and background in the video frame image are separated according to the transparent image Alpha matte to obtain the cutout image.

[0106] 2.2.1 Feature extraction of video frame images

[0107] Different from previous studies that only rely on RGB information for feature extraction, which limits the model's perception of local tiny details, this paper believes that the matching problem should not be limited to replicating human visual perception in a single RGB domain, but should combine additional frequency information to obtain better recognition capabilities. The specific operations are as follows:

[0108] In the first step, the video frame image is processed by two data streams, namely the RGB stream and the frequency stream.

[0109] For the RGB stream, the RGB features of the video frame image are extracted by the feature extractor HRNet48, and its expression is:

[0110] ;

[0111] in, I is the video frame image, is the RGB feature, is the feature extractor.

[0112] For the frequency stream, the frequency features of the video frame image are extracted through the discrete cosine transform module, including:

[0113] First, the input video frame image is divided into three color channels: R, G, and B, and then the RGB feature Divide into a set of 8×8 patches, and get .

[0114] Finally, for each block of a color channel Perform discrete cosine transform to get the spectrum After the above operation, each value corresponds to the intensity of a certain frequency band.

[0115] To combine all components of the same frequency into one channel, first concatenate each channel To obtain Then, the spectrum Perform flattening and reshape operations to obtain the frequency characteristics of the video frame image In this way, the signal is rearranged in a zigzag order within one patch, and the frequency characteristics Each channel of belongs to a frequency band. Therefore, the original RGB input is transformed into the frequency domain.

[0116] In the second step, the RGB features and frequency features have been obtained through the HRNet48 and discrete cosine transform modules respectively.

[0117] However, aggregating features from two different domains remains a challenge. To this end, the present invention designs a domain aggregation module to fuse these features, such as Figure 3 As shown in Figure 2. Feature aggregation is a complementary process, where frequency features are discriminative for local details, while RGB features have a larger receptive field to perceive global semantics. Since CNN is more sensitive to low-frequency channels, a filter is first applied to manually extract high-frequency channels. For frequency features , the network can automatically focus on the most important spectrum.

[0118] Specifically, the present invention uses binary basis filtering and convolutional modules Conv Adjust the frequency characteristics of the channel for cascading to obtain a frequency domain matrix Then, the RGB features are fed into another Sigmoid Convolutional module with activation function and two output channels Conv , get the RGB domain matrix Finally, the RGB domain matrix and frequency domain matrix and the learning vector Multiply to get the RGB domain aggregation feature and frequency domain aggregation features The calculation formula is:

[0119] ;

[0120] ;

[0121] The third step is to add the RGB domain aggregation features and the frequency domain aggregation features to obtain the fusion features. , the formula is:

[0122] ;

[0123] in, is the RGB feature, is the frequency characteristic, is the RGB domain aggregation feature, is the frequency domain aggregation feature.

[0124] In this way, the semantic information can be maintained while fully utilizing the distinguishing frequency information to ensure that both the integrity and details of the foreground are protected.

[0125] 2.2.2 Double branch structure

[0126] Different from previous work that uses a two-stage model to separate the target person from the background, which will lead to the problem of cumulative error, this invention uses a two-branch method to solve this problem. The specific operation is as follows:

[0127] The first step is to fusion the features obtained previously Add the spatial pyramid pooling module SPPELAN (Spatial Pyramid Pooling Enhanced with Efficient Local Aggregation Network) to obtain fine semantic features .

[0128] Specifically, the spatial pyramid pooling module SPPELAN combines SPP with the local attention mechanism, and assigns different weights to different regions by calculating the correlation between feature maps, thereby achieving local attention enhancement. And the fine semantic features output by SPPELAN The concatenation on the channel is performed and passed to the attention semantic estimation branch to extract the semantic map.

[0129] Specifically, the attention semantic estimation branch consists of five blocks, each of which consists of three ConvBNReLU (convolution, batch normalization (BN), and rectified linear unit (ReLU)) and one bilinear upsampling.

[0130] In addition, in order to improve the ability of the attention semantic estimation branch to learn discriminative semantic features, the present invention also builds an enhanced deformable attention mechanism DAS (Deformable Attention Salient) after each block to help focus attention on relevant information. DAS attention integrates deep separable convolution (DSC) and deformable convolution (DC) to focus and increase attention on salient areas and calculate dense attention (pixel direction) weights. Deformable convolution is used to locate relevant image areas, and separable convolution is used to improve efficiency.

[0131] The fusion features and fine semantic features P Splice to get semantic features , its specific operation can be expressed by the following formula:

[0132] ;

[0133] ;

[0134] in, is a fine semantic feature. is the convolution and Upsampling, To fusion features, For channel-level splicing, It is a convolutional neural network composed of 3×3 convolution, batch normalization BN and rectified linear unit ReLU. DAS It is a deformable attention mechanism.

[0135] Finally, the last layer outputs the features Input Softmax activation function to get semantic graph , the formula is as follows:

[0136] ;

[0137] in, is the Softmax activation function.

[0138] In the second step, in order to obtain a more detailed foreground, such as the hair of a student, the present invention designs a dual-stream detail prediction branch to capture more accurate detail information of the student.

[0139] Specifically, the two-stream detail prediction branch consists of a two-stream structure of a high-resolution feature extraction stream and a shallow feature extraction stream. Since the underlying texture information usually exists in the high-resolution feature map, however, the downsampling-upsampling in the existing encoder-decoder network architecture inevitably loses the high-resolution features.

[0140] Therefore, the present invention constructs a high-resolution feature extraction flow to obtain rich spatial features and contour boundary features in the high-resolution geometric cache, which is exactly what is needed for cutout. The initial input of the high-resolution feature extraction flow is the fusion feature extracted upstream. The fusion features Input the high-resolution feature extraction stream in the two-stream detail prediction branch to obtain high-resolution detail features Specifically, the high-resolution feature extraction flow consists of four blocks, each of which consists of a residual group and a convolutional layer. The residual group consists of three residual blocks, which can be expressed by the following formula:

[0141] ;

[0142] ;

[0143] in, is the enhanced high-resolution feature of the previous layer, is the residual group structure, is a 1×1 convolutional layer, i is the number of processing times.

[0144] Since the detail features in an image are usually low-level features, in order to better obtain shallow feature information, the present invention introduces the original image into the dual-stream detail prediction branch. While the high-resolution feature extraction stream extracts features, the shallow feature extraction stream retains the feature extraction of the original image to obtain finer-grained feature information.

[0145] Specifically, the shallow feature extraction flow consists of four blocks, each of which consists of a convolutional neural network ConvBNReLU and a bilinear upsampling UP. Input the shallow feature extraction stream in the dual-stream detail prediction branch and execute 4 times ConvBNReLU And bilinear upsampling to obtain shallow detail features , the formula is as follows:

[0146] ;

[0147] ;

[0148] in, It is a convolutional neural network composed of 3×3 convolution, batch normalization BN and rectified linear unit ReLU. is the original image, is bilinear upsampling.

[0149] High-resolution detail features and shallow detail features Perform channel-level splicing and fuse through the convolutional neural network ConvBNReLU to obtain detailed fusion features , the formula is as follows:

[0150] ;

[0151] in, Will be used as the output feature of the two-stream detail prediction branch Input into the cross-region attention progressive refinement module.

[0152] In the third step, in order to help the dual-stream detail prediction branch focus on the detail features of the transition area, the present invention designs an auxiliary module to guide the attention semantic estimation branch to assist the dual-stream detail prediction branch to strengthen the prediction of details.

[0153] In order to obtain more accurate semantic context features, a gated convolutional layer (GCL) is used to help the auxiliary stream enhance boundary-related information and filter out irrelevant information.

[0154] Specifically, the semantic features and detail fusion features Splicing is performed, and convolution, normalization and sigmoid function are performed in sequence to obtain auxiliary feature maps , the formula is:

[0155] ;

[0156] Generate detail feature map , the formula is as follows:

[0157] ;

[0158] in, Represents auxiliary stream, represents element-wise product, T represents transpose, Represents the channel weighted kernel. Specifically, this process is to multiply the shape feature obtained in the previous layer by the obtained attention feature, add the shape distribution, transpose it, and multiply it by the channel weight kernel.

[0159] The detail feature map output by the last layer As detail feature D_fea.

[0160] In this paper, attention semantics is used to estimate the branch generation (1 / 16), (1 / 4) and The intermediate features of (1 / 1) are fed into the auxiliary module to generate an auxiliary graph G to guide the two-stream detail prediction branch to form a detail graph.

[0161] 2.2.3 Cross-region attention gradual refinement of modules

[0162] After extracting features in the upstream stage, context and low-level texture information are inevitably lost. Therefore, a mechanism is needed to restore the context and low-level texture information after region extraction. This paper proposes a new cross-region attention step-by-step refinement module to capture the context information across adjacent regions and compensate for the underlying texture information lost by detail features to obtain a more refined detail map. It is divided into three steps:

[0163] In the first step, the present invention uses cross-region attention to capture contextual information across adjacent regions. Specifically, cross-region attention includes the K-nearest neighbors algorithm KNN (K-Nearest Neighbors) and the calculation of relative position deviation. The area where the bottom-level texture information is lost is located by the K-nearest neighbor algorithm KNN, and the closest area KNNs (under the metric of Euclidean distance) of each area where the bottom-level texture information is lost are located, and the KNNs are obtained from the lookup table Query the relative position deviation in .

[0164] The closest region KNNs may be scattered anywhere in the central region and distributed on an irregular grid, so a method is needed to encode their relative positions. The relative position on each axis should be within This ultimately results in a search range of possible relative positions, using a learnable lookup table Encodes relative position. The first entry encodes all possible in-range positions. The last entry encodes any out-of-range positions.

[0165] Specifically, after obtaining the detailed features of all extracted regions from the upstream decoder, the semantic map is identified. The K-nearest neighbor algorithm (KNN) is used to locate the closest KNNs of each area where the underlying texture information is lost, and the KNNs are obtained from the lookup table. Query the relative position deviation in .

[0166] In the second step, in order to capture accurate details more effectively, set Indicates area Features, Indicates area i of neighbors. For each region i , the features of the region and its KNNs Together with their relative position deviation The semantic graph processed by the cross-region attention step-by-step refinement module is defined as .

[0167] Specifically, the step-by-step refinement module consists of a convolutional stream and a progressive refinement block. The convolutional stream consists of a series of simple ConvBNReLU layers. The original image I and the semantic map processed by the cross-region attention step-by-step refinement module are combined. Stitching to get low-level texture features , the formula is as follows:

[0168] ;

[0169] ;

[0170] in, is the original image, Gradually refine the semantic graph processed by the module for cross-region attention With the original image Perform channel-level splicing. is bilinear upsampling, It is a convolutional neural network consisting of 3×3 convolution, batch normalization BN and rectified linear unit ReLU.

[0171] Detail features output by the dual-stream detail prediction branch It is used as one of the inputs of the step-by-step refinement module, which contains rich underlying texture information. Next, the output feature map D_fea of ​​the two-stream detail prediction branch and the low-level texture feature Send to fusion module , get the detailed features , the formula is:

[0172] ;

[0173] The equation shows the step-by-step refinement process. First, the output feature map of the two-stream detail prediction branch is With the last layer of convolutional flow features Fusion to obtain , and With convolutional flow Cascade fusion to obtain , to achieve gradual refinement.

[0174] Since image regions closer to object boundaries require lower-level features to depict the foreground, while identifying internal object regions requires higher-level guidance, the present invention designs a progressive refinement module to selectively fuse the outputs of the previous layer and the current layer. The previous layer of detail features after processing and Fusion Module The processed detail features of the current layer Input the progressive refinement module to fuse the output of the previous layer and the current layer to obtain the final layer of features .

[0175] Specifically, for the current level , use the following function to get the output from the previous level Generate a bootstrap mask :

[0176] ;

[0177] The self-guided mask defines transparent areas as unknown and The unknown area is replaced by the current original output to get the updated current level :

[0178] ;

[0179] In this way, confident regions predicted from previous higher-level features are preserved, while the current level only needs to focus on refining uncertain regions.

[0180] Finally, the last layer of features As the final refined output feature, the Sigmoid activation function is input to obtain the final detail map , the formula is as follows:

[0181] ;

[0182] in, is the Sigmoid activation function.

[0183] The third step is to fuse the semantic map and the detail map to generate the final transparent map Alpha matte to complete the background separation and clipping of the target person.

[0184] Specifically, in order to determine the transition area and foreground area in the semantic map, the semantic map is first converted from S_map Mapping to index map . Secondly, the position of the pixel in the video frame image is determined by the value in the index map. When index=0, the pixel at this position belongs to the foreground, and when index=1, the pixel at this position belongs to the transition area. All pixels are filtered and separated to obtain two index maps, which represent the index of the foreground and the index of the transition area respectively. Then, the index map of the transition area is used With details Perform a dot product operation to map the foreground index to the foreground image. Finally, add the foreground map image to the image after the dot product to generate the final transparent image Alpha matte. The formula is as follows:

[0185] ;

[0186] in, represents the feature map with index 1, This is a detailed diagram showing the GRM prediction above. represents a pure foreground image, is the set of pixels belonging to the transition region, is the set of pixels belonging to the foreground area.

[0187] 2.2.4 Separate the foreground and background in the video frame image according to the transparent image Alpha matte to obtain the cutout image.

[0188] 2.2.5 Self-distillation training

[0189] Different from the previous pipeline training method, the present invention observes that downstream features often incorporate more multi-scale features from upstream features, and downstream features are closer to the predicted values. Therefore, the present invention believes that feature information from downstream features can guide the representation of upstream features, and designs a self-distillation training method to find downstream features as a teacher and provide guidance for upstream features. Specifically, the self-distillation loss It is defined by the cosine similarity between the teacher's features and the student's features. Figure 4 As shown, the present invention compares three reference feature settings, as follows:

[0190] 1. Best Teacher Distillation: The cross-region attention gradually refines the module results as reference features to guide the two-stream detail prediction branch and the attention semantic estimation branch, such as Figure 4 As shown in (a) in .

[0191] 2. Dense distillation: Each downstream feature is used as a reference feature to guide the upstream features. For example, the result of the cross-region attention step-by-step refinement module guides the attention semantic estimation branch and the two-stream detail prediction branch. The two-stream detail prediction branch is used as a reference feature to guide the attention semantic estimation branch. Figure 4 As shown in (b) in .

[0192] 3. Transfer distillation: Each downstream feature is selected only as a reference feature of its adjacent upstream features, such as Figure 4 As shown in (c) in .

[0193] It can be seen from the self-distillation training that the present invention proposes a cutout model, which is based on a dual-branch structure consisting of a dual-stream detail prediction branch and an attention semantic estimation branch. In order to improve the performance of the cutout model, multiple modules are added to the input end of the dual-branch structure, including a feature extractor HRNet48, a discrete cosine transform module, and a domain aggregation module, which can extract richer and more detailed feature information from the input image. At the output end of the dual-branch structure, a cross-regional attention step-by-step refinement module and a fusion module Fusion are further added. The cross-regional attention step-by-step refinement module can pay attention to different areas in the image, and through step-by-step iteration and refinement processing, it can accurately capture and restore the underlying texture information lost due to detail features. The fusion module uses an advanced fusion algorithm to seamlessly integrate the output results of the two branches to ensure the smooth transition of the cutout edges and the overall consistency, thereby obtaining a more accurate and complete cutout result.

[0194] The above-mentioned embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. An image cutout method, characterized in that: include: Acquire video frame images of the teacher and each student during the class process in the remote teaching environment; The method of obtaining the video frame images of the teacher and each student specifically includes: obtaining the video data V of the teacher and each student in the classroom process of the remote teaching environment; performing multi-thread extraction on the video data V to obtain the video frame image I k ={I1,I2,I3,…,I k }, where k represents the total number of video frame images; A dual-branch structure including a dual-stream detail prediction branch and an attention semantic estimation branch is constructed, and a feature extractor HRNet48, a discrete cosine transform module, and a domain aggregation module are added before the input of the dual-branch structure; the output of the feature extractor HRNet48 and the discrete cosine transform module are connected to the input of the domain aggregation module, and a spatial pyramid pooling module SPPELAN is added between the output of the domain aggregation module and the attention semantic estimation branch; a cross-regional attention step-by-step refinement module is added to the output of the dual-branch structure, and a fusion module Fusion is added between the attention semantic estimation branch and the cross-regional attention step-by-step refinement module to construct a cutout model; The video frame image is input into the matting model, and the RGB features of the video frame image are extracted by the feature extractor HRNet48, and the frequency features of the video frame image are extracted by the discrete cosine transform module; the RGB features and the frequency features are fused by the domain aggregation module to obtain the fused features; the fused features are locally enhanced by the spatial pyramid pooling module SPPELAN to obtain fine semantic features; the fused features are spliced ​​with the fine semantic features to obtain semantic features; the semantic features are processed by multiple convolutions and Softmax activation functions by the attention semantic estimation branch to obtain the semantic map of the video frame image; the detail features of the fused features are extracted by the dual-stream detail prediction branch, and the underlying texture information lost by the detail features is compensated by the cross-regional attention step-by-step refinement module to obtain the detail map of the video frame image; the semantic map and the detail map are fused by the fusion module Fusion to obtain the transparent map Alpha matte; the foreground and background in the video frame image are separated according to the transparent map Alpha matte to obtain the matted image.

2. The image cutout method according to claim 1, wherein: The RGB features of the video frame image are extracted by the feature extractor HRNet48, and the expression form is: X rgb =HRNet48(I); Among them, I is the video frame image, X rgb is the RGB feature, HRNet48() is the feature extractor; The method of extracting the frequency characteristics of the video frame image by using a discrete cosine transform module specifically includes: Divide the input video frame image into three color channels: R, G, and B; The RGB feature X rgb Divide into a set of 8×8 patches, and get For each block of the color channel Perform discrete cosine transform to obtain the channel spectrum By concatenating each channel spectrum Get the image spectrum D i,j ∈R 8×8×3 ; Image spectrum D i,j Perform flattening and reshape operations to obtain the frequency characteristics of the video frame image 3. The image cutout method according to claim 1, wherein: The RGB features and the frequency features are fused through the domain aggregation module, which specifically includes: Use binary basis filtering f base The convolution module Conv adjusts the frequency characteristics of the channel to cascade and obtain the frequency domain matrix I freq ; The RGB features are fed into another convolution module Conv with a Sigmoid activation function and two output channels to obtain the RGB domain matrix I rgb ; The RGB domain matrix I rgb and frequency domain matrix I freq Respectively and learn vector v∈R 1×C Multiply to get the RGB domain aggregation feature and frequency domain aggregation features The calculation formula is: Add the RGB domain aggregation features and the frequency domain aggregation features to get the fusion feature X s , the formula is: Among them, X rgb is the RGB feature, X freq is the frequency characteristic, is the RGB domain aggregation feature, is the frequency domain aggregation feature.

4. The image cutout method according to claim 1, wherein: The obtaining of the semantic graph of the video frame image specifically includes: The fusion feature X s And the fine semantic feature P is concatenated to obtain the semantic feature The formula is; S1=DAS(ConvBNReLU((X s )||(P))); WITH i =DAS(ConvBNReLU((S i-1 )∥Conv up×2i (P))),i∈[2,5]; Among them, P is the fine semantic feature, Conv up×2i () is the convolution sum ×2 i Upsampling, X s is the fusion feature, ∥ is the channel-level concatenation, ConvBNReLU() is a convolutional neural network composed of 3×3 convolution, batch normalization BN and rectified linear unit ReLU, and DAS is a deformable attention mechanism; The last layer output feature S5 is input into the Softmax activation function to obtain the semantic graph S map , the formula is: S map =Softmax(S5); Among them, Softmax() is the Softmax activation function.

5. The image cutout method according to claim 1, characterized in that: The detailed features of the fusion features are extracted by the dual-stream detail prediction branch, which specifically includes: The fusion feature X s Input the high-resolution feature extraction stream in the two-stream detail prediction branch to obtain the high-resolution detail feature H i ,i∈[1,4], the formula is: H1=Conv 1×1 (Res ×3 (X s )); H i =Conv 1×1 (Res ×3 (H′ i-1 )),i∈[2,4]; Among them, H′ i-1 is the enhanced high-resolution feature of the previous layer, Res ×3 () is the residual group structure, Conv 1×1 () is a 1×1 convolutional layer, i is the number of processing times; The original image I is input into the shallow feature extraction stream in the two-stream detail prediction branch, and four ConvBNReLU and bilinear upsampling are performed to obtain the shallow detail feature C i , i∈[4,1], the formula is: C4 = UP(ConvBNReLU(I)); C i =UP(ConvBNReLU(C i+1 )),i∈[3,1]; Among them, ConvBNReLU() is a convolutional neural network composed of 3×3 convolution, batch normalization BN and rectified linear unit ReLU, I is the original image, and UP() is bilinear upsampling; The high-resolution detail feature H i and shallow detail features C i Perform channel-level splicing and fuse through the convolutional neural network ConvBNReLU to obtain detailed fusion features The formula is: The semantic feature S 2i-1 and detail fusion features Splicing is performed, and convolution, normalization and sigmoid function are performed in sequence to obtain the auxiliary feature map G∈R H×W , the formula is: Generate the detail feature map H′, the formula is: in, For auxiliary flow, is the element-by-element product, T is the transpose, and W is the channel weighted kernel; The detail feature map H′ output by the last layer is used as the detail feature D_fea.

6. The image cutout method according to claim 5, characterized in that: The cross-region attention step-by-step refinement module is used to compensate for the underlying texture information lost by the detail features, specifically including: Identify the semantic graph S map Areas where texture information is lost in the middle and bottom layers; The K-nearest neighbor algorithm KNN is used to locate the closest area KNNs of each area where the underlying texture information is lost, and the relative position deviation is queried from the lookup table P; Set f i ∈R d is the feature of region i, {i1,i1,…,i k } are the k nearest neighbors of region i. For each region i, the original image I and the semantic map processed by the cross-region attention step-by-step refinement module are Splicing to get the low-level texture feature K i ,i∈[3,0], the formula is: K i =UP(ConvBNReLU(K i+1 )),i∈[2,0]; Among them, I is the original image, ∥ is the semantic map processed by the cross-region attention step-by-step refinement module Channel-level splicing with the original image I, UP() is bilinear upsampling, ConvBNReLU() is a convolutional neural network composed of 3×3 convolution, batch normalization BN and rectified linear unit ReLU; The output feature map D_fea of ​​the two-stream detail prediction branch and the low-level texture feature K i Send it to the fusion module ConvBNReLU to get the detail features The formula is: in, The detail feature D_fea output by the dual-stream detail prediction branch; The previous layer of detail features processed by the fusion module ConvBNReLU And fusion module ConvBNRe LU The processed detail features of the current layer Input the progressive refinement module to fuse the output of the previous layer and the current layer to obtain the final layer of features The last layer of features As the final refined output feature, the Sigmoid activation function is input to obtain the final detail map The formula is: Among them, Sigmoid() is the Sigmoid activation function.

7. The image cutout method according to claim 1, characterized in that: The fusion module Fusion is used to fuse the semantic map and the detail map, which specifically includes: By taking the maximum index value of the video frame image, the semantic map is transformed from S_map∈R H×W×3 Mapped to index map F index ∈R H×W×1 ; Determine the position of the pixel in the video frame image by the value in the index map; All pixels are filtered and separated to obtain two index maps, which represent the index of the foreground area and the index of the transition area respectively; The index map F of the transition region index＝1 With details Perform a dot product operation to map the foreground index to the foreground image; Add the foreground map and the dot-producted image to generate the final transparent image Alpha matte, the formula is: in, is the feature map with index 1, For the detail picture, FG m is a pure foreground image, N t is the set of pixels belonging to the transition area, N f is the set of pixels belonging to the foreground area.

8. An image cutout method as claimed in claim 7, characterized in that: Determining the position of a pixel in a video frame image by using the value in the index map specifically includes: When index=0 in the index map, the pixel in the video frame image belongs to the foreground area; When index=1 in the index map, the pixel in the video frame image belongs to the transition area.

Images