Picture matting method, system and equipment based on fusion of time sequence features and object features, and medium

Through the method of fusion of timing features and object features, combined with deep learning models and attention mechanism, the problem of poor effect of video cutout in complex backgrounds and low-light environments is solved, and more efficient prospects and background segmentation and subtle parts are achieved. accurate cutout.

CN118941580APending Publication Date: 2024-11-12SHENZHEN GUANGJIAN TECH CO LTD +1
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202410994478.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-24
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Existing video cutout technology is not effective in complex backgrounds or low-light environments, and it is difficult to accurately cut out subtle parts such as hair or transparent clothing.

Method used

Using a method based on the fusion of timing features and object features, deep learning models such as semantic encoder, prompt encoder and decoder, combined with attention mechanism, the target mask is generated to improve the applicable scenarios and spatial stability of the cutout model.

Benefits of technology

It improves the space-time stability and processing efficiency of video clipping, is suitable for various image types, and can more accurately identify and segment the foreground and background, especially in complex scenes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941580A_ABST
    Figure CN118941580A_ABST
Patent Text Reader

Abstract

The invention provides a matting method, system and device based on time sequence feature and object feature fusion and a medium, and the method comprises the steps: S1, determining an initial frame, and obtaining a target mask of the initial frame; s2, for the current frame, obtaining a similarity matrix; s3, acquiring all values higher than a first threshold for each position of the similarity matrix, normalizing the values to obtain a normalized matrix, and multiplying the normalized matrix by historical value coding features to obtain a historical feature map; s4, fusing the historical feature map, the fourth feature map and the fifth feature map by using an attention mechanism to obtain a sixth feature map; s5, processing the sixth feature map by using the decoder to obtain a target mask of the current frame; and S6, extracting the target mask of the current frame by using the value encoder, and writing the target mask into a memory pool. The matting is more stable and continuous.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0002] Video cutout technology is an advanced image processing technology that uses computer vision and machine learning methods to separate the foreground and background from a video. Its technical principles include:

[0003] Deep learning algorithm: The core of video cutout technology is to use deep learning models, such as convolutional neural networks (CNN) and recurrent neural networks (RNN), to identify and separate the foreground and background in the video. These models usually need to be trained with a large amount of data to learn the characteristic differences between the foreground and background.

[0004] Video preprocessing: Before cutting out an image, it is usually necessary to preprocess the video, including noise reduction, color adjustment, etc., to improve the accuracy and efficiency of subsequent processing.

[0005] Feature extraction: The model extracts various features from the video, such as color, texture, and shape, to identify the foreground and background.

[0006] Cutout calculation: Based on the result of feature recognition, the algorithm generates a mask to distinguish the foreground from the background. This process requires high-precision calculation to ensure the natural and realistic cutout result.

[0007] Post-processing: After cutout, further post-processing may be required, including edge smoothing and color correction, to improve the final visual effect.

[0008] Video cutout technology is widely used in various scenarios such as movie special effects production, short video production, live broadcast interaction, etc.

[0009] The effect of video cutout is greatly affected by the background environment. In scenes with complex backgrounds or low light, the cutout effect may be greatly reduced. At the same time, accurate cutout of subtle parts such as hair or transparent clothing is still a major challenge in the technology. This is because existing technologies use the information given by trimap for cutout, or use semantic information for pure portrait cutout.

[0010] The disclosure of the above background technology content is only used to assist in understanding the inventive concept and technical solution of the present invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above content has been disclosed on the filing date of this patent application, the above background technology should not be used to evaluate the novelty and creativity of the present application. Summary of the invention

[0011] In view of the defects in the prior art, the purpose of the present invention is to provide a cutout method, system, device and medium based on the fusion of temporal features and object features, so as to effectively improve the applicable scenarios of the cutout model and the spatiotemporal stability of video cutout.

[0012] According to the present invention, a method for cutting out an image based on the fusion of temporal features and object features comprises the following steps:

[0013] Step S1: determining an initial frame, identifying a first feature map using a semantic encoder, obtaining a second feature map according to a depth value using a hint encoder, obtaining image features using a feature encoder, and processing a third feature map composed of the image features, the first feature map, and the second feature map using a decoder to obtain a target mask of the initial frame;

[0014] Step S2: for the current frame, using a feature encoder to obtain image features, comparing the image features of the current frame with the image features of the previous frame to obtain a similarity matrix;

[0015] Step S3: obtaining all values ​​above the first threshold for each position of the similarity matrix, and normalizing them to obtain a normalized matrix, and multiplying the normalized matrix with the historical value encoding feature to obtain a historical feature map;

[0016] Step S4: using the semantic encoder to identify a fourth feature map, using the hint encoder to obtain a fifth feature map according to the depth value, and using the attention mechanism to fuse the historical feature map, the fourth feature map and the fifth feature map to obtain a sixth feature map;

[0017] Step S5: using the decoder to process the sixth feature map to obtain a target mask of the current frame;

[0018] Step S6: Utilize the value encoder to extract the target mask of the current frame and write it into the memory pool.

[0019] Optionally, the cutout method based on the fusion of temporal features and object features is characterized in that, when the decoder processes the first feature map and the second feature map, or the fourth feature map and the fifth feature map, it splices the first feature map and the second feature map, or the fourth feature map and the fifth feature map, and then processes them as a whole.

[0020] Optionally, the above-mentioned image cutting method based on the fusion of time series features and object features is characterized in that step S1 comprises:

[0021] Step S11: determining an initial frame in the video; the initial frame is an RGBD image; the RGBD image includes an RGB image and a depth image;

[0022] Step S12: using the semantic encoder to identify a first target range on the RGB image to obtain a first feature map;

[0023] Step S13: using the hint encoder to obtain a second target range on the depth map according to the depth value to obtain a second feature map;

[0024] Step S14: obtaining image features using the feature encoder;

[0025] Step S15: After splicing the first feature map and the second feature map, the third feature map is fused with the image feature to obtain the third feature map, and the third feature map is processed by a decoder to obtain the target mask of the initial frame.

[0026] Optionally, the above-mentioned image cutting method based on the fusion of time series features and object features is characterized in that step S3 comprises:

[0027] Step S31: compare multiple values ​​at each position in the similarity matrix, determine a first threshold at each position, and obtain a correlation value higher than the first threshold;

[0028] Step S32: normalizing the correlation values ​​so that the sum of the weights at each position is 1, thereby obtaining a normalized matrix;

[0029] Step S33: multiplying the normalized matrix by the image feature value corresponding to the correlation value to obtain a historical feature map.

[0030] Optionally, the image cutout method based on the fusion of temporal features and object features is characterized in that the first threshold is determined according to the number of the similarity matrices.

[0031] Optionally, the above-mentioned image cutting method based on the fusion of time series features and object features is characterized in that step S4 comprises:

[0032] Step S41: using the semantic encoder to identify a fourth target range and obtain a fourth feature map;

[0033] Step S42: using the hint encoder to obtain a fifth target range according to the depth value, thereby obtaining a fifth feature map;

[0034] Step S43: A sixth feature map is obtained by fusing the historical feature map, the fourth feature map and the fifth feature map using an attention mechanism.

[0035] Optionally, the cutout method based on the fusion of temporal features and object features is characterized in that the depth value is acquired by a camera or predicted by a deep learning model.

[0036] According to the present invention, a cutout system based on the fusion of temporal features and object features includes the following modules:

[0037] An initial module is used to determine an initial frame, identify a first feature map using a semantic encoder, obtain a second feature map according to a depth value using a hint encoder, obtain image features using a feature encoder, and process a third feature map composed of the image features, the first feature map, and the second feature map using a decoder to obtain a target mask of the initial frame;

[0038] A similarity matrix module is used to obtain image features for a current frame using a feature encoder, and compare the image features of the current frame with the image features of a previous frame to obtain a similarity matrix;

[0039] A historical feature map module, used for obtaining all values ​​above a first threshold value for each position of the similarity matrix, normalizing them to obtain a normalized matrix, and multiplying the normalized matrix with the historical value coding feature to obtain a historical feature map;

[0040] A fusion module, configured to use the semantic encoder to identify a fourth feature map, use the hint encoder to obtain a fifth feature map according to the depth value, and use an attention mechanism to fuse the historical feature map, the fourth feature map and the fifth feature map to obtain a sixth feature map;

[0041] A target mask module, used to process the sixth feature map using the decoder to obtain a target mask of the current frame;

[0042] A writing module is used to extract the target mask of the current frame by using the value encoder and write it into a memory pool.

[0043] According to the present invention, a cutout device based on the fusion of temporal features and object features includes:

[0044] processor;

[0045] a memory storing executable instructions of the processor;

[0046] The processor is configured to execute the steps of the image cutout method based on the fusion of temporal features and object features by executing the executable instructions.

[0047] The computer-readable storage medium provided according to the present invention is used to store a program, and when the program is executed, the steps of the cutout method based on the fusion of temporal features and object features are implemented.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] The present invention utilizes a multimodal solution and uses a first encoder and a second encoder to process the image respectively, which can effectively improve the applicable scenarios of the cutout model and the spatiotemporal stability of video cutout.

[0050] The present invention utilizes a decoder to process a feature map, can be applied to various image types, has wide applicability and scalability, and has better processing efficiency.

[0051] The present invention adopts the attention mechanism to fuse the features of the current frame with the previous frame, updating the new foreground range in the scene while retaining historical information, ensuring that targets that have not appeared in the history can also be identified and predicted. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings in the following descriptions are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work. By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, purposes and advantages of the present invention will become more obvious:

[0053] Figure 1 A flowchart of a method for cutting out an image based on the fusion of temporal features and object features in an embodiment of the present invention;

[0054] Figure 2 A flowchart of steps for obtaining a target mask of an initial frame in an embodiment of the present invention;

[0055] Figure 3 A schematic diagram of a first target range and a second target range in an embodiment of the present invention;

[0056] Figure 4 A flowchart of the steps of obtaining a historical feature map in an embodiment of the present invention;

[0057] Figure 5 is a flow chart of steps for obtaining a sixth characteristic graph in an embodiment of the present invention;

[0058] Figure 6 Schematic diagram of a module of a cutout system based on the fusion of time series features and object features in an embodiment of the present invention;

[0059] Figure 7 is a structural schematic diagram of a cutout device based on the fusion of time series features and object features in an embodiment of the present invention; and

[0060] Figure 8 Schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0062] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein, for example, can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0063] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0064] The present invention provides a cutout method based on the fusion of temporal features and object features, aiming to solve the problems existing in the prior art.

[0065] The following specific embodiments are used to describe in detail the technical solutions of the present invention and how the technical solutions of the present application solve the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0066] Figure 1 FIG. 1 is a flowchart of a method for cutting out an image based on the fusion of temporal features and object features in an embodiment of the present invention. Figure 1 As shown, the present invention provides a method for cutting out an image based on the fusion of temporal features and object features, comprising the following steps:

[0067] Step S1: determine the initial frame, use the semantic encoder to identify the first feature map, use the hint encoder to obtain the second feature map according to the depth value, use the feature encoder to obtain the image features, and use the decoder to process the third feature map composed of the image features, the first feature map and the second feature map to obtain the target mask of the initial frame.

[0068] In this step, the initial frame can be any frame in the video sequence, which can be a frame at the beginning of the video or a randomly specified frame. The initial frame is the first frame that starts to be processed in this embodiment. The semantic encoder is a pre-trained neural network model used to extract object features in the image. The semantic encoder varies according to the target object and the type of image. The hint encoder uses the trimap obtained by the depth information to parse the target range. The semantic encoder can identify the range of a specific target object, while the hint encoder is used to identify other objects associated with the specific target object. For example, in a live broadcast scene, the semantic encoder is used to identify the range of a portrait, and the hint encoder is used to identify various items such as tables, chairs, and handheld objects related to people. Feature coding recognizes the image as a whole and can obtain the image features of the target. The image features, the first feature map, and the second feature map are combined into a third feature map, and the decoder is used to process it to obtain the target mask of the initial frame. This target mask represents the position and shape of the object in the initial frame. The decoder is another neural network model that takes the output of the semantic encoder and the hint encoder, that is, the first feature map, as input. The role of the decoder is to transform or reconstruct these feature maps to produce a more accurate object mask. The object mask is a binary image where the foreground object is labeled as 1 (or white) and the background is labeled as 0 (or black).

[0069] Step S2: For the current frame, use a feature encoder to obtain image features, compare the image features of the current frame with the image features of the previous frame, and obtain a similarity matrix.

[0070] In this step, a frame after the initial frame is selected as the current frame. This current frame can be any frame in the video sequence. Generally speaking, the video is processed in the order of frames. For this embodiment, step S1 is used to process the initial frame, and steps S2-S6 are used to process subsequent frames. The feature encoder recognizes the image and obtains the image features of the current frame. The similarity matrix represents the similarity between image features. Since the feature encoder processes the entire image, the size of the similarity matrix is ​​the same as the image size. The number of pixels on the current frame is the same as the number of elements in the similarity matrix. Each previous frame has a similarity matrix with the current frame. Multiple previous frames have multiple similarity matrices.

[0071] Step S3: for each position of the similarity matrix, all values ​​above the first threshold are obtained and normalized to obtain a normalized matrix, and the normalized matrix is ​​multiplied by the historical value coding feature to obtain a historical feature map.

[0072] In this step, each position in the similarity matrix is processed individually. When processing, the values in multiple similarity matrices at the same position are operated on. The first threshold is determined according to the number of the similarity matrices. The number of the first thresholds can be either fixed or floating. For example, when the number of similarity matrices is n, taking the top 20% of the similarity data, the first threshold is (1 - 0.2)*n. Another example, when the number of similarity matrices is m, if m > a, the first threshold is 0.8*m; when the number of similarity matrices is s, if s < a, the first threshold is 0.5*m. The normalization process can make the similarity at each position equal to 1, which is convenient for subsequent calculations. The historical coding feature is the coding value corresponding to the normalization matrix in the previous frame. For example, if the value in the normalization matrix corresponds to b pixel points in a certain frame, the feature values corresponding to the b pixel points are the historical coding features. The historical feature map contains current information and historical information, and has strong spatio-temporal stability and robustness.

[0073] Step S4: Use the semantic encoder to identify the fourth feature map, use the hint encoder to obtain the fifth feature map according to the depth value, and fuse the historical feature map, the fourth feature map and the fifth feature map using the attention mechanism to obtain the sixth feature map.

[0074] In this step, a semantic encoder is used to identify the fourth feature map. This fourth feature map represents the position and shape of the object in the current frame. The fourth feature map in this step may be different from the first feature map. Use the hint encoder to obtain the fifth feature map according to the depth value. The fifth feature map in this step may be different from the second feature map. The attention mechanism can help us selectively focus on the important parts in different feature maps and combine them into a new feature map. This new sixth feature map contains information from multiple frames, can provide a more accurate estimate of the object position and shape, can update the new foreground range in the scene while retaining historical information, and ensure that targets that have not appeared in history can also be recognized and predicted.

[0075] Step S5: Use the decoder to process the sixth feature map to obtain the target mask of the current frame.

[0076] In this step, the decoder is used to process the sixth feature map to obtain the target mask of the current frame. This target mask represents the position and shape of the object in the current frame and can be used for subsequent image processing or analysis tasks, such as foreground / background segmentation, object tracking, etc.

[0077] When the decoder processes the first feature map and the second feature map, or the fourth feature map and the fifth feature map, the first feature map and the second feature map, or the fourth feature map and the fifth feature map are spliced ​​and then processed as a whole.

[0078] Step S6: Utilize the value encoder to extract the target mask of the current frame and write it into the memory pool.

[0079] In this step, the value encoder is used to obtain the final target mask of the current frame and write it into the memory pool. The memory pool is used to store target mask information of different frames for subsequent frame calculations.

[0080] Figure 2 FIG. 1 is a flow chart of steps for obtaining a target mask of an initial frame in an embodiment of the present invention. Figure 2 As shown, in an embodiment of the present invention, a step of obtaining a target mask of an initial frame includes:

[0081] Step S11: determining an initial frame in the video; the initial frame is an RGBD image.

[0082] In this step, the initial frame is an RGBD image; the RGBD image includes an RGB image and a depth map. RGBD image is the most commonly used image format for depth cameras. In this embodiment, RGBD images are used for detailed description. RGBD image is a special image format, which not only includes a conventional RGB (red, green, and blue) image, but also includes a depth map (D). The RGB image provides color information of the scene, while the depth map provides distance information from each point in the scene to the camera plane.

[0083] Step S12: using the semantic encoder to identify a first target range on the RGB image to obtain a first feature map.

[0084] In this step, the semantic encoder is a target recognition model of the RGB image, which recognizes the target object on the RGB image to obtain a first feature map. Features are extracted from the first feature map to obtain the first feature map. For some target recognition models, feature extraction has been performed when performing target recognition, so it is only necessary to obtain the feature map area corresponding to the first feature map. Figure 3 A schematic diagram showing a first target range and a second target range is shown. Figure 3 The dotted box in the middle left picture is the first target range. Figure 3 The dotted line box in the middle right figure is the second target range. It should be noted that the dotted lines in the figure are only exemplary illustrations of the approximate area, not accurate area definitions. Figure 3 It can be seen that the first target range and the second target range are usually different.

[0085] Step S13: using the hint encoder to obtain a second target range on the depth map according to the depth value to obtain a second feature map.

[0086] In this step, if depth information is available (such as from a depth camera or depth estimation algorithm), the encoder is prompted to use these depth values ​​to further refine the target range. Depth information can help distinguish foreground from background, especially in complex scenes.

[0087] Step S14: using the feature encoder to obtain image features.

[0088] In this step, the feature encoder extracts features from the initial frame to obtain image features that contain rich information. The feature encoder has been trained to extract meaningful features from the image. The initial frame (which can be a combination of RGB images or RGBD images) is input into the feature encoder. The feature encoder outputs an image feature that contains abstract, compressed, and meaningful information about the initial frame.

[0089] Step S15: After splicing the first feature map and the second feature map, the third feature map is fused with the image feature to obtain the third feature map, and the third feature map is processed by a decoder to obtain the target mask of the initial frame.

[0090] In this step, the first feature map, the image features, and the second feature map are merged into a new feature map, namely the third feature map. Among them, the first feature map and the second feature map are spliced. When the first feature map and the second feature map are spliced ​​and fused with the image features, the attention mechanism is used for fusion. The decoder is a neural network model corresponding to the encoder, which can convert the feature map back to the image space or generate other forms of output. The decoder receives the first feature map as input and generates a target mask of the initial frame through a series of calculations and conversion operations. The target mask is a binary image in which the foreground object (i.e., the target object) is marked as 1 (or white) and the background is marked as 0 (or black). This mask can be used for subsequent image processing or analysis tasks.

[0091] This embodiment is described in detail using an RGBD image as an example, and the target mask can be obtained more clearly.

[0092] Figure 4 FIG. 1 is a flow chart of steps for obtaining a historical feature graph in an embodiment of the present invention. Figure 4 As shown, in an embodiment of the present invention, a step of obtaining a historical feature graph includes:

[0093] Step S31: compare multiple values ​​at each position in the similarity matrix, determine a first threshold at each position, and obtain a correlation value higher than the first threshold.

[0094] In this step, each element in the similarity matrix is ​​compared. The similarity matrix is ​​a two-dimensional matrix in which each element represents the similarity between two image features. The similarity can be calculated based on the distance between feature vectors (such as Euclidean distance, cosine similarity, etc.). Our goal is to find those elements with high similarity. For each position (i.e., each element) in the similarity matrix, a fixed global threshold can be selected, or a threshold can be dynamically calculated based on the distribution of values ​​at that position (for example, using the mean, median, or specific percentile as the threshold). This threshold is used to filter out values ​​with high similarity. The similarity value at each position is compared with the threshold, and those values ​​above the threshold are retained. These values ​​represent features or images with high similarity or correlation at that position.

[0095] Step S32: normalize the correlation values ​​so that the sum of the weights at each position is 1, thereby obtaining a normalized matrix.

[0096] In this step, the correlation values ​​obtained in the previous step are normalized. The purpose of normalization is to make the sum of the weights of each element equal to 1, so that in the subsequent steps, the weight of each element is relative, not absolute. The normalization process is usually to divide the value of each element by the sum of all element values. In this way, we get a normalized matrix in which the weights of each element have been adjusted so that their sum is 1.

[0097] Step S33: multiplying the normalized matrix by the image feature value corresponding to the correlation value to obtain a historical feature map.

[0098] In this step, the normalized matrix is ​​multiplied with the image eigenvalues ​​corresponding to the correlation values. Specifically, each element in the normalized matrix (i.e., the normalized weight) is multiplied with its corresponding image eigenvalue. This is usually done in the framework of matrix multiplication, where each row (or column, depending on the specific implementation) of the normalized matrix is ​​multiplied with the corresponding eigenvector. The eigenvalues ​​may come from multiple images or different parts of the image, and they describe the specific content or attributes of the image. The purpose of this is to combine the weight of each image feature with the corresponding correlation value to obtain a new feature map, namely the historical feature map. The historical feature map can be used to represent the similarity relationship between images and the importance of each image feature in the entire image set. In this way, we can better understand and analyze the image data, thereby providing useful information for subsequent tasks (such as classification, clustering, etc.).

[0099] Figure 5 FIG. 1 is a flow chart of steps for obtaining the sixth characteristic graph in an embodiment of the present invention. Figure 5As shown, in an embodiment of the present invention, a step of obtaining a sixth characteristic map includes:

[0100] Step S41: using the semantic encoder to identify a fourth target range and obtain a fourth feature map.

[0101] In this step, the role of the semantic encoder is to identify and process the semantic information in the input data. The semantic encoder is usually a deep learning model, such as a convolutional neural network (CNN) or a recurrent neural network (RNN), which can extract high-level features from images, text, or other types of data. In image processing tasks, these high-level features may include the edges, textures, shapes, etc. of objects. In this way, the semantic encoder can identify the fourth target range in the image and generate a corresponding feature map, called the fourth feature map. This feature map contains key information for further processing.

[0102] Step S42: using the hint encoder to obtain a fifth target range according to the depth value, thereby obtaining a fifth feature map.

[0103] In this step, the hint encoder is another deep learning model that is specifically designed to process data related to depth information. Depth values ​​usually refer to the distance information between pixels in an image and the camera, which is very important for understanding the three-dimensional structure of the scene. The hint encoder identifies the fifth target range in the image based on these depth values. This range usually corresponds to a specific object or area in the scene, and its depth information is crucial for subsequent tasks (such as object recognition, scene understanding, etc.). After processing by the hint encoder, we get the fifth feature map, which represents the image features based on depth information.

[0104] Step S43: A sixth feature map is obtained by fusing the historical feature map, the fourth feature map and the fifth feature map using an attention mechanism.

[0105] In this step, the attention mechanism is used to fuse the previously obtained historical feature maps, the fourth feature map, and the fifth feature map. The attention mechanism is a mechanism that simulates human visual attention, which allows the model to focus on certain parts of the input data when processing information while ignoring other less important information. During the feature fusion process, the attention mechanism dynamically adjusts the weights of different feature maps according to their importance, so that more critical information for the current task can receive more attention. In this way, we finally get a sixth feature map that fuses information from multiple feature maps. This feature map integrates information from different sources and provides rich feature basis for subsequent image processing tasks.

[0106] Figure 6FIG. 1 is a module diagram of a cutout system based on the fusion of time series features and object features in an embodiment of the present invention. Figure 6 As shown, the present invention provides a cutout system based on the fusion of temporal features and object features, comprising the following modules:

[0107] An initial module is used to determine an initial frame, identify a first feature map using a semantic encoder, obtain a second feature map according to a depth value using a hint encoder, obtain image features using a feature encoder, and process a third feature map composed of the image features, the first feature map, and the second feature map using a decoder to obtain a target mask of the initial frame;

[0108] A similarity matrix module is used to obtain image features for a current frame using a feature encoder, and compare the image features of the current frame with the image features of a previous frame to obtain a similarity matrix;

[0109] A historical feature map module, used for obtaining all values ​​above a first threshold value for each position of the similarity matrix, normalizing them to obtain a normalized matrix, and multiplying the normalized matrix with the historical value coding feature to obtain a historical feature map;

[0110] A fusion module, configured to use the semantic encoder to identify a fourth feature map, use the hint encoder to obtain a fifth feature map according to the depth value, and use an attention mechanism to fuse the historical feature map, the fourth feature map and the fifth feature map to obtain a sixth feature map;

[0111] A target mask module, used to process the sixth feature map using the decoder to obtain a target mask of the current frame;

[0112] A writing module is used to extract the target mask of the current frame by using the value encoder and write it into a memory pool.

[0113] In this embodiment, the first encoder and the second encoder are used to identify different areas respectively to better obtain the target area, and the information of the previous frame is combined to adjust the range of the current frame, effectively improving the applicable scenarios of the cutout model and the spatiotemporal stability of the video cutout.

[0114] The embodiment of the present invention also provides a cutout device based on the fusion of time series features and object features, including a processor and a memory, in which executable instructions of the processor are stored. The processor is configured to execute the steps of a cutout method based on the fusion of time series features and object features by executing the executable instructions.

[0115] As described above, in this embodiment, the first encoder and the second encoder are used to identify different areas respectively, so as to better obtain the target area, and the range of the current frame is adjusted by combining the information of the previous frame, so as to effectively improve the applicable scenarios of the cutout model and the spatiotemporal stability of the video cutout.

[0116] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "platforms".

[0117] Figure 7 Schematic diagram of a structure of a cutout device based on the fusion of temporal features and object features in an embodiment of the present invention. Figure 7 The electronic device 600 according to this embodiment of the present invention is described. Figure 7 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0118] like Figure 7 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0119] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 executes the steps of various exemplary embodiments of the present invention described in the above-mentioned part of the cutout method based on the fusion of temporal features and object features. For example, the processing unit 610 can execute the following steps: Figure 1 Follow the steps shown in .

[0120] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0121] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0122] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0123] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 through the bus 630. It should be understood that although Figure 7 Not shown, other hardware and / or software modules may be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0124] In an embodiment of the present invention, a computer-readable storage medium is also provided for storing a program, and when the program is executed, the steps of a cutout method based on the fusion of temporal features and object features are implemented. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary implementations of the present invention described in the above-mentioned cutout method based on the fusion of temporal features and object features section of this specification.

[0125] As shown above, in this embodiment, the first encoder and the second encoder are used to identify different areas respectively to better obtain the target area, and the information of the previous frame is combined to adjust the range of the current frame, effectively improving the applicable scenarios of the cutout model and the spatiotemporal stability of the video cutout.

[0126] Figure 8 Schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. Figure 8As shown, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0127] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0128] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0129] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0130] In this embodiment, the first encoder and the second encoder are used to identify different areas respectively to better obtain the target area, and the information of the previous frame is combined to adjust the range of the current frame, effectively improving the applicable scenarios of the cutout model and the spatiotemporal stability of the video cutout.

[0131] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other. The above description of the disclosed embodiments enables professionals and technicians in this field to implement or use the present invention. Various modifications to these embodiments will be obvious to professionals and technicians in this field, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown in this article, but will comply with the widest range consistent with the principles and novel features disclosed herein.

[0132] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A cutout method based on the fusion of temporal features and object features. It is characterized in that The steps include: Step S1: determining an initial frame, identifying a first feature map using a semantic encoder, obtaining a second feature map according to a depth value using a hint encoder, obtaining image features using a feature encoder, and processing a third feature map composed of the image features, the first feature map, and the second feature map using a decoder to obtain a target mask of the initial frame; Step S2: for the current frame, using a feature encoder to obtain image features, comparing the image features of the current frame with the image features of the previous frame to obtain a similarity matrix; Step S3: obtaining all values ​​above the first threshold for each position of the similarity matrix, and normalizing them to obtain a normalized matrix, and multiplying the normalized matrix with the historical value encoding feature to obtain a historical feature map; Step S4: using the semantic encoder to identify a fourth feature map, using the hint encoder to obtain a fifth feature map according to the depth value, and using the attention mechanism to fuse the historical feature map, the fourth feature map and the fifth feature map to obtain a sixth feature map; Step S5: using the decoder to process the sixth feature map to obtain a target mask of the current frame; Step S6: Utilize the value encoder to extract the target mask of the current frame and write it into the memory pool.

2. According to claim 1, a method for cutting out images based on the fusion of temporal features and object features, It is characterized in that When the decoder processes the first feature map and the second feature map, or the fourth feature map and the fifth feature map, the first feature map and the second feature map, or the fourth feature map and the fifth feature map are spliced ​​and then processed as a whole.

3. According to claim 1, a method for cutting out images based on the fusion of temporal features and object features, It is characterized in that Step S1 includes: Step S11: determining an initial frame in the video; the initial frame is an RGBD image; the RGBD image includes an RGB image and a depth image; Step S12: using the semantic encoder to identify a first target range on the RGB image to obtain a first feature map; Step S13: using the hint encoder to obtain a second target range on the depth map according to the depth value to obtain a second feature map; Step S14: obtaining image features using the feature encoder; Step S15: After splicing the first feature map and the second feature map, the third feature map is fused with the image feature to obtain the third feature map, and the third feature map is processed by a decoder to obtain the target mask of the initial frame.

4. The method for cutting out an image based on the fusion of temporal features and object features according to claim 1, It is characterized in that Step S3 includes: Step S31: compare multiple values ​​at each position in the similarity matrix, determine a first threshold at each position, and obtain a correlation value higher than the first threshold; Step S32: normalizing the correlation values ​​so that the sum of the weights at each position is 1, thereby obtaining a normalized matrix; Step S33: multiplying the normalized matrix by the image feature value corresponding to the correlation value to obtain a historical feature map.

5. The method for cutting out an image based on the fusion of temporal features and object features according to claim 1, It is characterized in that The first threshold is determined according to the number of the similarity matrix.

6. The image cutting method based on the fusion of temporal features and object features according to claim 1, It is characterized in that Step S4 includes: Step S41: using the semantic encoder to identify a fourth target range and obtain a fourth feature map; Step S42: using the hint encoder to obtain a fifth target range according to the depth value, thereby obtaining a fifth feature map; Step S43: A sixth feature map is obtained by fusing the historical feature map, the fourth feature map and the fifth feature map using an attention mechanism.

7. The image cutting method based on the fusion of temporal features and object features according to claim 1, It is characterized in that The depth value is acquired by a camera or predicted by a deep learning model.

8. A cutout system based on the fusion of temporal features and object features. It is characterized in that Includes the following modules: An initial module is used to determine an initial frame, identify a first feature map using a semantic encoder, obtain a second feature map according to a depth value using a hint encoder, obtain image features using a feature encoder, and process a third feature map composed of the image features, the first feature map, and the second feature map using a decoder to obtain a target mask of the initial frame; A similarity matrix module is used to obtain image features for a current frame using a feature encoder, and compare the image features of the current frame with the image features of a previous frame to obtain a similarity matrix; A historical feature map module, used for obtaining all values ​​above a first threshold value for each position of the similarity matrix, normalizing them to obtain a normalized matrix, and multiplying the normalized matrix with the historical value coding feature to obtain a historical feature map; A fusion module, configured to use the semantic encoder to identify a fourth feature map, use the hint encoder to obtain a fifth feature map according to the depth value, and use an attention mechanism to fuse the historical feature map, the fourth feature map and the fifth feature map to obtain a sixth feature map; A target mask module, used to process the sixth feature map using the decoder to obtain a target mask of the current frame; A writing module is used to extract the target mask of the current frame by using the value encoder and write it into a memory pool.

9. A cutout device based on the fusion of temporal features and object features, It is characterized in that include: processor; a memory storing executable instructions of the processor; The processor is configured to execute the steps of a cutout method based on fusion of temporal features and object features as described in any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium for storing a program, It is characterized in that When the program is executed, the steps of a cutout method based on the fusion of temporal features and object features as described in any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • Matting method and system based on RGBD camera

    CN111899266A

  • Foreground matting method and device and electronic equipment

    CN114419091A

  • Video target segmentation method based on space-time decoupling attention mechanism

    CN116416553A

  • Optical flow mask-based unsupervised monocular depth estimation method

    WO2024051184A1