Semi-supervised video frame segmentation method and device, medium and product
By combining a visual backbone network and a fine-grained memory readout module, the problems of missing spatial details and high computational complexity in existing semi-supervised video frame segmentation methods are solved, achieving efficient and accurate video frame segmentation, which is applicable to fields such as augmented reality and autonomous driving.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-12
- Publication Date
- 2026-03-31
AI Technical Summary
Existing semi-supervised video frame segmentation methods suffer from spatial detail loss, high computational complexity, and insufficient robustness over long time periods, making it difficult to meet real-time requirements.
A video object segmentation network is proposed, which consists of a visual backbone network, a fine-grained memory readout module, a global feature compensation module, and a linear fusion module. The fine-grained memory readout module enables shallow matching, and the global feature compensation module is used for lightweight global relationship modeling, thereby improving segmentation efficiency and accuracy.
It significantly reduces computational complexity, improves segmentation efficiency and accuracy, preserves spatial details of target boundaries, and is suitable for deployment in resource-constrained scenarios.
Smart Images
Figure CN121767906A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and in particular to a semi-supervised video frame segmentation method, apparatus, medium, and product. Background Technology
[0002] Semi-supervised video object segmentation aims to automatically segment target objects in videos using the annotations of the first frame. It is widely used in augmented reality, autonomous driving, and other fields, and has stringent requirements for accuracy and timeliness. Early methods, represented by Oneshot Video Object Segmentation (OSVOS), adopted an online fine-tuning strategy. During inference, the network parameters need to be optimized for backpropagation for each video, which is computationally expensive and difficult to meet real-time requirements.
[0003] Recent research has shifted towards non-fine-tuning frameworks, such as Space-Time Memory Networks (STM). The core of STM is storing historical frame features in a memory bank and achieving target propagation through global similarity matching, significantly improving efficiency. However, such methods generally suffer from the following drawbacks: 1. Loss of spatial detail: Matching at deep features (e.g., 1 / 16 resolution) results in severe loss of fine-grained information such as target boundaries, leading to blurred segmentation edges; 2. High computational complexity: The computational complexity of global similarity reaches O(C×N). 2 ), where C is the number of channels and N is the number of pixels, which increases twice with the resolution, forming a bottleneck for high-resolution video processing; 3. Insufficient robustness over long time: In long videos, due to target deformation, occlusion, etc., single deep features are prone to drift, matching stability decreases, and accuracy decays significantly.
[0004] In summary, there is an urgent need for a semi-supervised video frame segmentation method that combines local modeling capabilities with global dependency representation capabilities, and is computationally efficient and has high segmentation accuracy. Summary of the Invention
[0005] The purpose of this application is to provide a semi-supervised video frame segmentation method, apparatus, medium, and product to improve the segmentation efficiency and accuracy of video frames.
[0006] To achieve the above objectives, this application provides the following solution.
[0007] Firstly, this application provides a semi-supervised video frame segmentation method, including: Retrieve the video for the current time period; the video for the current time period includes video frames from multiple consecutive moments. Pixel-level annotation is performed on the video frame at the first moment in the current time period to obtain the true pixel mask of the video frame at the first moment; Using a pixel mask prediction model, based on the real pixel mask of the video frame at time 1 and the video frame at time 1, starting from time 2 of the current time period, the predicted pixel masks of the video frames at each time point of the current time period are determined sequentially, thus completing the segmentation of the video frames at each time point of the current time period. The pixel mask prediction model is obtained by training a video object segmentation network, which includes: a visual backbone network, a fine-grained memory readout module, a global feature compensation module, and a linear fusion module. The process of determining the predicted pixel masks of the video frames at each time point from time 2 of the current time period to the last time point includes: The actual pixel masks of the video frames at time 2, time 1, and time 1 are input into the pixel mask prediction model to obtain the predicted pixel mask of the video frame at time 2. The current time is determined from any time from the third time to the last time in the current time period. The time before the current time is determined as the previous time. The video frames of each time in the time period from the second time to the previous time are determined as the current predictive memory frames. The video frame of the first time is determined as the current real memory frame. The current video frame, the current real memory frame, the real pixel mask of the current real memory frame, each current predicted memory frame, and each current predicted memory frame's predicted pixel mask are input into the pixel mask prediction model to obtain the predicted pixel mask of the current video frame.
[0008] Secondly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the semi-supervised video frame segmentation method described above.
[0009] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the semi-supervised video frame segmentation method described above.
[0010] Fourthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the semi-supervised video frame segmentation method described above.
[0011] According to the specific embodiments provided in this application, the following technical effects are disclosed: This application discloses a semi-supervised video frame segmentation method, apparatus, medium, and product. First, the video of the current time period is acquired; the video of the current time period includes multiple consecutive video frames. Then, the video frame of the first moment in the current time period is pixel-level annotated to obtain the true pixel mask of the video frame of the first moment. Finally, using a pixel mask prediction model, based on the true pixel mask of the video frame of the first moment and the video frame of the first moment, the predicted pixel masks of the video frames of each moment in the current time period are determined sequentially, starting from the second moment, to complete the segmentation of the video frames of each moment in the current time period. The pixel mask prediction model is obtained by training a video object segmentation network, which includes a visual backbone network, a fine-grained memory readout module, a global feature compensation module, and a linear fusion module. This application achieves shallow matching through a fine-grained memory readout module, avoiding the high computational overhead of large-scale global matching. It achieves lightweight global relationship modeling through a global feature compensation module, making the overall inference speed significantly faster than other methods, while occupying less GPU memory and being more suitable for deployment in resource-constrained real-world scenarios, thus improving the segmentation efficiency of video frames. By fully utilizing the shallow high-resolution feature information obtained from the fine-grained memory readout module, clearer spatial details are maintained at the target boundary, making the predicted pixel mask more accurate. This effectively alleviates problems such as blurred target edges and omission of small targets, thereby improving the segmentation efficiency and accuracy of video frames. Attached Figure Description
[0012] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0013] Figure 1 A schematic flowchart of a semi-supervised video frame segmentation method provided in an embodiment of this application; Figure 2 A schematic diagram of a video object segmentation network structure; Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation
[0014] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0015] The purpose of this application is to provide a semi-supervised video frame segmentation method, apparatus, medium, and product, which aims to improve the segmentation efficiency and accuracy of video frames.
[0016] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0017] In one exemplary embodiment, such as Figure 1 As shown, a semi-supervised video frame segmentation method is provided, including steps 1-3.
[0018] Step 1: Obtain the video for the current time period; the video for the current time period includes video frames from multiple consecutive moments.
[0019] Step 2: Perform pixel-level annotation on the video frame at the first moment of the current time period to obtain the true pixel mask of the video frame at the first moment.
[0020] Step 3: Using the pixel mask prediction model, based on the real pixel mask of the video frame at time 1 and the video frame at time 1, starting from time 2 of the current time period, determine the predicted pixel mask of the video frame at each time of the current time period in sequence, and complete the segmentation of the video frame at each time of the current time period.
[0021] The pixel mask prediction model is obtained by training a video object segmentation network, such as... Figure 2 As shown, the video object segmentation network includes: a visual backbone network, a fine-grained memory readout module, a global feature compensation module, and a linear fusion module.
[0022] As an optional implementation, step 3, the process of determining the pixel mask prediction model, includes steps 311-315.
[0023] Step 311: Obtain video from historical time periods; the video from historical time periods includes video frames from T+1 consecutive historical moments; T≥1.
[0024] Step 312: Perform pixel-level annotation on the video frames at each historical moment to obtain the true pixel mask of the video frames at each historical moment.
[0025] Step 313: Construct a training set based on video frames and real pixel masks from T+1 historical moments.
[0026] Step 314: Initialize the video object segmentation network.
[0027] Step 315: Based on the training set and the total loss function, perform multiple iterations of training on the video object segmentation network to obtain the video object segmentation model; the total loss function includes: ; ; ; ; ; ; ; in, Total loss; This represents the coarse segmentation loss. To compensate for the losses from the division; The cross-entropy loss function; The Dice loss function; This is the initial prediction mask for historical query frames; The actual pixel mask for the historical query frame; This represents the total number of pixels in the historical query frames. The pixel value of the j-th pixel in the true pixel mask of the historical query frame; The pixel value of the j-th pixel in the preliminary prediction mask of the historical query frame; For smoothing terms used to maintain numerical stability; The pixel value of the j-th pixel in the predicted pixel mask of the historical query frame.
[0028] As an optional implementation, in step 315, the training process of the video object segmentation network at any current iteration number includes steps 31501-31511.
[0029] Step 31501: Determine the video frame at time T+1 in the historical time period as the historical query frame, and determine any video frame at the first T times in the historical time period as the historical memory frame.
[0030] Step 31502: Input the historical query frames into the visual backbone network at the current iteration number to obtain the multi-scale features of the historical query frames at the current iteration number; the multi-scale features include Layer features, When the current iteration number is the same as the initial iteration number, the video object segmentation network at the current iteration number is the initialized video object segmentation network.
[0031] Specifically, the calculation formula for the multi-scale features of historical query frames is as follows: ; in, For the first historical query frame Layer features, For visual backbone network; This is a historical query frame.
[0032] Step 31503: Input T historical memory frames into the visual backbone network at the current iteration number to obtain the multi-scale features of the T historical memory frames at the current iteration number.
[0033] Specifically, the formula for calculating the multi-scale features of historical memory frames is as follows: ; in, For the first The first historical memory frame Layer features; For the first Zhang's historical memory frames.
[0034] Step 31504: Input the real pixel masks of T historical memory frames into the visual backbone network at the current iteration number to obtain the multi-scale features of the real pixel masks of T historical memory frames at the current iteration number.
[0035] Specifically, the formula for calculating the multi-scale features of the true pixel mask of the historical query frame is as follows: ; in, For the first The first true pixel mask of Zhang's historical memory frames Layer features; For the first The true pixel mask of a historical memory frame.
[0036] Step 31505: Input the first layer features of the historical query frame at the current iteration number, the first layer features of T historical memory frames at the current iteration number, and the first layer features of the real pixel masks of T historical memory frames at the current iteration number into the fine-grained memory readout module at the current iteration number to obtain the preliminary prediction mask of the historical query frame at the current iteration number.
[0037] As an optional implementation, the fine-grained memory readout module includes: a splicing unit, a fusion wizard feature generation unit, an enhanced feature generation unit, a local readout feature generation unit, a fine-grained memory readout result generation unit, and a decoder.
[0038] Step 31505 includes steps S11-S17.
[0039] S11: Select any one of the T historical memory frames as the target memory frame.
[0040] S12: Input the first layer features of the historical query frame and the first layer features of the target memory frame at the current iteration number into the splicing unit at the current iteration number to obtain the splicing features of the historical query frame and the target memory frame at the current iteration number.
[0041] Specifically, the expression for the splicing unit is: ; in, For historical query frames and the first The splicing characteristics of Zhang's historical memory frames; For splicing operations; The first layer feature of the historical query frame; For the first The first layer of features of Zhang's historical memory frame.
[0042] S13: Input the splicing features of the historical query frame and the target memory frame at the current iteration number into the fusion wizard feature generation unit at the current iteration number to obtain the fusion wizard feature of the target memory frame at the current iteration number.
[0043] Specifically, the expression for the fusion wizard feature generation unit is: ; in, For the first The fusion guide feature of Zhang's historical memory frames; It is a non-linear activation function; for Convolutional layer.
[0044] S14: Input the fusion wizard features of the target memory frame at the current iteration number and the first layer features of the historical query frame at the current iteration number into the enhancement feature generation unit at the current iteration number to obtain the enhancement features of the target memory frame at the current iteration number.
[0045] Specifically, the expression for the enhanced feature generation unit is: ; in, For the first Enhanced features of Zhang's historical memory frames; for Convolutional layer.
[0046] S15: Input the enhanced features of the target memory frame at the current iteration number and the first layer features of the real pixel mask of the target memory frame at the current iteration number into the local readout feature generation unit at the current iteration number to obtain the local readout features of the target memory frame at the current iteration number.
[0047] Specifically, the expression for the local readout feature generation unit is: ; in, For the first Local readout features of Zhang's historical memory frames; This is a pixel-by-pixel multiplication; For the first The first layer of features of the true pixel mask of Zhang's historical memory frames.
[0048] S16: Input all local readout features of all historical memory frames at the current iteration number into the fine-grained memory readout result generation unit at the current iteration number to obtain the fine-grained memory readout result at the current iteration number.
[0049] Specifically, the expression for the fine-grained memory readout result generation unit is: ; in, To read out the results for fine-grained memory.
[0050] S17: Input the fine-grained memory readout result of the current iteration number into the decoder of the current iteration number to obtain the preliminary prediction mask of the historical query frame at the current iteration number.
[0051] Specifically, the decoder expression is: ; in, For decoders.
[0052] Step 31506: Retrieve the historical query frame at the current iteration number... Layer features and T historical memory frames at the current iteration number All layer features are input into the global feature compensation module at the current iteration number to obtain the global relationship features of the historical query frame at the current iteration number.
[0053] As an optional implementation, the global feature compensation module includes: a cross-attention aggregation unit and an attention mechanism unit.
[0054] Step 31506 includes steps S21-S22.
[0055] S21: Retrieve the T historical memory frames at the current iteration number. All layer features are input into the cross-attention aggregation unit at the current iteration number to obtain the global representation at the current iteration number.
[0056] Specifically, the expression for the cross-attention aggregation unit is: ; in, For global representation; This is a cross-attention aggregation operation; For the first The first historical memory frame Layer features.
[0057] S22: Combine the global representation and historical query frames at the current iteration number. All layer features are input into the attention mechanism unit at the current iteration number to obtain the global relation features of the historical query frame at the current iteration number.
[0058] Specifically, the expression for the attention mechanism unit is: ; ; in, The global relational features of historical query frames; For attention mechanisms; For query vector; For the first historical query frame Layer features; The key vector; It is a value vector.
[0059] Step 31507: Input the preliminary prediction mask of the historical query frame at the current iteration number and the global relation features of the historical query frame at the current iteration number into the linear fusion module at the current iteration number to obtain the predicted pixel mask of the historical query frame at the current iteration number.
[0060] Specifically, the expression for the linear fusion module is: ; in, This is a linear fusion operation; This is an upsampling operation.
[0061] Step 31508: Using the total loss function, determine the total loss for the current iteration number based on the true pixel mask of the historical query frame, the preliminary prediction mask of the historical query frame at the current iteration number, and the prediction pixel mask of the historical query frame at the current iteration number.
[0062] Step 31509: Determine whether the iteration stopping condition is met; the iteration stopping condition is that the preset number of iterations has been reached or the total loss under the current number of iterations is less than the preset loss.
[0063] Step 31510: If yes, then the video object segmentation network at the current iteration number is determined as the video object segmentation model.
[0064] Step 31511: If not, update the network parameters of the video object segmentation network at the current iteration number using the gradient descent algorithm, and return "Input the historical query frames into the visual backbone network at the current iteration number to obtain the multi-scale features of the historical query frames at the current iteration number".
[0065] Step 3, the process of determining the predicted pixel mask of the video frames at each time point from the second moment of the current time period to the last moment, includes steps 321-323.
[0066] Step 321: Input the video frame at time 2, the video frame at time 1, and the actual pixel mask of the video frame at time 1 into the pixel mask prediction model to obtain the predicted pixel mask of the video frame at time 2.
[0067] As an optional implementation, step 321 includes the following steps.
[0068] Step 3211: Input the video frame at the second time step into the visual backbone network of the pixel mask prediction model to obtain the multi-scale features of the video frame at the second time step; the multi-scale features include Layer features, .
[0069] Step 3212: Input the video frame at the first time step into the visual backbone network in the pixel mask prediction model to obtain the multi-scale features of the video frame at the first time step.
[0070] Step 3213: Input the real pixel mask of the video frame at the first time step into the visual backbone network in the pixel mask prediction model to obtain the multi-scale features of the real pixel mask of the video frame at the first time step.
[0071] Step 3214: Input the first layer features of the video frame at the second time step, the first layer features of the video frame at the first time step, and the first layer features of the real pixel mask of the video frame at the first time step into the fine-grained memory readout module in the pixel mask prediction model to obtain the preliminary predicted mask of the video frame at the second time step.
[0072] Step 3215: Transfer the video frame at time 2 to the first frame. Layer features and the first time frame of the video frame All layer features are input into the global feature compensation module in the pixel mask prediction model to obtain the global relation features of the video frame at the second time step.
[0073] Step 3216: Input the preliminary predicted mask of the video frame at the second time step and the global relational features of the video frame at the second time step into the linear fusion module in the pixel mask prediction model to obtain the predicted pixel mask of the video frame at the second time step.
[0074] Step 322: Determine any time from the 3rd moment to the last moment of the current time period as the current moment, determine the previous moment of the current moment as the previous moment, determine the video frames of each moment in the time period from the 2nd moment to the previous moment as the current predictive memory frames, and determine the video frame of the 1st moment as the current real memory frame.
[0075] Step 323: Input the current video frame, the current real memory frame, the real pixel mask of the current real memory frame, each current predicted memory frame, and the predicted pixel mask of each current predicted memory frame into the pixel mask prediction model to obtain the predicted pixel mask of the current video frame.
[0076] As an optional implementation, step 323 includes the following steps: Step 3231: Input the current video frame into the visual backbone network of the pixel mask prediction model to obtain the multi-scale features of the current video frame; the multi-scale features include Layer features, ; Step 3232: Input the current real memory frame into the visual backbone network in the pixel mask prediction model to obtain the multi-scale features of the current real memory frame; Step 3233: Input the real pixel mask of the current real memory frame into the visual backbone network in the pixel mask prediction model to obtain the multi-scale features of the real pixel mask of the current real memory frame; Step 3234: Input each current prediction memory frame into the visual backbone network in the pixel mask prediction model to obtain the multi-scale features of each current prediction memory frame; Step 3235: Input the real pixel mask of each current predicted memory frame into the visual backbone network in the pixel mask prediction model to obtain the multi-scale features of the real pixel mask of each current predicted memory frame. Step 3236: Input the first layer features of the current video frame, the first layer features of the current real memory frame, the first layer features of the real pixel mask of the current real memory frame, the first layer features of each current predicted memory frame, and the first layer features of the real pixel mask of each current predicted memory frame into the fine-grained memory readout module in the pixel mask prediction model to obtain the preliminary predicted mask of the current video frame. Step 3237: Set the current video frame to the [number]th [frame]. Layer features, the first real memory frame Layer features and the first layer of each current prediction memory frame All layer features are input into the global feature compensation module in the pixel mask prediction model to obtain the global relation features of the video frame at the current moment; Step 3238: Input the preliminary predicted mask of the video frame at the current moment and the global relational features of the video frame at the current moment into the linear fusion module in the pixel mask prediction model to obtain the predicted pixel mask of the video frame at the current moment.
[0077] This application has the following advantages: 1) Computational complexity is significantly reduced, and reasoning efficiency is higher: Shallow (i.e., layer 1) feature matching is achieved through local convolutional attention in the fine-grained memory readout module, avoiding the high computational cost of large-scale global matching. In the global feature compensation module, high-level features are aggregated to achieve lightweight global relationship modeling, making the overall inference speed significantly faster than traditional methods, while occupying less GPU memory, making it more suitable for deployment in resource-constrained real-world scenarios. 2) Superior segmentation accuracy and detail fidelity: Compared to global attention methods that rely solely on high-level features, this method fully utilizes shallow, high-resolution feature information to maintain clearer spatial details at the target boundary, making the predicted pixel mask more accurate and effectively alleviating problems such as blurred target edges and omission of small targets.
[0078] In one exemplary embodiment, a computer device is provided, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a semi-supervised video frame segmentation method.
[0079] In one exemplary embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements a semi-supervised video frame segmentation method.
[0080] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements a semi-supervised video frame segmentation method.
[0081] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 3As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and databases. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a semi-supervised video frame segmentation method.
[0082] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0083] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).
[0084] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.
[0085] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.
[0086] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0087] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A semi-supervised video frame segmentation method, characterized in that, The semi-supervised video frame segmentation method comprises: acquiring a current period video; the current period video comprises a plurality of continuous time point video frames; pixel-level labeling is performed on a first time point video frame in the current period video to obtain a real pixel mask of the first time point video frame; a pixel mask prediction model is used to determine, based on the real pixel mask of the first time point video frame and the first time point video frame, a predicted pixel mask of each time point video frame in the current period from a second time point in the current period, to complete segmentation of the video frames in each time point in the current period; the pixel mask prediction model is obtained by training a video object segmentation network, and the video object segmentation network comprises a visual backbone network, a fine-grained memory reading module, a global feature compensation module and a linear fusion module; wherein the process of determining the predicted pixel mask of each time point video frame in the period from the second time point to the last time point in the current period comprises: inputting the second time point video frame, the first time point video frame and the real pixel mask of the first time point video frame into the pixel mask prediction model to obtain the predicted pixel mask of the second time point video frame; determining any time point from a third time point to the last time point in the current period as a current time point, determining the previous time point of the current time point as a previous time point, determining each time point video frame in the period from the second time point to the previous time point as a current predicted memory frame, and determining the first time point video frame as a current real memory frame; inputting the current time point video frame, the current real memory frame, the real pixel mask of the current real memory frame, each current predicted memory frame and the predicted pixel mask of each current predicted memory frame into the pixel mask prediction model to obtain the predicted pixel mask of the current time point video frame.
2. The semi-supervised video frame segmentation method of claim 1, wherein, The process of determining the pixel mask prediction model comprises: acquiring a historical period video; the historical period video comprises T+1 continuous historical time point video frames; T≥1; pixel-level labeling is performed on each historical time point video frame to obtain a real pixel mask of each historical time point video frame; based on the T+1 historical time point video frames and the real pixel masks, a training set is constructed; initializing a video object segmentation network; based on the training set and a total loss function, the video object segmentation network is iteratively trained multiple times to obtain a video object segmentation model; the total loss function comprises: ； ； ； ； ； ； ； wherein, is the total loss; is the coarse segmentation loss; is the compensation segmentation loss; is the cross-entropy loss function; is the Dice loss function; is the preliminary predicted mask of the historical query frame; is the true pixel mask of the historical query frame; is the total number of pixels in the historical query frame; is the pixel value of the jth pixel in the true pixel mask of the historical query frame; is the pixel value of the jth pixel in the preliminary predicted mask of the historical query frame; is the smoothing term for maintaining numerical stability; is the pixel value of the jth pixel in the predicted pixel mask of the historical query frame.
3. The semi-supervised video frame segmentation method of claim 2, wherein, during the process of iteratively training the video object segmentation network multiple times based on the training set and the total loss function to obtain the video object segmentation model, the training process of the video object segmentation network at any current iteration number comprises: determining any iteration number as a current iteration number; the training process of the video object segmentation network at the current iteration number comprises: determining the T+1 time point video frame in the historical period as a historical query frame and any video frame in the first T time points in the historical period as a historical memory frame; The historical query frame is input into the visual backbone network at the current iteration number to obtain multi-scale features of the historical query frame at the current iteration number; the multi-scale features include layer features, When the current iteration number is the initial iteration number, the video object segmentation network at the current iteration number is an initialized video object segmentation network; inputting the T historical memory frames into the visual backbone network at the current iteration number to obtain multi-scale features of the T historical memory frames at the current iteration number; input the real pixel masks of the T historical memory frames into the visual backbone network at the current iteration number respectively to obtain multi-scale features of the real pixel masks of the T historical memory frames at the current iteration number; input the first layer feature of the historical query frame at the current iteration number, the first layer feature of the T historical memory frames at the current iteration number, and the first layer feature of the real pixel masks of the T historical memory frames at the current iteration number into the fine-grained memory reading module at the current iteration number to obtain the preliminary prediction mask of the historical query frame at the current iteration number; the first layer features and the T historical memory frames at the current iteration number are input into a global feature compensation module at the current iteration number to obtain global relationship features of the historical query frame at the current iteration number. layer features are input into a global feature compensation module at the current iteration number to obtain global relationship features of the historical query frame at the current iteration number. input the preliminary prediction mask of the historical query frame at the current iteration number and the global relationship feature of the historical query frame at the current iteration number into the linear fusion module at the current iteration number to obtain the predicted pixel mask of the historical query frame at the current iteration number; determine the total loss at the current iteration number based on the real pixel mask of the historical query frame, the preliminary prediction mask of the historical query frame at the current iteration number, and the predicted pixel mask of the historical query frame at the current iteration number by using the total loss function; determine whether the iteration stop condition is met; the iteration stop condition is that the preset iteration number is reached or the total loss at the current iteration number is less than the preset loss; if yes, determine the video object segmentation network at the current iteration number as the video object segmentation model; if no, update the network parameters of the video object segmentation network at the current iteration number by using the gradient descent algorithm, and return to "input the historical query frame into the visual backbone network at the current iteration number to obtain the multi-scale feature of the historical query frame at the current iteration number".
4. The semi-supervised video frame segmentation method of claim 3, wherein, the fine-grained memory reading module comprises a splicing unit, a fusion guide feature generation unit, an enhanced feature generation unit, a local reading feature generation unit, a fine-grained memory reading result generation unit, and a decoder; input the first layer feature of the historical query frame at the current iteration number, the first layer feature of the T historical memory frames at the current iteration number, and the first layer feature of the real pixel masks of the T historical memory frames at the current iteration number into the fine-grained memory reading module at the current iteration number to obtain the preliminary prediction mask of the historical query frame at the current iteration number, comprising: determine any one of the T historical memory frames as a target memory frame; input the first layer feature of the historical query frame at the current iteration number and the first layer feature of the target memory frame at the current iteration number into the splicing unit at the current iteration number to obtain the splicing feature of the historical query frame and the target memory frame at the current iteration number; input the splicing feature of the historical query frame and the target memory frame at the current iteration number into the fusion guide feature generation unit at the current iteration number to obtain the fusion guide feature of the target memory frame at the current iteration number; input the fusion guide feature of the target memory frame at the current iteration number and the first layer feature of the historical query frame at the current iteration number into the enhanced feature generation unit at the current iteration number to obtain the enhanced feature of the target memory frame at the current iteration number; The enhanced feature of the target memory frame at the current iteration number and the first layer feature of the real pixel mask of the target memory frame at the current iteration number are input into a local readout feature generation unit at the current iteration number to obtain a local readout feature of the target memory frame at the current iteration number; The local readout features of all the historical memory frames at the current iteration number are input into a fine-grained memory readout result generation unit at the current iteration number to obtain a fine-grained memory readout result at the current iteration number; The fine-grained memory readout result at the current iteration number is input into a decoder at the current iteration number to obtain a preliminary prediction mask of the historical query frame at the current iteration number.
5. The semi-supervised video frame segmentation method of claim 4, wherein, The global feature compensation module comprises a cross-attention aggregation unit and an attention mechanism unit; the first layer feature of the historical query frame at the current iteration number the first layer feature of the T historical memory frames at the current iteration number The first layer feature and the T historical memory frames at the current iteration number are input into a global feature compensation module at the current iteration number to obtain global relationship features of the historical query frame at the current iteration number, including: T historical memory frames at the current iteration number are input into the cross-attention aggregation unit at the current iteration number to obtain a global representation at the current iteration number. The layer features are all input into the cross-attention aggregation unit at the current iteration number to obtain a global representation at the current iteration number. The global representation at the current iteration number and the historical query frame at the current iteration number are input into the attention mechanism unit at the current iteration number to obtain the global relationship feature of the historical query frame at the current iteration number. The layer features are all input into the attention mechanism unit at the current iteration number to obtain the global relationship feature of the historical query frame at the current iteration number.
6. The method of semi-supervised video frame segmentation according to claim 1, wherein, The video frame at the second time, the video frame at the first time and the real pixel mask of the video frame at the first time are input into the pixel mask prediction model to obtain the predicted pixel mask of the video frame at the second time, comprising: The video frame at the second time is input into a visual backbone network in the pixel mask prediction model to obtain a multi-scale feature of the video frame at the second time; the multi-scale feature includes layer features, ; The video frame at the first time is input into the visual backbone network in the pixel mask prediction model to obtain the multi-scale feature of the video frame at the first time; The real pixel mask of the video frame at the first time is input into the visual backbone network in the pixel mask prediction model to obtain the multi-scale feature of the real pixel mask of the video frame at the first time; The first layer feature of the video frame at the second time, the first layer feature of the video frame at the first time and the first layer feature of the real pixel mask of the video frame at the first time are input into the fine-grained memory readout module in the pixel mask prediction model to obtain the preliminary prediction mask of the video frame at the second time; the first layer features of the video frame at the second time instant the first layer features of the video frame at the first time instant the first layer features of the video frame at the second time instant The preliminary prediction mask of the video frame at the second time and the global relationship feature of the video frame at the second time are input into the linear fusion module in the pixel mask prediction model to obtain the predicted pixel mask of the video frame at the second time.
7. The method of semi-supervised video frame segmentation according to claim 1, wherein, The video frame at the current time, the current real memory frame, the real pixel mask of the current real memory frame, each current predicted memory frame and the predicted pixel mask of each current predicted memory frame are input into the pixel mask prediction model to obtain the predicted pixel mask of the video frame at the current time, comprising: The video frame at the current moment is input into a visual backbone network in the pixel mask prediction model to obtain multi-scale features of the video frame at the current moment; the multi-scale features include layer features, The current real memory frame is input into the visual backbone network in the pixel mask prediction model to obtain the multi-scale feature of the current real memory frame; The real pixel mask of the current real memory frame is input into the visual backbone network in the pixel mask prediction model to obtain the multi-scale feature of the real pixel mask of the current real memory frame; Each current predicted memory frame is input into the visual backbone network in the pixel mask prediction model to obtain the multi-scale feature of each current predicted memory frame; The real pixel mask of each current predicted memory frame is input into the visual backbone network in the pixel mask prediction model to obtain the multi-scale feature of the real pixel mask of each current predicted memory frame; The first layer feature of the current time video frame, the first layer feature of the current real memory frame, the first layer feature of the real pixel mask of the current real memory frame, the first layer feature of each current prediction memory frame, and the first layer feature of the real pixel mask of each current prediction memory frame are input into a fine-grained memory reading module in the pixel mask prediction model to obtain a preliminary prediction mask of the current time video frame. The first layer feature of the video frame at the current moment The first layer feature of the current real memory frame The first layer feature of each current prediction memory frame The first layer feature of the video frame at the current moment The preliminary prediction mask of the current time video frame and the global relationship feature of the current time video frame are input into a linear fusion module in the pixel mask prediction model to obtain a prediction pixel mask of the current time video frame.
8. A computer apparatus comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the semi-supervised video frame segmentation method of any one of claims 1-7.
9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the semi-supervised video frame segmentation method of any one of claims 1-7.
10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the semi-supervised video frame segmentation method of any one of claims 1-7.