Microscopic cell video instance segmentation method and device

Abstract

The invention provides a microscopic cell video instance segmentation method and device, and relates to the technical field of biomedical image processing, and the method comprises the steps: obtaining a target microscopic cell video; inputting the target microscopic cell video into an instance segmentation model to obtain a single-cell video and single-cell trajectory information output by the instance segmentation model; wherein the instance segmentation model comprises a video input coding module, a prompt input coding module, a memory module, a memory attention module, a mask decoding module and a video segmentation module. According to the method and the device provided by the invention, the accuracy and the stability of instance segmentation in a complex microscopic cell video are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of biomedical image processing, and in particular to a method and device for segmenting microscopic cell video instances. Background Art

[0002] Microscopic cell video analysis is a key link in biomedical research. It not only supports early diagnosis and treatment of diseases, but also plays an important role in drug development, cell biology, genomics and other fields. However, as one of the core technologies in this field, microscopic cell video instance segmentation still faces many challenges. Factors such as the diversity of cell morphology, the complexity of the background environment and the uncertainty of the motion trajectory all increase the difficulty of implementing this task.

[0003] The methods in the related art are difficult to maintain stable tracking performance in long-term videos. Especially when the cell morphology changes significantly or is blocked or overlapped, these methods often encounter tracking failures or produce erroneous tracking results, and cannot accurately segment microscopic cell videos.

[0004] Therefore, how to improve the accuracy and stability of microscopic cell video instance segmentation has become a technical problem that needs to be urgently solved in the industry. Summary of the invention

[0005] The invention provides a microscopic cell video instance segmentation method and device, which are used to solve the technical problem of how to improve the accuracy and stability of microscopic cell video instance segmentation.

[0006] The present invention provides a microscopic cell video instance segmentation method, comprising: Acquire target microscopic cell video; Inputting the target microscopic cell video into an instance segmentation model to obtain a single cell video and single cell trajectory information output by the instance segmentation model; The instance segmentation model includes a video input encoding module, a prompt input encoding module, a memory module, a memory attention module, a mask decoding module and a video segmentation module; The video input encoding module is used to extract features from each video frame in the target microscopic cell video to obtain initial input features; the prompt input encoding module is used to extract features from cell image segmentation prompt information to obtain segmentation prompt features; the memory module is used to determine the segmentation prediction features of the next video frame based on the initial image features and the instance segmentation mask of the current video frame; the memory attention module is used to determine the enhanced input features based on the initial input features and the segmentation prediction features; the mask decoding module is used to determine the instance segmentation mask based on the enhanced input features and the segmentation prompt features; the video segmentation module is used to perform instance segmentation on each video frame in the target microscopic cell video based on the instance segmentation mask to obtain the single cell video and the single cell trajectory information.

[0007] In some embodiments, the prompt input encoding module is used to: Based on the prompt marks in each video frame in the target microscopic cell video, and / or based on the image detection model, target detection is performed on each video frame in the target microscopic cell video to determine the position point prompt feature, the position frame prompt feature and the position mask prompt feature; The segmentation hint feature is determined based on at least one of the position point hint feature, the position box hint feature and the position mask hint feature.

[0008] In some embodiments, the memory module includes a memory encoding submodule and a memory storage submodule; The memory encoding submodule is used to determine the object pointer feature of the current video frame based on the initial image feature and the instance segmentation mask of the current video frame in the target microscopic cell video; The memory storage submodule is used to store the object pointer features of the segmented video frames; the segmented video frames include the current video frame and a plurality of adjacent video frames before the current video frame; A graph network is established by taking single cells in each segmented video frame as nodes and taking object pointer features and instance position features in each segmented video frame as node features; The segmentation prediction features of the next video frame are determined based on the node features of each node in the graph network.

[0009] In some embodiments, the memory encoding submodule is used to: Downsampling the instance segmentation mask of the current video frame to obtain a segmentation mask feature of the current video frame; Based on the initial image features and the segmentation mask features of the current video frame, an object pointer feature of the current video frame is determined.

[0010] In some embodiments, the memory encoding submodule is used to: Processing the initial image features and segmentation mask features of the current video frame based on multiple stacked attention layers to determine a query vector matrix, a key vector matrix, and a value vector matrix of the current video frame at the current attention layer, as well as mask representation features and image representation features of the previous attention layer; Based on the mask representation feature, the query vector matrix, the key vector matrix, the value vector matrix, and the image representation feature, an object pointer feature of the current video frame is determined.

[0011] In some embodiments, the memory storage submodule is used to: In the current video frame, based on the node feature of any node and the node feature of any neighboring node of any node, determine the node association feature of any node and any neighboring node; Performing weighted aggregation on the node association features of any node and each adjacent node, and updating the node feature of any node; Based on the updated node features of each node, the segmentation prediction features of the next video frame are determined.

[0012] The present invention provides a microscopic cell video instance segmentation device, comprising: An acquisition unit, used for acquiring a target microscopic cell video; A segmentation unit, used for inputting the target microscopic cell video into an instance segmentation model to obtain a single cell video and single cell trajectory information output by the instance segmentation model; The instance segmentation model includes a video input encoding module, a prompt input encoding module, a memory module, a memory attention module, a mask decoding module and a video segmentation module; The video input encoding module is used to extract features from each video frame in the target microscopic cell video to obtain initial input features; the prompt input encoding module is used to extract features from cell image segmentation prompt information to obtain segmentation prompt features; the memory module is used to determine the segmentation prediction features of the next video frame based on the initial image features and the instance segmentation mask of the current video frame; the memory attention module is used to determine the enhanced input features based on the initial input features and the segmentation prediction features; the mask decoding module is used to determine the instance segmentation mask based on the enhanced input features and the segmentation prompt features; the video segmentation module is used to perform instance segmentation on each video frame in the target microscopic cell video based on the instance segmentation mask to obtain the single cell video and the single cell trajectory information.

[0013] The present invention provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and operable on the processor. When the processor executes the computer program, the microscopic cell video instance segmentation method is implemented.

[0014] The present invention provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the microscopic cell video instance segmentation method is implemented.

[0015] The present invention provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the microscopic cell video instance segmentation method is implemented.

[0016] The present invention provides a method and device for instance segmentation of a microscopic cell video. The target microscopic cell video is input into an instance segmentation model to obtain a single cell video and single cell trajectory information output by the instance segmentation model. The instance segmentation model includes a video input encoding module, a prompt input encoding module, a memory module, a memory attention module, a mask decoding module and a video segmentation module. The video input encoding module is used to extract features from each video frame in the target microscopic cell video to obtain initial input features. The prompt input encoding module is used to extract features from cell image segmentation prompt information to obtain segmentation prompt features. The memory module is used to determine the segmentation prediction of the next video frame based on the initial image features of the current video frame and the instance segmentation mask. Features; the memory attention module is used to determine the enhanced input features based on the initial input features and the segmentation prediction features; the mask decoding module is used to determine the instance segmentation mask based on the enhanced input features and the segmentation hint features; the video segmentation module is used to perform instance segmentation on each video frame in the target microscopic cell video based on the instance segmentation mask to obtain the single cell video and single cell trajectory information; due to the construction of an end-to-end network architecture, by integrating the information between video frames, instance segmentation and instance tracking are carried out synchronously, thus avoiding information loss and inconsistency, realizing video-level instance segmentation, improving the consistency and accuracy of instance segmentation, and improving the accuracy and stability of instance segmentation in complex microscopic cell videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 This is one of the flow charts of the microscopic cell video instance segmentation method provided by the present invention.

[0020] Figure 2 It is a model structure diagram of the instance segmentation model provided by the present invention.

[0021] Figure 3 This is a model structure diagram of the shielded attention mechanism provided by the present invention.

[0022] Figure 4 This is the second flow chart of the microscopic cell video instance segmentation method provided by the present invention.

[0023] Figure 5 It is a schematic diagram of the structure of the microscopic cell video instance segmentation device provided by the present invention.

[0024] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units or modules is not necessarily limited to those steps or units or modules that are clearly listed, but may include other steps or units or modules that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] The multi-target cell image segmentation and tracking methods in related technologies show certain limitations when processing microscopic cell images. For example, semantic segmentation algorithms based on deep learning (such as the U-Net model), although they can provide good segmentation results on a single frame image, fail to effectively integrate inter-frame information, resulting in poor performance when tracking dynamic changes in cells. Bounding box detection algorithms (such as the YOLO model), although they can quickly determine the location of cells, are insufficient in accurately segmenting cell details and are difficult to meet the needs of in-depth analysis.

[0028] In terms of target tracking, traditional tracking technologies (such as optical flow and Kalman filtering) also have limitations. Although the optical flow method can capture the movement path of cells, its tracking accuracy will drop significantly when the image quality is poor or the cell morphology changes. Prediction methods such as Kalman filtering rely on accurate prior knowledge and motion models, and the tracking effect is poor when faced with complex and changeable cell behavior patterns.

[0029] In order to solve the shortcomings of related technologies, Figure 1 FIG. 1 is one of the flow charts of the microscopic cell video instance segmentation method provided by the present invention, such as Figure 1 As shown, the method includes step 110 and step 120.

[0030] Step 110: Acquire the target microscopic cell video.

[0031] Specifically, the execution subject of the microscopic cell video instance segmentation method provided in the embodiment of the present invention is a microscopic cell video instance segmentation device. The device can be implemented by software, such as a microscopic cell video instance segmentation program; or by hardware, such as a computer or server that executes the microscopic cell video instance segmentation method.

[0032] Microscopic cell video refers to a dynamic video shot by microscope technology to show cell activities. The target microscopic cell video is the microscopic cell video that needs to be processed.

[0033] Step 120: Input the target microscopic cell video into the instance segmentation model to obtain the single cell video and single cell trajectory information output by the instance segmentation model.

[0034] Among them, the instance segmentation model includes a video input encoding module, a prompt input encoding module, a memory module, a memory attention module, a mask decoding module and a video segmentation module; the video input encoding module is used to extract features of each video frame in the target microscopic cell video to obtain initial input features; the prompt input encoding module is used to extract features of cell image segmentation prompt information to obtain segmentation prompt features; the memory module is used to determine the segmentation prediction features of the next video frame based on the initial image features of the current video frame and the instance segmentation mask; the memory attention module is used to determine the enhanced input features based on the initial input features and the segmentation prediction features; the mask decoding module is used to determine the instance segmentation mask based on the enhanced input features and the segmentation prompt features; the video segmentation module is used to perform instance segmentation on each video frame in the target microscopic cell video based on the instance segmentation mask to obtain single cell video and single cell trajectory information.

[0035] Specifically, instance segmentation refers to the use of computer vision technology to identify each object in the image and accurately segment the specific pixel area of ​​each object. In the application scenario of the embodiment of the present invention, instance segmentation refers to the identification and segmentation of pixel areas of each single cell in the target microscopic cell video.

[0036] An instance segmentation model can be constructed through a neural network to process each video frame in the target microscopic cell video to generate single-cell videos of each single cell and single-cell trajectory information of each single cell.

[0037] Single-cell video refers to a video containing only a single cell. Single-cell trajectory information refers to the activity trajectory of a single cell in the target microscopic cell video, which can be determined by the activity position of a single cell in the video.

[0038] A large number of microscopic cell video samples can be collected in advance, and the pixel areas of single cells in each video frame in the video samples can be marked to obtain the data mask of each single cell (to identify the position of the single cell in the image), cell markers (to distinguish different single cells) and trajectory information (activity trajectory of single cells). The neural network is used as the initial model, and the initial model is trained using microscopic cell video samples to obtain an instance segmentation model.

[0039] Figure 2 is a model structure diagram of the instance segmentation model provided by the present invention, such as Figure 2 As shown, the instance segmentation model 200 may include a video input encoding module 210 , a hint input encoding module 220 , a memory module 230 , a memory attention module 240 , a mask decoding module 250 , and a video segmentation module 260 .

[0040] The video input encoding module is used to extract features from each video frame in the target microscopic cell video to obtain initial input features. The initial input features are used to represent image features in each video frame.

[0041] The prompt input encoding module is used to extract features of the cell image segmentation prompt information to obtain segmentation prompt features. The cell image segmentation prompt information refers to information used to prompt the pixel position or spatial coordinates of each single cell in the video frame, which may include position points, position boxes, and position masks. The segmentation prompt features are used to represent the cell image segmentation prompt information.

[0042] The memory module is used to determine the segmentation prediction features of the next video frame based on the initial image features and instance segmentation mask of the current video frame. The instance segmentation mask refers to a binary image or matrix that distinguishes the pixel area where a single cell is located from the foreground pixel area (excluding a single cell) in a video frame, and can be used to filter, extract, modify or block specific parts of a video frame. The segmentation prediction feature refers to the feature that predicts the pixel area where a single cell is located in a video frame. The memory module can use the current video frame (historical information) to predict the next video frame (future information).

[0043] The memory attention module is used to determine the enhanced input features based on the initial input features and the segmentation prediction features. The enhanced input features refer to the input features obtained by enhancing the position information of the single cell in the initial input features. Compared with the initial input features, the position information of the single cell is more prominent. The enhanced input features can be obtained by fusing the initial input features and the segmentation prediction features.

[0044] The mask decoding module is used to fuse the enhanced input features and segmentation hint features to obtain the instance segmentation mask.

[0045] The video segmentation module is used to perform instance segmentation on each video frame in the target microscopic cell video according to the instance segmentation mask, combine the video frames containing only single cells obtained after instance segmentation to obtain single cell videos; and combine the position information of single cells in the video frames to obtain single cell trajectory information.

[0046] The embodiment of the present invention provides a method for instance segmentation of a microscopic cell video, wherein a target microscopic cell video is input into an instance segmentation model to obtain a single cell video and single cell trajectory information output by the instance segmentation model; the instance segmentation model comprises a video input encoding module, a prompt input encoding module, a memory module, a memory attention module, a mask decoding module and a video segmentation module; the video input encoding module is used to extract features from each video frame in the target microscopic cell video to obtain initial input features; the prompt input encoding module is used to extract features from cell image segmentation prompt information to obtain segmentation prompt features; the memory module is used to determine the segmentation prediction of the next video frame based on the initial image features of the current video frame and the instance segmentation mask Features; the memory attention module is used to determine the enhanced input features based on the initial input features and the segmentation prediction features; the mask decoding module is used to determine the instance segmentation mask based on the enhanced input features and the segmentation hint features; the video segmentation module is used to perform instance segmentation on each video frame in the target microscopic cell video based on the instance segmentation mask to obtain the single cell video and single cell trajectory information; due to the construction of an end-to-end network architecture, by integrating the information between video frames, instance segmentation and instance tracking are carried out synchronously, thus avoiding information loss and inconsistency, realizing video-level instance segmentation, improving the consistency and accuracy of instance segmentation, and improving the accuracy and stability of instance segmentation in complex microscopic cell videos.

[0047] It should be noted that each implementation of the present invention can be freely combined, the order can be changed, or it can be executed separately, and does not need to rely on or depend on a fixed execution order.

[0048] In some embodiments, the prompt input encoding module is used to: Based on the prompt marks in each video frame in the target microscopic cell video, and / or based on the image detection model, target detection is performed on each video frame in the target microscopic cell video to determine the position point prompt feature, the position frame prompt feature and the position mask prompt feature; The segmentation hint feature is determined based on at least one of the position point hint feature, the position box hint feature and the position mask hint feature.

[0049] Specifically, the prompt input encoding module is mainly used to process the input prompt information and convert it into a feature representation that can be used for subsequent processing.

[0050] The prompt features may include position point prompt features, position frame prompt features and position mask prompt features. The position point prompt feature can prompt the specific position point of a single cell in a video frame; the position frame prompt feature can prompt the pixel position area where a single cell is located in a video frame; the position mask prompt feature can mask each pixel position in a video frame and prompt the cell area and foreground area.

[0051] The location point hint feature and the location box hint feature can be considered as sparse input; the location mask hint feature can be considered as dense input. Sparse input is suitable for identifying a small number of key locations or areas; dense input is used to accurately describe the distribution of features in a larger range. Through a variety of input methods, it can flexibly respond to the needs of different application scenarios and ensure the effective processing and encoding of various hint information.

[0052] There are two ways to obtain prompt features, or the two ways can be combined to obtain prompt features. The first way is to manually mark each video frame in the human-computer interaction interface to determine the prompt mark, including position points, position boxes, and position masks, so as to perform manual error correction in the subsequent frame segmentation process; the second way is to use an image detection model (such as the YOLO model) to perform target detection on each video frame in the target microscopic cell video to achieve automatic marking of the target area, thereby improving efficiency and accuracy. The combination of these two prompt methods not only enhances the flexibility and adaptability of the system, but also provides strong support for target recognition in complex scenes.

[0053] The location point prompt feature, the location box prompt feature and the location mask prompt feature can be fused to obtain the segmentation prompt feature.

[0054] In the process of processing the location point prompt feature and the location box prompt feature, the original dimension of the location point prompt feature can be 1*1*2 (width*height*channel), which is padded to the standard format dimension using the Pad method, for example, it can be 1*2*2 dimension, and feature encoding is performed after adding the rotation position encoding, and then the feature encoding of 1*2*256 dimension is obtained through multi-layer perceptron network mapping. For the location box prompt feature, its original dimension is 1*2*2, and no additional dimension padding step is required. The rotation position encoding is directly added and the multi-layer perceptron network mapping is created to obtain the feature encoding of 1*2*256 dimension. The feature encodings of the location point prompt feature and the location box prompt feature are merged based on the dimension channel to form a comprehensive and complete sparse position encoding, which can contain the information of the location point and the location box at the same time, providing a richer and more comprehensive feature representation for the subsequent processing steps.

[0055] In the process of processing the position mask hint feature, a method based on depthwise separable convolution can be used to process the position mask hint feature. This method first downsamples the input image to reduce the amount of calculation and memory consumption while maintaining important spatial information. The output is a dense feature encoding of 1*256*64*64 (batch*width*height*channel) dimensions, which is suitable for application scenarios that require high-precision feature representation.

[0056] The microscopic cell video instance segmentation method provided by the embodiment of the present invention determines the segmentation prompt feature according to at least one of the position point prompt feature, the position frame prompt feature and the position mask prompt feature, thereby improving the accuracy of the segmentation prompt feature and the accuracy of the instance segmentation.

[0057] In some embodiments, the memory module includes a memory encoding submodule and a memory storage submodule; The memory encoding submodule is used to determine the object pointer features of the current video frame based on the initial image features and the instance segmentation mask of the current video frame in the target microscopic cell video; The memory storage submodule is used to store the object pointer features of the segmented video frames; the segmented video frames include the current video frame and a plurality of neighboring video frames before the current video frame; A graph network is established by taking single cells in each segmented video frame as nodes and taking object pointer features and instance position features in each segmented video frame as node features; The segmentation prediction features of the next video frame are determined based on the node features of each node in the graph network.

[0058] Specifically, from a functional point of view, the memory module may include a memory encoding submodule and a memory storage submodule.

[0059] The memory encoding submodule is used to perform feature fusion on the initial image features and instance segmentation mask of the current video frame in the target microscopic cell video to obtain the object pointer feature that determines the current video frame. The object pointer feature refers to the feature vector used to point to the location of a single cell. The object pointer feature is also the feature of the object of interest (single cell). The initial image feature can retain the overall features of the current frame, providing a basis for subsequent feature comparison and fusion. The object pointer feature obtained after fusion with the instance segmentation mask not only reduces the dimension of the feature, but also retains the main structural information of the mask, which is convenient for subsequent feature matching and tracking.

[0060] The memory storage submodule is used to store the object pointer features of the segmented video frames. The segmented video frames include the current video frame and multiple neighboring video frames. The neighboring video frames refer to multiple video frames whose instance segmentation order is before the current video frame, which can provide historical information.

[0061] The memory storage submodule can establish a graph network with single cells in each segmented video frame as nodes and object pointer features and instance position features in each segmented video frame as node features. Instance position features refer to features extracted from information such as the spatial position coordinates of a single cell in a video frame. After establishing the graph network, the memory storage submodule can capture temporal correlation and spatial proximity based on the node features of each node, thereby better understanding the movement and changes of the object (single cell), thereby generating segmentation prediction features for the next video frame.

[0062] In the microscopic cell video instance segmentation method provided by an embodiment of the present invention, the memory encoding submodule can reduce the feature dimensions that need to be stored, and the memory storage submodule can capture the temporal correlation and spatial proximity by constructing a graph network, so as to better predict the segmentation prediction features of the next video frame, thereby improving the accuracy of instance segmentation.

[0063] In some embodiments, the memory encoding submodule is used to: Down-sampling the instance segmentation mask of the current video frame to obtain the segmentation mask feature of the current video frame; Based on the initial image features and the segmentation mask features of the current video frame, the object pointer features of the current video frame are determined.

[0064] Specifically, the memory encoding submodule can downsample the instance segmentation mask of the current video frame by using the convolution module to generate the segmentation mask feature of the current video frame, and fuse the initial image feature and the segmentation mask feature of the current video frame to obtain the object pointer feature of the current video frame.

[0065] The microscopic cell video instance segmentation method provided by the embodiment of the present invention can reduce the dimension of the feature by generating object pointer features, retain the main structural information of the mask, and facilitate subsequent feature matching and tracking.

[0066] In some embodiments, the memory encoding submodule is used to: Based on multiple stacked attention layers, the initial image features and segmentation mask features of the current video frame are processed to determine the query vector matrix, key vector matrix and value vector matrix of the current video frame in the current attention layer, as well as the mask representation features and image representation features of the previous attention layer; Based on the mask representation feature, the query vector matrix, the key vector matrix, the value vector matrix, and the image representation feature, an object pointer feature of the current video frame is determined.

[0067] Specifically, the memory encoding submodule can adopt a masked attention mechanism, which utilizes the foreground area identified by each segmentation mask feature to filter out the local features of the object of current interest (object pointer features) in the initial image features instead of extracting the entire feature map.

[0068] Figure 3 is a model structure diagram of the shielded attention mechanism provided by the present invention, such as Figure 3As shown in the figure, the model of the masked attention mechanism can include a masked attention layer, a first normalization layer (Add&Norm), a feedforward neural network layer (Feed Forward), and a second normalization layer (Add&Norm) connected in sequence. The masked attention layer can be composed of multiple stacked attention layers. The feature update process adopts a residual connection method so that the original features can be directly passed to the next layer, thereby maintaining information integrity.

[0069] According to the current attention layer in the masked attention layer By processing the current video frame, we can get the query vector matrix , key vector matrix Sum vector matrix The calculation formula of the current attention layer can be expressed as: .

[0070] in, is the current attention layer The output image represents the features, For the previous attention layer The output image represents the features, is the transposition operator symbol, For the previous attention layer The output mask represents the feature.

[0071] .

[0072] in, is the pixel coordinate, For the previous attention layer The mask prediction output obtained after processing the segmentation mask features is used to indicate whether to mask the feature channel of the current layer, where 1 represents the foreground area and other values ​​represent the background area, which can be distinguished by a threshold of 0.5.

[0073] The microscopic cell video instance segmentation method provided by an embodiment of the present invention uses a shielded attention mechanism to prevent the model from being disturbed by global background information when processing local features, thereby improving the accuracy of segmentation and reducing unnecessary computational burden. The feature information of each segmentation instance is embedded as an object pointer and saved in a memory storage submodule to facilitate the transmission of segmentation information of subsequent frames. In this way, the model can effectively utilize historical information and improve object tracking and segmentation performance in the video.

[0074] In some embodiments, the memory storage submodule is used to: In the current video frame, based on the node feature of any node and the node feature of any neighboring node of any node, determine the node association feature of any node and any neighboring node; Perform weighted aggregation on the node-related features of any node and each neighboring node, and update the node features of any node; Based on the updated node features of each node, the segmentation prediction features of the next video frame are determined.

[0075] Specifically, the memory storage submodule is mainly used to save and transmit feature information to improve the continuity and accuracy of segmentation information transmission of subsequent frames.

[0076] The memory storage submodule can maintain two first-in-first-out queues (FIFO). The first FIFO queue is used to store the maximum The object pointer features of the nearest video frames ensure that the model can access the features of the most recent frames and provide context information for the segmentation and tracking of the next frame; the second first-in-first-out queue is used to store the most recent frames. The object pointer features of neighboring video frames with hints retain the past prediction information of the target object in the video, which helps the model to use historical hints when processing the next frame and improve the accuracy of segmentation and tracking.

[0077] Each object pointer feature is combined with the corresponding spatial coordinate location information (instance location feature) as the node feature of the graph network. Each node represents an object instance, and the feature includes the object pointer feature and the spatial location information. Then, connect each adjacent The nodes of the frame and The model can capture temporal correlation and spatial proximity to better understand the movement and change of objects.

[0078] In order to update node feature information, in the graph network, each node exchanges information with its connected neighboring nodes and updates the features of the current node by aggregating the features of the neighboring nodes. The specific implementation method is as follows: Defining Nodes Initial node features Includes the object pointer features and instance position features in the current frame. A multi-layer perceptron network (MLP) is used for processing. The node association features are the messages that need to be transmitted between nodes, which can be used with the message function Indicates that it is used to compute nodes To Node The information transmission can be expressed as: .

[0079] in, Respectively Node in layer and nodes The feature vector of . The attention coefficient of each node is calculated using the attention mechanism , the expression is: .

[0080] in, For the Node in layer and nodes The attention coefficient between is the activation function, is the natural exponential function, is the weight matrix, For all nodes The nodes.

[0081] Use the attention coefficient to weight the node-related features (messages) of neighboring nodes and update the node The specific calculation of the eigenvector is as follows: .

[0082] In order to maintain the integrity of the information, a residual connection is used to add the updated feature vector to the original feature vector. The specific calculation formula is as follows: .

[0083] Finally, the prediction layer of the multi-layer perceptron is used to determine the segmentation prediction features of the next video frame based on the updated node features of each node in the current frame. , expressed as: .

[0084] In the microscopic cell video instance segmentation method provided by the embodiment of the present invention, due to the structural characteristics of the graph network, each node contains the feature information of the adjacent nodes. Therefore, the node feature of the current frame is the result derived by referring to the feature information of all related nodes in the past. Through the above method, the node feature information can be updated efficiently, and the continuity and accuracy of the segmentation information transmission of subsequent frames can be improved, thereby enhancing the object tracking and segmentation performance of the model in the video sequence.

[0085] In some embodiments, the video input encoding module is used to map the image of each frame in the video into a feature space to obtain initial input features.

[0086] The video input encoding module can adopt a streaming method to ensure that the video input encoding module is only run once during the entire interaction process. This method can not only generate feature codes for each frame of the image, but also significantly reduce the hardware burden and improve the processing speed.

[0087] The video input encoding module uses the Hiera image encoder pre-trained by masked autoencoder (MAE), which can provide multi-layer multi-scale features and extract rich feature information from the input video frames. These multi-scale features help capture different details and dynamic changes of the target object, thereby improving the accuracy of segmentation and tracking.

[0088] The video input encoding module receives video frames in a streaming manner, and processes each received frame in real time to generate the corresponding feature code. Using the pre-trained Hiera image encoder, each frame is multi-level encoded, and features of different scales are gradually extracted through multiple convolutional layers and pooling layers to generate multi-layer feature maps. The generated multi-scale feature map contains rich information from local details to global structures, and can more accurately describe the target objects in the video frame and their dynamic changes.

[0089] In some embodiments, the memory attention module is implemented by stacking multiple Transformer blocks. Among them, the first Transformer block receives the image encoding of the current frame to be predicted as input. Each Transformer block first performs a self-attention calculation to capture the dependencies within the current frame, ensuring that the model can understand the interactions between the various parts of the current frame. Subsequently, cross-attention processing is performed on the features of adjacent frames and object pointer features stored in the memory module to integrate historical information. In this way, the model can not only utilize the information of the current frame, but also effectively utilize the context and object pointer features of past frames to better understand and predict the content of the current frame.

[0090] The updated feature information is passed to the mask decoding module through a multi-layer perceptron. The multi-layer perceptron is responsible for further processing and transforming the features to make them more suitable for the subsequent mask generation task. The whole process ensures that the model can accurately adjust and optimize the feature representation of the current frame based on historical information and new cues, thereby improving the overall prediction accuracy and robustness.

[0091] In some embodiments, the mask decoding module interactively decodes the enhanced input features and segmentation hint features by stacking cross-attention blocks and self-attention blocks, and outputs instance segmentation masks, IOU (Intersection over Union, a measure of the accuracy of detecting corresponding objects in a specific data set) and occlusion predictions. The specific implementation method is as follows: The mask decoding module stacks a Transformer-based bidirectional cross-attention module to alternately update the enhanced input features and segmentation hint features to achieve deep information interaction. This process enables the model to perform image instance segmentation tasks through hint embedding, ensuring sufficient communication and fusion between hint information and image features.

[0092] When the augmented input features may have multiple compatible target masks, multiple mask outputs are predicted simultaneously to ensure that the model output must include a valid output mask. This step helps handle multi-target scenarios and improves the robustness and accuracy of the model.

[0093] The mask decoding module uses the token embedding output by the cross-attention block to predict the intersection-over-union ratio of the current mask through a multi-layer perceptron network and outputs a prediction score. This mechanism helps to evaluate the quality of each predicted mask and select the best mask output.

[0094] The mask decoding module adds an extra header to predict whether the object of interest exists in the current frame, and handles the problem that there is no valid object in the current frame due to occlusion. This module effectively helps the model better understand and handle occlusion when processing video prediction tasks, improving the adaptability and accuracy of the model.

[0095] The mask decoding module uses the feature image encoding from the memory attention module and the current frame image encoding to merge high-resolution information through skip connections for mask decoding. This fusion of high-resolution information helps the model retain more details when generating masks and improves segmentation accuracy.

[0096] In some embodiments, the video segmentation module performs instance segmentation on each video frame in the target microscopic cell video according to the instance segmentation mask to obtain single cell video and single cell trajectory information. Specifically: First, the video frames are input into the model, which generates segmentation masks and trajectory information for each frame. The instance segmentation mask is used to identify the location of the single cell, and the trajectory information is used to record the movement path of each cell in the video sequence. In this way, the model can accurately identify and track the dynamic changes of each single cell in the video.

[0097] The video segmentation module segments the single cells in each frame based on the generated segmentation mask. Specifically, the mask is used to separate each cell from the background to generate a single cell image segment. This process ensures that the image information of each single cell is extracted independently, which is convenient for subsequent analysis and processing. The single cell image segment in each frame is saved to form a single cell video sequence.

[0098] The video segmentation module outputs single-cell video sequences and their trajectory information. The image segments of each single cell are arranged in chronological order to generate a single-cell video sequence. At the same time, the trajectory information of each cell is output together to record the movement path of each cell in the video sequence. These trajectory information include the position, speed and direction of the cell, etc., providing detailed data support for subsequent biological analysis and research.

[0099] Through the above method, the model can efficiently segment and extract single-cell images, output high-quality single-cell video sequences and their trajectory information, and provide strong support for single-cell dynamic analysis and biological research.

[0100] Figure 4 FIG. 2 is a flow chart of the microscopic cell video instance segmentation method provided by the present invention. Figure 4 As shown, the method includes: Step 410: collect high-quality microscopic cell video datasets to ensure that the data covers various cell types and their dynamic changes, and use semi-automatic semantic segmentation and annotation methods to annotate data masks, identification (ID) information, and trajectory information. Preprocess the data through standardized preprocessing methods, including image size standardization, color correction, noise removal, etc., to eliminate differences between data from different sources. Finally, divide the dataset into 70% training set, 15% validation set, and 15% test set to ensure the effectiveness of model learning and the accuracy of evaluation.

[0101] Step 420: Encode the input video sequence in combination with the Hiera video encoder pre-trained with MAE to provide multi-layer and multi-scale embedding features including the entire video.

[0102] Step 430: Use a sparse encoder and a dense encoder to encode the bounding box, positive and negative marker points, and mask prompt information of the current segmentation frame, and concatenate them in the feature dimension to obtain complete segmentation prompt embedded feature information.

[0103] Step 440: Set up a memory attention mechanism to adjust the features of the current frame, and optimize the feature representation of the current frame by integrating the features and prediction results of past frames and any new hints.

[0104] Step 450: For the feature vectors obtained by video encoding and hint encoding, interactive decoding is performed by stacking cross-attention blocks and self-attention blocks to obtain mask output, output IOU, and output occlusion. The output IOU and output occlusion are directly output through the multi-layer perceptron to output the object prediction IOU score and object occlusion prediction score. The mask output is then multiplied with the image embedding feature to finally obtain the instance segmentation mask output.

[0105] Step 460: extract and encode instance segmentation information of the current frame. Use the masked attention mechanism to restrict each predicted mask foreground area to extract local features instead of extracting the complete feature map. In this way, feature information of each segmentation instance is obtained and saved.

[0106] Step 470: Establish a graph relationship network through the segmentation results of each frame, use the segmentation result of each instance as the node information of the graph network, and predict the segmentation prompt information of the next frame through the message passing network.

[0107] Step 480: According to the model prediction results, single-cell image segmentation and extraction are performed, and single-cell video sequences and their trajectory information are output.

[0108] The method provided by the embodiment of the present invention establishes an end-to-end network architecture and avoids the disadvantage of the traditional segmentation and tracking method being performed step by step by integrating the information flow between video frames. The traditional method often performs segmentation first and then tracks, which may lead to information loss and inconsistency. However, when processing video data, the method of the present invention can simultaneously consider the information of the current frame and the historical frame, realize the instance segmentation at the video level, and improve the consistency and accuracy of the segmentation.

[0109] The present invention introduces a memory architecture based on the message passing mechanism of the graph network to achieve efficient transmission of information between frames. The segmentation result of each frame is used as node information through the graph network, and the segmentation prompt information of the next frame is predicted through the message passing network, avoiding the error accumulation caused by the isolation of inter-frame information in the traditional method. In addition, this memory architecture enables the model to have zero-shot generalization ability, and can work on unseen data without additional training, which enhances the adaptability and flexibility of the model.

[0110] The present invention uses a masked attention mechanism to extract only the feature parameters of the region of interest rather than the complete feature map. This method not only reduces unnecessary computational burden, but also improves processing speed and segmentation accuracy. Because it only focuses on the cells themselves and their changes, it eliminates the influence of background noise, making the segmentation results more accurate and reliable.

[0111] The following describes an apparatus provided by an embodiment of the present invention. The apparatus described below and the method described above can refer to each other.

[0112] Figure 5 Schematic diagram of the structure of the microscopic cell video segmentation device provided by the present invention. Figure 5 As shown, the device comprises: An acquisition unit 510 is used to acquire a target microscopic cell video; A segmentation unit 520, used to input the target microscopic cell video into the instance segmentation model to obtain a single cell video and single cell trajectory information output by the instance segmentation model; Among them, the instance segmentation model includes a video input encoding module, a prompt input encoding module, a memory module, a memory attention module, a mask decoding module and a video segmentation module; The video input encoding module is used to extract features from each video frame in the target microscopic cell video to obtain initial input features; the prompt input encoding module is used to extract features from the cell image segmentation prompt information to obtain segmentation prompt features; the memory module is used to determine the segmentation prediction features of the next video frame based on the initial image features and the instance segmentation mask of the current video frame; the memory attention module is used to determine the enhanced input features based on the initial input features and the segmentation prediction features; the mask decoding module is used to determine the instance segmentation mask based on the enhanced input features and the segmentation prompt features; the video segmentation module is used to perform instance segmentation on each video frame in the target microscopic cell video based on the instance segmentation mask to obtain single cell video and single cell trajectory information.

[0113] The microscopic cell video instance segmentation device provided by the embodiment of the present invention inputs the target microscopic cell video into the instance segmentation model to obtain the single cell video and the single cell trajectory information output by the instance segmentation model; the instance segmentation model includes a video input encoding module, a prompt input encoding module, a memory module, a memory attention module, a mask decoding module and a video segmentation module; the video input encoding module is used to extract features of each video frame in the target microscopic cell video to obtain initial input features; the prompt input encoding module is used to extract features of the cell image segmentation prompt information to obtain segmentation prompt features; the memory module is used to determine the segmentation prediction of the next video frame based on the initial image features of the current video frame and the instance segmentation mask Features; the memory attention module is used to determine the enhanced input features based on the initial input features and the segmentation prediction features; the mask decoding module is used to determine the instance segmentation mask based on the enhanced input features and the segmentation hint features; the video segmentation module is used to perform instance segmentation on each video frame in the target microscopic cell video based on the instance segmentation mask to obtain the single cell video and single cell trajectory information; due to the construction of an end-to-end network architecture, by integrating the information between video frames, instance segmentation and instance tracking are carried out synchronously, thus avoiding information loss and inconsistency, realizing video-level instance segmentation, improving the consistency and accuracy of instance segmentation, and improving the accuracy and stability of instance segmentation in complex microscopic cell videos.

[0114] Figure 6 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 6As shown, the electronic device may include: a processor (Processor) 610, a communication interface (Communications Interface) 620, a memory (Memory) 630 and a communication bus (Communications Bus) 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic command in the memory 630 to execute the method described in the above embodiment, for example: Obtain a target microscopic cell video; input the target microscopic cell video into an instance segmentation model to obtain a single cell video and single cell trajectory information output by the instance segmentation model; wherein the instance segmentation model includes a video input encoding module, a prompt input encoding module, a memory module, a memory attention module, a mask decoding module and a video segmentation module; the video input encoding module is used to extract features from each video frame in the target microscopic cell video to obtain an initial input feature; the prompt input encoding module is used to extract features from cell image segmentation prompt information to obtain a segmentation prompt feature; the memory module is used to determine the segmentation prediction feature of the next video frame based on the initial image feature of the current video frame and the instance segmentation mask; the memory attention module is used to determine the enhanced input feature based on the initial input feature and the segmentation prediction feature; the mask decoding module is used to determine the instance segmentation mask based on the enhanced input feature and the segmentation prompt feature; the video segmentation module is used to perform instance segmentation on each video frame in the target microscopic cell video based on the instance segmentation mask to obtain a single cell video and single cell trajectory information.

[0115] In addition, the logic commands in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several commands to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0116] The processor in the electronic device provided in the embodiment of the present invention can call the logic instructions in the memory to implement the above method. Its specific implementation method is consistent with the implementation method of the aforementioned method and can achieve the same beneficial effects, which will not be repeated here.

[0117] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method provided in the above embodiments is implemented.

[0118] Its specific implementation is consistent with the aforementioned method implementation and can achieve the same beneficial effects, so it will not be repeated here.

[0119] An embodiment of the present invention provides a computer program product, including a computer program. When the computer program is executed by a processor, the method described above is implemented.

[0120] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0121] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for segmenting microscopic cell video instances, characterized in that: include: Acquire target microscopic cell video; Inputting the target microscopic cell video into an instance segmentation model to obtain a single cell video and single cell trajectory information output by the instance segmentation model; The instance segmentation model includes a video input encoding module, a prompt input encoding module, a memory module, a memory attention module, a mask decoding module and a video segmentation module; The video input encoding module is used to extract features from each video frame in the target microscopic cell video to obtain initial input features; the prompt input encoding module is used to extract features from the cell image segmentation prompt information to obtain segmentation prompt features; the memory module is used to determine the segmentation prediction features of the next video frame based on the initial image features of the current video frame and the instance segmentation mask; The memory attention module is used to determine the enhanced input features based on the initial input features and the segmentation prediction features; the mask decoding module is used to determine the instance segmentation mask based on the enhanced input features and the segmentation hint features; the video segmentation module is used to perform instance segmentation on each video frame in the target microscopic cell video based on the instance segmentation mask to obtain the single cell video and the single cell trajectory information.

2. The microscopic cell video instance segmentation method according to claim 1, characterized in that: The prompt input encoding module is used for: Based on the prompt marks in each video frame in the target microscopic cell video, and / or based on the image detection model, target detection is performed on each video frame in the target microscopic cell video to determine the position point prompt feature, the position frame prompt feature and the position mask prompt feature; The segmentation hint feature is determined based on at least one of the position point hint feature, the position box hint feature and the position mask hint feature.

3. The microscopic cell video instance segmentation method according to claim 1, characterized in that: The memory module includes a memory encoding submodule and a memory storage submodule; The memory encoding submodule is used to determine the object pointer feature of the current video frame based on the initial image feature and the instance segmentation mask of the current video frame in the target microscopic cell video; The memory storage submodule is used to store the object pointer features of the segmented video frames; the segmented video frames include the current video frame and a plurality of adjacent video frames before the current video frame; A graph network is established by taking single cells in each segmented video frame as nodes and taking object pointer features and instance position features in each segmented video frame as node features; The segmentation prediction features of the next video frame are determined based on the node features of each node in the graph network.

4. The microscopic cell video instance segmentation method according to claim 3, characterized in that: The memory encoding submodule is used for: Downsampling the instance segmentation mask of the current video frame to obtain a segmentation mask feature of the current video frame; Based on the initial image features and the segmentation mask features of the current video frame, an object pointer feature of the current video frame is determined.

5. The microscopic cell video instance segmentation method according to claim 4, characterized in that: The memory encoding submodule is used for: Processing the initial image features and segmentation mask features of the current video frame based on multiple stacked attention layers to determine a query vector matrix, a key vector matrix, and a value vector matrix of the current video frame at the current attention layer, as well as mask representation features and image representation features of the previous attention layer; Based on the mask representation feature, the query vector matrix, the key vector matrix, the value vector matrix, and the image representation feature, an object pointer feature of the current video frame is determined.

6. The microscopic cell video instance segmentation method according to claim 3, characterized in that: The memory storage submodule is used for: In the current video frame, based on the node feature of any node and the node feature of any neighboring node of any node, determine the node association feature of any node and any neighboring node; Performing weighted aggregation on the node association features of any node and each adjacent node, and updating the node feature of any node; Based on the updated node features of each node, the segmentation prediction features of the next video frame are determined.

7. A microscopic cell video instance segmentation device, characterized in that: include: An acquisition unit, used for acquiring a target microscopic cell video; A segmentation unit, used for inputting the target microscopic cell video into an instance segmentation model to obtain a single cell video and single cell trajectory information output by the instance segmentation model; The instance segmentation model includes a video input encoding module, a prompt input encoding module, a memory module, a memory attention module, a mask decoding module and a video segmentation module; The video input encoding module is used to extract features from each video frame in the target microscopic cell video to obtain initial input features; the prompt input encoding module is used to extract features from the cell image segmentation prompt information to obtain segmentation prompt features; the memory module is used to determine the segmentation prediction features of the next video frame based on the initial image features of the current video frame and the instance segmentation mask; The memory attention module is used to determine the enhanced input features based on the initial input features and the segmentation prediction features; the mask decoding module is used to determine the instance segmentation mask based on the enhanced input features and the segmentation hint features; the video segmentation module is used to perform instance segmentation on each video frame in the target microscopic cell video based on the instance segmentation mask to obtain the single cell video and the single cell trajectory information.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the microscopic cell video instance segmentation method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the microscopic cell video instance segmentation method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the microscopic cell video instance segmentation method according to any one of claims 1 to 6 is implemented.

Images