A semi-supervised optical flow prediction method for cell migration path tracking

Abstract

A semi-supervised optical flow prediction method for cell migration path tracking is proposed. It addresses the technical challenge of directly applying models trained using existing synthetic optical flow data to cell tracking. First, recognizing the limitations of existing convolutional kernels fixed in handling non-rigid cell motion, a novel heterogeneous convolutional block with a specific convolutional mode is proposed to accurately model the details of motion edges in cell images. Second, to address the difficulty in distinguishing motion in cell image sequences due to dense textures, a multi-dimensional feature fusion mechanism is established. This mechanism integrates multi-dimensional feature relationships across horizontal, vertical, and channel dimensions, ensuring the model can more comprehensively and accurately capture dynamic features in cell images. Third, to address the issue of missing cell image labels, a semi-supervised learning method is explored to reduce reliance on accurate labels.

Description

Technical Field

[0001] The invention belongs to the technical field of cell migration path tracking, and in particular relates to a semi-supervised optical flow prediction method for cell migration path tracking. Background Art

[0002] In cell tracking, dense optical flow estimation can not only provide pixel-level motion vectors for adjacent frames, but also reveal the details of cell motion between consecutive frames. This motion information can be used to analyze the motion patterns of cells, including speed, direction, and acceleration, to assist doctors in analyzing the growth and diffusion of cells. By estimating the optical flow of multiple cells, we can study the interactions and group behaviors between cells and explore the dynamic relationships between cells in tissues, development, and diseases. Therefore, accurately estimating dense optical flow in cell image sequences is of great significance for understanding the biological characteristics and pathological mechanisms of specific diseases and developing potential treatments.

[0003] Traditional dense optical flow estimation estimates the optical flow field based on brightness constancy and minimization of the objective function. Although dense optical flow estimation has advantages in providing global motion information, its high computational cost and dependence on the assumption of brightness constancy limit its applicability in some scenarios. With the continuous development of deep learning methods, convolutional neural networks have gradually surpassed the performance of traditional methods in dense optical flow estimation. In convolutional neural network optical flow estimation, supervised learning, unsupervised learning and semi-supervised learning are three different learning paradigms.

[0004] Supervised learning can achieve high accuracy by training on labeled datasets, such as Sintel, FlyingChairs, and FlyingThings. Such models can learn the mapping relationship between input and output. However, supervised learning requires a large amount of labeled data, and its generalization ability is limited when faced with unseen data. Unsupervised learning does not rely on labels and can use a large amount of unlabeled data for training, reducing the cost of data labeling. However, the lack of clear labels makes it more difficult to evaluate the performance of the model. Semi-supervised learning is particularly useful when data labeling is expensive. It reduces the overall labeling cost by making full use of unlabeled data. This method achieves good performance with relatively limited labeled data, while also making full use of unlabeled data for model training.

[0005] Analysis of the medical cell dataset shows that it is significantly different from existing synthetic datasets (Sintel, FlyingChairs, and FlyingThings). Medical cell images usually involve complex cell structures and motion patterns, which are significantly different from the characteristics and distribution of synthetic datasets. Medical cell images contain more abundant cell structures, non-rigid deformations, noise, and dense textures, making it difficult for existing optical flow estimation models to adapt to this special data distribution. Due to the complexity of cell images, the existing optical flow estimation methods are insufficient in modeling capabilities, resulting in a significant decrease in their tracking accuracy in medical scenarios.

[0006] Due to the above differences, the model trained directly using existing synthetic optical flow data is difficult to be directly applied to the field of cell tracking. Therefore, in order to pursue high accuracy in cell optical flow estimation, it is crucial to bridge the gap between cell image sequences and synthetic datasets. Summary of the invention

[0007] In order to solve the technical problem that a model trained by directly using existing synthetic optical flow data is difficult to be directly applied to the field of cell tracking, the present invention provides a semi-supervised optical flow prediction method for cell migration path tracking, the method comprising the following steps:

[0008] S1, collecting two adjacent frames of images in a video image sequence, recording them as an original image and a target image respectively, and calculating the optical flow value between the original image and the target image to obtain a real optical flow map of the original image and the target image; constructing an image data set, wherein the image data set includes a plurality of data pairs, and the data pairs are in the form of original image-target image-real optical flow map of the original image and the target image;

[0009] S2. Construct a cyclic optical flow prediction model. The operations performed in the cyclic optical flow prediction model are as follows:

[0010] S21, the original image and the target image are respectively input into the independently designed anisotropic context encoder, and after processing, the feature map AC_Ces corresponding to the original image and the feature map AC_Cet corresponding to the target image are output;

[0011] S22, the input AC_CEs and AC_CEt are processed respectively through the multi-dimensional feature fusion mechanism, and the three dimensions of the feature map are respectively extracted and jointly modeled to form representative feature representations AC_CEsA and AC_CEtA;

[0012] S23, performing cyclic optical flow prediction on AC_CEsA and AC_CEtA to obtain a predicted optical flow map. When performing cyclic optical flow prediction, a loss function is obtained by calculating the distance between the real optical flow map of the original image and the target image and the predicted optical flow map;

[0013] S3, using the data set constructed in step S1 to train the cyclic optical flow prediction model constructed in step S2. After the training is completed, the original image and the target image are input into the cyclic optical flow prediction model to obtain a predicted optical flow map. The cyclic optical flow prediction model at this time is called a teacher model.

[0014] S4, collecting two adjacent frames of images in the cell motion image sequence, recording them as a cell original image and a cell target image respectively, and constructing a cell data set, wherein the cell data set includes a plurality of data pairs, and the data pairs are in the form of a cell original image-a cell target image;

[0015] S5, inputting the cell data set constructed in step S4 into the teacher model, obtaining several pairs of predicted optical flow maps corresponding to the cell original image-cell target image, naming the predicted optical flow map corresponding to the cell original image-cell target image as the cell predicted optical flow map, and naming the set consisting of several pairs of cell original image-cell target image-cell predicted optical flow map as the cell image sequence optical flow estimation value with the prepared label;

[0016] S6. Use the DSC evaluation index to filter the optical flow estimation values ​​of the cell image sequence with prepared labels. The filtered data are combined into a labeled cell data set. The labeled cell data set is used to train the teacher model. The trained model is called the student model. The original cell image and the target cell image are input to the student model. The student model can output the cell prediction optical flow map. The student model can be used to predict the optical flow of the cell image sequence.

[0017] Furthermore, the structure of the independently designed anisotropic context encoder is: from input to output, it includes a convolution layer, six residual units and anisotropic convolution blocks.

[0018] Furthermore, the heterogeneous convolution block includes three branches AC1, AC2, and AC3 connected in parallel; the output of the heterogeneous convolution block is: ， in, , and denote the outputs of AC1, AC2 and AC3 respectively, and The data connection relationship between them is:

[0019] ;

[0020] in ( ) indicates the data processing method in the AC1 branch. Processing is performed, and the adaptive offset set during the processing is , Indicates the data processing method in the AC2 branch. Processing is performed, and the adaptive offset set during the processing is , Indicates the data processing method in the AC3 branch. Processing is performed, and the adaptive offset set during the processing is .

[0021] Furthermore, the data processing method in the three branches AC1, AC2 and AC3 is as follows: after the data is input, the receptive field is expanded by defining an expanded convolution to generate a feature map, and then an adaptive offset is assigned to each feature vector in the feature map to obtain a mixed expanded feature vector output with an adaptive offset.

[0022] Furthermore, the three dimensions of the feature map are respectively extracted as follows:

[0023] S51. Extract features of AC_CEs in dimensions of height H and width W. Specifically, AC_CEs is a three-dimensional tensor. When extracting features, keep the channel dimension unchanged and convert the width The spatial coordinates of the direction are compressed to 1, and the first two-dimensional tensor is obtained , compress the spatial coordinates in the height H direction to 1, and obtain the second two-dimensional tensor ;

[0024] S52, yes Perform average pooling and maximum pooling operations respectively to obtain two feature tensors and ;right Perform the same process to obtain two feature tensors and ;

[0025] S53, will and Combine to get ,Will and Combine to get ,

[0026] Then the combined feature tensor and Perform 2D convolution, batch normalization and activation function processing respectively to form the output of feature representation and ;

[0027] S54, yes and The splitting method is to The height H and width W are split into two feature tensors and ,Will Split into two feature tensors according to height H and width W and , respectively and Transpose the dimension and then , , and To add channels:

[0028] ;

[0029] right and Process and get the attention weight and , specifically: ,in, The operation represented by is a one-dimensional convolution operation. is the activation function;

[0030] The feature representation of S55 and AC_CEs after height H and width W feature extraction is: ,in Represents a two-dimensional tensor consisting of height H and width W in AC_CEs;

[0031] S56, extract channel C dimension features from AC_CEs. During feature extraction, only the channel C dimension of AC_CEs is processed. Specifically, the input feature map Perform a Reshape operation, perform a Transpose operation on the reshaped feature map, multiply the two transformed feature maps, and generate an intermediate matrix Map: ; Pass the intermediate matrix Map through Operation processing, the number of channels is obtained as a C×C matrix Map C , the generated Map C and Perform element-by-element product operation, and then perform element-by-element addition operation with AC_CEs to generate the final output feature map AC_CEsC;

[0032] S57, perform the same processing on AC_CEt as on AC_CEs, i.e., perform steps S51-S55, and obtain the feature of AC_CEt after extracting the height H and width W features. , perform step S56 to obtain AC_CEt and generate the final output feature map AC_CEtC.

[0033] Furthermore, the joint modeling is specifically as follows: and Fusion is performed to finally obtain a more representative feature map :

[0034] ;

[0035] Will and Fusion is performed to finally obtain a more representative feature map : .

[0036] Furthermore, the recurrent optical flow prediction is performed using a convolutional gated recurrent unit, specifically: using the dot product similarity between each pixel between AC_CEsA and AC_CEtA to construct a size of H × W × H × W The 4D cost volume of ,Will Input the convolution gated recurrent unit to get the predicted optical flow residual value The operations performed in the convolutional gated recurrent unit are as follows:

[0037] ;

[0038] in, represents a convolutional gated recurrent unit, Represents an aggregation operation, Indicates AC_CEs, Indicates AC_CEt, Indicates the time index of adjacent frames, used to specify ConvGRU The time step of the cyclic optical flow prediction outputs the optical flow map , where n represents the number of loops. When the loop starts for the first time, n=1 and ;

[0039] Optical flow map of the output of cyclic optical flow prediction Up-sample to obtain the predicted optical flow map.

[0040] Furthermore, the optical flow estimation values ​​of the cell image sequence with the prepared labels are screened by using the DSC evaluation index, specifically:

[0041] S81, segmenting the original cell image and the target cell image respectively to obtain segmented mask images M1 and M2;

[0042] S82, multiply the original cell image by M1 to obtain the foreground image If1, multiply the cell target image by M2 to get the foreground image If2 ;

[0043] S83, the foreground image If1 and foreground image If2 Every pixel in Perform a difference operation to obtain a 0cc image, specifically:

[0044] ;

[0045] Among them, 0 means occlusion, and 1 means no occlusion;

[0046] S84. The optical flow estimation value of the cell prediction optical flow map is recorded as F s→t ,right F s→t Perform a distortion operation to obtain a distorted mask image ;

[0047] S85, compare the 0cc image with the distorted mask image Multiply them to get the reconstructed mask map

[0048] S86, using the formula: ;

[0049] Get the DSC evaluation index; among them, Represents the original image of the cell;

[0050] S87. Design a threshold t=0.75. If the DSC evaluation index is greater than the threshold t, the corresponding cell original image-cell target image-cell predicted optical flow map is retained and combined into a labeled cell data set. Otherwise, it is deleted.

[0051] The beneficial effects of the method of the present invention are:

[0052] The method described in the present invention can improve the pixel-level motion estimation accuracy of cell image sequences. First, in view of the limitations of the existing convolution kernel being fixed in processing non-rigid cell motion, a new convolution mode-heterotropic convolution block is proposed, which uses dilated convolutions with different dilation rates to expand the receptive field range, and assigns an adaptive offset to each feature vector. Finally, the outputs after parallel processing are combined through the concat operation, which can learn the geometric deformation of the target at different scales and accurately model the details of the moving edges in the cell image. Then, in order to solve the problem that the optical flow in the cell image sequence is difficult to distinguish the motion due to dense textures, a multi-dimensional feature fusion mechanism is built, which integrates the multi-dimensional feature relationships in the horizontal, vertical and channel directions to ensure that the model can capture the dynamic features in the cell image more comprehensively and accurately. Secondly, in order to solve the problem of missing cell image labels, we explored semi-supervised learning methods to reduce the reliance on accurate labels. By using data with existing optical flow estimation labels to train the teacher model, the model can predict the unlabeled cell dataset, thereby generating optical flow estimation values ​​of cell image sequences with prepared labels. The Dice Similarity Coefficient (DSC) method was introduced to screen out qualified optical flow estimation values ​​to form a labeled cell training set. Finally, the student model was trained using the labeled cell dataset to obtain a model that can effectively predict the optical flow values ​​of cell image sequences. The model's ability to capture subtle movements in cell image sequences was analyzed through quantitative indicators and qualitative visualization. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a flow chart of the semi-supervised optical flow prediction method described in an embodiment of the present invention;

[0054] Figure 2 : is a structural diagram of the heterogeneous convolution block described in an embodiment of the present invention;

[0055] Figure 3 4 is a flow chart of the DSC evaluation index described in the embodiment of the present invention. DETAILED DESCRIPTION

[0056] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0057] Embodiment 1,

[0058] This embodiment provides a semi-supervised optical flow prediction method for cell migration path tracking, and its flow chart is as follows: Figure 1 As shown, the method comprises the following steps:

[0059] S1, collecting two adjacent frames of images in a video image sequence, recording them as an original image and a target image respectively, and calculating the optical flow value between the original image and the target image to obtain a real optical flow map of the original image and the target image; constructing an image data set, wherein the image data set includes a plurality of data pairs, and the data pairs are in the form of original image-target image-real optical flow map of the original image and the target image;

[0060] S2. Construct a cyclic optical flow prediction model. The operations performed in the cyclic optical flow prediction model are as follows:

[0061] S21, the original image and the target image are respectively input into the independently designed anisotropic context encoder, and after processing, the feature map AC_Ces corresponding to the original image and the feature map AC_Cet corresponding to the target image are output;

[0062] S22, the input AC_CEs and AC_CEt are processed respectively through the multi-dimensional feature fusion mechanism, and the three dimensions of the feature map are respectively extracted and jointly modeled to form representative feature representations AC_CEsA and AC_CEtA;

[0063] S23, performing cyclic optical flow prediction on AC_CEsA and AC_CEtA to obtain a predicted optical flow map. When performing cyclic optical flow prediction, a loss function is obtained by calculating the distance between the real optical flow map of the original image and the target image and the predicted optical flow map;

[0064] S3, using the data set constructed in step S1 to train the cyclic optical flow prediction model constructed in step S2. After the training is completed, the original image and the target image are input into the cyclic optical flow prediction model to obtain a predicted optical flow map. The cyclic optical flow prediction model at this time is called a teacher model.

[0065] S4, collecting two adjacent frames of images in the cell motion image sequence, recording them as a cell original image and a cell target image respectively, and constructing a cell data set, wherein the cell data set includes a plurality of data pairs, and the data pairs are in the form of a cell original image-a cell target image;

[0066] S5, inputting the cell data set constructed in step S4 into the teacher model, obtaining several pairs of predicted optical flow maps corresponding to the cell original image-cell target image, naming the predicted optical flow map corresponding to the cell original image-cell target image as the cell predicted optical flow map, and naming the set consisting of several pairs of cell original image-cell target image-cell predicted optical flow map as the cell image sequence optical flow estimation value with the prepared label;

[0067] S6. Use the DSC evaluation index to filter the optical flow estimation values ​​of the cell image sequence with prepared labels. The filtered data are combined into a labeled cell data set. The labeled cell data set is used to train the teacher model. The trained model is called the student model. The original cell image and the target cell image are input to the student model. The student model can output the cell prediction optical flow map. The student model can be used to predict the optical flow of the cell image sequence.

[0068] Embodiment 2,

[0069] This embodiment further limits Embodiment 1 and introduces the anisotropic context encoder in detail.

[0070] The anisotropic context encoder includes a convolution layer (Conv convolution kernel 7×7, channel number 64), six residual units (Res_Unit, channel numbers are 64, 64, 96, 96, 128, 128) and an anisotropic convolution block (ACN) from input to output. The anisotropic convolution block is designed independently. The structure diagram is as follows: Figure 2 shown.

[0071] The heterogeneous convolution block includes three branches AC1, AC2, and AC3 connected in parallel; the output of the heterogeneous convolution block is: ， in , and denote the outputs of AC1, AC2 and AC3 respectively, and The data connection relationship between them is:

[0072] ;

[0073] in ( ) indicates the data processing method in the AC1 branch. Processing is performed, and the adaptive offset set during the processing is , Indicates the data processing method in the AC2 branch. Processing is performed, and the adaptive offset set during the processing is , Indicates the data processing method in the AC3 branch. Processing is performed, and the adaptive offset set during the processing is .

[0074] The specific way of processing data in the three branches AC1, AC2, and AC3 is as follows: after the data is input, the receptive field is expanded by defining a dilated convolution (HDC). The 2N in the figure means that after the dilated convolution, the channel dimension of the generated feature map is 2N, and N is determined by the size of the convolution kernel. Then, an adaptive offset is assigned to each feature vector in the feature map to obtain a mixed dilated feature vector output with an adaptive offset; the DCNv2 in the figure represents the obtained mixed dilated feature vector with an adaptive offset.

[0075] Among them, the convolution kernel size of the dilated convolution in AC1, AC2, and AC3 is 3×3, and the dilation rate is 1, 3, and 5 respectively. Therefore, the corresponding dilated convolution enumeration grid G They are:

[0076] ; ; .

[0077] In the three branches AC1, AC2, and AC3, let the input feature map be x , x Upper center point The feature map for:

[0078]

[0079] in: To enumerate the points in the grid G, is the sampling point weight, is the learned adaptive offset, are the learned weights, is the feature map sampling position, where , which is determined by the size of the convolution kernel. For example, if the convolution kernel size is 3×3, then .when When it is a fraction, quadratic linear interpolation is used to reduce the error.

[0080] Embodiment 3,

[0081] This embodiment is a further limitation of the embodiment 1. The input AC_CEs and AC_CEt are processed respectively through a multi-dimensional feature fusion mechanism, and the three dimensions (height H, width W and number of channels C) of the feature map are respectively subjected to feature extraction and joint modeling to form representative feature representations AC_CEsA and AC_CEtA. The specific process is as follows:

[0082] AC_CEs is a three-dimensional tensor, denoted by F =[ F 1, F 2,…, FC ]∈ R C×H×W .

[0083] The height (H) and width (W) dimension features of AC_CEs are extracted as follows:

[0084] Will F =[ F 1, F 2,…, F C ]∈ R C×H×W Split into two 2D tensors and , the splitting method is: when splitting, keep the channel dimension unchanged, compress the spatial coordinates in the width W direction to 1, and get the first two-dimensional tensor , compress the spatial coordinates in the high H direction to 1, and obtain the second two-dimensional tensor .

[0085] right Perform average pooling (APL) and maximum pooling (MPL) operations respectively to obtain two feature tensors and ;

[0086] right Perform the same process to obtain two feature tensors and ;Will and Combine to get ,Will and Combine to get , and then the combined feature tensor and 2D convolution, batch normalization and activation function processing are performed separately to form a richer output with higher-level feature representation and , this design helps the model better capture the structure and patterns in the image. The processing flow is as follows:

[0087] ;

[0088] in, represents the activation function, represents batch normalization, represents 2D convolution, Represents aggregation.

[0089] right and To split ( Split ), the splitting method is to The height H and width W are split into two feature tensors and ,Will Split into two feature tensors according to height H and width W and ,

[0090] Respectively and Transpose the dimensions, that is, swap the height and width dimensions, and then , , and To add channels:

[0091] ;

[0092] right and Process and get the attention weight and , specifically: ,in, The operation represented by is a one-dimensional convolution operation. is the activation function.

[0093] The feature representation of AC_CEs after height H and width W feature extraction is: ,in Represents a two-dimensional tensor consisting of height H and width W in AC_CEs;

[0094] Features of AC_CEt after height H and width W feature extraction How to obtain same.

[0095] The channel C dimension feature extraction is performed on AC_CEs. During feature extraction, only the channel C dimension of AC_CEs is processed. Specifically, the input feature map Perform the Reshape operation ( Figure 1 Medium Re Reshape stands for Reshape, adjusts the shape of the feature map to adapt to subsequent matrix operations, and performs Transpose operation on the reshaped feature map ( Figure 2 Medium Tr Represents Transpose) to change the order of its dimensions, multiply the two transformed feature maps, and generate an intermediate matrix Map: ;

[0096] The intermediate matrix Map is passed through operate( Figure 1 Medium S represent ) to obtain a C×C matrix Map with a channel number of C ,The C×C matrix maps the relationship between each channel and measures the relevance and importance of each channel to other channels.

[0097] In order to enhance the feature expression of important channels, the generated Map C and Perform element-by-element product operation, and then perform element-by-element addition operation with AC_CEs to generate the final output feature map AC_CEsC;

[0098] The same processing process is performed on AC_CEt as AC_CEs to obtain the features of AC_CEt after extracting the height H and width W features. , and generate the final output feature map AC_CEtC.

[0099] The joint modeling is specifically as follows: and Fusion is performed to finally obtain a more representative feature map :

[0100] ;

[0101] Will and Fusion is performed to finally obtain a more representative feature map : .

[0102] In the case of complex background, the fused feature map helps the network capture the key features of the target more accurately and improves the robustness and generalization ability of the model.

[0103] This mechanism effectively utilizes information of different dimensions in a coordinated manner, enabling the network to better adapt to complex and changing visual tasks, and ultimately achieve higher pixel-level motion estimation accuracy in various challenging scenarios.

[0104] Embodiment 4,

[0105] This embodiment is a further limitation of Embodiment 1. The cyclic optical flow prediction is performed using a convolutional gated recurrent unit (ConvGRU), specifically: using the dot product similarity between each pixel point between AC_CEsA and AC_CEtA to construct a size of H × W × H × W The 4D cost volume of ,Will Input the convolution gated recurrent unit to get the predicted optical flow residual value The operations performed in the convolutional gated recurrent unit are as follows:

[0106] ;

[0107] in, represents a convolutional gated recurrent unit, Represents an aggregation operation, Indicates AC_CEs, Indicates AC_CEt, Indicates the time index of adjacent frames, used to specify ConvGRU The time step of the cyclic optical flow prediction outputs the optical flow map , where n represents the number of loops. When the loop starts for the first time, n=1 and ;

[0108] Optical flow map of the output of cyclic optical flow prediction Up-sample to obtain the predicted optical flow map.

[0109] When performing cyclic optical flow prediction, the loss function is obtained by calculating the distance between the real optical flow map of the original image and the target image and the predicted optical flow map, specifically: ;

[0110] in, Represents the predicted optical flow map of the source image and the target image, Represents the real optical flow map of the original image and the target image, N represents the logarithm of the original image and the target image, γ = 0.8, is the L1 distance function.

[0111] Embodiment 5,

[0112] This embodiment is a further explanation of embodiment 1. The process of using the DSC evaluation index to screen the optical flow estimation values ​​of the cell image sequence with the prepared label is as follows: Figure 3 As shown, specifically:

[0113] S81, segmenting the original cell image and the target cell image respectively to obtain segmented mask images M1 and M2;

[0114] S82, multiply the original cell image by M1 to obtain the foreground image If1 , multiply the cell target image by M2 to get the foreground image If2 ;

[0115] S83, the foreground image If1 and foreground image If2 Every pixel in Perform a difference operation to obtain a 0cc image, specifically:

[0116] ;

[0117] Among them, 0 means occlusion, and 1 means no occlusion;

[0118] S84. The optical flow estimation value of the cell prediction optical flow map is recorded as F s→t ,right F s→t Perform a distortion operation to obtain a distorted mask image ;

[0119] S85, compare the 0cc image with the distorted mask image Multiply them to get the reconstructed mask map

[0120] S86, using the formula: ;

[0121] Get the DSC evaluation index; among them, Represents the original image of the cell;

[0122] S87. Design a threshold t=0.75. If the DSC evaluation index is greater than the threshold t, the corresponding cell original image-cell target image-cell predicted optical flow map is retained and combined into a labeled cell data set. Otherwise, it is deleted.

[0123] When the teacher model is trained using a labeled cell dataset for cyclic optical flow prediction, the loss function is obtained by calculating the distance between the cell prediction optical flow map in the labeled cell dataset and the cell prediction optical flow map predicted by the teacher model, specifically: ;

[0124] in, Represents the predicted optical flow map of cells in the labeled cell dataset, represents the cell prediction optical flow map predicted by the teacher model, N represents the logarithm of the cell original image and the cell target image, γ = 0.8, is the L1 distance function.

[0125] In order to estimate the accuracy of the cell region optical flow, we define AEPE oMeasurement:

[0126] ;

[0127] In the formula, m is the cell mask on the original cell image of the first frame ( ), where 0 represents background and 1 represents cells; represents element-wise multiplication operation, Represents each pixel in the feature map with height H and width W.

Claims

1. A semi-supervised optical flow prediction method for cell migration path tracking, characterized in that: The method comprises the following steps: S1, collecting two adjacent frames of images in a video image sequence, recording them as an original image and a target image respectively, and calculating the optical flow value between the original image and the target image to obtain a real optical flow map of the original image and the target image; constructing an image data set, wherein the image data set includes a plurality of data pairs, and the data pairs are in the form of original image-target image-real optical flow map of the original image and the target image; S2. Construct a cyclic optical flow prediction model. The operations performed in the cyclic optical flow prediction model are as follows: S21, the original image and the target image are respectively input into the independently designed anisotropic context encoder, and after processing, the feature map AC_Ces corresponding to the original image and the feature map AC_Cet corresponding to the target image are output; S22, the input AC_CEs and AC_CEt are processed respectively through the multi-dimensional feature fusion mechanism, and the three dimensions of the feature map are respectively extracted and jointly modeled to form representative feature representations AC_CEsA and AC_CEtA; S23, performing cyclic optical flow prediction on AC_CEsA and AC_CEtA to obtain a predicted optical flow map. When performing cyclic optical flow prediction, a loss function is obtained by calculating the distance between the real optical flow map of the original image and the target image and the predicted optical flow map; S3, using the data set constructed in step S1 to train the cyclic optical flow prediction model constructed in step S2. After the training is completed, the original image and the target image are input into the cyclic optical flow prediction model to obtain a predicted optical flow map. The cyclic optical flow prediction model at this time is called a teacher model. S4, collecting two adjacent frames of images in the cell motion image sequence, recording them as a cell original image and a cell target image respectively, and constructing a cell data set, wherein the cell data set includes a plurality of data pairs, and the data pairs are in the form of a cell original image-a cell target image; S5, inputting the cell data set constructed in step S4 into the teacher model, obtaining several pairs of predicted optical flow maps corresponding to the cell original image-cell target image, naming the predicted optical flow map corresponding to the cell original image-cell target image as the cell predicted optical flow map, and naming the set consisting of several pairs of cell original image-cell target image-cell predicted optical flow map as the cell image sequence optical flow estimation value with the prepared label; S6. Use the DSC evaluation index to filter the optical flow estimation values ​​of the cell image sequence with prepared labels. The filtered data are combined into a labeled cell data set. The labeled cell data set is used to train the teacher model. The trained model is called the student model. The original cell image and the target cell image are input to the student model. The student model can output the cell prediction optical flow map. The student model can be used to predict the optical flow of the cell image sequence.

2. The semi-supervised optical flow prediction method for cell migration path tracking according to claim 1, characterized in that: The structure of the independently designed anisotropic context encoder is: from input to output, it includes a convolution layer, six residual units and anisotropic convolution blocks.

3. The semi-supervised optical flow prediction method for cell migration path tracking according to claim 2, characterized in that: The heterogeneous convolution block includes three branches AC1, AC2, and AC3 connected in parallel; the output of the heterogeneous convolution block is: ， in, , and denote the outputs of AC1, AC2 and AC3 respectively, and The data connection relationship between them is: ； in ( ) indicates the data processing method in the AC1 branch. Processing is performed, and the adaptive offset set during the processing is , Indicates the data processing method in the AC2 branch. Processing is performed, and the adaptive offset set during the processing is , Indicates the data processing method in the AC3 branch. Processing is performed, and the adaptive offset set during the processing is .

4. The semi-supervised optical flow prediction method for cell migration path tracking according to claim 3, characterized in that: The specific way of processing data in the three branches AC1, AC2 and AC3 is as follows: after the data is input, the receptive field is expanded by defining an expanded convolution to generate a feature map, and then an adaptive offset is assigned to each feature vector in the feature map to obtain a mixed expanded feature vector output with an adaptive offset.

5. The semi-supervised optical flow prediction method for cell migration path tracking according to claim 4, characterized in that: The three dimensions of the feature map are extracted separately as follows: S51. Extract features of AC_CEs in dimensions of height H and width W. Specifically, AC_CEs is a three-dimensional tensor. When extracting features, keep the channel dimension unchanged and convert the width The spatial coordinates of the direction are compressed to 1, and the first two-dimensional tensor is obtained , compress the spatial coordinates in the height H direction to 1, and obtain the second two-dimensional tensor ; S52, yes Perform average pooling and maximum pooling operations respectively to obtain two feature tensors and ;right Perform the same process to obtain two feature tensors and ; S53, will and Combine to get ,Will and Combine to get , Then the combined feature tensor and Perform 2D convolution, batch normalization and activation function processing respectively to form the output of feature representation and ; S54, yes and The splitting method is to The height H and width W are split into two feature tensors and ,Will Split into two feature tensors according to height H and width W and , respectively and Transpose the dimension and then , , and To add channels: ； right and Process and get the attention weight and , specifically: ,in, The operation represented by is a one-dimensional convolution operation. is the activation function; The feature representation of S55 and AC_CEs after height H and width W feature extraction is: ,in Represents a two-dimensional tensor consisting of height H and width W in AC_CEs; S56, extract channel C dimension features from AC_CEs. During feature extraction, only the channel C dimension of AC_CEs is processed. Specifically, the input feature map Perform a Reshape operation, perform a Transpose operation on the reshaped feature map, multiply the two transformed feature maps, and generate an intermediate matrix Map: ; Pass the intermediate matrix Map through Operation processing, the number of channels is obtained as a C×C matrix Map C , the generated Map C and Perform element-by-element product operation, and then perform element-by-element addition operation with AC_CEs to generate the final output feature map AC_CEsC; S57, perform the same processing on AC_CEt as on AC_CEs, i.e., perform steps S51-S55, and obtain the feature of AC_CEt after extracting the height H and width W features. , perform step S56 to obtain AC_CEt and generate the final output feature map AC_CEtC.

6. The semi-supervised optical flow prediction method for cell migration path tracking according to claim 5, characterized in that: The joint modeling is specifically as follows: and Fusion is performed to finally obtain a more representative feature map : ； Will and Fusion is performed to finally obtain a more representative feature map : .

7. The semi-supervised optical flow prediction method for cell migration path tracking according to claim 6, characterized in that: The cyclic optical flow prediction is performed using a convolutional gated cyclic unit, specifically: using the dot product similarity between each pixel between AC_CEsA and AC_CEtA to construct a size of H × W × H × W The 4D cost volume of ,Will Input the convolution gated recurrent unit to get the predicted optical flow residual value The operations performed in the convolutional gated recurrent unit are as follows: ； in, represents a convolutional gated recurrent unit, Represents an aggregation operation, Indicates AC_CEs, Indicates AC_CEt, Represents the time index of adjacent frames, used to specify ConvGRU The time step of the cyclic optical flow prediction outputs the optical flow map , where n represents the number of loops. When the loop starts for the first time, n=1 and ; Optical flow map of the output of cyclic optical flow prediction Up-sample to obtain the predicted optical flow map.

8. The semi-supervised optical flow prediction method for cell migration path tracking according to claim 7, characterized in that: The optical flow estimation values ​​of the cell image sequence with the prepared labels are screened by using the DSC evaluation index, specifically: S81, segmenting the original cell image and the target cell image respectively to obtain segmented mask images M1 and M2; S82, multiply the original cell image by M1 to obtain the foreground image If1 , multiply the cell target image by M2 to get the foreground image If2 ; S83, the foreground image If1 and foreground image If2 Every pixel in Perform difference operation to obtain the 0cc image, specifically: ； Among them, 0 means occlusion, and 1 means no occlusion; S84. The optical flow estimation value of the cell prediction optical flow map is recorded as F s→t ,right F s→t Perform a distortion operation to obtain a distorted mask image ; S85, compare the 0cc image with the distorted mask image Multiply them to get the reconstructed mask map S86, using the formula: ; Get the DSC evaluation index; among them, Represents the original image of the cell; S87. Design a threshold t=0.

75. If the DSC evaluation index is greater than the threshold t, the corresponding cell original image-cell target image-cell predicted optical flow map is retained and combined into a labeled cell data set. Otherwise, it is deleted.

Images