Motion cell depth information estimation method and robot micro-operation equipment

Abstract

The invention relates to the technical field of robot technology and image processing, in particular to a motor cell depth information estimation method and robot microoperation equipment, and the method comprises the steps: collecting a plurality of cell images at the depth value of each depth label, constructing a data set, and dividing the data set into a training set and a test set; sequentially inputting each training image in the training set into an FGAF module of the depth estimation network, extracting a weighted feature map of each training image by using the FG module, and obtaining a final feature map of each training image by using the AF module; according to a channel-based feature enhancement method, obtaining an enhanced feature map of each training image, and finally obtaining a depth value of the motor cells in each training image; and training the depth estimation network by minimizing the weight loss function to obtain a trained depth estimation network. According to the invention, the problem complexity is reduced, the estimation precision of the depth value of the motor cell is improved, and the robustness and generalization ability of the depth estimation network are enhanced.

Description

Technical Field

[0001] The present invention relates to the fields of robotics and image processing technology, and in particular to a method for estimating depth information of moving cells and a robotic micro-manipulation device. Background Art

[0002] The development of microscopic cell manipulation techniques is crucial in modern biomedical research and clinical treatment. These techniques are widely used in a variety of key areas, including in vitro fertilization, cell therapy, tissue engineering, and various types of biological experiments. Acquiring three-dimensional positional information of motile cells (such as sperm and bacteria) under a microscope using robotic-assisted cell micromanipulation is a critical and challenging task, primarily due to the challenge of obtaining Z-depth information about motile cells under a monocular microscope.

[0003] Existing methods for estimating the Z-axis depth information of moving cells under a monocular microscope include image-based depth estimation algorithms and classification estimation methods.

[0004] Image-based depth estimation algorithms include the DFF (Depth from Focus) algorithm and the DFD (Depth from Defocus) algorithm; among them, the essence of the DFF algorithm is autofocus, which determines the three-dimensional position of the target object by moving the focal plane up and down and using a focus measurement algorithm to find the clearest image. However, this process based on repeated experiments limits the speed of the algorithm, making it unsuitable for quickly locating moving cells; the DFD algorithm constructs a defocus model of the imaged object and establishes a mapping relationship between the blurred features in the image and the corresponding depth, thereby avoiding the time-consuming process of finding the plane. However, the algorithm is heavily dependent on the accuracy of the defocus model, and the rapidly changing projection shape during cell movement increases the difficulty of building a reliable defocus model. Therefore, existing image-based depth estimation algorithms are not suitable for obtaining depth information of moving cells.

[0005] Classification estimation methods include research on achieving automatic focusing by classifying the depth of silicone filaments, and estimating the offset of micropipette and adherent cells relative to the focus position through the network. However, these methods have only been tested on stationary target objects with fixed shapes. In order to avoid the problem of multiple morphologies corresponding to a single depth value caused by rapid morphological changes during cell movement, the modeling process is simplified, making it unsuitable for practical applications and resulting in low accuracy in estimating the depth information of moving cells.

[0006] In addition, although the existing technology shows that deep networks can establish a mapping relationship between blurred image features and corresponding depth values ​​at different depths, it does not consider improving the ability of deep networks to capture subtle differences between adjacent focal planes, and thus cannot solve the problem that the projection features of motion cells between different focal planes are very different due to the small size of motion cells and the lack of texture and structural information, resulting in low accuracy in estimating the depth information of motion cells. Summary of the Invention

[0007] To this end, the technical problem to be solved by the present invention is to overcome the problem that the existing technology has a low ability to capture subtle differences in the projection characteristics of moving cells between adjacent focal planes, and there is a many-to-one relationship between the morphology of moving cells and the depth value, resulting in low accuracy in estimating the depth information of moving cells.

[0008] To solve the above technical problems, the present invention provides a method for estimating depth information of moving cells, comprising:

[0009] According to the depth interval threshold, the continuous depth value range is discretized into multiple depth labels and their depth values; at the depth value of each depth label, multiple cell images are collected; based on each cell image and the depth value of its corresponding depth label, a dataset is constructed and divided into a training set and a test set;

[0010] Each training image in the training set is sequentially input into the fine-grained attention fusion module of the depth estimation network, and features are extracted for each training image to obtain the final feature map of each training image, including:

[0011] Input the current training image into the fine-grained feature extraction module of the fine-grained attention fusion module, and use the grouped convolution layer of the fine-grained feature extraction module to extract each scale feature map of the current training image; input each scale feature map of the current training image into the global pooling layer of the fine-grained feature extraction module, and output the feature weight vector corresponding to each scale feature map of the current training image; input each scale feature map of the current training image and its corresponding feature weight vector into the activation layer of the fine-grained feature extraction module, and output the weighted feature map of the current training image;

[0012] The weighted feature map of the current training image is input into the attention fusion module of the fine-grained attention fusion module. The fusion pooling layer of the attention fusion module performs average pooling and maximum pooling operations on each channel in the weighted feature map of the current training image, and then the average pooling matrix and the maximum pooling matrix of the weighted feature map are spliced ​​together to output the fusion pooling matrix of the weighted feature map of the current training image; the fusion pooling matrix of the weighted feature map of the current training image is input into the fully connected layer of the attention fusion module, and the channel importance weight matrix of the weighted feature map of the current training image is output; the weighted feature map of the current training image and its channel importance weight matrix are input into the weighted layer of the attention fusion module, and the corrected feature map of the current training image is output; the corrected feature map of the current training image is input into the pooling convolution layer of the attention fusion module, and the final feature map of the current training image is output, thereby obtaining the final feature map of each training image;

[0013] The final feature map of each training image is input into the processing layer of the depth estimation network. After performing an affine transformation on each channel in the final feature map of each training image using the channel cross normalization method, the features of any two channels are randomly swapped to obtain the enhanced feature map of each training image.

[0014] The enhanced feature map of each training image is input into the output layer of the depth estimation network, and the depth prediction value of the corresponding depth label of each training image is output as the depth value of the motion cell in each training image;

[0015] The depth estimation network is trained by minimizing the weighted loss function using the training set to obtain a trained depth estimation network for estimating the depth value of the motion cell.

[0016] Preferably, using the fine-grained feature extraction module of the fine-grained attention fusion module to output a weighted feature map of the current training image according to the input current training image includes:

[0017] Input the current training image into the fine-grained feature extraction module of the fine-grained attention fusion module, and use the grouped convolution layer of the fine-grained feature extraction module to extract the feature map of each scale of the current training image;

[0018] Input each scale feature map of the current training image into the global pooling layer of the fine-grained feature extraction module, and output the feature weight vector corresponding to each scale feature map of the current training image, which is expressed as:

[0019] ;

[0020] in, Indicates the current training image The feature weight vector corresponding to the scale feature map; Indicates the current training image Scale feature map; Represents the GeLU activation function; Represents the Sigmoid activation function; Represents a global pooling operation; represents the global pooling weight; Represents the GeLU activation function weight;

[0021] Each scale feature map of the current training image and its corresponding feature weight vector are input into the activation layer of the fine-grained feature extraction module, and the weighted feature map of the current training image is output. Its expression is:

[0022] ;

[0023] in, Represents the weighted feature map of the current training image; represents the softmax function; Represents element-wise multiplication.

[0024] Preferably, using the attention fusion module of the fine-grained attention fusion module, outputting the final feature map of the current training image according to the input weighted feature map of the current training image includes:

[0025] The weighted feature map of the current training image is input into the attention fusion module of the fine-grained attention fusion module. The fusion pooling layer of the attention fusion module is used to perform average pooling and maximum pooling operations on each channel in the weighted feature map of the current training image, and the average pooling value and maximum pooling value of each channel are obtained. Their expressions are:

[0026] ;

[0027] ;

[0028] in, Represents the weighted feature map of the current training image The average pooling value of channels; Represents the weighted feature map of the current training image The maximum pooling value of channels; Represents the weighted feature map of the current training image; Indicates that the average pooling operation and the maximum pooling operation are at a height of and width is It is carried out in two dimensions;

[0029] The average pooling matrix and the maximum pooling matrix of the weighted feature map obtained by splicing are used to output the fusion pooling matrix of the weighted feature map of the current training image. Its expression is:

[0030] ;

[0031] in, Represents the fusion pooling matrix of the weighted feature map of the current training image; represents the fusion function; Represents the average pooling matrix of the weighted feature map of the current training image; Represents the fusion pooling matrix of the weighted feature map of the current training image;

[0032] The fusion pooling matrix of the weighted feature map of the current training image is input into the fully connected layer of the attention fusion module, and the channel importance weight matrix of the weighted feature map of the current training image is output. Its expression is:

[0033] ;

[0034] in, The channel importance weight matrix representing the weighted feature map of the current training image; Represents a fully connected operation;

[0035] The weighted feature map of the current training image and its channel importance weight matrix are input into the weighted layer of the attention fusion module, and the corrected feature map of the current training image is output. Its expression is:

[0036] ;

[0037] in, Represents the corrected feature map of the current training image; Represents element-wise multiplication;

[0038] The corrected feature map of the current training image is input into the pooling convolution layer of the attention fusion module, and the final feature map of the current training image is output. Its expression is:

[0039] ;

[0040] in, Represents a pooling operation; Represents the convolution operation; Represents the final feature map of the current training image.

[0041] Preferably, the final feature map of the current training image is input into the processing layer of the depth estimation network, and after performing an affine transformation on each channel in the final feature map of the current training image using the channel cross normalization method, the features of any two channels are randomly exchanged to obtain the enhanced feature map of the current training image, including:

[0042] The final feature map of the current training image Input the processing layer of the depth estimation network, the final feature map of the current training image Perform affine transformation on each channel in to obtain the new features of each channel in the final feature map of the current training image, which is expressed as:

[0043] ;

[0044] in, Represents the final feature map of the current training image Middle New features for each channel; Represents the final feature map of the current training image Middle The characteristics of each channel; Represents the final feature map of the current training image Middle The mean of the channels; Represents the final feature map of the current training image Middle The variance of the channels; represents the first transformation parameter; represents the second transformation parameter;

[0045] Randomly select the final feature map of the current training image Middle Channel and Channel, exchange Channel and The characteristic statistical properties of the channel are obtained Channel and The target features of each channel are expressed as follows:

[0046] ;

[0047] ;

[0048] in, Represents the final feature map of the current training image Middle Target features of each channel; Represents the final feature map of the current training image Middle Target features of each channel; Represents the final feature map of the current training image Middle New features for each channel; Represents the final feature map of the current training image Middle New features for each channel; Represents the final feature map of the current training image Middle The mean of the channels; Represents the final feature map of the current training image Middle The variance of the channels; Represents the final feature map of the current training image Middle The mean of the channels; Represents the final feature map of the current training image Middle The variance of the channels;

[0049] Based on the target features of all channels in the final feature map of the current training image, an enhanced feature map of the current training image is obtained.

[0050] Preferably, the expression of the weighted loss function is:

[0051] ;

[0052] in, represents the weighted loss function; Represents the total number of all training images; Indicates the The weights of the training images, ; Indicates the The true depth value of the depth label corresponding to each training image; Indicates the The depth prediction value of the depth label corresponding to the training image; Indicates the The predicted probability distribution corresponding to the training images.

[0053] Preferably, after obtaining the trained depth estimation network, the method further includes:

[0054] Each test image in the test set is sequentially input into the fine-grained attention fusion module of the trained depth estimation network, and features are extracted for each test image to obtain the final feature map of each test image;

[0055] The final feature map of each test image is input into the processing layer of the trained depth estimation network. The adaptive normalization method is used to perform an affine transformation on each channel in the final feature map of each test image to obtain an enhanced feature map of each test image.

[0056] The enhanced feature map of each test image is input into the output layer of the trained depth estimation network, and the depth prediction value of the depth label corresponding to each test image is output as the depth value of the motion cell in each test image.

[0057] Preferably, the final feature map of the current test image is input into the processing layer of the trained depth estimation network, and an adaptive normalization method is used to perform an affine transformation on each channel in the final feature map of the current test image to obtain an enhanced feature map of the current test image, including:

[0058] The final feature map of the current test image is input into the processing layer of the trained depth estimation network. The adaptive normalization method is used to perform an affine transformation on each channel in the final feature map of the current test image to obtain the new features of each channel in the final feature map of the current test image. The expression is:

[0059] ;

[0060] in, Indicates the final feature map of the current test image. New features for each channel; Indicates the final feature map of the current test image. The characteristics of each channel; represents the first attention parameter; Second attention parameter; Indicates the final feature map of the current test image. The mean of the channels; Indicates the final feature map of the current test image. The variance of the channels;

[0061] Based on the new features of all channels in the final feature map of the current test image, an enhanced feature map of the current test image is obtained.

[0062] Preferably, the depth interval threshold is 0.5μm; the continuous depth value range is -2μm to 2μm; the depth labels are the first to ninth depth labels; the depth values ​​of the first to ninth depth labels are -2μm, -1.5μm, -1μm, -0.5μm, 0μm, 0.5μm, 1μm, 1.5μm, and 2μm respectively.

[0063] Preferably, before constructing the data set, all collected cell images need to be preprocessed to obtain preprocessed cell images; wherein the preprocessing includes denoising and contrast enhancement operations.

[0064] The present invention also provides a robot micro-manipulation device, comprising:

[0065] Inverted microscope for magnified viewing of motile cells at different focal planes on the stage;

[0066] A CCD camera is placed at the imaging port of the inverted microscope and connected to the computer via a cable; it is used to collect cell images at different focal planes under the inverted microscope and transmit the cell images to the computer;

[0067] The XY motorized stage is placed under the stage of the inverted microscope and is connected to the computer via a cable. It is used to receive control signals from the computer and control the movement of the microinjection needle in the X-axis and Y-axis directions.

[0068] A Z-axis motorized micromanipulator is placed on the XY motorized stage and connected to the computer via a cable; it is used to move the microinjection needle in the Z-axis direction according to the depth value of the moving cell estimated by the computer;

[0069] Freedom micromanipulator arm, fixed on the Z-axis motorized micromanipulator, adjacent to the inverted microscope; used to fix the microinjection needle;

[0070] Microinjection needle, placed on a free-degree-of-freedom micromanipulator arm; used for micromanipulation of motile cells;

[0071] A vacuum suction pump is adjacent to the stage and connected to the computer via a cable; it is used to aspirate motile cells;

[0072] Computer, connected to the inverted microscope via a cable, including:

[0073] A microscope focusing module, used to control the focus of an inverted microscope according to a built-in focusing mechanism program;

[0074] An image processing module is used to perform denoising and contrast enhancement preprocessing operations on each cell image captured by the CCD camera to obtain each preprocessed cell image; based on each preprocessed cell image, execute the built-in program of the above-mentioned method for estimating the depth information of moving cells to estimate the depth value of moving cells in each cell image;

[0075] a control module for receiving the estimated depth value of the moving cells in each cell image, and controlling the XY motorized stage and the Z-axis motorized micromanipulator according to the depth value of the moving cells in each cell image, so as to move the microinjection needle to the depth value of the moving cells, thereby achieving micromanipulation and aspiration of the moving cells;

[0076] The operation interface display module is used to display the depth value of the moving cells in each cell image and micro-operation guidance information.

[0077] The above technical solution of the present invention has the following beneficial effects compared with the prior art:

[0078] The present invention discloses a method for estimating the depth information of moving cells, which discretizes the continuous depth value range into a series of depth labels according to the depth interval threshold, and then converts the depth estimation problem into a multi-category depth classification problem, thereby reducing the complexity of the problem, enhancing the robustness and generalization ability of the depth estimation network, and facilitating data annotation and understanding of the results; introducing a fine-grained attention fusion module (FGAF module), and extracting multi-scale fine-grained features according to grouped convolutions of different scales through the fine-grained feature extraction module (FG module) in the FGAF module; integrating spatial attention and channel attention mechanisms through the attention fusion module (AF module) in the FGAF module, and enhancing the features of significant regions, thereby effectively overcoming the depth estimation difficulties caused by the subtle differences in cell projection features between adjacent focal planes and the rapid morphological changes caused by cell movement, and thereby improving the estimation accuracy of the depth value of moving cells; in the training stage, the channel cross normalization method is used to normalize the training images. The final feature map of the training image is enhanced by exchanging statistical properties between different channels of the high-dimensional feature map, expanding the data feature distribution, and enabling the model to learn the fine-grained feature differences in cell images; in the testing phase, the adaptive normalization method is used to enhance the final feature map of the test image, and the affine transformation parameters are adjusted by learning a fully connected network and two attention mechanisms, so that the feature distribution of the test image is closer to that of the training image, thereby improving the generalization ability and processing speed of the model in practical applications; a weighted loss function is introduced, and a penalty coefficient for erroneous prediction is incorporated into the loss function, giving higher weights to samples with large depth category differences, so that the depth estimation network focuses on distinguishing the fine-grained features of adjacent depth categories, improving the ability of the depth estimation network to accurately classify sperm images at different depth gradients, and further improving the accuracy of estimating the depth information of moving cells, thereby providing accurate and reliable visual feedback of moving cells for robotic micro-manipulation equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0080] Figure 1 This is a flow chart of a method for estimating depth information of moving cells provided by the present invention;

[0081] Figure 2 is a schematic diagram of the fine-grained attention fusion module;

[0082] Figure 3 is a schematic diagram of the feature enhancement method; where Figure 3 (a) in the figure shows a schematic diagram of feature enhancement using the channel cross normalization method; Figure 3 (b) in the figure shows a schematic diagram of feature enhancement using the adaptive normalization method;

[0083] Figure 4 is a schematic diagram of a robotic micro-manipulation device provided by the present invention;

[0084] Figure 5 This is a flow chart of the depth estimation of moving cells based on a robotic micromanipulation device. DETAILED DESCRIPTION

[0085] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention.

[0086] Reference Figure 1 As shown, Figure 1 This is a flow chart of a method for estimating the depth information of moving cells provided by the present invention. This method aims to address the challenge of estimating the Z-axis depth of moving cells (such as sperm and bacteria) under an inverted microscope during robot-assisted cell micromanipulation, and provide accurate and reliable visual feedback for the robot micromanipulation device. Specifically, it includes:

[0087] S1: According to the depth interval threshold, the continuous depth value range is discretized into multiple depth labels and their depth values; at the depth value of each depth label, multiple cell images are collected; based on each cell image and the depth value of its corresponding depth label, a data set is constructed, and the data set is divided into a training set and a test set; wherein, the depth interval threshold is 0.5μm; the continuous depth value range is -2μm to 2μm; the depth labels are the first to ninth depth labels; the depth values ​​of the first to ninth depth labels are -2μm, -1.5μm, -1μm, -0.5μm, 0μm, 0.5μm, 1μm, 1.5μm, and 2μm, respectively; before constructing the data set, all collected cell images need to be preprocessed to obtain preprocessed cell images; the preprocessing includes denoising and contrast enhancement operations;

[0088] Before determining to discretize the continuous depth value range into a plurality of depth labels and their depth values, the method further includes:

[0089] By utilizing the depth of field characteristics of a microscope, the continuous depth values ​​are discretized into a finite series of depth labels, each of which corresponds to a depth focal plane. This allows the original depth estimation problem to be reshaped into a multi-class depth classification problem.

[0090] The depth of field formula of the microscope is:

[0091] ;

[0092] in, Indicates the wavelength of light in the microscope; It represents the refractive index of the medium between the microscope objective and the object; Indicates the numerical aperture of the microscope objective; Indicates the magnification of the objective lens; represents aberration;

[0093] In a specific embodiment of the present invention, the numerical aperture is 0.65 and the magnification of the objective lens is 40. At this time, the depth of field of the microscope is about 1.0 μm. This 1.0 μm depth of field determines the resolution of the microscope in the Z-axis direction, that is, within a range of 1.0 μm, the object on the stage can be clearly imaged;

[0094] However, during sperm motility, the sperm head moves up and down rapidly along the Z axis, so that it cannot be completely in the same focal plane, resulting in the final image being partially clear and partially blurred;

[0095] In order to capture the subtle depth changes of sperm along the Z axis and further improve the depth information resolution, the present invention discretizes the continuous depth values ​​into a series of depth labels at intervals of 0.5 μm. With the help of these depth labels, the depth estimation problem under the microscope is naturally transformed into a multi-class depth classification problem.

[0096] Therefore, the entire image acquisition process was performed within the range of -2 to +2 μm, discretized into nine depth categories at a resolution of 0.5 μm (depth interval threshold), and a total of 2,088 images were collected, each with a resolution of 96 × 96 pixels. The automated acquisition method ensured data diversity and accuracy, providing a stable and reliable foundation for multi-classification studies of fine-grained feature differences.

[0097] In summary, by generating depth labels, the depth estimation problem is converted into a multi-class depth classification problem, and classification is performed by learning the mapping relationship between image features and discrete depth labels; the multi-class depth classification problem is performed using supervised learning, using a labeled training set to train the depth estimation network to complete the classification task;

[0098] S2: Input each training image in the training set into the fine-grained attention fusion module of the depth estimation network in turn, extract features from each training image, and obtain the final feature map of each training image, including:

[0099] S21: Input the current training image into the fine-grained feature extraction module (FG module) of the fine-grained attention fusion module, and use the grouped convolution layer of the fine-grained feature extraction module to extract the feature map of each scale of the current training image; wherein, through grouped convolution, feature maps of different scales are obtained, which facilitates the independent processing of feature maps of different scales in subsequent steps and avoids interference between different scales; at the same time, considering that the fine-grained feature extraction module is a plug-and-play module and does not require too many depth estimation network parameters, global pooling is used after grouped convolution to compress the spatial information of each scale feature map to reduce the number of parameters;

[0100] Input each scale feature map of the current training image into the global pooling layer of the fine-grained feature extraction module, and output the feature weight vector corresponding to each scale feature map of the current training image, which is expressed as:

[0101] ;

[0102] in, Indicates the current training image The feature weight vector corresponding to the scale feature map; Indicates the current training image Scale feature map; Represents the GeLU activation function; Represents the Sigmoid activation function; Represents a global pooling operation; represents the global pooling weight; Represents the weight of the GeLU activation function; the feature weight vector contains global semantic information;

[0103] Each scale feature map of the current training image and its corresponding feature weight vector are input into the activation layer of the fine-grained feature extraction module, and the weighted feature map of the current training image is output. Its expression is:

[0104] ;

[0105] in, Represents the weighted feature map of the current training image; represents the softmax function; Represents element-by-element multiplication; wherein the softmax function is used to balance the importance of features at different scales and recalibrate the feature weight vector to ensure that the sum of the weight vectors of features at different scales is 1, which facilitates the integration of multi-scale feature information;

[0106] S22: Input the weighted feature map of the current training image into the attention fusion module (AF module) of the fine-grained attention fusion module, and use the fusion pooling layer of the attention fusion module to perform average pooling and maximum pooling operations on each channel in the weighted feature map of the current training image, and obtain the average pooling value and maximum pooling value of each channel, whose expressions are:

[0107] ;

[0108] ;

[0109] in, Represents the weighted feature map of the current training image The average pooling value of channels; Represents the weighted feature map of the current training image The maximum pooling value of channels; Represents the weighted feature map of the current training image; Indicates that the average pooling operation and the maximum pooling operation are at a height of and width is The average pooling operation is used to capture the typical features in each channel, and the maximum pooling operation is used to capture the extreme features in each channel.

[0110] The average pooling matrix and the maximum pooling matrix of the weighted feature map obtained by splicing are used to output the fusion pooling matrix of the weighted feature map of the current training image. Its expression is:

[0111] ;

[0112] in, Represents the fusion pooling matrix of the weighted feature map of the current training image; represents the fusion function; Represents the average pooling matrix of the weighted feature map of the current training image; Represents the fusion pooling matrix of the weighted feature map of the current training image;

[0113] The fusion pooling matrix of the weighted feature map of the current training image is input into the fully connected layer of the attention fusion module, and the channel importance weight matrix of the weighted feature map of the current training image is output. Its expression is:

[0114] ;

[0115] in, The channel importance weight matrix representing the weighted feature map of the current training image; Represents a fully connected operation; the concatenated fusion pooling matrix is ​​fed into the fully connected layer to analyze and learn the dependencies between features of different channels;

[0116] The weighted feature map of the current training image and its channel importance weight matrix are input into the weighted layer of the attention fusion module, and the corrected feature map of the current training image is output. Its expression is:

[0117] ;

[0118] in, Represents the corrected feature map of the current training image;

[0119] The corrected feature map of the current training image is input into the pooling convolution layer of the attention fusion module, and the final feature map of the current training image is output. Then, the final feature map of each training image is obtained, which is expressed as:

[0120] ;

[0121] in, Represents a pooling operation; Represents the convolution operation; Represents the final feature map of the current training image; in order to build a bridge between the local and global feature spaces in the cell image, the corrected feature map is spatially rescaled; the rescaling process involves a pooling operation to reduce the spatial dimension of the corrected feature map, which helps to more significantly emphasize global features; then, through a convolution operation, a complex connection is established between the local fine-grained features of the cell image and the broader global features, so that the final feature map of the output encapsulates the fine-grained local features of the cell image and the enhanced global features;

[0122] Based on S22, it can be seen that the attention fusion module focuses on extracting key features within a single channel and adjusts the spatial relationship to build a bridge between local and global feature spaces in the cell image; the AF module, for the FG module, fuses the multi-scale weighted feature maps, performs average pooling and maximum pooling on each channel, captures typical features and extreme features, and splices the statistical information after pooling, inputs the fully connected layer to learn the dependency relationship between features of different channels, forms a channel importance weight matrix, multiplies it element by element with the weighted feature map, recalibrates the channel weights, and obtains the corrected feature map; then the corrected feature map is spatially adjusted, and the pooling operation is used to compress the spatial dimension and highlight the global features. After the convolution operation, a complex relationship is established between the local fine-grained features and the global features, and the final feature map with both fine-grained local features and enhanced global features is output;

[0123] In summary, the fine-grained attention fusion module (FGAF module) consists of a fine-grained feature extraction module and an attention fusion module, which is used to identify subtle differences between different focal planes. Its structure is as follows: Figure 2 As shown in the figure, the fine-grained feature extraction module extracts feature information such as texture, edge, and morphology from the images of the moving cells at different focal planes by analyzing them, i.e., it uses grouped convolution at different scales to extract multi-scale fine-grained features. The attention fusion module is used to weight the contribution of features at different focal planes to highlight the features that are most critical for depth judgment, i.e., it integrates spatial attention and channel attention mechanisms to enhance salient regional features.

[0124] S3: The final feature map of each training image is input into the processing layer of the depth estimation network. After performing an affine transformation on each channel in the final feature map of each training image using the channel cross normalization method, the features of any two channels are randomly swapped to obtain an enhanced feature map of each training image, including:

[0125] The final feature map of the current training image Input the processing layer of the depth estimation network, the final feature map of the current training image Perform affine transformation on each channel in to obtain the new features of each channel in the final feature map of the current training image, which is expressed as:

[0126] ;

[0127] in, Represents the final feature map of the current training image Middle New features for each channel; Represents the final feature map of the current training image Middle The characteristics of each channel; Represents the final feature map of the current training image Middle The mean of the channels; Represents the final feature map of the current training image Middle The variance of the channels; represents the first transformation parameter; represents the second transformation parameter;

[0128] Randomly select the final feature map of the current training image Middle Channel and Channel, exchange Channel and The characteristic statistical properties of the channel are obtained Channel and The target features of each channel are expressed as follows:

[0129] ;

[0130] ;

[0131] in, Represents the final feature map of the current training image Middle Target features of each channel; Represents the final feature map of the current training image Middle Target features of each channel; Represents the final feature map of the current training image Middle New features for each channel; Represents the final feature map of the current training image Middle New features for each channel; Represents the final feature map of the current training image Middle The mean of the channels; Represents the final feature map of the current training image Middle The variance of the channels; Represents the final feature map of the current training image Middle The mean of the channels; Represents the final feature map of the current training image Middle The variance of the channels;

[0132] Based on the target features of all channels in the final feature map of the current training image, an enhanced feature map of the current training image is obtained;

[0133] In summary, during the training phase, the channel cross normalization method (CrossNorm) is used to enhance the fine-grained features in the cell image; by exchanging feature statistical properties between different channels of the high-dimensional feature map of the training image, the data feature distribution is expanded, so that the depth estimation network can learn the fine-grained differences in the cell image; the process of the channel cross normalization method is as follows Figure 3 As shown in (a);

[0134] S4: Input the enhanced feature map of each training image into the output layer of the depth estimation network, and output the depth prediction value corresponding to the depth label of each training image as the depth value of the motion cell in each training image;

[0135] S5: Using the training set, the depth estimation network is trained by minimizing the weighted loss function to obtain a trained depth estimation network for estimating the depth value of the motion cell;

[0136] Among them, in the task of fine-grained classification of cell images with different depth labels, the traditional cross-entropy loss function is not effective in capturing subtle differences between adjacent depth labels. Since the fine-grained feature differences between cell images corresponding to adjacent depth labels are very small, if the cross-entropy loss function is used to train the depth estimation network, misclassification problems are likely to occur in the initial training stage. In order to address this limitation and improve the classification accuracy of the depth estimation network, the present invention introduces a weighted loss function, which incorporates a penalty coefficient for incorrect predictions into the loss function, encouraging the model to focus on the fine-grained feature differences between adjacent depth labels. The expression of the weighted loss function is:

[0137] ;

[0138] in, represents the weighted loss function; Represents the total number of all training images; Indicates the The weights of the training images, ; Indicates the The true depth value of the depth label corresponding to each training image; Indicates the The depth prediction value of the depth label corresponding to the training image; Indicates the The predicted probability distribution corresponding to the training images;

[0139] The depth estimation network parameters are optimized through stochastic gradient descent, and momentum and weight decay strategies are used to accelerate convergence and prevent overfitting.

[0140] After completing steps S1-S5 and obtaining a trained depth estimation network, the following steps are also included:

[0141] Each test image in the test set is sequentially input into the fine-grained attention fusion module of the trained depth estimation network, and features are extracted for each test image to obtain the final feature map of each test image;

[0142] The final feature map of each test image is input into the processing layer of the trained depth estimation network. Using the adaptive normalization method, each channel in the final feature map of each test image is affine transformed to obtain the enhanced feature map of each test image, including:

[0143] The final feature map of the current test image is input into the processing layer of the trained depth estimation network. The adaptive normalization method is used to perform an affine transformation on each channel in the final feature map of the current test image to obtain the new features of each channel in the final feature map of the current test image. The expression is:

[0144] ;

[0145] in, Indicates the final feature map of the current test image. New features for each channel; Indicates the final feature map of the current test image. The characteristics of each channel; represents the first attention parameter; Second attention parameter; Indicates the final feature map of the current test image. The mean of the channels; Indicates the final feature map of the current test image. The variance of the channels;

[0146] Based on the new features of all channels in the final feature map of the current test image, an enhanced feature map of the current test image is obtained;

[0147] In summary, during the test phase, the adaptive normalization method (SelfNorm) is used to enhance the fine-grained features in the cell images; by learning the fully connected network and two attention mechanisms to adjust the affine change parameters, the data distribution in the test phase is made closer to the training data, improving the robustness of the depth estimation network and the generalization ability in practical applications; at the same time, the stability and generalization ability of the depth estimation network are enhanced, so that the depth estimation network can also extract key fine-grained features when facing unseen cell images; the process of the adaptive normalization method is as follows Figure 3 As shown in (b); the adaptive normalization method can be applied not only to the testing stage, but also to the training stage;

[0148] From the comprehensive training and testing phases, we can see that channel-based feature enhancement methods include channel cross-normalization and adaptive normalization. By normalizing and gain-adjusting the features of each channel to adapt to different imaging conditions and environmental changes, we can enhance fine-grained features and improve the generalization ability of the model.

[0149] The enhanced feature map of each test image is input into the output layer of the trained depth estimation network, and the depth prediction value of the depth label corresponding to each test image is output as the depth value of the motion cell in each test image.

[0150] The present invention also provides a robot micro-manipulation device, such as Figure 4 Shown, including:

[0151] Inverted microscope for magnified viewing of motile cells at different focal planes on the stage;

[0152] A CCD camera is placed at the imaging port of an inverted microscope and connected to a computer via a cable. It is used to capture cell images at different focal planes under the inverted microscope and transmit the cell images to the computer. The CCD camera can capture the dynamic activities of moving cells and micro-manipulation processes.

[0153] The XY motorized stage is placed under the stage of the inverted microscope and is connected to the computer via a cable. It is used to receive control signals from the computer and control the movement of the microinjection needle in the X-axis and Y-axis directions.

[0154] A Z-axis motorized micromanipulator is placed on the XY motorized stage and connected to the computer via a cable; it is used to move the microinjection needle in the Z-axis direction according to the depth value of the moving cell estimated by the computer;

[0155] A free-degree micromanipulator arm is fixed on the Z-axis electric micromanipulator and is adjacent to the inverted microscope, i.e., close to the inverted microscope; and is used to fix the microinjection needle;

[0156] Microinjection needle, placed on a free-degree-of-freedom micromanipulator arm; used for micromanipulation of motile cells;

[0157] A vacuum suction pump is adjacent to the stage and connected to the computer via a cable; it is used to aspirate motile cells;

[0158] Computer, connected to the inverted microscope via a cable, including:

[0159] A microscope focusing module is used to control the focus of the inverted microscope according to the built-in focusing mechanism program to obtain clear cell images at different focal planes;

[0160] The image processing module is used to perform denoising, contrast enhancement and other image optimization preprocessing operations on each cell image acquired by the CCD camera to obtain each preprocessed cell image; based on each preprocessed cell image, the built-in program of the above-mentioned method for estimating the depth information of moving cells is executed to estimate the depth value of moving cells in each cell image; wherein, the process of estimating the depth information of moving cells based on the robot micro-manipulation device is as follows Figure 5 As shown;

[0161] a control module for receiving the estimated depth value of the moving cells in each cell image, and controlling the XY motorized stage and the Z-axis motorized micromanipulator according to the depth value of the moving cells in each cell image, so as to move the microinjection needle to the depth value of the moving cells, thereby achieving micromanipulation and aspiration of the moving cells;

[0162] The operation interface display module is used to display the depth value of moving cells in each cell image and micro-operation guidance information to assist operators in precise control.

[0163] In the robot micromanipulation system, by combining the depth estimation algorithm with the XY plane positioning algorithm, the three-dimensional spatial information of sperm is obtained, and automatic tracking and absorption of sperm is achieved.

[0164] In a specific embodiment of the present invention, taking the sperm aspiration task as an example, a motorized XY stage is moved to maintain a constant distance of 100 pixels between the target sperm and the microinjection needle tip. The XY plane position information of the target sperm head is provided to the microinjection needle through the pre-trained UNeXt semantic segmentation model. Simultaneously, the depth estimation network predicts the Z-axis depth information of the sperm. After obtaining the depth information, the microinjection needle tip is moved to the corresponding depth. Throughout the aspiration process, the injection pump maintains a constant flow rate (1.2nL / s).

[0165] In summary, the present invention provides a method for estimating the depth information of moving cells and a robotic micromanipulation device. By adopting a multi-class classification strategy, the depth estimation problem is converted into a multi-category depth classification problem. By introducing a fine-grained attention fusion module (FGAF module), the fine-grained feature extraction module and the attention fusion module are combined to effectively identify subtle differences between focal planes, thereby solving the problem of difficulty in accurately obtaining depth information due to the limited depth of field of the microscope and the large dynamic changes of the cells themselves. The present invention applies a channel-based feature enhancement method to improve the generalization ability and processing speed of the depth estimation network. In addition, the present invention also demonstrates its application potential and practicality in robotic micromanipulation systems.

[0166] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications arising therefrom remain within the scope of protection of the present invention.

Claims

1. A method for estimating depth information of moving cells, characterized in that: include: According to the depth interval threshold, the continuous depth value range is discretized into multiple depth labels and their depth values; At each depth value of the depth label, multiple cell images are collected; Based on the depth value of each cell image and its corresponding depth label, a dataset is constructed and divided into a training set and a test set; Each training image in the training set is input into the fine-grained attention fusion module of the depth estimation network in turn, and features are extracted for each training image to obtain the final feature map of each training image, including: Input the current training image into the fine-grained feature extraction module of the fine-grained attention fusion module, and use the grouped convolution layer of the fine-grained feature extraction module to extract each scale feature map of the current training image; input each scale feature map of the current training image into the global pooling layer of the fine-grained feature extraction module, and output the feature weight vector corresponding to each scale feature map of the current training image; input each scale feature map of the current training image and its corresponding feature weight vector into the activation layer of the fine-grained feature extraction module, and output the weighted feature map of the current training image; The weighted feature map of the current training image is input into the attention fusion module of the fine-grained attention fusion module, and the fusion pooling layer of the attention fusion module is used to perform average pooling and maximum pooling operations on each channel in the weighted feature map of the current training image, and the average pooling matrix and the maximum pooling matrix of the weighted feature map are spliced ​​to output the fusion pooling matrix of the weighted feature map of the current training image; the fusion pooling matrix of the weighted feature map of the current training image is input into the fully connected layer of the attention fusion module, and the channel importance weight matrix of the weighted feature map of the current training image is output; the weighted feature map of the current training image and its channel importance weight matrix are input into the weighted layer of the attention fusion module, and the corrected feature map of the current training image is output; the corrected feature map of the current training image is input into the pooling convolution layer of the attention fusion module, and the final feature map of the current training image is output, thereby obtaining the final feature map of each training image; The final feature map of each training image is input into the processing layer of the depth estimation network. After performing affine transformation on each channel in the final feature map of each training image using the channel cross normalization method, the features of any two channels are randomly exchanged to obtain the enhanced feature map of each training image. The enhanced feature map of each training image is input into the output layer of the depth estimation network, and the depth prediction value of the depth label corresponding to each training image is output as the depth value of the motion cell in each training image; The depth estimation network is trained by minimizing the weighted loss function using the training set to obtain a trained depth estimation network for estimating the depth value of the motion cell.

2. A method for estimating depth information of moving cells according to claim 1, characterized in that: The fine-grained feature extraction module of the fine-grained attention fusion module outputs a weighted feature map of the current training image according to the input current training image, including: Input the current training image into the fine-grained feature extraction module of the fine-grained attention fusion module, and use the grouped convolution layer of the fine-grained feature extraction module to extract each scale feature map of the current training image; Each scale feature map of the current training image is input into the global pooling layer of the fine-grained feature extraction module, and the feature weight vector corresponding to each scale feature map of the current training image is output, which is expressed as: ； in, Represents the current training image The feature weight vector corresponding to the scale feature map; Represents the current training image Scale feature map; represents the GeLU activation function; Represents the Sigmoid activation function; Represents a global pooling operation; represents the global pooling weight; Represents the GeLU activation function weight; Each scale feature map of the current training image and its corresponding feature weight vector are input into the activation layer of the fine-grained feature extraction module, and the weighted feature map of the current training image is output. Its expression is: ； in, Represents the weighted feature map of the current training image; represents the softmax function; Represents element-wise multiplication.

3. A method for estimating depth information of moving cells according to claim 1, characterized in that: The attention fusion module of the fine-grained attention fusion module outputs the final feature map of the current training image according to the weighted feature map of the input current training image, including: The weighted feature map of the current training image is input into the attention fusion module of the fine-grained attention fusion module. The fusion pooling layer of the attention fusion module is used to perform average pooling and maximum pooling operations on each channel in the weighted feature map of the current training image, and the average pooling value and maximum pooling value of each channel are obtained. The expressions are: ； ； in, Represents the weighted feature map of the current training image. The average pooling value of channels; Represents the weighted feature map of the current training image. The maximum pooling value of channels; Represents the weighted feature map of the current training image; Indicates that the average pooling operation and the maximum pooling operation are at a height of and width is in two dimensions; The average pooling matrix and the maximum pooling matrix of the weighted feature map obtained by splicing are used to output the fusion pooling matrix of the weighted feature map of the current training image, which is expressed as: ； in, Represents the fusion pooling matrix of the weighted feature map of the current training image; represents the fusion function; Represents the average pooling matrix of the weighted feature map of the current training image; Represents the fusion pooling matrix of the weighted feature map of the current training image; The fusion pooling matrix of the weighted feature map of the current training image is input into the fully connected layer of the attention fusion module, and the channel importance weight matrix of the weighted feature map of the current training image is output, which is expressed as: ； in, The channel importance weight matrix representing the weighted feature map of the current training image; Represents a fully connected operation; The weighted feature map of the current training image and its channel importance weight matrix are input into the weighted layer of the attention fusion module, and the corrected feature map of the current training image is output. Its expression is: ； in, Represents the corrected feature map of the current training image; Represents element-wise multiplication; The corrected feature map of the current training image is input into the pooling convolution layer of the attention fusion module, and the final feature map of the current training image is output, which is expressed as: ； in, Represents a pooling operation; Represents the convolution operation; Represents the final feature map of the current training image.

4. A method for estimating depth information of moving cells according to claim 1, characterized in that: The final feature map of the current training image is input into the processing layer of the depth estimation network. After performing affine transformation on each channel in the final feature map of the current training image using the channel cross normalization method, the features of any two channels are randomly exchanged to obtain the enhanced feature map of the current training image, including: The final feature map of the current training image Input the processing layer of the depth estimation network, the final feature map of the current training image Perform an affine transformation on each channel in to obtain the new features of each channel in the final feature map of the current training image, which is expressed as: ； in, Represents the final feature map of the current training image Middle New features for each channel; Represents the final feature map of the current training image Middle The characteristics of each channel; Represents the final feature map of the current training image Middle The mean of the channels; Represents the final feature map of the current training image Middle The variance of the channels; represents the first transformation parameter; represents the second transformation parameter; Randomly select the final feature map of the current training image Middle Channels and Channels, exchange Channel and The characteristic statistical properties of the channel are obtained Channels and The target features of each channel are expressed as: ； ； in, Represents the final feature map of the current training image Middle The target features of each channel; Represents the final feature map of the current training image Middle The target features of each channel; Represents the final feature map of the current training image Middle New features for each channel; Represents the final feature map of the current training image Middle New features for each channel; Represents the final feature map of the current training image Middle The mean of the channels; Represents the final feature map of the current training image Middle The variance of the channels; Represents the final feature map of the current training image Middle The mean of the channels; Represents the final feature map of the current training image Middle The variance of the channels; Based on the target features of all channels in the final feature map of the current training image, an enhanced feature map of the current training image is obtained.

5. The method for estimating depth information of moving cells according to claim 1, characterized in that: The expression of the weighted loss function is: ； in, represents the weighted loss function; Represents the total number of all training images; Indicates The weights of the training images, ; Indicates The true depth value of the depth label corresponding to each training image; Indicates The depth prediction value of the depth label corresponding to the training image; Indicates The predicted probability distribution corresponding to the training images.

6. A method for estimating depth information of moving cells according to claim 1, characterized in that: After obtaining the trained depth estimation network, it also includes: Input each test image in the test set into the fine-grained attention fusion module of the trained depth estimation network in turn, extract features from each test image, and obtain the final feature map of each test image; The final feature map of each test image is input into the processing layer of the trained depth estimation network, and an adaptive normalization method is used to perform an affine transformation on each channel in the final feature map of each test image to obtain an enhanced feature map of each test image. The enhanced feature map of each test image is input into the output layer of the trained depth estimation network, and the depth prediction value of the depth label corresponding to each test image is output as the depth value of the motion cell in each test image.

7. A method for estimating depth information of moving cells according to claim 6, characterized in that: The final feature map of the current test image is input into the processing layer of the trained depth estimation network. The adaptive normalization method is used to perform an affine transformation on each channel in the final feature map of the current test image. The enhanced feature map of the current test image is obtained, including: The final feature map of the current test image is input into the processing layer of the trained depth estimation network. The adaptive normalization method is used to perform an affine transformation on each channel in the final feature map of the current test image to obtain a new feature for each channel in the final feature map of the current test image. The expression is: ； in, Indicates the final feature map of the current test image. New features for each channel; Indicates the final feature map of the current test image. The characteristics of each channel; represents the first attention parameter; Second attention parameter; Indicates the final feature map of the current test image. The mean of the channels; Indicates the final feature map of the current test image. The variance of the channels; Based on the new features of all channels in the final feature map of the current test image, an enhanced feature map of the current test image is obtained.

8. The method for estimating depth information of moving cells according to claim 1, characterized in that: The depth interval threshold is 0.5μm; the continuous depth value range is -2μm to 2μm; the depth labels are the first to ninth depth labels; the depth values ​​of the first to ninth depth labels are -2μm, -1.5μm, -1μm, -0.5μm, 0μm, 0.5μm, 1μm, 1.5μm, and 2μm respectively.

9. A method for estimating depth information of moving cells according to claim 1, characterized in that: Before constructing the data set, all collected cell images need to be preprocessed to obtain preprocessed cell images; the preprocessing includes denoising and contrast enhancement operations.

10. A robot micro-manipulation device, characterized in that: include: Inverted microscope for magnified viewing of moving cells at different focal planes on the stage; The CCD camera is placed at the imaging port of the inverted microscope and connected to the computer via a cable; Used to collect cell images at different focal planes under an inverted microscope and transmit the cell images to a computer; The XY electric platform is placed under the stage of the inverted microscope and connected to the computer through a cable; it is used to receive control signals from the computer and control the movement of the microinjection needle in the X-axis and Y-axis directions; A Z-axis electric micromanipulator is placed on the XY electric platform and connected to the computer through a cable; it is used to move the microinjection needle in the Z-axis direction according to the depth value of the moving cell estimated by the computer; The 3-DOF micromanipulator arm is fixed on the Z-axis motorized micromanipulator and is adjacent to the inverted microscope; Used to fix microinjection needles; The microinjection needle is placed on a micromanipulator with a degree of freedom; For micromanipulation of motor cells; A vacuum suction pump is adjacent to the stage and connected to the computer via a cable; Used to absorb motile cells; Computer, connected to the inverted microscope via a cable, including: A microscope focusing module, used to control the focal length of an inverted microscope according to a built-in focusing mechanism program; An image processing module, used for performing denoising and contrast enhancement preprocessing operations on each cell image captured by a CCD camera to obtain each preprocessed cell image; executing a built-in program of a method for estimating the depth information of moving cells as described in any one of claims 1 to 9 according to each preprocessed cell image, to estimate the depth value of the moving cells in each cell image; A control module is used to receive the estimated depth value of the moving cells in each cell image, and control the XY electric platform and the Z-axis electric micromanipulator according to the depth value of the moving cells in each cell image, so as to move the microinjection needle to the depth value of the moving cells, thereby realizing micro-manipulation and aspiration of the moving cells; The operation interface display module is used to display the depth value of the moving cells in each cell image and the micro-operation guidance information.

Images