A method of pose estimation of a cell preparation material and related devices

Abstract

This application relates to the field of machine vision technology, and provides a method and related equipment for pose estimation of cell preparation materials. The method includes: extracting features from the left-eye image of the cell preparation material to obtain a left-eye feature map, and extracting features from the right-eye image to obtain a right-eye feature map; constructing a global energy optimization function based on the left-eye and right-eye feature maps, and solving the global energy optimization function to obtain an optimal disparity map; combining the left-eye and right-eye feature maps to obtain a final feature map; generating an initial point cloud of the cell preparation material based on the optimal disparity map and the final feature map, and generating a point cloud feature map based on the initial point cloud; fusing the final feature map and the point cloud feature map to obtain a fused feature map of the cell preparation material; and estimating the pose of the cell preparation material based on the fused feature map to obtain the pose estimation result of the cell preparation material. The method of this application can improve the accuracy of pose estimation for cell preparation materials.

Description

Technical Field

[0001] This application relates to the field of machine vision technology, and in particular to a method and related equipment for estimating the pose of cell preparation materials. Background Technology

[0002] Automation of cell preparation is a core direction for the industrialization of biomanufacturing. One of its core bottlenecks lies in the accurate estimation of the 6D pose of the material (translation along the X / Y / Z axes + rotation around the three axes) – this parameter directly determines the stability of the robotic arm operation and the safety of the cell preparation process.

[0003] The existing technological approach has two major flaws:

[0004] 1. The contradiction between dependence on structured light sources and cell damage: Traditional pose estimation often uses structured light sources such as lasers and infrared to assist in the acquisition of depth information. However, such light sources can increase the cell apoptosis rate by 2-3 times and the DNA damage rate by 15%-20%, which is completely unsuitable for the biosafety requirements of cell preparation. If a binocular red-green-blue (RGB) camera is used to acquire depth information, the problem of insufficient depth accuracy under unstructured lighting will be faced.

[0005] 2. Module Fragmentation and Error Accumulation: Traditional methods separate point cloud optimization, bimodal fusion, and pose regression into independent non-learning processes. For example, a depth map is first generated using a traditional semi-global block matching (SGBM) algorithm, then principal component analysis (PCA) is used for coarse point cloud alignment, and finally, pose regression is performed using an independent model. The parameters of each step cannot be optimized collaboratively, and errors from preceding steps (such as point cloud noise and feature matching bias) continue to propagate to subsequent steps, ultimately resulting in pose estimation accuracy that fails to meet the micrometer-level operational requirements of cell preparation.

[0006] Therefore, current methods for estimating the pose of cell preparation materials suffer from low accuracy. Summary of the Invention

[0007] This application provides a method and related equipment for estimating the pose of cell preparation materials, which can solve the problem of low pose estimation accuracy of cell preparation materials.

[0008] In a first aspect, embodiments of this application provide a pose estimation method for cell preparation materials, the pose estimation method comprising:

[0009] The left and right eye images of the cell preparation materials were acquired. Feature extraction was performed on the left eye image to obtain the left eye feature map, and feature extraction was performed on the right eye image to obtain the right eye feature map.

[0010] A global energy optimization function is constructed based on the left and right eye feature maps, and the optimal disparity map is obtained by solving the global energy optimization function. The global energy optimization function is used to describe the accuracy of the disparity between the left and right eye feature maps.

[0011] The left and right eye feature maps are combined to obtain the final feature map. The initial point cloud of the cell preparation material is generated based on the optimal disparity map and the final feature map, and the point cloud feature map is generated based on the initial point cloud.

[0012] The final feature map and the point cloud feature map are fused to obtain the fused features of the cell preparation material;

[0013] Pose estimation of cell preparation materials is performed based on fusion features to obtain the pose estimation results of cell preparation materials.

[0014] Optionally, the global energy optimization function is:

[0015] ;

[0016] in, This represents the value of the global energy optimization function. Indicates parallax. This represents the pixel coordinates in the horizontal direction of the feature map. Represents the pixel coordinates in the vertical direction of the feature map. Indicates the weight of the smoothing term. Indicates the cost of local block matching:

[0017] ;

[0018] in, This represents the feature map of the left eye. This represents the feature map of the right eye.

[0019] Optionally, an initial point cloud for cell preparation materials is generated based on the optimal disparity map and the final feature map, including:

[0020] Convert the optimal disparity map into a depth feature map;

[0021] The initial point cloud of the cell preparation material is generated by combining the depth feature map and the final feature map.

[0022] Optionally, a point cloud feature map is calculated based on the initial point cloud, including:

[0023] Perform coarse alignment on the initial point cloud to obtain a coarsely aligned point cloud;

[0024] Denoise the coarsely aligned point cloud to obtain the denoised point cloud;

[0025] The denoised point cloud is filtered using the differentiable expectation-maximum algorithm to obtain an optimized point cloud;

[0026] The optimized point cloud is upgraded to a point cloud feature map.

[0027] Optionally, the initial point cloud is coarsely aligned to obtain a coarsely aligned point cloud, including:

[0028] Through the formula:

[0029] ;

[0030] Calculate coarsely aligned point clouds ;

[0031] in, Represents the initial rotation matrix. Represents the initial point cloud. Indicates the initial translation vector:

[0032] ;

[0033] ;

[0034] in, This represents the centroid of the preset target point cloud. Indicates the centroid of the point cloud. Represents the covariance matrix Feature vectors:

[0035] ;

[0036] in, This indicates the number of points in the initial point cloud. Represents the first point in the initial point cloud. One point, This indicates the transpose operation.

[0037] Optionally, the coarsely aligned point cloud is denoised to obtain a denoised point cloud, including:

[0038] Through the formula:

[0039] ;

[0040] ;

[0041] Computational Denoising of Point Clouds ;

[0042] in, This indicates that the Gaussian function is used to assign matching weights. Indicates learnable parameters, This represents the average distance between two points in a coarsely aligned point cloud:

[0043] ;

[0044] in, This represents points in a coarsely aligned point cloud. Indicates the first coarsely aligned point cloud Points The set of nearest neighbors, express The number of midpoints , This indicates the number of points in the coarsely aligned point cloud.

[0045] Optionally, the denoised point cloud can be filtered using a differentiable expectation-maximum algorithm to obtain an optimized point cloud, including:

[0046] An expression for the interior point probability of each point in the denoised point cloud is established using the differentiable expectation-maximum algorithm.

[0047] Solve the expression for the inlier probability of each point to obtain the final inlier probability of each point in the denoised point cloud;

[0048] The optimized point cloud is obtained by retaining all points in the denoised point cloud whose final interior point probability is greater than a preset probability threshold.

[0049] Optionally, the expression for the interior point probability is:

[0050] ;

[0051] ;

[0052] in, Indicating the first point in the denoised cloud The probability of an interior point. , This indicates the number of points in the denoised point cloud. Indicates the scale of interior points. Represents statistical distribution. Indicates the first The distance between each point and the target point, where the target point is a reference point for calculating the distance between points. This represents the mean distance between inliers in the denoised point cloud. This represents the standard deviation of the inlier distance in the denoised point cloud. This represents the uniform distribution function.

[0053] Optionally, the final feature map and the point cloud feature map are fused to obtain the fused features of the cell preparation material, including:

[0054] Through the formula:

[0055] ;

[0056] ;

[0057] ;

[0058] ;

[0059] Computational fusion features ;

[0060] in, Represents the final feature map The weight, Point cloud feature map The weight, This represents the pixel-level cosine similarity between the final feature map and the point cloud feature map. This represents the coordinates of a point in the final feature map. This represents the coordinates of a point in the point cloud feature map. Represents the numerical stability parameter. This indicates the transpose operation.

[0061] Secondly, embodiments of this application provide a pose estimation device for cell preparation materials, comprising:

[0062] The acquisition module is used to acquire left and right eye images of cell preparation materials, extract features from the left eye image to obtain a left eye feature map, and extract features from the right eye image to obtain a right eye feature map;

[0063] The module is used to construct a global energy optimization function based on the left and right eye feature maps, and to solve the global energy optimization function to obtain the optimal disparity map; the global energy optimization function is used to describe the accuracy of the disparity between the left and right eye feature maps.

[0064] The combination module is used to combine the left and right eye feature maps to obtain the final feature map. Based on the optimal disparity map and the final feature map, the initial point cloud of the cell preparation material is generated, and the point cloud feature map is generated based on the initial point cloud.

[0065] The fusion module is used to fuse the final feature map and the point cloud feature map to obtain the fused features of the cell preparation material;

[0066] The pose estimation module is used to estimate the pose of cell preparation materials based on fusion features, and obtain the pose estimation results of the cell preparation materials.

[0067] Thirdly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described pose estimation method for cell preparation materials.

[0068] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for estimating the pose of cell preparation materials.

[0069] The above-mentioned solution in this application has the following beneficial effects:

[0070] In the embodiments of this application, left and right eye images of the cell preparation material are acquired. Features are extracted from the left eye image to obtain a left eye feature map, and features are extracted from the right eye image to obtain a right eye feature map. Then, a global energy optimization function is constructed based on the left and right eye feature maps, and the global energy optimization function is solved to obtain an optimal disparity map. The left and right eye feature maps are then combined to obtain a final feature map. An initial point cloud of the cell preparation material is generated based on the optimal disparity map and the final feature map, and a point cloud feature map is generated based on the initial point cloud. Then, the final feature map and the point cloud feature map are fused to obtain the fused features of the cell preparation material. Finally, the pose of the cell preparation material is estimated based on the fused features to obtain the pose estimation result of the cell preparation material. Specifically, a global energy optimization function is constructed to represent the disparity between the left and right eye feature maps, capture disparity features, and improve the accuracy of the disparity map. The initial point cloud generated based on the high-accuracy disparity map has high precision, which in turn improves the accuracy of the point cloud feature map obtained based on the initial point cloud. Pose estimation is performed based on the point cloud feature map and the final feature map, taking into account the image features and point cloud features of the cell preparation materials, improving the comprehensiveness of the pose estimation information, and thus improving the accuracy of the pose estimation results.

[0071] Other beneficial effects of this application will be described in detail in the following detailed description section. Attached Figure Description

[0072] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0073] Figure 1 A flowchart illustrating a method for estimating the pose of cell preparation materials according to an embodiment of this application;

[0074] Figure 2 A schematic diagram of the pose estimation device for cell preparation materials provided in an embodiment of this application;

[0075] Figure 3 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0076] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0077] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0078] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0079] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0080] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0081] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0082] To address the low accuracy of pose estimation for existing cell preparation materials, this application provides a pose estimation method for cell preparation materials. This method acquires left and right eye images of the cell preparation material, extracts features from the left eye image to obtain a left eye feature map, and extracts features from the right eye image to obtain a right eye feature map. Then, a global energy optimization function is constructed based on the left and right eye feature maps, and the optimal disparity map is obtained by solving the global energy optimization function. The left and right eye feature maps are then combined to obtain a final feature map. An initial point cloud of the cell preparation material is generated based on the optimal disparity map and the final feature map, and a point cloud feature map is generated based on the initial point cloud. The final feature map and the point cloud feature map are then fused to obtain a fused feature map of the cell preparation material. Finally, pose estimation of the cell preparation material is performed based on the fused feature map to obtain the pose estimation result of the cell preparation material. Specifically, a global energy optimization function is constructed to represent the disparity between the left and right eye feature maps, capture disparity features, and improve the accuracy of the disparity map. The initial point cloud generated based on the high-accuracy disparity map has high precision, which in turn improves the accuracy of the point cloud feature map obtained based on the initial point cloud. Pose estimation is performed based on the point cloud feature map and the final feature map, taking into account the image features and point cloud features of the cell preparation materials, improving the comprehensiveness of the pose estimation information, and thus improving the accuracy of the pose estimation results.

[0083] The pose estimation method for cell preparation materials provided in this application will be described by way of example below.

[0084] like Figure 1 As shown, the pose estimation method for cell preparation materials provided in this application includes the following steps:

[0085] Step 11: Obtain the left and right eye images of the cell preparation materials, extract features from the left eye image to obtain the left eye feature map, and extract features from the right eye image to obtain the right eye feature map.

[0086] The cell preparation materials mentioned above include transparent culture flasks, stainless steel trays, centrifuge tubes, etc. The left-eye image is the left-side image of the cell preparation materials, and the right-eye image is the right-side image of the cell preparation materials.

[0087] In some embodiments of this application, a binocular camera can be used to capture left and right eye images of cell preparation materials. The steps described above, including feature extraction of the left eye image to obtain a left eye feature map and feature extraction of the right eye image to obtain a right eye feature map, include:

[0088] The first step is to extract features from the left eye image to obtain the left eye feature map.

[0089] For example, a feature extraction model (such as ResNet18) can be used to extract features from the left eye image to obtain the left eye feature map. In the embodiments of this application, ResNet18 can be improved, such as by: a lightweight network structure: removing redundant convolutional blocks in stages 3 and 4 of the traditional ResNet18, retaining 5 convolutional stages (conv1-conv5), and concatenating batch normalization (BN) layers, ReLU activation functions, and max pooling layers in each stage, thereby reducing computational complexity while ensuring feature extraction capabilities.

[0090] Edge feature enhancement sublayer: A differentiable edge attention sublayer is embedded after the conv5 output, and the gradients of the feature map in the X / Y directions are calculated using the Sobel operator. , And solve for the gradient magnitude. The gradient magnitude is converted into edge attention weights using the Sigmoid function. The final RGB feature map is obtained through element-wise multiplication. (dimension is) This enhances the expression of material edge features.

[0091] Parameter Collaborative Learning: The parameters of all convolutional layers, BN layers, and edge attention sublayers are updated uniformly through backpropagation, achieving collaborative optimization of texture features and edge features, and avoiding the feature disconnect problem caused by traditional independent feature extraction.

[0092] The second step is to extract features from the right eye image to obtain the right eye feature map.

[0093] For example, ResNet18 mentioned above can be used to extract features from the right eye image to obtain the right eye feature map.

[0094] Step 12: Construct a global energy optimization function based on the left and right eye feature maps, and solve the global energy optimization function to obtain the optimal disparity map.

[0095] The global energy optimization function described above is used to describe the accuracy of the disparity between the left and right eye feature maps.

[0096] The global energy optimization function is as follows:

[0097] ;

[0098] in, This represents the value of the global energy optimization function. Indicates parallax. This represents the pixel coordinates in the horizontal direction of the feature map. Represents the pixel coordinates in the vertical direction of the feature map. Indicates the weight of the smoothing term. Indicates the cost of local block matching:

[0099] ;

[0100] in, This represents the feature map of the left eye. This represents the feature map of the right eye.

[0101] For example, the forward-backward algorithm can be used to solve the global energy optimization function to obtain the disparity of each pixel between the left and right feature maps, and the disparities of all pixels can be integrated into a single image according to the coordinates of the corresponding pixels to obtain the optimal disparity map.

[0102] Step 13: Combine the left and right feature maps to obtain the final feature map. Generate the initial point cloud of the cell preparation material based on the optimal disparity map and the final feature map, and then generate the point cloud feature map based on the initial point cloud.

[0103] In some embodiments of this application, the final feature map can be obtained by combining the left and right eye feature maps pixel-by-pixel multiplication and concatenation. The steps described above for generating the initial point cloud of cell preparation materials based on the optimal disparity map and the final feature map, and for generating the point cloud feature map based on the initial point cloud, include:

[0104] The first step is to convert the optimal disparity map into a depth feature map.

[0105] Specifically, based on the fixed intrinsic parameters (baseline) of the stereo camera ,focal length Main point ), through differentiable formula ( For parallax, (For depth values) Convert the disparity map into a depth feature map .

[0106] The second step is to combine the depth feature map and the final feature map to generate the initial point cloud of the cell preparation material.

[0107] For example, combining the pixel coordinates of the depth feature map and the final feature map. Through camera intrinsic parameter formula , ( Converting 3D spatial coordinates to raw point cloud .

[0108] The third step is to perform coarse alignment on the initial point cloud to obtain a coarsely aligned point cloud.

[0109] Through the formula:

[0110] ;

[0111] Calculate coarsely aligned point clouds .

[0112] in, Represents the initial rotation matrix. Represents the initial point cloud. Indicates the initial translation vector:

[0113] ;

[0114] ;

[0115] in, This represents the centroid of the preset target point cloud. Indicates the centroid of the point cloud. Represents the covariance matrix Feature vectors:

[0116] ;

[0117] in, This indicates the number of points in the initial point cloud. Represents the first point in the initial point cloud. One point, This indicates the transpose operation.

[0118] The fourth step is to denoise the coarsely aligned point cloud to obtain a denoised point cloud.

[0119] Through the formula:

[0120] ;

[0121] ;

[0122] Computational Denoising of Point Clouds .

[0123] in, This indicates that the Gaussian function is used to assign matching weights. Indicates learnable parameters, This represents the average distance between two points in a coarsely aligned point cloud:

[0124] ;

[0125] in, This represents points in a coarsely aligned point cloud. Indicates the first coarsely aligned point cloud Points The set of nearest neighbors, express The number of midpoints , This indicates the number of points in the coarsely aligned point cloud.

[0126] The fifth step is to use the differentiable expectation-maximum algorithm to filter the denoised point cloud and obtain the optimized point cloud.

[0127] Specifically, the expression for the interior probability of each point in the denoised point cloud is established using the differentiable expectation-maximum algorithm; the expression for the interior probability of each point is solved to obtain the final interior probability of each point in the denoised point cloud; all points in the denoised point cloud whose final interior probability is greater than a preset probability threshold (such as 0.6) are retained to obtain the optimized point cloud.

[0128] For example, the expression for the interior point probability is:

[0129] ;

[0130] ;

[0131] in, Indicating the first point in the denoised cloud The probability of an interior point. , This indicates the number of points in the denoised point cloud. Indicates the scale of interior points. Represents statistical distribution. Indicates the first The distance between each point and the target point, where the target point is a reference point (pre-selected) used to calculate the distance between points. This represents the mean distance between inliers in the denoised point cloud. This represents the standard deviation of the inlier distance in the denoised point cloud. This represents the uniform distribution function.

[0132] The expression for the inlier probability is solved using the expectation-maximization algorithm to obtain the final inlier probability for each point.

[0133] The sixth step is to upgrade the optimized point cloud to a point cloud feature map.

[0134] For example, differential convolutional layers can be used to upscale an optimized point cloud into a point cloud feature map.

[0135] Step 14: Fuse the final feature map and the point cloud feature map to obtain the fused features of the cell preparation material.

[0136] Specifically, through the formula:

[0137] ;

[0138] ;

[0139] ;

[0140] ;

[0141] Computational fusion features .

[0142] in, Represents the final feature map The weight, Point cloud feature map The weight, This represents the pixel-level cosine similarity between the final feature map and the point cloud feature map. This represents the coordinates of a point in the final feature map. This represents the coordinates of a point in the point cloud feature map. Represents the numerical stability parameter. This indicates the transpose operation.

[0143] Step 15: Based on the fusion features, perform pose estimation on the cell preparation material to obtain the pose estimation result of the cell preparation material.

[0144] The above pose estimation results are 6D pose data (rotation matrix and translation vector) of the cell preparation material.

[0145] For example, a three-layer fully connected network can be used to calculate the fusion features to achieve pose estimation of the cell preparation material and obtain the pose estimation result.

[0146] It should be noted that before performing this step, the fully connected network and the parameters in the formulas used above can be trained using the cell preparation material as a sample. For example, the pose estimation result of the cell preparation material as a sample can be calculated through the above process. Then, a loss function can be constructed based on the pose estimation result. The fully connected network and the parameters in the formulas used above can be trained with the goal of minimizing the loss function. The loss function is:

[0147] ;

[0148] ;

[0149] ;

[0150] in, This represents the value of the loss function. , This is the balance coefficient (preset value, which can be set to 0.4 or 0.6). For point cloud registration loss, For pose return loss, The number of points in the point cloud corresponding to the material used as a sample cell. This represents the first point in the point cloud corresponding to the cell preparation material used as a sample. Point cloud registration loss Calculated parameters, This represents the first point in the optimized point cloud corresponding to the cell preparation material used as a sample. One point, This represents the first point in the target point cloud corresponding to the cell preparation material used as a sample. One point, This represents the predicted point cloud generated based on the pose estimation results. The first in One point, , Represents the rotation matrix. This represents the translation vector.

[0151] By updating all learnable parameters of the network through backpropagation, end-to-end optimization of the entire process from RGB input to 6D pose output is achieved.

[0152] For example, after obtaining the pose estimation results of the cell preparation materials, it can be used for precision operations such as material grasping and transportation.

[0153] In some embodiments of this application, the methods of this application can be deployed according to the following process:

[0154] 1. Hardware configuration: It adopts an Intel Core i7-10700K CPU and an NVIDIA RTX 3090 GPU (24GB video memory). The binocular camera is an industrial camera with a resolution of 1280×720 and a baseline of 120mm to ensure the real-time performance of image acquisition and network inference.

[0155] 2. Data preparation: Binocular RGB images of 40 types of target materials (transparent culture flasks, stainless steel trays, centrifuge tubes, etc.) in the cell preparation scenario were collected. 500 frames were collected for each type of material, and 6D poses were labeled as training datasets. Data augmentation techniques such as image flipping and brightness perturbation were used to expand the dataset and improve the network generalization.

[0156] 3. Network Training: The AdamW optimizer was used (initial learning rate 0.001, weight decay 0.0001), batch size was set to 8, and training epochs were 100 (the learning rate decayed linearly for the first 50 epochs, and was fixed at 1e-5 for the last 50 epochs); global collaborative loss was used. To optimize the objective, iterate until the loss converges (loss fluctuation < 0.005 for 100 consecutive rounds).

[0157] 4. Deployment and Application: Export the trained model to ONNX format and deploy it in the edge computing module of the automated cell preparation system; acquire binocular RGB images in real time, output 6D pose through model inference, and transmit it to the robotic arm controller to achieve precise material grasping and transfer.

[0158] It is worth mentioning that constructing a global energy optimization function can describe the disparity between the left and right eye feature maps, capture disparity features, and improve the accuracy of the disparity map. The initial point cloud generated based on the high-accuracy disparity map has high precision, which in turn improves the accuracy of the point cloud feature map obtained based on the initial point cloud. Pose estimation is performed based on the point cloud feature map and the final feature map, taking into account the image features and point cloud features of the cell preparation materials, improving the comprehensiveness of pose estimation information, and thus improving the accuracy of the pose estimation results.

[0159] Furthermore, the method of this application has the following advantages:

[0160] 1. Differentiable reconstruction of traditional algorithms: For the first time, all non-learning algorithms are reconstructed into differentiable neural network layers, so that the parameters of these steps can be uniformly optimized with the subsequent pose regression module through backpropagation, completely eliminating the key defect of "preceding error propagation accumulation" in the traditional process.

[0161] 2. Dual-modal collaborative attention fusion mechanism: Unlike traditional fixed-weight dual-modal fusion, this invention dynamically allocates the weights of RGB features and point cloud features through pixel-level cosine similarity—strengthening the proportion of RGB features in texture-rich areas and strengthening the proportion of point cloud features in areas with complex spatial structures, thereby achieving adaptive fusion that "emphasizes whichever feature is needed where it is needed," thus improving the accuracy of feature representation.

[0162] 3. High-precision adaptation to unstructured light sources: To meet the biosafety requirements of cell preparation, structured light sources such as lasers and infrared are abandoned, and 6D pose estimation is achieved solely through binocular RGB images; through the collaborative design of an embedded SGBM depth generation layer and a point cloud error optimization layer, depth accuracy can still be guaranteed under unstructured lighting, solving the industry pain point that "biosafety and pose accuracy cannot be achieved simultaneously".

[0163] The pose estimation device for cell preparation materials provided in this application will be described by way of example below.

[0164] like Figure 2 As shown, this application embodiment provides a pose estimation device for cell preparation materials. The pose estimation device 200 for cell preparation materials includes:

[0165] The acquisition module 201 is used to acquire the left and right eye images of the cell preparation materials, extract features from the left eye image to obtain the left eye feature map, and extract features from the right eye image to obtain the right eye feature map;

[0166] Module 202 is used to construct a global energy optimization function based on the left and right eye feature maps, and to solve the global energy optimization function to obtain the optimal disparity map; the global energy optimization function is used to describe the accuracy of the disparity between the left and right eye feature maps.

[0167] The combination module 203 is used to combine the left eye feature map and the right eye feature map to obtain the final feature map, generate the initial point cloud of the cell preparation material based on the optimal disparity map and the final feature map, and generate the point cloud feature map based on the initial point cloud.

[0168] The fusion module 204 is used to fuse the final feature map and the point cloud feature map to obtain the fused features of the cell preparation material;

[0169] The pose estimation module 205 is used to estimate the pose of cell preparation materials based on fusion features, and obtain the pose estimation results of cell preparation materials.

[0170] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0171] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0172] like Figure 3 As shown, an embodiment of this application provides a terminal device, wherein the terminal device D10 of this embodiment includes: at least one processor D100 ( Figure 3 The diagram shows only one processor, a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100, wherein the processor D100 executes the computer program D102 to implement the steps in any of the above method embodiments.

[0173] Specifically, when the processor D100 executes the computer program D102, it acquires the left and right eye images of the cell preparation material, extracts features from the left eye image to obtain a left eye feature map, and extracts features from the right eye image to obtain a right eye feature map. Then, it constructs a global energy optimization function based on the left and right eye feature maps, solves the global energy optimization function to obtain an optimal disparity map, and combines the left and right eye feature maps to obtain a final feature map. Based on the optimal disparity map and the final feature map, it generates an initial point cloud of the cell preparation material, and generates a point cloud feature map based on the initial point cloud. Then, it fuses the final feature map and the point cloud feature map to obtain the fused features of the cell preparation material. Finally, it performs pose estimation on the cell preparation material based on the fused features to obtain the pose estimation result of the cell preparation material. Specifically, a global energy optimization function is constructed to represent the disparity between the left and right eye feature maps, capture disparity features, and improve the accuracy of the disparity map. The initial point cloud generated based on the high-accuracy disparity map has high precision, which in turn improves the accuracy of the point cloud feature map obtained based on the initial point cloud. Pose estimation is performed based on the point cloud feature map and the final feature map, taking into account the image features and point cloud features of the cell preparation materials, improving the comprehensiveness of the pose estimation information, and thus improving the accuracy of the pose estimation results.

[0174] The processor D100 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0175] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as a hard disk or memory of the terminal device D10. In other embodiments, the memory D101 may be an external storage device of the terminal device D10, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the terminal device D10. Furthermore, the memory D101 may include both internal and external storage units of the terminal device D10. The memory D101 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory D101 can also be used to temporarily store data that has been output or will be output.

[0176] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0177] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the above-described method embodiments.

[0178] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to the pose estimation method device / terminal device for cell preparation materials, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, such as a USB flash drive, a portable hard drive, a magnetic disk, or an optical disk.

[0179] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0180] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0181] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention.

Claims

1. A method for estimating the pose of cell preparation materials, characterized in that, include: Acquire left and right eye images of the cell preparation material, extract features from the left eye image to obtain a left eye feature map, and extract features from the right eye image to obtain a right eye feature map; A global energy optimization function is constructed based on the left and right eye feature maps, and the optimal disparity map is obtained by solving the global energy optimization function. The global energy optimization function is used to describe the accuracy of the disparity between the left and right eye feature maps. The left and right feature maps are combined to obtain a final feature map. An initial point cloud of the cell preparation material is generated based on the optimal disparity map and the final feature map, and a point cloud feature map is generated based on the initial point cloud. The final feature map and the point cloud feature map are fused to obtain the fused features of the cell preparation material; Based on the fusion features, the pose of the cell preparation material is estimated to obtain the pose estimation result of the cell preparation material; The step of fusing the final feature map and the point cloud feature map to obtain the fused features of the cell preparation material includes: Through the formula: Computational fusion features ; in, Represents the final feature map The weight, Point cloud feature map The weight, This represents the pixel-level cosine similarity between the final feature map and the point cloud feature map. This represents the coordinates of a point in the final feature map. This represents the coordinates of a point in the point cloud feature map. Represents the numerical stability parameter. This indicates the transpose operation.

2. The pose estimation method according to claim 1, characterized in that, The global energy optimization function is: in, This represents the value of the global energy optimization function. Indicates parallax. This represents the pixel coordinates in the horizontal direction of the feature map. Represents the pixel coordinates in the vertical direction of the feature map. Indicates the weight of the smoothing term. Indicates the cost of local block matching: in, This represents the feature map of the left eye. This represents the feature map of the right eye.

3. The pose estimation method according to claim 1, characterized in that, The process of generating the initial point cloud of the cell preparation material based on the optimal disparity map and the final feature map includes: The optimal disparity map is converted into a depth feature map; The initial point cloud of the cell preparation material is generated by combining the depth feature map and the final feature map.

4. The pose estimation method according to claim 1, characterized in that, The step of computing the point cloud feature map based on the initial point cloud includes: The initial point cloud is coarsely aligned to obtain a coarsely aligned point cloud; The coarsely aligned point cloud is denoised to obtain a denoised point cloud; The denoised point cloud is filtered using the differentiable expectation-maximum algorithm to obtain an optimized point cloud; The optimized point cloud is then upgraded to a point cloud feature map.

5. The pose estimation method according to claim 4, characterized in that, The step of coarsely aligning the initial point cloud to obtain a coarsely aligned point cloud includes: Through the formula: Calculate coarsely aligned point clouds ; in, Represents the initial rotation matrix. Represents the initial point cloud. Indicates the initial translation vector: in, This represents the centroid of the preset target point cloud. Indicates the centroid of the point cloud. Represents the covariance matrix Feature vectors: in, This indicates the number of points in the initial point cloud. Represents the first point in the initial point cloud. One point, This indicates the transpose operation.

6. The pose estimation method according to claim 5, characterized in that, The step of denoising the coarsely aligned point cloud to obtain a denoised point cloud includes: Through the formula: Computational Denoising of Point Clouds ; in, This indicates that the Gaussian function is used to assign matching weights. Indicates learnable parameters, This represents the average distance between two points in a coarsely aligned point cloud: in, This represents points in a coarsely aligned point cloud. Indicates the first coarsely aligned point cloud Points The set of nearest neighbors, express The number of midpoints , This indicates the number of points in the coarsely aligned point cloud.

7. The pose estimation method according to claim 6, characterized in that, The step of using a differentiable expectation-maximum algorithm to filter the denoised point cloud to obtain an optimized point cloud includes: An expression for the interior point probability of each point in the denoised point cloud is established using the differentiable expectation-maximum algorithm; The expression for the inlier probability of each point is solved to obtain the final inlier probability of each point in the denoised point cloud; The optimized point cloud is obtained by retaining all points in the denoised point cloud whose final interior point probability is greater than a preset probability threshold.

8. The pose estimation method according to claim 7, characterized in that, The expression for the interior point probability is: in, Indicating the first point in the denoised cloud The probability of an interior point. , This indicates the number of points in the denoised point cloud. Indicates the scale of interior points. Represents statistical distribution. Indicates the first The distance between each point and the target point, where the target point is a reference point for calculating the distance between points. This represents the mean distance between inliers in the denoised point cloud. This represents the standard deviation of the inlier distance in the denoised point cloud. This represents the uniform distribution function.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the pose estimation method for cell preparation materials as described in any one of claims 1 to 8.

Images