Two-stage vision detection method, system, device and apparatus for micro-nano operation

Abstract

The embodiment of the present application provides a two-stage visual detection method, system, device and equipment for micro-nano operation, which comprises the following steps: inputting a picture to be detected into a first-stage object detection network for position detection to obtain an initial position of a micro object; and a second-stage position correction network corrects the initial position of the micro object based on a Kalman filtering algorithm and by using a position offset of the micro object to obtain a real-time position of the micro object, wherein the position offset of the micro object is determined according to a pre-trained Kalman gain matrix. In the embodiment of the present application, the initial position of the micro object is quickly output in the first stage, thereby guaranteeing the real-time performance of detection; meanwhile, the real-time motion position transformation problem is solved by using a position correction method in the second stage, thereby guaranteeing the accuracy of the detection result, so that the method can better balance the balance between the detection accuracy and the real-time performance, and the detection result has high accuracy and robustness.

Description

Technical Field

[0001] This invention relates to the field of visual inspection technology for micro-nano manipulation, and in particular to a two-stage visual inspection method, system, device and equipment for micro-nano manipulation. Background Technology

[0002] Unlike macroscopic robotic manipulation, micromanipulation relies heavily on vision, which is often the only effective means of detecting microscopic objects and providing feedback for subsequent tasks, such as optical capture, microfluidic cell selection and manipulation. In these tasks, accurate localization of micro-objects is crucial for the success of micromanipulation.

[0003] Because micro-objects exist in a dynamic and complex micromanipulation environment—that is, micro-objects are located in three-dimensional space at different depths, multiple objects may overlap and occlude each other, and the micromanipulation environment is usually highly dynamic, with multiple micro-objects entering or leaving the field of view, significant and frequent flow, random perturbations of Brownian forces, etc.—existing methods struggle to obtain accurate and robust detection results in such dynamic and complex micromanipulation environments. Summary of the Invention

[0004] In view of the above problems, embodiments of the present invention provide a two-stage visual inspection method, system, apparatus and device for micro-nano operations, so as to overcome the above problems or at least partially solve the above problems.

[0005] A first aspect of this invention discloses a two-stage visual inspection method for micro / nano manipulation, the method comprising:

[0006] The image to be detected is input into the first-stage object detection network for position detection to obtain the initial position of the micro-object;

[0007] The second-stage position correction network is based on the Kalman filter algorithm. It uses the position offset of the micro-object to correct the initial position of the micro-object and obtain the real-time position of the micro-object. The position offset of the micro-object is determined according to the pre-trained Kalman gain matrix.

[0008] Optionally, the first-stage object detection network is constructed in the following manner:

[0009] The first-stage object detection network is designed with the same CSP feature extraction framework as the YOLOX model as its backbone network;

[0010] The CIoU loss in the YOLOX model is replaced with the EIoU loss function, and the BCE loss function in the YOLOX model is replaced with the focus loss function.

[0011] Optionally, the second-stage position correction network is based on the Kalman filter algorithm, which uses the position offset of the micro-object to correct the initial position of the micro-object to obtain the real-time position of the micro-object, including:

[0012] State-space equations are constructed based on the Kalman filter algorithm;

[0013] The initial position of the micro-object is corrected using the state-space equation and the position offset of the micro-object to obtain the real-time position of the micro-object.

[0014] Optionally, the construction of the state-space equations based on the Kalman filter algorithm includes:

[0015] Determine the prior covariance, which is the covariance corresponding to the initial position of the micro-object;

[0016] Determine the posterior covariance, which is the covariance corresponding to the real-time position of the micro-object;

[0017] A real-time position prediction equation is determined, which is used to predict the real-time position of the micro-object based on its initial position and the Kalman gain matrix, wherein the Kalman gain matrix characterizes the position offset of the micro-object.

[0018] Optionally, the first-stage object detection network and the second-stage position correction network constitute a complete object detection network, which is trained in the following manner:

[0019] Obtain a training dataset, wherein the training data in the training dataset includes microscope images under different media and different lighting conditions;

[0020] The training dataset is input into the object detection network for learning;

[0021] For each piece of training data, the first-stage object detection network performs position detection on the training data to obtain an initial predicted position, and the second-stage position correction network corrects the initial predicted position to obtain a predicted position.

[0022] The state update difference and information difference are calculated based on the initial predicted position and the predicted position. The loss function value is calculated using the EIoU loss function and the focus loss function. The Kalman gain loss is calculated using the squared error loss function.

[0023] The network parameters of the first-stage object detection network are updated based on the state update difference, the information difference, and the loss function value, and the coefficients of the Kalman gain matrix of the second-stage position correction network are updated based on the Kalman gain loss.

[0024] After the training termination condition is met, the trained object detection network and Kalman gain matrix are obtained.

[0025] Optionally, calculating the state update difference and information difference based on the initial predicted position and the predicted position includes:

[0026] Subtracting the initial predicted position from the predicted position yields the state update difference, which is expressed as:

[0027]

[0028] in, This represents the state update difference. Indicates the predicted location. Indicates the initial predicted position;

[0029] The information difference is obtained by subtracting the measured value of the initial predicted position from the measured value of the predicted position. The information difference is expressed as follows:

[0030]

[0031] Where, Δy t Represents the information difference, y t|t The measured value represents the predicted location. The measurement value represents the initial predicted location.

[0032] Optionally, the EIoU loss function L EIoU Represented as:

[0033]

[0034] Among them, L IoU L represents the overlap loss. dis L represents the center distance loss. asp Let c represent the width and height loss, and w represent the diagonal length of the smallest bounding box that encloses the two boxes. c and h c Let b and b represent the width and height of the minimum bounding box that covers the two boxes, respectively. gt These represent the predicted frame and the actual frame, respectively, where ρ represents the Euclidean distance and w gt and h gt These represent the width and height of the truth box, respectively.

[0035] The focus loss function FL(p) t ) is represented as:

[0036] FL(p t )=-α(1-p t )γ log(p t )

[0037]

[0038] Where, p t α represents the probability that a sample belongs to the true category, α represents the weighting factor between 0 and 1, β represents the focusing parameter, and y represents the true label.

[0039] A second aspect of the present invention discloses a two-stage visual inspection system for micro-nano manipulation, used to execute the two-stage visual inspection method for micro-nano manipulation described in the first aspect of the present invention, the system comprising:

[0040] An electric worktable, the electric worktable being used to place micro-objects;

[0041] A joystick that controls the movement of the motorized stage to bring the micro-object into the field of view of the microscope;

[0042] A microscope and a camera, wherein the microscope and the camera are used to measure the position of the micro-object and to acquire an image of the micro-object to be detected;

[0043] A computer is used to display the image to be detected and to perform two-stage visual detection on the image to obtain the real-time position of the micro-object.

[0044] A third aspect of the present invention discloses a two-stage visual inspection device for micro-nano operations, the device comprising:

[0045] The detection module is used to input the image to be detected into the first-stage object detection network for position detection, and obtain the initial position of the micro-object;

[0046] The correction module is used in the second-stage position correction network based on the Kalman filter algorithm to correct the initial position of the micro-object using the position offset of the micro-object, thereby obtaining the real-time position of the micro-object. The position offset of the micro-object is determined based on a pre-trained Kalman gain matrix.

[0047] A fourth aspect of the present invention discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executed, implements the two-stage visual inspection method for micro-nano operations as described in the first aspect of the present invention.

[0048] The embodiments of the present invention have the following advantages:

[0049] In this embodiment of the invention, a two-stage visual detection method for micro- and nano-manipulation is provided, which follows a two-stage detection process from coarse to fine. First, the image to be detected is input into a first-stage object detection network for position detection to obtain the initial position of the micro-object. Then, a second-stage position correction network, based on the Kalman filter algorithm, corrects the initial position of the micro-object using the position offset of the micro-object to obtain the real-time position of the micro-object. The position offset of the micro-object is determined based on a pre-trained Kalman gain matrix.

[0050] Because the process is divided into two stages, the initial position of the micro-object is quickly output in the first stage, thus ensuring the real-time performance of the detection. At the same time, the position correction method is used in the second stage to handle the real-time motion position transformation problem and fine-tune the initial position to obtain a more accurate result, thus ensuring the accuracy of the detection result. Therefore, the two-stage visual detection method for micro-nano operations provided in this embodiment can better balance the balance between detection accuracy and real-time performance, and the obtained detection result has high accuracy and robustness. Attached Figure Description

[0051] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is an example of the deviation between the predicted position and the true position provided in an embodiment of the present invention;

[0053] Figure 2 This is a flowchart illustrating the steps of a two-stage visual inspection method for micro / nano manipulation provided in an embodiment of the present invention.

[0054] Figure 3 This is a schematic diagram of the framework of an object detection network provided in an embodiment of the present invention;

[0055] Figure 4 This is an example of training sample data provided in an embodiment of the present invention;

[0056] Figure 5 This invention provides a comparison of micro-object detection results in different scenarios.

[0057] Figure 6 This is a schematic diagram of the structure of a two-stage visual inspection system for micro-nano operations provided in an embodiment of the present invention;

[0058] Figure 7This is a physical example of a two-stage visual inspection system for micro-nano operations provided in this embodiment of the invention;

[0059] Figure 8 This is a schematic diagram of a two-stage visual inspection device for micro-nano operations provided in an embodiment of the present invention. Detailed Implementation

[0060] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] In related technologies, the center point coordinates of the bounding box regressed by object detection networks cannot well fit the requirements of optical tweezers control. For example... Figure 1 As shown, when the size and quality of the dataset are relatively low, the detection results are biased, which can significantly affect the accuracy of the detection and even lead to the failure of micro-operations. Therefore, in order to achieve high-precision position detection of micro-objects and to balance detection accuracy and real-time performance, the applicant proposes the following technical concept: a two-stage detection method from coarse to fine. First, a first-stage object detection network is used to perform preliminary position detection of the micro-object to quickly output the initial position; then, a second-stage position correction network is used to correct the initial position obtained in the first stage, further improving the result to obtain the accurate position.

[0062] Based on the above technical concept, embodiments of the present invention provide a two-stage visual inspection method for micro-nano operations, such as... Figure 2 As shown, Figure 2 This invention provides a flowchart of a two-stage visual inspection method for micro / nano manipulation, comprising steps S201 to S202:

[0063] Step S201: Input the image to be detected into the first-stage object detection network for position detection to obtain the initial position of the micro-object.

[0064] In this embodiment, the image to be detected is a microscope image of a micro-object, acquired using a microscope and camera. Specifically, the image to be detected can be a microscope image of a cell. The first-stage object detection network is a target detection network, and the initial position of the micro-object is a coarse estimate. Therefore, by processing the image to be detected through the first-stage object detection network, the initial position of the micro-object can be quickly output, thus obtaining a coarse detection result. The rapid output of the initial position of the micro-object ensures, to a certain extent, the real-time performance of the detection.

[0065] In an optional embodiment, the first-stage object detection network is constructed as follows:

[0066] The first-stage object detection network is designed with the same CSP feature extraction framework as the YOLOX model as its backbone network; the CIoU loss in the YOLOX model is replaced by the EIoU loss function, and the BCE loss function in the YOLOX model is replaced by the focus loss function.

[0067] In this embodiment, considering that the YOLOX model is an anchorless model—meaning it can directly predict the center of an object without needing to pre-use known anchor box offsets—it can significantly accelerate the convergence speed of training. Therefore, to speed up network training convergence, this embodiment constructs a first-stage object detection network based on the YOLOX model, using the same CSP feature extraction framework as the backbone network. Furthermore, to obtain a network more suitable for micro-object detection, the original loss function in the YOLOX model is replaced.

[0068] Because the SiLU activation function is unbounded by upper and lower limits, smooth, and non-monotonic, it outperforms Leaky ReLU in the model. Furthermore, considering that the bounding box after regression will affect the initial localization coordinates, the original CIoU loss function in the YOLOX model is replaced with the EIoU loss function. Using the replaced CIoU loss function can comprehensively consider the overlapping region, the distance between the center points, and the aspect ratio of the bounding box regression.

[0069] Specifically, the EIoU loss function L EIoU Represented as:

[0070]

[0071] Among them, L IoU L represents the overlap loss. dis L represents the center distance loss. asp Let c represent the width and height loss, and w represent the diagonal length of the smallest bounding box that encloses the two boxes.c and h c Let b and b represent the width and height of the minimum bounding box that covers the two boxes, respectively. gt These represent the predicted frame and the actual frame, respectively, where ρ represents the Euclidean distance and w gt and h gt These represent the width and height of the ground truth bounding box, respectively. Furthermore, the loss function in the first-stage object detection network makes it easier for the regression process to generate high-quality anchor points, improving the detection accuracy of micro-objects.

[0072] Furthermore, the BCE loss function in the YOLOX model is replaced with a focus loss function to reduce the weight of easily classified samples, allowing the first-stage object detection network to focus more on difficult-to-classify samples during training. The focus loss function FL(p t ) is represented as:

[0073] FL(p t )=-α(1-p t ) γ log(p t )

[0074]

[0075] Where, p t Let α represent the probability that a sample belongs to the true class, β represent the weighting factor between 0 and 1, β represent the focusing parameter, and y represent the true label. A y value of 1 indicates that the true label of the sample is positive. Furthermore, the focusing loss function in the first-stage object detection network allows it to pay more attention to the detection of micro-objects.

[0076] In this embodiment, since the first-stage object detection network is built based on the YOLOX model, the training and convergence speed of the first-stage object detection network is fast. At the same time, the EIoU loss function and the focus loss function enable the first-stage object detection network to generate high-quality anchor points to realize the detection of micro-object positions.

[0077] Step S202: The second-stage position correction network is based on the Kalman filter algorithm. It uses the position offset of the micro-object to correct the initial position of the micro-object and obtain the real-time position of the micro-object. The position offset of the micro-object is determined according to the pre-trained Kalman gain matrix.

[0078] In this embodiment, since the micro-object is dynamically changing, position offset is used to handle real-time movement and position changes in order to obtain an accurate micro-object position. The second-stage position correction network used for micro-object position correction is a pre-trained network. The Kalman gain matrix in the second-stage position correction network can provide the predicted position offset of the micro-object. Therefore, the initial position of the micro-object is corrected using the second-stage position correction network to obtain an accurate real-time position of the micro-object, thereby ensuring the accuracy of the detection result. Therefore, the two-stage visual detection method for micro-nano manipulation described in steps S201 and S202 above can better balance the balance between detection accuracy and real-time performance, and the obtained detection result has high accuracy and robustness.

[0079] In an optional embodiment, the second-stage position correction network is based on the Kalman filter algorithm, which uses the position offset of the micro-object to correct the initial position of the micro-object to obtain the real-time position of the micro-object, including steps S202-1 and S202-2:

[0080] Step S202-1: Construct the state-space equation based on the Kalman filter algorithm.

[0081] In this embodiment, the Kalman filter algorithm, given a known measurement variance, can estimate the state of a dynamic system from a series of data containing measurement noise. Therefore, based on the Kalman filter algorithm, the real-time position of the micro-object is predicted from its initial position.

[0082] Specifically, the construction of the state-space equation based on the Kalman filter algorithm includes steps A1 to A3:

[0083] Step A1: Determine the prior covariance, which is the covariance corresponding to the initial position of the micro-object.

[0084] Specifically, the covariance corresponding to the initial position of the micro-object is also called the prior covariance of the state vector at time t, which is expressed as:

[0085]

[0086] in, Let x represent the prior mean, which characterizes the initial position of the micro-object. t This represents the prior value.

[0087] Step A2: Determine the posterior covariance, which is the covariance corresponding to the real-time position of the micro-object.

[0088] Specifically, the covariance corresponding to the real-time position of the micro-object is also called the posterior covariance of the state vector at time t, which is expressed as:

[0089]

[0090] in, denoted as the posterior mean, which represents the real-time position of the micro-object.

[0091] Step A3: Determine the real-time position prediction equation, which is used to predict the real-time position of the micro-object based on its initial position and the Kalman gain matrix, wherein the Kalman gain matrix characterizes the position offset of the micro-object.

[0092] Specifically, the real-time location prediction equation is expressed as:

[0093]

[0094] Where θ represents the trainable parameters, and K t (θ) represents the Kalman gain matrix. The Kalman gain matrix obtained after training is a fixed matrix that characterizes the positional offset of the micro-object. y represents the mean of the measurements. t Indicates the measured value. This represents the information difference, which is a fixed value obtained through training.

[0095] In this embodiment, the prior covariance and posterior covariance are used to update the Kalman gain matrix during the training of the second-stage position correction network. After training, a definite Kalman gain matrix is ​​obtained to characterize the position offset of the micro-object. The real-time position prediction equation is used to calculate the real-time position of the micro-object in subsequent steps.

[0096] Step S202-2: Correct the initial position of the micro-object using the state-space equation and the position offset of the micro-object to obtain the real-time position of the micro-object.

[0097] In this embodiment, the position offset of the micro-object is a fixed value obtained through training, and the information difference in the real-time position prediction equation is a fixed value. The position offset and the initial position of the micro-object are substituted into the real-time position prediction equation for calculation to obtain the real-time position of the micro-object. Furthermore, the initial position in the first stage is effectively corrected based on the Kalman filter algorithm, compensating for the error caused by the real-time movement and position change of the micro-object, and ensuring the accuracy and robustness of the micro-object position detection.

[0098] In an optional embodiment, the first-stage object detection network and the second-stage position correction network constitute a complete object detection network, such as... Figure 3 As shown, Figure 3The schematic diagram illustrates the framework of the object detection network in this embodiment. After the image to be detected is input into the object detection network, the first-stage object detection network quickly outputs the initial position of a micro-object from the image. Then, the second-stage position correction network corrects the position based on the initial position, finally obtaining the real-time position of the micro-object. The object detection network is trained in the following manner, specifically including steps B1 to B6:

[0099] Step B1: Obtain the training dataset, which includes microscopic images under different media and lighting conditions.

[0100] In this embodiment, a training dataset is constructed to train the object detection network. For example... Figure 4 As shown, unlike existing datasets used for supervised network training of cell image enhancement models, the training data in this embodiment includes microscope images under different media and lighting conditions. Specifically, when actually acquiring microscope images, the light intensity and focal plane position are adjusted to obtain microscope images under different lighting conditions, and ground truth labels are added to each microscope image to obtain the training data.

[0101] Step B2: Input the training dataset into the object detection network for learning.

[0102] Step B3: For each training data, the first-stage object detection network performs position detection on the training data to obtain an initial predicted position, and the second-stage position correction network corrects the initial predicted position to obtain a predicted position.

[0103] Step B4: Calculate the state update difference and information difference based on the initial predicted position and the predicted position, calculate the loss function value using the EIoU loss function and the focus loss function, and calculate the Kalman gain loss using the squared error loss function.

[0104] In this embodiment, the Kalman gain is trained by optimizing the squared error loss function of the training parameters θ in an end-to-end manner. Therefore, the Kalman gain loss is calculated using the squared error loss function, so that the coefficients of the Kalman gain matrix of the second-stage position correction network can be updated based on the Kalman gain loss in subsequent steps. Simultaneously, in order to implicitly track the covariance matrix, the object detection network updates the network parameters of the first-stage object detection network using the state update difference and information difference as inputs.

[0105] Specifically, calculating the state update difference and information difference based on the initial predicted position and the predicted position includes: subtracting the initial predicted position from the predicted position to obtain the state update difference, wherein the state update difference is expressed as:

[0106]

[0107] in, This represents the state update difference. Indicates the predicted location. This indicates the initial predicted position.

[0108] The information difference is obtained by subtracting the measured value of the initial predicted position from the measured value of the predicted position. The information difference is expressed as follows:

[0109]

[0110] Where, Δy t Represents the information difference, y t|t The measured value represents the predicted location. The measurement value represents the initial predicted location.

[0111] Step B5: Update the network parameters of the first-stage object detection network based on the state update difference, the information difference, and the loss function value, and update the coefficients of the Kalman gain matrix of the second-stage position correction network based on the Kalman gain loss.

[0112] Step B6: After the training termination condition is met, the trained object detection network and Kalman gain matrix are obtained.

[0113] In this embodiment, the object detection network trained on the training dataset exhibits good detection accuracy and robustness. The network can receive images of micro-objects and automatically detect them, obtaining their initial positions. Simultaneously, the trained Kalman gain matrix provides the predicted position offset of the micro-object. The object detection network then uses this position offset to correct the initial position of the micro-object, obtaining its real-time position.

[0114] To further illustrate the performance of the proposed two-stage visual detection method for micro / nano manipulation, a comparative study was conducted on two baselines (i.e., the original YOLOX-S base model and the base model with EIoU loss). The verification results are shown in Table 1. As can be seen from Table 1, the network using EIoU loss achieves a 2.14% improvement, and the method provided in this embodiment achieves a further 2.35% improvement compared to the method using the base model and EIoU loss. It is worth noting that the significant improvement in the average accuracy of the object detection network in this embodiment leads to a slight increase in the parameter size (i.e., 9.73M) and GLOPs (i.e., 31.4G).

[0115] Table 1. Performance comparison results of different methods

[0116] Basic Model 38.4 9.0 26.8 Base model + EIoU loss 40.18 9.59 29.7 The method in this embodiment 42.53 9.73 31.4

[0117] Figure 5 The illustration shows a comparison of detection results under different scenarios. In four different scenarios (i.e., normal micro-objects, dark environment, defocused micro-objects, and entering micro-objects), the two-stage visual detection method for micro-nano operations proposed in this embodiment has higher accuracy than the detection method without position correction.

[0118] This invention provides a two-stage visual detection method for micro- and nano-manipulation, which follows a coarse-to-fine two-stage detection process. First, the image to be detected is input into a first-stage object detection network for position detection, obtaining the initial position of the micro-object. Then, a second-stage position correction network, based on a Kalman filter algorithm, corrects the initial position of the micro-object using its position offset, obtaining the real-time position of the micro-object. The position offset is determined based on a pre-trained Kalman gain matrix. Because the process is divided into two stages, the initial position of the micro-object is quickly output in the first stage, ensuring real-time detection. Simultaneously, the position correction method in the second stage handles real-time motion position changes, fine-tuning the initial position to obtain a more accurate result, thus ensuring the accuracy of the detection result. Therefore, the two-stage visual detection method for micro- and nano-manipulation provided in this embodiment better balances detection accuracy and real-time performance, resulting in detection results with high accuracy and robustness.

[0119] This invention also provides a two-stage visual inspection system for micro / nano manipulation, used to execute the two-stage visual inspection method for micro / nano manipulation described in the first aspect of this invention, such as... Figure 6 As shown, Figure 6 This is a schematic diagram of a two-stage visual inspection system for micro / nano manipulation provided in an embodiment of the present invention. The system includes:

[0120] An electric worktable, the electric worktable being used to place micro-objects;

[0121] A joystick that controls the movement of the motorized stage to bring the micro-object into the field of view of the microscope;

[0122] A microscope and a camera, wherein the microscope and the camera are used to measure the position of the micro-object and to acquire an image of the micro-object to be detected;

[0123] A computer is used to display the image to be detected and to perform two-stage visual detection on the image to obtain the real-time position of the micro-object.

[0124] In practical applications, the microscope can be a Nikon ECLIPSE Ti2-U, the motorized stage is a THORLABS MLS203-1 high-speed XY scanning stage, and the camera is a BASLER PLA640-210gm camera. Figure 7 This is a physical example of the two-stage visual inspection system provided in this embodiment. When inspecting micro-objects, the micro-object is placed on an electric worktable, and the micro-object is brought into the field of view of the microscope by manipulating the joystick. Then, the microscope and camera are used to acquire microscope images of the micro-object. The acquired microscope images can also be displayed on the computer. Finally, the computer performs two-stage visual inspection on the acquired images to obtain the real-time position of the micro-object.

[0125] This invention also provides a two-stage visual inspection device for micro-nano operations, such as... Figure 8 As shown, Figure 8 This is a schematic diagram of a two-stage visual inspection device for micro-nano operations provided in an embodiment of the present invention. The device includes:

[0126] The detection module 81 is used to input the image to be detected into the first-stage object detection network for position detection to obtain the initial position of the micro-object;

[0127] The correction module 82 is used in the second-stage position correction network based on the Kalman filter algorithm to correct the initial position of the micro-object using the position offset of the micro-object, thereby obtaining the real-time position of the micro-object. The position offset of the micro-object is determined based on a pre-trained Kalman gain matrix.

[0128] In an optional embodiment, the first-stage object detection network is constructed using a network construction module, the network construction module comprising:

[0129] The network design module is used to design the first-stage object detection network using the same CSP feature extraction framework as the YOLOX model as the backbone network.

[0130] The function replacement module is used to replace the CIoU loss in the YOLOX model with the EIoU loss function and to replace the BCE loss function in the YOLOX model with the focus loss function.

[0131] In one optional embodiment, the correction module includes:

[0132] The equation construction module is used to construct state-space equations based on the Kalman filter algorithm.

[0133] The position correction module is used to correct the initial position of the micro-object using the state space equation and the position offset of the micro-object, so as to obtain the real-time position of the micro-object.

[0134] In one optional embodiment, the equation construction module includes:

[0135] The first construction module is used to determine the prior covariance, which is the covariance corresponding to the initial position of the micro-object;

[0136] The second construction module is used to determine the posterior covariance, which is the covariance corresponding to the real-time position of the micro-object.

[0137] The third construction module is used to determine the real-time position prediction equation, which is used to predict the real-time position of the micro-object based on the initial position of the micro-object and the Kalman gain matrix, wherein the Kalman gain matrix represents the position offset of the micro-object.

[0138] In an optional embodiment, the first-stage object detection network and the second-stage position correction network constitute a complete object detection network. The object detection network is trained by a training module, which includes:

[0139] The first training submodule is used to acquire a training dataset, wherein the training data in the training dataset includes microscope images under different media and different lighting conditions;

[0140] The second training submodule is used to input the training dataset into the object detection network for learning;

[0141] The third training submodule is used for each training data point, whereby the first-stage object detection network performs position detection on the training data to obtain an initial predicted position, and the second-stage position correction network corrects the initial predicted position to obtain a predicted position.

[0142] The fourth training submodule is used to calculate the state update difference and information difference based on the initial predicted position and the predicted position, calculate the loss function value through the EIoU loss function and the focus loss function, and calculate the Kalman gain loss through the squared error loss function.

[0143] The fifth training submodule is used to update the network parameters of the first-stage object detection network based on the state update difference, the information difference, and the loss function value, and to update the coefficients of the Kalman gain matrix of the second-stage position correction network based on the Kalman gain loss.

[0144] The sixth training submodule is used to obtain the trained object detection network and Kalman gain matrix after the training termination conditions are met.

[0145] In one optional embodiment, the fourth training submodule includes:

[0146] The state update difference unit is used to subtract the initial prediction position from the predicted position to obtain the state update difference.

[0147] The information difference unit is used to subtract the measured value of the initial predicted position from the measured value of the predicted position to obtain the information difference.

[0148] This invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When executed by the processor, the program implements the two-stage visual inspection method for micro-nano operations described in this invention.

[0149] This invention also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the two-stage visual inspection method for micro-nano operations described in this invention.

[0150] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0151] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, electronic devices, and storage media according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0152] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0153] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0154] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0155] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0156] The foregoing has provided a detailed description of a two-stage visual inspection method, system, apparatus, and device for micro-nano operations provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A two-stage visual inspection method for micro / nano manipulation, characterized in that, The method includes: The image to be detected is input into the first-stage object detection network for position detection to obtain the initial position of the micro-object; The second-stage position correction network is based on the Kalman filter algorithm. It uses the position offset of the micro-object to correct the initial position of the micro-object and obtain the real-time position of the micro-object. The position offset of the micro-object is determined according to the pre-trained Kalman gain matrix. The first-stage object detection network and the second-stage position correction network constitute a complete object detection network, which is trained in the following manner: Obtain a training dataset, wherein the training data in the training dataset includes microscope images under different media and different lighting conditions; The training dataset is input into the object detection network for learning; For each piece of training data, the first-stage object detection network performs position detection on the training data to obtain an initial predicted position, and the second-stage position correction network corrects the initial predicted position to obtain a predicted position. The state update difference and information difference are calculated based on the initial predicted position and the predicted position. The loss function value is calculated using the EIoU loss function and the focus loss function. The Kalman gain loss is calculated using the squared error loss function. The network parameters of the first-stage object detection network are updated based on the state update difference, the information difference, and the loss function value, and the coefficients of the Kalman gain matrix of the second-stage position correction network are updated based on the Kalman gain loss. After the training termination condition is met, the trained object detection network and Kalman gain matrix are obtained; Among them, the Kalman gain matrix obtained after training is a fixed matrix to characterize the positional offset of the micro-object; The second-stage position correction network, based on the Kalman filter algorithm, corrects the initial position of the micro-object using its position offset to obtain its real-time position, including: A state-space equation is constructed based on the Kalman filter algorithm. The state-space equation is used to predict the real-time position of the micro-object based on its initial position, Kalman gain matrix, and information difference. The information difference is a fixed value obtained through training. The initial position of the micro-object is corrected using the state-space equation and the position offset of the micro-object to obtain the real-time position of the micro-object. The position offset is a fixed value obtained through training.

2. The method according to claim 1, characterized in that, The first-stage object detection network is constructed in the following way: The first-stage object detection network is designed with the same CSP feature extraction framework as the YOLOX model as its backbone network; The CIoU loss in the YOLOX model is replaced with the EIoU loss function, and the BCE loss function in the YOLOX model is replaced with the focus loss function.

3. The method according to claim 1, characterized in that, The construction of the state-space equations based on the Kalman filter algorithm includes: Determine the prior covariance, which is the covariance corresponding to the initial position of the micro-object; Determine the posterior covariance, which is the covariance corresponding to the real-time position of the micro-object; A real-time position prediction equation is determined, which is used to predict the real-time position of the micro-object based on its initial position and the Kalman gain matrix, wherein the Kalman gain matrix characterizes the position offset of the micro-object.

4. The method according to claim 1, characterized in that, The step of calculating the state update difference and information difference based on the initial predicted position and the predicted position includes: Subtracting the initial predicted position from the predicted position yields the state update difference, which is expressed as: in, This represents the state update difference. Indicates the predicted location. Indicates the initial predicted position; The information difference is obtained by subtracting the measured value of the initial predicted position from the measured value of the predicted position. The information difference is expressed as follows: in, Indicates the information difference. The measured value represents the predicted location. The measurement value represents the initial predicted location.

5. The method according to claim 2, characterized in that, The EIoU loss function Represented as: in, Indicates overlap loss. Indicates the center distance loss. Let represent the width and height loss, and 'c' represent the diagonal length of the smallest bounding box that encloses the two boxes. and These represent the width and height of the minimum bounding box that covers the two boxes, respectively. and These represent the predicted frame and the actual frame, respectively. Represents Euclidean distance. and These represent the width and height of the truth box, respectively. The focus loss function Represented as: in, This represents the probability that a sample belongs to the true class. The weight factor is between 0 and 1. Indicates the focus parameter, This indicates the actual label.

6. A two-stage visual inspection system for micro / nano manipulation, characterized in that, The system is used to perform the two-stage visual inspection method for micro / nano manipulation as described in any one of claims 1-5, the system comprising: An electric worktable, the electric worktable being used to place micro-objects; A joystick that controls the movement of the motorized stage to bring the micro-object into the field of view of the microscope; A microscope and a camera, wherein the microscope and the camera are used to measure the position of the micro-object and to acquire an image of the micro-object to be detected; A computer is used to display the image to be detected and to perform two-stage visual detection on the image to obtain the real-time position of the micro-object.

7. A two-stage visual inspection device for micro / nano manipulation, characterized in that, The device includes: The detection module is used to input the image to be detected into the first-stage object detection network for position detection, and obtain the initial position of the micro-object; The correction module, used in the second-stage position correction network based on the Kalman filter algorithm, corrects the initial position of the micro-object using its position offset to obtain the real-time position of the micro-object. The position offset of the micro-object is determined based on a pre-trained Kalman gain matrix. The first-stage object detection network and the second-stage position correction network constitute a complete object detection network. The object detection network is trained as follows: acquiring a training dataset, which includes microscope images under different media and lighting conditions; inputting the training dataset into the object detection network for learning; and for each training data point, the first-stage object detection network performs position detection to obtain an initial predicted position. The second-stage position correction network corrects the initial predicted position to obtain a predicted position; it calculates the state update difference and information difference based on the initial predicted position and the predicted position, calculates the loss function value using the EIoU loss function and the focus loss function, and calculates the Kalman gain loss using the squared error loss function; it updates the network parameters of the first-stage object detection network based on the state update difference, the information difference, and the loss function value, and updates the coefficients of the Kalman gain matrix of the second-stage position correction network based on the Kalman gain loss; after the training termination condition is met, the trained object detection network and Kalman gain matrix are obtained; wherein, the Kalman gain matrix obtained after training is a fixed matrix to represent the positional offset of the micro-object; The correction module is further configured to construct a state-space equation based on a Kalman filter algorithm. The state-space equation is used to predict the real-time position of the micro-object based on its initial position, Kalman gain matrix, and information difference, where the information difference is a fixed value obtained through training. The initial position of the micro-object is corrected using the state-space equation and the position offset of the micro-object to obtain the real-time position of the micro-object, where the position offset is a fixed value obtained through training.

8. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executed, implements the two-stage visual inspection method for micro-nano operations as described in any one of claims 1-5.

Images