Two-stage visual detection method, system and apparatus for micro-nano operation, and device
Through the two-stage visual detection method, combined with the YOLOX model and the Kalman filtering algorithm, the accuracy and real-time problems of micro-object detection in the micro-operating environment are solved, and high-precision and robust detection in dynamic and complex environments are achieved.
Patent Information
- Application Number
- PCT/CN2023/122706
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-09
- Filing Date
- 2023-09-28
- Publication Date
- 2024-11-14
AI Technical Summary
In dynamic and complex microoperation environments, existing methods are difficult to obtain accurate and robust microobject detection results, especially when microobjects are located in three-dimensional spaces of different depths and have occlusion and frequent flow.
Using a two-stage visual detection method, the first stage uses the CSP feature extraction framework of the YOLOX model for initial position detection, and the second stage uses the Kalman filtering algorithm to perform position correction based on the pre-trained Kalman gain matrix, and uses the EIoU loss function and focus loss. Function optimization network parameters.
It achieves higher detection accuracy and robustness in micro-nano operation, balances detection accuracy and real-time performance, and can accurately track the real-time position of micro-objects in a dynamic environment.
Smart Images

Figure CN2023122706_14112024_PF_FP_ABST
Abstract
Description
Two-stage visual inspection method, system, device and equipment for micro-nano operations
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on May 9, 2023, with application number 202310519515.1 and entitled “Two-stage visual inspection method, system, device and equipment for micro-nano operations”, the entire contents of which are incorporated by reference into this application. Technical Field
[0003] The present application relates to the field of micro-nano operation visual detection technology, and in particular to a two-stage visual detection method, system, device and equipment for micro-nano operation. Background Art
[0004] Unlike macro-scale robotic manipulation, micromanipulation relies heavily on vision, which is often the only effective measure to detect microscopic objects and provide feedback information for subsequent tasks, such as optical trapping, microfluidic cell selection and processing. In these tasks, accurate positioning of micro-objects is the key to successful micromanipulation.
[0005] Because micro-objects exist in a dynamic and complex micro-manipulation environment, that is, micro-objects are located in three-dimensional space at different depths, multiple objects may overlap and occlude each other, and the micro-manipulation environment is usually highly dynamic, with multiple micro-objects entering or leaving the field of view, significant and frequent flow, and random perturbations of Brownian forces. Therefore, existing methods have difficulty obtaining accurate and robust detection results in such a dynamic and complex micro-manipulation environment.
[0006] Summary of the Invention
[0007] In view of the above problems, the embodiments of the present application provide a two-stage visual inspection method, system, device and equipment for micro-nano operations to overcome the above problems or at least partially solve the above problems.
[0008] In a first aspect of the embodiments of the present application, a two-stage visual inspection method for micro-nano operations is disclosed, the method comprising:
[0009] The image to be detected is input into the first-stage object detection network for position detection to obtain the initial position of the micro-object;
[0010] The second-stage position correction network is based on the Kalman filter algorithm, which uses the position offset of the micro-object to correct the initial position of the micro-object to obtain the real-time position of the micro-object, wherein the position offset of the micro-object is determined according to the pre-trained Kalman gain matrix.
[0011] Optionally, the first-stage object detection network is constructed in the following manner:
[0012] The first-stage object detection network is designed using the same CSP feature extraction framework as the YOLOX model as the backbone network;
[0013] The CIoU loss in the YOLOX model is replaced by the EIoU loss function, and the BCE loss function in the YOLOX model is replaced by the focal loss function.
[0014] Optionally, the second-stage position correction network is based on a Kalman filter algorithm and uses the position offset of the micro-object to correct the initial position of the micro-object to obtain the real-time position of the micro-object, including:
[0015] Construct state space equations based on Kalman filter algorithm;
[0016] The initial position of the micro-object is corrected using the state space equation and the position offset of the micro-object to obtain the real-time position of the micro-object.
[0017] Optionally, constructing the state space equation based on the Kalman filter algorithm includes:
[0018] Determining a priori covariance, where the priori covariance is the covariance corresponding to the initial position of the micro-object;
[0019] Determining a posterior covariance, where the posterior covariance is the covariance corresponding to the real-time position of the micro-object;
[0020] A real-time position prediction equation is determined, wherein the real-time position prediction equation is used to predict the real-time position of the micro-object based on the initial position of the micro-object and a Kalman gain matrix, wherein the Kalman gain matrix represents the position offset of the micro-object.
[0021] Optionally, the first-stage object detection network and the second-stage position correction network constitute a complete object detection network, and the object detection network is trained in the following manner:
[0022] Acquire a training data set, wherein the training data in the training data set includes microscope images in different media and different lighting conditions;
[0023] Inputting the training data set into the object detection network for learning;
[0024] For each of the training data, the first-stage object detection network performs position detection on the training data to obtain an initial predicted position, and the second-stage position correction network corrects the initial predicted position to obtain a predicted position;
[0025] Calculate the state update difference and the information difference according to the initial predicted position and the predicted position, calculate the loss function value by the EIoU loss function and the focus loss function, and calculate the Kalman gain loss by the square error loss function;
[0026] updating the network parameters of the first-stage object detection network based on the state update difference, the information difference, and the loss function value, and updating the coefficients of the Kalman gain matrix of the second-stage position correction network based on the Kalman gain loss;
[0027] After the training end conditions are met, the trained object detection network and Kalman gain matrix are obtained.
[0028] Optionally, the calculating a state update difference and an information difference according to the initial predicted position and the predicted position includes:
[0029] The initial predicted position is subtracted from the predicted position to obtain the state update difference, which is expressed as:
[0030] in, Indicates the status update difference, represents the predicted position, represents the initial predicted position;
[0031] The measured value of the predicted position is subtracted from the measured value of the initial predicted position to obtain the information difference value, which is expressed as:
[0032] Where Δy t represents the information difference, y t|t represents the measured value at the predicted location, The measured values representing the initial prediction locations.
[0033] Optionally, the EIoU loss function L EIoU Expressed as:
[0034] Among them, L IoU represents the overlap loss, L dis represents the center distance loss, L asp represents the width and height loss, c represents the diagonal length of the minimum bounding box surrounding the two boxes, w c and h c The width and height of the minimum bounding box covering the two boxes are b and b respectively. gt Represent the predicted frame and the actual frame respectively, ρ represents the Euclidean distance, w gt and h gtRepresent the width and height of the true value bounding box respectively;
[0035] The focal loss function FL(p t ) is expressed as: FL(p t )=-α(1-p t ) γ log(p t )
[0036] Among them, p t represents the probability that the sample belongs to the true category, α represents the weight factor between 0 and 1, γ represents the focus parameter, and y represents the true label.
[0037] A second aspect of the embodiments of the present application discloses a two-stage visual inspection system for micro-nano operations, for performing the two-stage visual inspection method for micro-nano operations described in the first aspect of the embodiments of the present application, the system comprising:
[0038] An electric workbench, wherein the electric workbench is used to place micro-objects;
[0039] a joystick for manipulating the electric worktable to move so that the micro-object appears within the field of view of the microscope;
[0040] A microscope and a camera, wherein the microscope and the camera are used to measure the position of the micro-object and obtain a picture to be detected corresponding to the micro-object;
[0041] A computer is used to display the image to be detected and perform two-stage visual detection on the image to be detected to obtain the real-time position of the micro-object.
[0042] In a third aspect of the embodiments of the present application, a two-stage visual inspection device for micro-nano operations is disclosed, the device comprising:
[0043] The detection module is used to input the image to be detected into the first-stage object detection network for position detection to obtain the initial position of the micro-object;
[0044] The correction module is used for the second-stage position correction network based on the Kalman filter algorithm, which uses the position offset of the micro-object to correct the initial position of the micro-object to obtain the real-time position of the micro-object, wherein the position offset of the micro-object is determined according to the pre-trained Kalman gain matrix.
[0045] The fourth aspect of the embodiments of the present application discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor. When executed, the processor implements the two-stage visual inspection method for micro-nano operations as described in the first aspect of the embodiments of the present application.
[0046] The embodiments of the present application include the following advantages:
[0047] In an embodiment of the present application, a two-stage visual inspection method for micro-nano operations is provided. The method follows a two-stage inspection process from coarse to fine. First, the image to be inspected is input into the first-stage object detection network for position detection, which determines the initial position of the micro-object. Then, the second-stage position correction network, based on the Kalman filter algorithm, uses the micro-object's position offset to correct the initial position of the micro-object to obtain the micro-object's real-time position. The micro-object's position offset is determined based on a pre-trained Kalman gain matrix.
[0048] Since the processing is divided into two stages, the initial position of the micro-object is quickly output in the first stage, thereby ensuring the real-time detection; at the same time, the position correction method is used in the second stage to deal with the real-time motion position transformation problem, and the initial position is fine-tuned to obtain more accurate results, thereby ensuring the accuracy of the detection results. Therefore, the two-stage visual detection method for micro-nano operations provided in this embodiment can better balance the balance between detection accuracy and real-time performance, and the obtained detection results have higher accuracy and robustness. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following is a brief introduction to the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0050] FIG1 is an example of the deviation between a predicted position and a true position provided by an embodiment of the present application;
[0051] FIG2 is a flowchart of a two-stage visual inspection method for micro-nano operations provided by an embodiment of the present application;
[0052] FIG3 is a schematic diagram of a framework of an object detection network provided in an embodiment of the present application;
[0053] FIG4 is an example of training sample data provided in an embodiment of the present application;
[0054] FIG5 is a comparison of micro-object detection results in different scenarios provided by an embodiment of the present application;
[0055] FIG6 is a schematic structural diagram of a two-stage visual inspection system for micro-nano operations provided in an embodiment of the present application;
[0056] FIG7 is a physical example of a two-stage visual inspection system for micro-nano operations provided by an embodiment of the present application;
[0057] FIG8 is a schematic structural diagram of a two-stage visual inspection device for micro-nano operations provided in an embodiment of the present application. Specific embodiments
[0058] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.
[0059] In the related art, the center point coordinates of the bounding box regression obtained by the target detection network cannot well fit the requirements of optical tweezers control. As shown in Figure 1, when the scale and quality of the data set are relatively low, there is a deviation in the detection results. This deviation will significantly affect the accuracy of the detection and even cause the failure of micro-operation. Therefore, in order to achieve high-precision position detection of micro-objects and balance detection accuracy and real-time performance, the applicant proposes the following technical concept: a two-stage detection method from coarse to fine is proposed. First, the first-stage object detection network is used to perform preliminary position detection on the micro-object to quickly output the initial position; then, the second-stage position correction network is used to correct the initial position obtained in the first stage, further improving the result to obtain an accurate position.
[0060] Based on the above technical concept, the embodiment of the present application provides a two-stage visual inspection method for micro-nano operations, as shown in Figure 2. Figure 2 is a flowchart of the steps of a two-stage visual inspection method for micro-nano operations provided by the embodiment of the present application. The method includes steps S201 to S202:
[0061] Step S201: Input the image to be detected into the first-stage object detection network for position detection to obtain the initial position of the micro-object.
[0062] In this embodiment, the image to be detected is a microscopic image of a micro-object, which is acquired using a microscope and a camera. Specifically, the image to be detected can be a microscopic image of a cell. The first-stage object detection network is a target detection network, and the initial position of the micro-object is a rough estimate. Therefore, by processing the image to be detected through the first-stage object detection network, the initial position of the micro-object can be quickly output, that is, a rough detection result can be obtained. The rapid output of the initial position of the micro-object ensures the real-time detection to a certain extent.
[0063] In an optional embodiment, the first-stage object detection network is constructed in the following manner:
[0064] The first-stage object detection network is designed using the same CSP feature extraction framework as the YOLOX model as the backbone network; the EIoU loss function is used to replace the CIoU loss in the YOLOX model, and the BCE loss function in the YOLOX model is used to replace the focal loss function.
[0065] In this embodiment, considering that the YOLOX model is an anchor-free model, that is, the YOLOX model can directly predict the center of the object without the need to use the known anchor box offset in advance, which can greatly accelerate the convergence speed of training. Therefore, in order to accelerate the convergence speed of network training, this embodiment constructs a first-stage object detection network based on the YOLOX model, that is, uses the same CSP feature extraction framework as the YOLOX model as the backbone network of the first-stage object detection network. Furthermore, in order to obtain a network more suitable for micro-object detection, the original loss function in the YOLOX model is replaced.
[0066] Because the SiLU activation function has the characteristics of no upper or lower bounds, smoothness, and non-monotonicity, it is superior to the Leaky ReLU in the model. In addition, considering that the bounding box after regression affects the coordinates of the initial positioning, the original CIoU loss function in the YOLOX model is replaced with the EIoU loss function. The replacement CIoU loss function comprehensively considers the overlapping area, center point distance, and aspect ratio of the bounding box regression.
[0067] Specifically, the EIoU loss function L EIoU Expressed as:
[0068] Among them, L IoU represents the overlap loss, l dis represents the center distance loss, L asp represents the width and height loss, c represents the diagonal length of the minimum bounding box surrounding the two boxes, w c and h c The width and height of the minimum bounding box covering the two boxes are b and b respectively. gt Represent the predicted frame and the actual frame respectively, ρ represents the Euclidean distance, w gt and h gt Represent the width and height of the ground-truth bounding box, respectively. Furthermore, the loss function in the first-stage object detection network makes it easier for the regression process to generate high-quality anchors, thereby improving the detection accuracy of micro-objects.
[0069] In addition, the BCE loss function in the YOLOX model is replaced by the focus loss function to reduce the weight of easy-to-classify samples, so that the first-stage object detection network can focus more on difficult-to-classify samples during training. t ) is expressed as: FL(p t )=-α(1-p t ) γ log(p t )
[0070] Among them, p t represents the probability that the sample belongs to the true category, α represents a weight factor between 0 and 1, γ represents the focus parameter, y represents the true label, and y equal to 1 indicates that the true label of the sample is a positive sample. Furthermore, the focus loss function in the first-stage object detection network can make the first-stage object detection network pay more attention to the detection of micro-objects.
[0071] In this embodiment, since the first-stage object detection network is built based on the YOLOX model, the training convergence speed of the first-stage object detection network is fast. At the same time, the EIoU loss function and the focus loss function enable the first-stage object detection network to generate high-quality anchor points to detect the position of micro-objects.
[0072] Step S202: The second-stage position correction network uses the position offset of the micro-object based on the Kalman filter algorithm to correct the initial position of the micro-object to obtain the real-time position of the micro-object, wherein the position offset of the micro-object is determined according to the pre-trained Kalman gain matrix.
[0073] In this embodiment, since the micro-objects are dynamically changing, in order to obtain the accurate micro-object positions, position offsets are used to handle the real-time mobile position transformation problem. The second-stage position correction network used for micro-object position correction is a pre-trained network. The Kalman gain matrix in the second-stage position correction network can provide the predicted position offset of the micro-objects. Therefore, the second-stage position correction network is used to correct the initial position of the micro-objects to obtain the accurate real-time position of the micro-objects, thereby ensuring the accuracy of the detection results. Therefore, the two-stage visual inspection method for micro-nano operations described in steps S201 and S202 can better balance the balance between detection accuracy and real-time performance, and the obtained detection results have higher accuracy and robustness.
[0074] In an optional embodiment, the second-stage position correction network is based on a Kalman filter algorithm and uses the position offset of the micro-object to correct the initial position of the micro-object to obtain the real-time position of the micro-object, including steps S202-1 and S202-2:
[0075] Step S202-1: Constructing a state space equation based on the Kalman filter algorithm.
[0076] In this embodiment, the Kalman filter algorithm can estimate the state of a dynamic system from a series of data with measurement noise when the measurement variance is known. Therefore, based on the Kalman filter algorithm, the real-time position of the micro-object is estimated based on the initial position of the micro-object.
[0077] Specifically, the state space equation is constructed based on the Kalman filter algorithm, including steps A1 to A3:
[0078] Step A1: Determine a priori covariance, where the priori covariance is the covariance corresponding to the initial position of the micro-object.
[0079] Specifically, the covariance corresponding to the initial position of the micro-object is also called the prior covariance of the state vector at time t. The prior covariance is expressed as:
[0080] in, Represents the prior mean, which represents the initial position of the micro-object, x t Represents a priori value.
[0081] Step A2: Determine a posterior covariance, where the posterior covariance is the covariance corresponding to the real-time position of the micro-object.
[0082] Specifically, the covariance corresponding to the real-time position of the micro-object is also called the posterior covariance of the state vector at time t, and the posterior covariance is expressed as:
[0083] in, Represents the posterior mean, which represents the real-time position of the micro-object.
[0084] Step A3: Determine a real-time position prediction equation, wherein the real-time position prediction equation is used to predict the real-time position of the micro-object based on the initial position of the micro-object and the Kalman gain matrix, wherein the Kalman gain matrix represents the position offset of the micro-object.
[0085] Specifically, the real-time position prediction equation is expressed as:
[0086] Among them, θ represents the trainable parameters, K t (θ) represents the Kalman gain matrix. The Kalman gain matrix obtained after training is a fixed matrix to characterize the position offset of the micro-object. represents the measured mean, y t Indicates the measured value, Represents the information difference, which is a fixed value obtained through training.
[0087] In this embodiment, the a priori covariance and posterior covariance are used to update the Kalman gain matrix during the training process of the second-stage position correction network. After training, a fixed Kalman gain matrix is obtained to represent the position offset of the micro-object. The real-time position prediction equation is used to calculate the real-time position of the micro-object in subsequent steps.
[0088] Step S202-2: using the state-space equation and the position offset of the micro-object to correct the initial position of the micro-object to obtain the real-time position of the micro-object.
[0089] In this embodiment, the micro-object's position offset is a fixed value obtained through training, while the information difference in the real-time position prediction equation is a fixed value. The micro-object's position offset and initial position are then incorporated into the real-time position prediction equation to calculate the micro-object's real-time position. Furthermore, the Kalman filter algorithm effectively corrects the initial position from the first stage, compensating for errors caused by the micro-object's real-time movement and position shifts, ensuring the accuracy and robustness of micro-object position detection.
[0090] In an optional embodiment, the first-stage object detection network and the second-stage position correction network constitute a complete object detection network, as shown in FIG3 , which illustrates a schematic diagram of the framework of the object detection network in this embodiment. After the image to be detected is input into the object detection network, the first-stage object detection network quickly outputs an initial position of a micro-object for the image to be detected, and then the second-stage position correction network performs position correction based on the initial position, ultimately obtaining the real-time position of the micro-object. The object detection network is trained in the following manner, specifically including steps B1 to B6:
[0091] Step B1: Acquire a training data set, where the training data in the training data set includes microscope images in different media and different lighting conditions.
[0092] In this embodiment, a training dataset is constructed to train the object detection network. As shown in Figure 4, unlike existing datasets used for supervised network training of cell image enhancement models, the training data in this embodiment includes microscope images captured in different media and under different lighting conditions. Specifically, when acquiring microscope images, the light intensity and focal plane position are adjusted to obtain microscope images under different lighting conditions. A ground-truth label is then added to each microscope image to generate the training data.
[0093] Step B2: Input the training data set into the object detection network for learning.
[0094] Step B3: For each of the training data, the first-stage object detection network performs position detection on the training data to obtain an initial predicted position, and the second-stage position correction network corrects the initial predicted position to obtain a predicted position.
[0095] Step B4: Calculate the state update difference and the information difference based on the initial predicted position and the predicted position, calculate the loss function value through the EIoU loss function and the focus loss function, and calculate the Kalman gain loss through the square error loss function.
[0096] In this embodiment, the Kalman gain is trained by optimizing the squared error loss function of the training parameter θ in an end-to-end manner. Therefore, the Kalman gain loss is calculated using the squared error loss function. In subsequent steps, the coefficients of the Kalman gain matrix of the second-stage position correction network are updated based on the Kalman gain loss. At the same time, in order to implicitly track the covariance matrix, the object detection network uses the state update difference and information difference as input to update the network parameters of the first-stage object detection network.
[0097] Specifically, the calculating of the state update difference and the information difference according to the initial predicted position and the predicted position includes: subtracting the initial predicted position from the predicted position to obtain the state update difference, and the state update difference is expressed as:
[0098] in, Indicates the status update difference, represents the predicted position, Represents the initial predicted position.
[0099] The measured value of the predicted position is subtracted from the measured value of the initial predicted position to obtain the information difference value, which is expressed as:
[0100] Where Δy t represents the information difference, y t|t represents the measured value at the predicted location, The measured values representing the initial prediction locations.
[0101] Step B5: Update the network parameters of the first-stage object detection network based on the state update difference, the information difference and the loss function value, and update the coefficients of the Kalman gain matrix of the second-stage position correction network based on the Kalman gain loss.
[0102] Step B6: After the training end conditions are met, the trained object detection network and Kalman gain matrix are obtained.
[0103] In this embodiment, the object detection network trained with the training dataset exhibits good detection accuracy and robustness. The object detection network can automatically detect micro-objects in the image to be detected and determine their initial positions. Furthermore, the trained Kalman gain matrix provides a predicted position offset for the micro-objects. The object detection network then uses this position offset to correct the initial position of the micro-objects, determining their real-time positions.
[0104] To further illustrate the performance of the proposed two-stage visual detection method for micro-nano operations, a comparative study was conducted on two baselines (i.e., the original YOLOX-S base model and the base model plus EIoU loss). The verification results are shown in Table 1. As can be seen from Table 1, the network using EIoU loss can achieve a 2.14% improvement. The method provided in this embodiment can further achieve a 2.35% improvement compared to the method using the base model and EIoU loss. It is worth noting that the significant improvement in the average precision of the object detection network of this embodiment results in a slight increase in parameter size (i.e., 9.73M) and GLOPs (i.e., 31.4G).
[0105] Table 1 Performance comparison results of different methods
[0106] Figure 5 illustrates a comparison of detection results in different scenarios. In four different scenarios (i.e., normal micro-objects, dark environments, defocused micro-objects, and entering micro-objects), the accuracy of the two-stage visual detection method for micro-nano operations proposed in this embodiment is higher than that of the detection method without position correction.
[0107] In an embodiment of the present application, a two-stage visual detection method for micro-nano operations is provided, which follows a two-stage detection process from coarse to fine. First, the image to be detected is input into the first-stage object detection network for position detection to obtain the initial position of the micro-object; then the second-stage position correction network uses the position offset of the micro-object based on the Kalman filter algorithm to correct the initial position of the micro-object to obtain the real-time position of the micro-object, wherein the position offset of the micro-object is determined according to the pre-trained Kalman gain matrix. Since it is divided into two stages of processing, the initial position of the micro-object is quickly output in the first stage, thereby ensuring the real-time performance of the detection; at the same time, the position correction method is used to handle the real-time motion position transformation problem in the second stage, and the initial position is fine-tuned to obtain a more accurate result, thereby ensuring the accuracy of the detection result. Therefore, the two-stage visual detection method for micro-nano operations provided by this embodiment can better balance the balance between detection accuracy and real-time performance, and the obtained detection result has high accuracy and robustness.
[0108] The present application also provides a two-stage visual inspection system for micro-nano operations, which is used to perform the two-stage visual inspection method for micro-nano operations described in the first aspect of the present application. As shown in FIG6 , FIG6 is a schematic structural diagram of a two-stage visual inspection system for micro-nano operations provided in the present application. The system includes:
[0109] An electric workbench, wherein the electric workbench is used to place micro-objects;
[0110] a joystick for manipulating the electric worktable to move so that the micro-object appears within the field of view of the microscope;
[0111] A microscope and a camera, wherein the microscope and the camera are used to measure the position of the micro-object and obtain a picture to be detected corresponding to the micro-object;
[0112] A computer is used to display the image to be detected and perform two-stage visual detection on the image to be detected to obtain the real-time position of the micro-object.
[0113] In practical applications, the microscope can be a Nikon ECLIPSE Ti2-U, the motorized stage can be a THORLABS MLS203-1 fast XY scanning stage, and the camera can be a BASLER PLA640-210gm camera. Figure 7 shows a physical example of the two-stage visual inspection system provided by this embodiment. When inspecting a micro-object, the micro-object is placed on the motorized stage. By manipulating the joystick, the micro-object appears within the microscope's field of view. The microscope and camera then capture a microscopic image of the object. This captured image can also be displayed on a computer. Finally, the computer performs two-stage visual inspection on the captured image to determine the micro-object's real-time position.
[0114] The present application also provides a two-stage visual inspection device for micro-nano operations, as shown in FIG8 . FIG8 is a schematic structural diagram of a two-stage visual inspection device for micro-nano operations provided in the present application. The device includes:
[0115] Detection module 81, used to input the image to be detected into the first stage object detection network for position detection to obtain the initial position of the micro-object;
[0116] The correction module 82 is used for the second-stage position correction network based on the Kalman filter algorithm, which uses the position offset of the micro-object to correct the initial position of the micro-object to obtain the real-time position of the micro-object, wherein the position offset of the micro-object is determined according to the pre-trained Kalman gain matrix.
[0117] In an optional embodiment, the first-stage object detection network is constructed by a network construction module, and the network construction module includes:
[0118] The network design module is used to design the first-stage object detection network using the same CSP feature extraction framework as the YOLOX model as the backbone network;
[0119] A function replacement module is used to replace the CIoU loss in the YOLOX model with the EIoU loss function, and to replace the BCE loss function in the YOLOX model with the focal loss function.
[0120] In an optional embodiment, the correction module includes:
[0121] Equation building module, used to build state space equations based on Kalman filter algorithm;
[0122] The position correction module is used to correct the initial position of the micro-object by using the state space equation and the position offset of the micro-object to obtain the real-time position of the micro-object.
[0123] In an optional embodiment, the equation building module includes:
[0124] A first building module is configured to determine a priori covariance, where the priori covariance is the covariance corresponding to the initial position of the micro-object;
[0125] A second building block is configured to determine a posterior covariance, where the posterior covariance is the covariance corresponding to the real-time position of the micro-object;
[0126] The third building block is used to determine a real-time position prediction equation, wherein the real-time position prediction equation is used to predict the real-time position of the micro-object based on the initial position of the micro-object and the Kalman gain matrix, and the Kalman gain matrix represents the position offset of the micro-object.
[0127] In an optional embodiment, the first-stage object detection network and the second-stage position correction network constitute a complete object detection network, and the object detection network is trained by a training module, and the training module includes:
[0128] A first training submodule is configured to obtain a training data set, wherein the training data in the training data set includes microscope images under different media and different illumination conditions;
[0129] A second training submodule is configured to input the training data set into the object detection network for learning;
[0130] a third training submodule, configured to, for each of the training data, perform position detection on the training data by the first-stage object detection network to obtain an initial predicted position, and to correct the initial predicted position by the second-stage position correction network to obtain a predicted position;
[0131] A fourth training submodule is used to calculate a state update difference and an information difference based on the initial predicted position and the predicted position, calculate a loss function value through an EIoU loss function and the focus loss function, and calculate a Kalman gain loss through a square error loss function;
[0132] a fifth training submodule, configured to update the network parameters of the first-stage object detection network based on the state update difference, the information difference, and the loss function value, and to update the coefficients of the Kalman gain matrix of the second-stage position correction network based on the Kalman gain loss;
[0133] The sixth training submodule is used to obtain a trained object detection network and a Kalman gain matrix after the training end condition is met.
[0134] In an optional embodiment, the fourth training submodule includes:
[0135] a state update difference unit, configured to subtract the initial predicted position from the predicted position to obtain the state update difference;
[0136] An information difference unit is configured to subtract the measurement value of the initial predicted position from the measurement value of the predicted position to obtain the information difference value.
[0137] An embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When executed by the processor, the two-stage visual inspection method for micro-nano operations described in the embodiment of the present application is implemented.
[0138] An embodiment of the present application also provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the two-stage visual inspection method for micro-nano operations described in the embodiment of the present application is implemented.
[0139] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0140] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices, electronic devices, and storage media according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0141] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0142] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce computer-implemented processing, so that the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0143] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.
[0144] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.
[0145] The above is a detailed introduction to the two-stage visual inspection method, system, device and equipment for micro-nano operations provided by this application. Specific examples are used in this article to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method of this application and its core idea; at the same time, for general technical personnel in this field, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on this application.
Claims
1. A two-stage visual inspection method for micro-nano operations, characterized in that: The method comprises: Input the image to be detected into the first-stage object detection network for position detection to obtain the initial position of the micro-object; The second-stage position correction network is based on the Kalman filter algorithm and uses the position offset of the micro-object to correct the initial position of the micro-object to obtain the real-time position of the micro-object, wherein the position offset of the micro-object is determined according to the pre-trained Kalman gain matrix.
2. The method according to claim 1, characterized in that The first-stage object detection network is constructed in the following way: The first-stage object detection network is designed using the same CSP feature extraction framework as the YOLOX model as the backbone network; The CIoU loss in the YOLOX model is replaced by the EIoU loss function, and the BCE loss function in the YOLOX model is replaced by the focal loss function.
3. The method according to claim 1, characterized in that The second-stage position correction network is based on the Kalman filter algorithm and uses the position offset of the micro-object to correct the initial position of the micro-object to obtain the real-time position of the micro-object, including: Construct state space equations based on Kalman filter algorithm; The initial position of the micro-object is corrected using the state space equation and the position offset of the micro-object to obtain the real-time position of the micro-object.
4. The method according to claim 3, characterized in that The state space equation is constructed based on the Kalman filter algorithm, including: Determine a priori covariance, where the priori covariance is the covariance corresponding to the initial position of the micro-object; Determine a posterior covariance, where the posterior covariance is the covariance corresponding to the real-time position of the micro-object; A real-time position prediction equation is determined, wherein the real-time position prediction equation is used to predict the real-time position of the micro-object according to the initial position of the micro-object and a Kalman gain matrix, wherein the Kalman gain matrix represents the position offset of the micro-object.
5. The method according to claim 1, characterized in that The first-stage object detection network and the second-stage position correction network constitute a complete object detection network, and the object detection network is trained in the following manner: Acquire a training data set, wherein the training data in the training data set includes microscope images under different media and different illumination conditions; Inputting the training data set into the object detection network for learning; For each of the training data, the first-stage object detection network performs position detection on the training data to obtain an initial predicted position, and the second-stage position correction network corrects the initial predicted position to obtain a predicted position; Calculate the state update difference and the information difference according to the initial predicted position and the predicted position, calculate the loss function value by the EIoU loss function and the focus loss function, and calculate the Kalman gain loss by the square error loss function; Based on the state update difference, the information difference and the loss function value, the network parameters of the first-stage object detection network are updated, and based on the Kalman gain loss, the coefficients of the Kalman gain matrix of the second-stage position correction network are updated; After the training end conditions are met, the trained object detection network and Kalman gain matrix are obtained.
6. The method according to claim 5, characterized in that The calculating the state update difference and the information difference according to the initial predicted position and the predicted position includes: The predicted position is subtracted from the initial predicted position to obtain the state update difference, which is expressed as: in, Indicates the state update difference, represents the predicted position, represents the initial predicted position; The measured value of the predicted position is subtracted from the measured value of the initial predicted position to obtain the information difference value, which is expressed as: Among them, Δy t represents the information difference, y t|t represents the measured value at the predicted location, The measured values representing the initial prediction locations.
7. The method according to claim 2, characterized in that The EIoU loss function L EIoU It is expressed as: Among them, L IoU represents the overlap loss, L dis represents the center distance loss, L asp represents the width and height loss, c represents the diagonal length of the minimum bounding box that encloses the two boxes, and w c and h c The width and height of the minimum bounding box covering the two boxes are b and b respectively. gt Represent the predicted frame and the actual frame respectively, ρ represents the Euclidean distance, w gt and h gt Represent the width and height of the true value bounding box respectively; The focal loss function FL(p t ) is expressed as: FL(p t )=-α(1-p t ) γ log(p t ) Among them, p t represents the probability that the sample belongs to the true category, α represents the weight factor between 0 and 1, γ represents the focusing parameter, and y represents the true label.
8. A two-stage visual inspection system for micro-nano operations, characterized in that: The system is used to perform the two-stage visual inspection method for micro-nano operations according to any one of claims 1 to 7, comprising: An electric workbench, wherein the electric workbench is used to place micro-objects; A joystick, wherein the joystick controls the electric worktable to move so that the micro-object appears in the field of view of the microscope; A microscope and a camera, wherein the microscope and the camera are used to measure the position of the micro-object and obtain a picture to be detected corresponding to the micro-object; A computer is used to display the image to be detected and perform two-stage visual detection on the image to be detected to obtain the real-time position of the micro-object.
9. A two-stage visual inspection device for micro-nano operations, characterized in that: The device comprises: The detection module is used to input the image to be detected into the first-stage object detection network for position detection to obtain the initial position of the micro-object; The correction module is used for the second stage position correction network based on the Kalman filter algorithm, and uses the position offset of the micro-object to correct the initial position of the micro-object to obtain the real-time position of the micro-object, wherein the position offset of the micro-object is determined according to the pre-trained Kalman gain matrix.
10. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor is executed, the two-stage visual inspection method for micro-nano operations as claimed in any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
A training method for ship detection by using a convolutional neural network and a ship detection method thereof
CN109583412A
Spherical rotor attitude detection method based on Kalman filtering FDSST algorithm
CN115937266A
Two-stage visual detection method, system, device and equipment for micro-nano operation
CN116543049A
Air curtain type spray gun for preventing scattering and method for painting using the same
KR102353471B1
Cited By
Method for automatically capturing biological particles through optical tweezers based on machine vision
CN120451258A
