Coordinate calibration methods, apparatus, microscopy systems and media for microscopic scanning

By using an automated microscopic scanning method, the pixel coordinates of calibration points in the calibration image are determined and the mechanical coordinates are obtained, which solves the problems of low efficiency and insufficient accuracy of manual calibration and achieves efficient and accurate coordinate calibration.

CN117537710BActive Publication Date: 2026-11-17QINGDAO NOVELBEAM TECH
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202311490957.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-11-09
Publication Date
2026-11-17
Estimated Expiration
2043-11-09

AI Technical Summary

Technical Problem

The coordinate calibration of existing microscopic scanning systems mainly relies on manual operation, which is inefficient, difficult to meet the accuracy requirements, and prone to introducing human error.

Method used

By using an automated microscopic scanning method, the pixel coordinates of calibration points in the calibration image are determined. A displacement device is used to bring the calibration object closer to the center of the field of view of the microscope camera to obtain mechanical coordinates. Multiple calibration points are iteratively calibrated, and coordinate transformation relationships are established to achieve fully automated calibration.

Benefits of technology

It improves calibration efficiency, avoids human error, enhances calibration accuracy, and shortens calibration time, and is only related to camera resolution.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117537710B_ABST
    Figure CN117537710B_ABST
Patent Text Reader

Abstract

This disclosure provides a coordinate calibration method, apparatus, system, and medium for microscopic scanning. The method first calculates the pixel coordinates of calibration points using calibration images acquired by a navigation camera (or a first camera). Then, it controls a displacement system to move, causing the center of the microscope camera's field of view to sequentially coincide with each calibration point on the calibration object. Whenever the center of the field of view coincides with a calibration point, the mechanical coordinates of the displacement device are recorded. Finally, a coordinate calibration matrix is ​​obtained based on the pixel coordinates and mechanical coordinates of multiple calibration points, revealing the transformation relationship between the center of the microscope's field of view and the position of the displacement device. This achieves a fully automated coordinate calibration process, eliminating the need for manual intervention and professional training, thus improving calibration efficiency and eliminating errors caused by manual calibration, thereby enhancing calibration accuracy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to the field of microscopy scanning technology, and in particular to coordinate calibration methods, apparatus, microscopy scanning systems and media for microscopy scanning. Background Technology

[0002] Microscopic scanning is primarily used for capturing images of minute sizes; the field of view of a microscope camera is typically measured in micrometers. Microscopic scanning systems require coordinate calibration to ensure the captured image content matches the user's expectations. Currently, however, coordinate calibration is mainly performed manually. This involves manually determining the position of calibration points in the captured calibration image, and visually locating these points to align the microscope camera with them, ultimately achieving calibration between pixel positions and mechanical positions.

[0003] Because the field of view in a microscope is very small, the accuracy requirements for calibration are high. Manual calibration requires the human eye to find the calibration point under the microscope, which is time-consuming, requires repeated calibration, and is inefficient. Furthermore, the calibration process demands highly skilled operators who have undergone professional training, and is prone to introducing human error, making it difficult to meet the required accuracy. Summary of the Invention

[0004] To address at least one of the aforementioned technical problems, this disclosure provides a coordinate calibration method, apparatus, microscopy system, and medium for microscopic scanning.

[0005] The first aspect of this disclosure proposes a coordinate calibration method for microscopic scanning, comprising: determining the first pixel coordinates of calibration points of a plurality of calibration patterns in a calibration image included in a first captured image in the first captured image; a microscopic imaging step, wherein a second captured image is obtained by imaging a calibration object located at a current microscopic imaging position, the second captured image being an image captured by a microscopic camera, the microscopic imaging position being such that the second captured image includes at least a portion of the pattern content of the current target calibration pattern; determining the second pixel coordinates of the calibration points of the current target calibration pattern in the coordinate system of the second captured image based on the at least a portion of the pattern content; and driving a displacement device to move the calibration object in water. The calibration point moves towards the center of the field of view of the microscope camera in a horizontal direction until the current calibration point coincides with the center of the field of view, and obtains the mechanical coordinates of the displacement device when they coincide; in the calibration point iteration step, the displacement device is driven to move the calibration object to the microscope imaging position of the next calibration pattern, and the microscope imaging position of the next calibration pattern is used as the current microscope imaging position and substituted into the microscope imaging step to obtain the mechanical coordinates of the displacement device corresponding to the calibration point of the next calibration pattern, until the mechanical coordinates of the displacement device of the calibration points of the multiple calibration patterns are obtained; and the coordinate transformation relationship is obtained based on the pixel coordinates of the calibration points of the multiple calibration patterns and the corresponding mechanical coordinates of the displacement device.

[0006] According to one embodiment of this disclosure, the plurality of calibration patterns are at least five calibration patterns, and the number of first pixel coordinates of the determined calibration points is at least five.

[0007] According to one embodiment of this disclosure, each calibration pattern includes only one calibration point.

[0008] According to one embodiment of this disclosure, each of the calibration patterns has the same shape, and each of the calibration patterns is divided into multiple sectors, the multiple sectors having the same center and being distributed circumferentially around the center.

[0009] According to one embodiment of this disclosure, the plurality of calibration patterns includes a first calibration pattern and a plurality of second calibration patterns, wherein the size of the first calibration pattern is larger than the size of the second calibration patterns, and the current target calibration pattern included in the first acquired second captured image is the first calibration pattern.

[0010] According to one embodiment of this disclosure, determining the first pixel coordinates of calibration points of a plurality of calibration patterns in a calibration image included in a first captured image in the first captured image includes: capturing a calibration object located at a first shooting position to obtain a first captured image, the first captured image including a calibration image on the calibration object, the calibration image including a plurality of calibration patterns; and determining the first pixel coordinates of calibration points in the plurality of calibration patterns in the first captured image.

[0011] According to one embodiment of this disclosure, taking a picture of a calibration object located at a first shooting position to obtain a first shooting image includes: driving a displacement device to move the calibration object to the first shooting position, the first shooting position corresponding to a first camera; and controlling the first camera to take a picture of the calibration object to obtain the first shooting image.

[0012] According to one embodiment of this disclosure, the driving displacement device moves a calibration object to a first shooting position, including: the driving displacement device moves a stage to an initial preset position, the stage being provided with a calibration object, and when the stage is at the initial preset position, the calibration object is at the first shooting position.

[0013] According to one embodiment of this disclosure, the calibrator located at the first shooting position is directly facing the first camera.

[0014] According to one embodiment of this disclosure, the calibration point is located at the center of the calibration pattern. Determining the first pixel coordinates of the calibration point in the first captured image in the plurality of calibration patterns includes: performing threshold segmentation on the first captured image to obtain a plurality of sub-regions; for each sub-region, identifying a calibration pattern region from the sub-region based on the shape information of the calibration pattern; and determining the center coordinates of the calibration pattern region and using them as the first pixel coordinates of the calibration point in the calibration pattern in the first captured image.

[0015] According to one embodiment of this disclosure, taking a second image of a calibration object located at the current microscopic imaging position includes: driving a displacement device to move the calibration object to the current microscopic imaging position; and controlling a microscope camera to take an image of the calibration object to obtain the second image.

[0016] According to one embodiment of this disclosure, controlling a microscope camera to capture images of the calibration object to obtain a second image includes: a focusing and capturing step, controlling the microscope camera to capture images of the calibration object at multiple different focal lengths to obtain multiple candidate images; a sharpness evaluation step, evaluating the sharpness of each of the multiple candidate images to obtain corresponding sharpness values; and an image determination step, determining the candidate image with the optimal sharpness value as the second image.

[0017] According to one embodiment of this disclosure, controlling a microscope camera to photograph the calibration object at multiple different focal lengths includes: controlling the microscope camera to move multiple times in the depth of field direction with a preset step size, and controlling the microscope camera to photograph the calibration object after each movement.

[0018] According to one embodiment of this disclosure, controlling a microscope camera to capture images of the calibration object to obtain a second image includes: performing a focusing and capturing step with a first step length as a preset step length to obtain a plurality of first candidate images; performing a sharpness evaluation step on the plurality of first candidate images to obtain a plurality of first sharpness values; determining a first movement position corresponding to the optimal first sharpness value; within a movement range including the first movement position, performing the focusing and capturing step with a second step length as a preset step length to obtain a plurality of second candidate images, wherein the second step length is smaller than the first step length; performing the sharpness evaluation step on the plurality of second candidate images to obtain a plurality of second sharpness values; and determining a second image from the plurality of second sharpness values ​​through an image determination step.

[0019] According to one embodiment of this disclosure, the sharpness evaluation step includes: taking the candidate captured image as the image to be evaluated, converting the image to be evaluated to grayscale to obtain a grayscale image; determining the image gradient and variance of the grayscale image; and performing a normalization operation based on the variance and the image gradient to obtain a sharpness value.

[0020] According to one embodiment of this disclosure, determining the second pixel coordinates of the calibration point of the current target calibration pattern in the coordinate system of the second captured image based on the at least part of the pattern content includes: performing line fitting on the at least part of the pattern content to obtain a plurality of line segments, wherein the at least part of the pattern content includes at least a portion of a plurality of fan-shaped regions, and the line segments are the boundaries of the fan-shaped regions; and determining the second pixel coordinates of the calibration point of the current target calibration pattern in the coordinate system of the second captured image based on the plurality of line segments.

[0021] According to one embodiment of this disclosure, determining the second pixel coordinates of the calibration point of the current target calibration pattern in the coordinate system of the second captured image based on the plurality of straight line segments includes: taking two of the straight line segments as a group of straight lines, determining the intersection point of each group of straight lines in different groups of straight lines to obtain a plurality of intersection points; and calculating the average coordinate value of the plurality of intersection points, and using the obtained average coordinate value as the second pixel coordinates of the calibration point of the current target calibration pattern, wherein the coordinates of the intersection points are coordinates in the coordinate system of the second captured image.

[0022] According to one embodiment of this disclosure, the driving displacement device moves the calibration object horizontally toward the center of the field of view of the microscope camera until the current calibration point coincides with the center of the field of view, including: a first moving step, in which the driving displacement device moves the calibration object toward the center of the field of view of the microscope camera; and a first iteration step, in which, each time the first moving step is completed, the new microscope imaging position formed by the movement is substituted into the microscope imaging step as the current microscope imaging position to obtain a new second pixel coordinate, until the new second pixel coordinate coincides with the center of the field of view.

[0023] According to one embodiment of this disclosure, in the first moving step, when moving towards the center of the field of view of the microscope camera, the direction of movement is obtained based on the relative positional relationship between the second pixel coordinates and the center of the field of view in the field of view coordinate system, and the distance of movement is obtained based on the current pixel distance between the second pixel coordinates and the center of the field of view.

[0024] According to one embodiment of this disclosure, the distance moved is obtained based on the estimated physical distance corresponding to the current pixel distance.

[0025] A second aspect of this disclosure provides a coordinate calibration device for microscopic scanning, comprising: a first pixel coordinate determining module, configured to determine the first pixel coordinates of calibration points of a plurality of calibration patterns in a calibration image included in a first captured image, in the first captured image; a microscopic imaging module, configured to perform a microscopic imaging step, the microscopic imaging step comprising: capturing a calibration object located at a current microscopic imaging position to obtain a second captured image, the second captured image being an image captured by a microscopic camera, the microscopic imaging position being such that the second captured image includes at least a portion of the pattern content in the current target calibration pattern; a second pixel coordinate determining module, configured to determine the second pixel coordinates of the calibration points of the current target calibration pattern in the coordinate system of the second captured image based on the at least a portion of the pattern content; and a mechanical coordinate acquisition module, configured to drive... The displacement device moves the calibration object horizontally toward the center of the field of view of the microscope camera until the current calibration point coincides with the center of the field of view, and obtains the mechanical coordinates of the displacement device when they coincide; the calibration point iteration module is used to execute the calibration point iteration step, which includes: driving the displacement device to move the calibration object to the microscope imaging position of the next calibration pattern, and substituting the microscope imaging position of the next calibration pattern as the current microscope imaging position into the microscope imaging step to obtain the mechanical coordinates of the displacement device corresponding to the calibration point of the next calibration pattern, until the mechanical coordinates of the displacement device of the calibration points of the multiple calibration patterns are obtained; and the coordinate relationship determination module is used to obtain the coordinate transformation relationship based on the pixel coordinates of the calibration points of the multiple calibration patterns and the corresponding mechanical coordinates of the displacement device.

[0026] A third aspect of this disclosure provides a microscopic scanning system, comprising: a coordinate calibration device for microscopic scanning as described in claim 21; an imaging system including a microscopic camera for capturing images of an object to be photographed under the control of the coordinate calibration device, thereby obtaining an image; and a displacement device for moving under the control of the coordinate calibration device to synchronously move the object to be photographed.

[0027] The fourth aspect of this disclosure provides a readable storage medium storing executable instructions, which, when executed by a processor, are used to implement the coordinate calibration method for microscopic scanning as described in any of the above embodiments. Attached Figure Description

[0028] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0029] Figure 1This is a schematic flowchart of a coordinate calibration method for microscopic scanning according to one embodiment of the present disclosure.

[0030] Figure 2 This is a schematic diagram of a first captured image according to one embodiment of the present disclosure.

[0031] Figure 3 This is a schematic diagram of a calibration image according to one embodiment of the present disclosure.

[0032] Figure 4 This is a schematic diagram of the process for determining the coordinates of a first pixel according to one embodiment of the present disclosure.

[0033] Figure 5 This is a schematic diagram of the process of capturing a first captured image according to one embodiment of the present disclosure.

[0034] Figure 6 This is a schematic flowchart illustrating the determination of the coordinates of the first pixel according to another embodiment of the present disclosure.

[0035] Figure 7 This is a schematic diagram of the process of capturing a second image according to one embodiment of the present disclosure.

[0036] Figure 8 This is a schematic diagram of a second captured image according to one embodiment of the present disclosure.

[0037] Figure 9 This is a schematic diagram of the process of capturing a second image according to another embodiment of the present disclosure.

[0038] Figure 10 This is a schematic diagram of the process of capturing a second image according to yet another embodiment of the present disclosure.

[0039] Figure 11 This is a schematic flowchart illustrating the process of determining the coordinates of a second pixel according to one embodiment of the present disclosure.

[0040] Figure 12 Yes Figure 8 A schematic diagram obtained after linear fitting.

[0041] Figure 13 This is a schematic diagram of the process for determining the coordinates of a second pixel according to another embodiment of the present disclosure.

[0042] Figure 14 This is a schematic diagram of a process for aligning the current calibration point with the center of the field of view according to one embodiment of the present disclosure.

[0043] Figure 15 This is a schematic diagram showing the alignment of the calibration point with the center of the field of view according to one embodiment of this disclosure.

[0044] Figure 16 This is a schematic diagram of a coordinate calibration device for microscopic scanning using a hardware implementation of a processing system according to one embodiment of the present disclosure.

[0045] Figure 17 This is a schematic diagram of a microscopic scanning system according to one embodiment of the present disclosure. Detailed Implementation

[0046] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0047] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0048] Unless otherwise stated, the exemplary implementations / embodiments shown are to be understood as providing exemplary features of various details that provide ways in which the technical concepts of this disclosure can be implemented in practice. Therefore, unless otherwise stated, the features of various implementations / embodiments may be additionally combined, separated, interchanged and / or rearranged without departing from the technical concepts of this disclosure.

[0049] The terminology used herein is for the purpose of describing particular embodiments and is not restrictive. As used herein, unless the context clearly indicates otherwise, the singular forms “a” and “the” are intended to include the plural forms as well. Furthermore, when the terms “comprising” and / or “including” and variations thereof are used in this specification, it indicates the presence of the stated features, integrals, steps, operations, parts, components, and / or groups thereof, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, parts, components, and / or groups thereof. It should also be noted that, as used herein, the terms “substantially,” “about,” and other similar terms are used as approximate terms rather than as terms of degree, thus explaining the inherent biases in measurements, calculated values, and / or provided values ​​that would be recognized by one of ordinary skill in the art.

[0050] The following description, with reference to the accompanying drawings, outlines a method, apparatus, system, and readable storage medium for coordinate calibration in microscopy.

[0051] Figure 1 This is a schematic flowchart of a coordinate calibration method for microscopic scanning according to one embodiment of the present disclosure. (See attached diagram) Figure 1The coordinate calibration method M10 for microscopic scanning in this embodiment may include the following steps: S100, S200, S300, S400, S500 and S600.

[0052] S100, determine the first pixel coordinates of the calibration points of multiple calibration patterns in the calibration image contained in the first captured image in the first captured image.

[0053] Microscopic imaging step S200: The calibration object located at the current microscopic imaging position is photographed to obtain a second image. The second image is an image captured by a microscopic camera. The microscopic imaging position is such that the second image includes at least a portion of the pattern content in the current target calibration pattern.

[0054] S300, based on at least part of the pattern content mentioned above, determine the second pixel coordinates of the calibration point of the current target calibration pattern in the coordinate system of the second captured image.

[0055] S400: The drive displacement device moves the calibration object horizontally toward the center of the field of view of the microscope until the current calibration point coincides with the center of the field of view, and obtains the mechanical coordinates of the displacement device when they coincide.

[0056] Calibration point iteration step S500: Drive the displacement device to move the calibration object to the microscopic imaging position of the next calibration pattern, and substitute the microscopic imaging position of the next calibration pattern as the current microscopic imaging position into the microscopic imaging step to obtain the displacement device mechanical coordinates corresponding to the calibration point of the next calibration pattern, until the displacement device mechanical coordinates of the calibration points of multiple calibration patterns are obtained.

[0057] S600 obtains the coordinate transformation relationship based on the pixel coordinates of the calibration points of multiple calibration patterns and the mechanical coordinates of the corresponding displacement devices.

[0058] According to the microscopic scanning coordinate calibration method proposed in the embodiments of this disclosure, the pixel coordinates of multiple calibration points on the calibration object are extracted from the first captured image. The mechanical coordinates corresponding to each calibration point are obtained from multiple captured second images. Coordinate calibration is performed using the pixel coordinates of the calibration points and the mechanical coordinates of the displacement device when the microscope camera is aligned with the calibration points. The transformation relationship between the center of the microscopic field of view and the moving position of the displacement device is obtained. This realizes a fully automated coordinate calibration process, which does not require manual intervention or professional training of personnel, improves calibration efficiency, and eliminates errors caused by manual calibration, thereby improving calibration accuracy.

[0059] The first image is obtained by photographing the calibration object. The calibration image is set on the calibration object, for example, it can be printed or pasted on the calibration object. The calibration object can be a calibration plate. The calibration image includes multiple calibration patterns, and each calibration pattern can have a calibration point, which can be located at or near the center of the calibration pattern. The position of the calibration point in the entire first image is the first pixel coordinate. The coordinate calibration of the microscopic scanning system is achieved by identifying and analyzing the position of the calibration patterns in step S100.

[0060] The first image can be obtained by photographing the calibration object using a first camera, also known as a navigation camera. The calibration object is pre-placed on a stage before the photographing begins. During the photographing and calibration process, the navigation camera remains stationary; changes in the relative position between the calibration object and the navigation camera are achieved by controlling a displacement device. When coordinate calibration begins, the displacement device can be moved to a preset position A, thereby bringing the calibration object into the navigation camera's field of view. Position A ensures that the first image obtained after the navigation camera photographs the calibration object includes the calibration image on the calibration object, thus obtaining the various calibration patterns on the calibration image.

[0061] After obtaining the first captured image, the calibration patterns in the calibration image can be identified and their positions analyzed using image processing algorithms to determine the coordinates of the calibration points of N calibration patterns, i.e., the pixel coordinates of N calibration points. These pixel coordinates of calibration points will be used as one of the data sets for subsequent determination of coordinate transformation relationships.

[0062] The other data used in determining the coordinate transformation relationship is the mechanical coordinates of the displacement device. The mechanical coordinates of the displacement device refer to the mechanical coordinates of the displacement device in its own motion coordinate system when the field of view of the microscope camera is aligned with the calibration point of the calibration pattern (i.e., when the center of the field of view coincides with the calibration point). The mechanical coordinates of the displacement device are obtained through the microscope imaging step S200 to the calibration point iteration step S500. Determining the coordinate transformation relationship requires multiple sets of coordinate data. Each set of coordinate data includes the pixel coordinates of a calibration point and the corresponding mechanical coordinates of the displacement device at that calibration point. Therefore, steps S200 to S500 are used to process each calibration pattern sequentially to obtain the corresponding N mechanical coordinates of the displacement device. Then, step S600 establishes the coordinate transformation relationship between the N pixel coordinates of the calibration points and the N mechanical coordinates of the displacement device, obtaining the coordinate calibration matrix. The coordinate calibration matrix allows the calculation of the mechanical coordinates corresponding to any pixel coordinate, enabling the microscope camera to capture images at the accurately aligned desired coordinate points, resulting in an ideal image.

[0063] In the microscopic imaging step S200, the second image is obtained by photographing the calibration object using a microscope camera. The magnification of the microscope camera is lower than that of the navigation camera, specifically only about one percent of the navigation camera's magnification. The microscope camera remains stationary during the imaging and calibration process; changes in the relative position between the calibration object and the microscope camera are achieved by controlling a displacement device. After obtaining the first image, the displacement device can be moved to another preset position B, thereby bringing the calibration object into the microscope camera's field of view. The preset position B ensures that the second image obtained after the microscope camera photographs the calibration object contains at least a portion of the pattern content of one of the calibration patterns on the calibration object.

[0064] Since it is necessary to determine the mechanical coordinates of the displacement device corresponding to the calibration points of multiple calibration patterns, the microscope camera will take pictures of multiple calibration patterns in sequence. Among them, the calibration pattern currently captured by the microscope camera is the current target calibration pattern.

[0065] The pattern of the calibration pattern is directional towards the calibration point when observed; for example, the pattern content may include line segments pointing towards the calibration point. Regardless of whether the calibration point of the current target calibration pattern (hereinafter referred to as the current calibration point) is located within the field of view of the microscope camera, that is, regardless of whether the current calibration point can be observed by the microscope camera, the pixel coordinates of the current calibration point in the coordinate system of the second captured image (i.e., the field of view coordinate system of the microscope camera) can be determined by analyzing the pattern content within the field of view; that is, the second pixel coordinates. It can be understood that the coordinate values ​​of the second pixel coordinates are obtained with reference to the origin of the second captured image, which can be the coordinate point of the upper left corner of the second captured image.

[0066] It should be noted that both the first pixel coordinate and the second pixel coordinate are coordinates of the calibration point, but the coordinate systems of these two coordinates are different. The first pixel coordinate is the position coordinate in the image captured by the navigation camera, while the second pixel coordinate is the position coordinate in the field of view coordinate system of the microscope camera. The origins of the two are different, and the ratio of the magnification of the navigation camera to that of the microscope camera may be on the order of hundreds of times. Therefore, when obtaining the mechanical coordinates of the displacement device, determining the second pixel coordinate can reduce the error generated by the pixel coordinate during the calculation compared to using the first pixel coordinate.

[0067] The second pixel coordinates and the center point of the field of view share the same coordinate system, with their origins both being the origin of the current field of view image. Therefore, after obtaining the second pixel coordinates, the pixel distance between the center point of the field of view and the second pixel coordinates can be calculated. The displacement device is controlled to move according to the pixel distance until the two coincide. At this point, the microscope camera is aligned with the calibration point of the current calibration pattern, and the center of the field of view is the current calibration point. The mechanical coordinates of the displacement device are then the mechanical coordinates corresponding to the current calibration point. At this point, the first set of coordinate data is obtained (including the pixel coordinates of the first calibration point and the corresponding mechanical coordinates of the displacement device).

[0068] Then, the pixel coordinates of the calibration points for other calibration patterns are determined through the calibration point iteration step S500. Specifically, the moving device is first moved from the mechanical coordinates corresponding to the first calibration point to another preset position C. The physical distance between the calibration points of each calibration pattern is known, so the preset position C can be set by the physical distance between the first calibration point and the next calibration point. When the moving device moves to the preset position C, the stage moves accordingly, and at least part of the pattern content of the next calibration pattern (the new current calibration pattern) in the calibration image of the calibration object appears in the field of view of the microscope camera. At this time, the microscope imaging step S200 is executed to obtain the first second imaging image for the second calibration pattern.

[0069] Then, the second pixel coordinates of the calibration point of the current calibration pattern can be determined. The displacement device is then moved so that the center of the microscope's field of view coincides with the current calibration point, thus obtaining the mechanical coordinates of the displacement device for the current calibration point. This yields the coordinate data corresponding to the calibration point of the second calibration pattern, i.e., the second set of coordinate data. The process of capturing images of other calibration patterns and identifying the corresponding second pixel coordinates of calibration points, as well as determining the mechanical coordinates of the displacement device, is repeated sequentially. After obtaining N sets of coordinate data, the coordinate calibration matrix can be generated.

[0070] The coordinate calibration method M10 of this embodiment reduces the overall calibration time to about 0.5 hours and saves a lot of labor costs. The accuracy of the calibration process is only related to the camera resolution, avoiding the introduction of human error and improving the calibration accuracy. It has the characteristics of short calibration time, high calibration accuracy and reduced subjective error of manual calibration.

[0071] Figure 2 This is a schematic diagram of a first captured image according to one embodiment of the present disclosure. Figure 3 This is a schematic diagram of a calibration image according to one embodiment of the present disclosure. (See also...) Figure 2 and Figure 3The calibration image can contain at least five calibration patterns. The number of first pixel coordinates of each calibration point must be at least five, i.e., N ≥ 5. Correspondingly, at least five sets of coordinate data are required. The more sets of coordinate data obtained, the more accurate the final coordinate calibration matrix, which is more beneficial for the precise positioning of the microscope camera during actual microscopic scanning.

[0072] Figure 2 and Figure 3 The calibration object shown has a rectangular surface facing the navigation camera, which is entirely set as the calibration image. The calibration image contains five circular calibration patterns. It is understood that during calibration, it is not necessary to acquire coordinate data sets for all calibration patterns in the calibration image. For example, if the calibration image contains 10 calibration patterns, coordinate data sets can be acquired for only 5 patterns, or all 10 patterns can be acquired.

[0073] Each calibration pattern may include only one calibration point. The shape of each calibration pattern may be identical. Each calibration pattern may be divided into multiple sectors, the centers of which may be the same and distributed circumferentially around the center. For example, the calibration pattern may be a pattern formed by multiple circles with the same center but different radii. The boundary shape of the calibration pattern may be elliptical, rectangular, or other shapes. This embodiment adopts... Figure 2 and Figure 3 The calibration patterns shown are all circular, and each circular pattern has a calibration point, such that... Figure 2 and Figure 3 It contains 5 calibration points. The calibration point can be the center of the circular pattern.

[0074] The calibration pattern can be a Siemens star pattern, which consists of multiple sectors evenly distributed around the center of the pattern. The dimensions of each calibration pattern can be the same or different, and the radii of the sectors within the same calibration pattern are identical. Using a Siemens star pattern, the sector boundaries radiate outwards from the center, providing stronger directional guidance for the center's position compared to concentric circle patterns. When the center is used as the calibration point, this directional guidance allows for accurate determination of the center's relative position within the current field of view, improving the accuracy of the calculated second pixel coordinates. Furthermore, as the calibration point moves towards the center of the field of view, it facilitates the correct direction of movement of the displacement device.

[0075] The calibration images may include a first calibration pattern and multiple second calibration patterns. The first calibration pattern is larger than the second calibration patterns, and the current target calibration pattern included in the first acquired second image may be the first calibration pattern. For example, among the five calibration patterns, the larger calibration pattern is the first calibration pattern, and the four calibration patterns surrounding the first calibration pattern are the second calibration patterns. The first and second calibration patterns may use Siemens star patterns of different sizes. For example, the diameter of the first calibration pattern may be set to 8 mm. Due to the small assembly error between the navigation camera and the microscope camera, for example, less than 2 mm, the microscope camera's field of view will fall within the pattern range of the first calibration pattern when the calibration object moves to the microscope imaging position.

[0076] When the first image is captured and the displacement device moves to the preset position B for the first time, the preset position B corresponds to the first calibration pattern. In other words, the first calibration pattern that appears in the field of view of the microscope camera is the first calibration pattern. This is to ensure that the calibration pattern will definitely appear in the field of view of the microscope camera by using the larger first calibration pattern, and to avoid the situation where no calibration pattern appears in the field of view of the microscope camera after the movement due to errors or other reasons.

[0077] Figure 4 This is a schematic flowchart illustrating the process of determining the coordinates of a first pixel according to one embodiment of this disclosure. (See also...) Figure 4 Step S100 may include steps S110 and S120.

[0078] S110, a first image is obtained by taking a picture of the calibration object located at the first shooting position. The first image includes a calibration image on the calibration object, and the calibration image includes multiple calibration patterns.

[0079] S120, determine the first pixel coordinates of the calibration point in the first captured image among multiple calibration patterns.

[0080] The shooting position is the location of the calibration object, and the preset positions A, B, and C are the positions of the displacement device; there is a corresponding relationship between the two. Preset position A corresponds to the first shooting position. If the displacement device is initially located at preset position A, the navigation camera can start shooting the calibration object without moving it. If the displacement device is initially located in another position, it needs to be moved to preset position A first to bring the calibration object into the navigation camera's field of view before shooting.

[0081] by Figure 2 and Figure 3For example, the calibration image is the square portion of the first captured image, and the five circles within it represent five calibration patterns. The style of the calibration patterns and the positions of the calibration points are known. The first pixel coordinates of the calibration points in the first captured image can be obtained by calculating the relative positional relationship between the calibration points and the origin, with the upper left corner of the first captured image as the origin. For example, if the size of the first captured image is l1*h1, and the size of the calibration image is l2*h2, l2... <l1,l2=h2,h2

[0082] Figure 5 This is a schematic diagram illustrating the process of capturing a first image according to one embodiment of this disclosure. (See also...) Figure 5 Step S110 may include steps S111 and S112.

[0083] S111, the driving displacement device moves the calibration object to the first shooting position, which corresponds to the first camera.

[0084] S112, control the first camera to take a picture of the calibration object and obtain the first captured image.

[0085] For example, step S111 may include: driving the displacement device to move the stage to an initial preset position, wherein a calibration object is provided on the stage, and when the stage is at the initial preset position, the calibration object is at a first shooting position. The calibration object at the first shooting position may be directly facing the first camera.

[0086] The calibration object can be detachably fixed to the platform, or the calibration object can be placed directly on the platform, and the movement speed of the displacement device can be slow to avoid relative positional changes between the calibration object and the platform due to movement. By sending a movement command to the displacement device, the displacement device moves to the initial preset position (i.e., preset position A). At this time, the calibration object is correspondingly located at the first shooting position, and the calibration image of the calibration object is facing the lens of the navigation camera. The first shooting image includes the overall calibration image of the side of the calibration object facing the navigation camera.

[0087] Figure 6 This is a flowchart illustrating the determination of the coordinates of a first pixel according to another embodiment of this disclosure. The calibration point can be located at the center of the calibration pattern. When the calibration point is the center point of the calibration pattern, refer to... Figure 6 Step S120 may include the following steps: S121, S122 and S123.

[0088] ​S121, Threshold segmentation is performed on the first captured image to obtain multiple sub-regions.

[0089] S122, For each sub-region, identify the calibration pattern region from the sub-region based on the shape information of the calibration pattern.

[0090] S122, determine the center coordinates of the calibration pattern area and use them as the first pixel coordinates of the calibration point in the calibration pattern in the first captured image.

[0091] The pixel coordinates of the first captured image are extracted using a feature extraction algorithm to facilitate the subsequent acquisition of coordinate data sets. Figure 2 For example, through the analysis of Figure 2 The calibration image shown is segmented using a threshold to obtain five sub-regions, each containing a calibration pattern. Since the calibration pattern is circular, and the roundness information of the circles is known, as well as the size relationship of the five circles, five circular regions can be identified from the five sub-regions, thus obtaining the calibration pattern regions. Outliers within the calibration pattern regions are then removed. The positions of the circle centers are determined from the calibration pattern regions, thereby obtaining the pixel coordinates of the calibration points in the calibration image.

[0092] Figure 7 This is a schematic diagram illustrating the process of capturing a second image according to one embodiment of this disclosure. (See also...) Figure 7 The microscopic imaging step S200 may include the following steps S210 and S220.

[0093] S210, the drive displacement device moves the calibration object to the current photomicrograph position.

[0094] S220 controls the microscope camera to take a picture of the calibration object to obtain a second image.

[0095] The microscopic imaging position corresponds to the microscope camera. After obtaining the first image, a movement command can be sent to the displacement device, causing the displacement device to move to a preset position B. The preset position B corresponds to the first calibration pattern to be identified, at which point the calibration object is located at the corresponding microscopic imaging position, and the current microscopic imaging position also corresponds to the first calibration pattern to be identified. Different calibration patterns to be identified correspond to different preset positions, and correspondingly, the current microscopic imaging position also changes accordingly.

[0096] Figure 8 This is a schematic diagram of a second captured image according to one embodiment of the present disclosure. (See also...) Figure 8The second image obtained after moving to the preset position B represents a portion of the first calibration pattern. At this point, the calibration point of the first calibration pattern does not appear in the second image; it is located outside the field of view of the microscope camera. It is understandable that the calibration point might initially appear in the second image. Regardless of whether the calibration point appears in the first second image of the first calibration pattern, subsequent steps will ensure that the calibration point coincides with the center of the field of view.

[0097] Figure 9 This is a schematic diagram illustrating the process of capturing a second image according to another embodiment of this disclosure. (See also...) Figure 9 Step S220 may include the following steps S10, S20 and S30.

[0098] Focusing and shooting step S10: Control the microscope camera to take pictures of the calibration object at multiple different focal lengths to obtain multiple candidate images.

[0099] Sharpness evaluation step S20: Sharpness evaluation is performed on the multiple candidate images obtained, and the corresponding sharpness values ​​are obtained.

[0100] Image determination step S30: Determine the candidate image with the best sharpness value and use it as the second image.

[0101] By adjusting the height distance between the microscope camera and the calibration object, the focal length is adjusted, and images are captured at different focal lengths to obtain images with different levels of clarity. The clarity values ​​are calculated to identify the clearest image, ensuring the best clarity for the second captured image and improving the accuracy of determining the coordinates of the second pixel in subsequent processes.

[0102] Since the microscope camera is set to remain stationary, the focal length is changed through a displacement device that can move along three axes (XYZ). This device can move in the height direction (i.e., the depth of field direction), thereby changing the height distance between the calibration object and the microscope camera, and thus adjusting the focal length.

[0103] The focusing and shooting step S10 may include: controlling the microscope camera to move multiple times in the depth of field direction with a preset step size, and controlling the microscope camera to take a picture of the calibration object after each movement. During focusing and shooting, m1 different height distances can be pre-selected, and the difference between adjacent height distances is the preset step size. The values ​​of each preset step size can be the same, for example, the same and small step size is used for each Z-axis movement. The values ​​of each preset step size can also be different, for example, starting from the lowest point, moving along the Z-axis with a larger step size, moving a certain number of times, then moving along the Z-axis with a smaller step size, moving a certain number of times, and then moving along the Z-axis with a larger step size until the highest point. The control displacement device moves along the Z-axis sequentially according to the preset step size to the corresponding height distance, and immediately controls the microscope camera to take a picture after each movement, obtaining m1 candidate images. Specifically, the displacement device is first controlled to move the stage in the Z-axis direction according to the preset step size, and the microscope camera is controlled to take a picture after each movement to obtain a candidate image. Once a candidate image is obtained, the sharpness value of the candidate image can be determined. Once all movement tasks in the Z-axis direction are completed, all candidate images are obtained, and the second image can be selected.

[0104] It is understandable that in the focusing and shooting step S10, the adjustment range of the focusing distance (i.e., the Z-axis movement step size) can be determined by the sharpness value obtained in the current sharpness evaluation step S20. That is, the movement step size is not a preset step size but can be adjusted at any time according to the candidate images. Specifically, the current calibration pattern can be photographed for the first time at the lowest point or a preset position. After obtaining the first candidate image, the sharpness value of the image is identified through the sharpness evaluation step S20, and the distance (i.e., the movement step size) between the image and the next shooting position is determined based on the sharpness value. A higher sharpness value indicates a closer distance to the position corresponding to the sharpest image, and the next movement step size is smaller; conversely, a higher sharpness value indicates a farther distance to the position corresponding to the sharpest image, and the next movement step size is larger. That is, there is a non-linear inverse relationship between the sharpness value and the Z-axis movement step size. After obtaining the next movement step size, the displacement device is controlled to move accordingly until the stopping condition is met. The stopping condition can be: the sharpness value of the candidate image obtained this time is less than the sharpness value of the candidate image obtained in the previous time.

[0105] After obtaining multiple candidate images corresponding to various focal lengths, a sharpness evaluation algorithm is used to evaluate the sharpness of these candidate images, calculating a sharpness value for each candidate image. Assuming that a higher sharpness value indicates higher image sharpness, the candidate image with the highest sharpness value is determined, and this candidate image can then be used as the second image for subsequent calculations.

[0106] The sharpness evaluation step S20 may include: taking the candidate captured image as the image to be evaluated, converting the image to grayscale to obtain a grayscale image; determining the image gradient and variance of the grayscale image; and performing a normalization operation based on the variance and image gradient to obtain a sharpness value. For example, using gradient and variance as sharpness evaluation indicators, for each candidate captured image, the image is first converted to grayscale, a filtering operator is used to perform a convolution operation on the image to extract the image edge regions, then mathematical operations are used to amplify the image edge gradient, the gradient of the entire image is calculated, then the variance of the grayscale image is calculated, and the image gradient and image grayscale variance are normalized according to weights, with the result used as the sharpness evaluation value.

[0107] Figure 10 This is a schematic diagram illustrating the process of capturing a second image according to yet another embodiment of this disclosure. See also... Figure 10 Step S220 may include the following steps: S221, S222, S223, S224, S225 and S226.

[0108] S221, the first step length is used as the preset step length to perform the focusing and shooting step S10, and multiple first candidate shooting images are obtained.

[0109] S222, perform the sharpness evaluation step S20 on multiple first candidate captured images to obtain multiple first sharpness values.

[0110] S223, determine the first movement position corresponding to the optimal first resolution value.

[0111] S224, within the movement range including the first moving position, the second step length is used as the preset step length to perform the focusing and shooting step S10, and multiple second candidate shooting images are obtained, where the second step length is smaller than the first step length.

[0112] S225, perform the sharpness evaluation step S20 on multiple second candidate captured images to obtain multiple second sharpness values.

[0113] S226, through image determination step S30, a second captured image is determined from a plurality of second sharpness values.

[0114] By dividing the second image capture process into coarse focusing and fine focusing, coarse focusing corresponds to a larger depth-of-field movement step and fine focusing corresponds to a smaller depth-of-field movement step. Coarse focusing is used to roughly determine the approximate location range of the shooting position that can capture the sharpest image, and then fine focusing is used to obtain the sharpest image. This reduces the number of times focusing and capturing candidate images, and improves the efficiency of the entire focusing process.

[0115] Assuming the focusing range is 50 micrometers, meaning the displacement device's movement range for focusing along the Z-axis is 50 micrometers, the first step size can be 5 micrometers, and the second step size can be 0.2 micrometers. Coarse focusing begins by moving 10 times in large 5-micrometer steps, stopping and taking a picture after every 5-micrometer movement, resulting in 11 first candidate images. These 11 first candidate images are then evaluated for sharpness, yielding 11 first sharpness values. If the highest value is the 6th first sharpness value, the first movement position is 25 micrometers. The fine focusing range is then selected based on this 25-micrometer height value; for example, the range can be set to [20, 30]. Fine focusing then begins by moving 50 times in small 0.2-micrometer steps from 20 micrometers to 30 micrometers, stopping and taking a picture after every 0.2-micrometer movement, resulting in 51 second candidate images. A sharpness evaluation calculation was performed on these 51 second candidate images to obtain 51 second sharpness values. The second candidate image with the highest sharpness value was selected as the second image.

[0116] Figure 11 This is a schematic flowchart illustrating the process of determining the coordinates of a second pixel according to one embodiment of this disclosure. (See also...) Figure 11 Step S300 may include steps S310 and S320.

[0117] S310, perform line fitting on the above-mentioned at least part of the pattern content to obtain multiple line segments, the above-mentioned at least part of the pattern content includes at least part of the region of multiple sectors, and the line segments are the boundaries of the above-mentioned sectors.

[0118] S320, based on the above multiple line segments, determine the second pixel coordinates of the calibration point of the current target calibration pattern in the coordinate system of the second captured image.

[0119] The sector boundary lines in the second captured image are extracted by straight line fitting. The intersection points of these boundary lines are then determined, yielding the center of the sector. Using the sector center as the calibration point, the intersection points of the boundary lines can be used as the calibration points themselves. Figure 8 Taking the second captured image shown as an example, a straight line fitting algorithm is used to... Figure 8Multiple sector boundary lines are extracted from the image to obtain straight line segments of the sector boundaries. The intersection points of these straight line segments are then calculated. Using the center of the Siemens star as the calibration point, the intersection points of the straight line segments are the center of the star, thus obtaining the calibration point. The pixel coordinates of the calibration point share the same coordinate system as the second captured image, which is equivalent to sharing the same coordinate system as the current field of view of the microscope camera. If the origin of the current field of view of the microscope camera is the upper left corner of the field of view, then the upper left corner of the second captured image is the origin, and the pixel coordinates of the calibration point are obtained based on the distance between the calibration point and this origin.

[0120] The line fitting algorithm is used to extract the coordinate points on each line, which are then located in the coordinate system of the second captured image. The least squares method can then be used to fit these coordinate points to obtain the lines; this least squares method can be the least squares M-estimator (M-estimator) method. The extracted line segments can be a subset of the aforementioned pattern content, or they can be all the line segments from the aforementioned pattern content.

[0121] Figure 12 Yes Figure 8 A schematic diagram obtained after linear fitting. (See attached image.) Figure 12 Assuming for Figure 8 By extracting all the line segments in the equation, we can obtain... Figure 12 The image shown is a straight line fitting image. Figure 12 The line segments in the image are the extracted line segments.

[0122] Figure 13 This is a schematic flowchart illustrating the determination of the second pixel coordinates according to another embodiment of this disclosure. (See also...) Figure 13 Step S320 may include steps S321 and S322.

[0123] S321: Using two line segments as a group of lines, determine the intersection point of each group of lines in different groups of lines to obtain multiple intersection points.

[0124] S322, calculate the average coordinates of the above multiple intersection points, and use the obtained average coordinates as the second pixel coordinates of the calibration point of the current target calibration pattern. The coordinates of the intersection points are the coordinates in the coordinate system of the second captured image.

[0125] for Figure 13For a portion or all of the line segments, the intersection points can be determined by identifying the intersection points of any two adjacent line segments, thus obtaining multiple intersection points. Alternatively, the intersection points can be determined by identifying the intersection points of any two adjacent line segments, resulting in even more intersection points. After obtaining multiple intersection points, the average X-axis coordinate and average Y-axis coordinate of these intersection points are calculated. The origin of the X-axis and Y-axis can be the top-left corner of the second captured image. The average value yields the coordinates of the calibration point, which is also the coordinates of the second pixel.

[0126] Figure 14 This is a schematic diagram illustrating the process of aligning the current calibration point with the center of the field of view according to one embodiment of this disclosure. See also... Figure 14 Step S400 may include a first moving step S410 and a first iterating step S420.

[0127] First moving step S410: The driving displacement device moves the calibration object toward the center of the field of view of the microscope camera.

[0128] First iteration step S420: Each time the first movement step is completed, the new microscopic imaging position formed by the movement is substituted into the current microscopic imaging position to obtain the new second pixel coordinates, until the new second pixel coordinates coincide with the center of the field of view.

[0129] In actual microscopic scanning using a microscope camera, the object is aligned with the center of the field of view. Therefore, during calibration, the alignment of the field of view center with the calibration point indicates that the displacement device has reached its correct position. After obtaining the second pixel coordinates of the calibration point, since the imaging size of the field of view is known, the orientation and distance of the calibration point at the current field of view center can be determined using the second pixel coordinates and the imaging size of the field of view.

[0130] For example, in the first moving step S410, when moving closer to the center of the field of view of the microscope camera, the direction of movement is obtained based on the relative position relationship between the second pixel coordinates and the center of the field of view in the field of view coordinate system, and the distance of movement is obtained based on the pixel distance between the second pixel coordinates and the center of the field of view.

[0131] Continue with Figure 12 For example, Figure 12 The coordinate system is the same as the current field-of-view coordinate system. Figure 12Within the field of view, the calibration point is located outside the field of view. Taking the upper left corner of the field of view as the origin, the pixel coordinates of the field of view center can be obtained from the length and width of the field of view. The length of the second pixel coordinate in the X-axis direction is greater than the length of the current field of view center in the X-axis direction, while the length of the second pixel coordinate in the Y-axis direction is less than the length of the current field of view center in the Y-axis direction. This reveals the pixel distance (pixel deviation) and relative positional relationship between the two. The displacement device is then controlled to move based on the pixel distance and relative positional relationship.

[0132] The aforementioned movement distance can be obtained based on the estimated physical distance corresponding to the current pixel distance. In the first movement step S410, the change in the mechanical coordinates of the displacement device during movement is obtained through pixel distance conversion. Since calibration has not yet been completed, the conversion between mechanical coordinates and pixel coordinates uses the estimated physical distance. Therefore, when the first movement step S410 of the current calibration image is completed for the first time, the current calibration point may directly coincide with the center of the field of view, or it may not completely coincide with the center of the field of view, but compared to before the first movement step S410, the distance between the current calibration point and the center of the field of view is closer. Assuming that the pixel distance between the current second pixel coordinate and the coordinate of the center of the field of view is 10 pixels, if the estimated physical distance corresponding to 10 pixels is 10 micrometers, then the displacement device is controlled to move 10 micrometers.

[0133] Each time the first movement step S410 of the current calibration image is completed, it is determined whether the second pixel coordinate of the calibration point coincides with the current field of view center. The method for determining coincidence is that the second pixel coordinate and the coordinate value of the field of view center are the same. If they coincide, the displacement device mechanical coordinates of the current calibration point can be obtained. If they do not coincide, the new field of view and the new microscopic imaging position formed after the movement are taken as the current field of view center and the current microscopic imaging position, and the microscopic imaging step S200 is performed again to obtain new pattern content. Compared to before executing the first movement step S410, the captured pattern content will change; for example, the calibration point may change from being outside the field of view to appearing within the field of view.

[0134] It is understandable that, for the same calibration pattern, when performing the second and subsequent photomicrographing steps during the process of aligning the current calibration point with the center of the field of view, there is no need to perform the focusing and shooting step S10, the sharpness evaluation step S20, and the image determination step S30. The photo can be taken directly after the calibration object is moved to the new photomicrographing position.

[0135] After obtaining the new pattern content, the new second pixel coordinates are determined by straight line fitting under the new field of view based on the new pattern content. The new second pixel coordinates correspond to the field of view coordinate system of the new field of view. Then, it is determined whether the new second pixel coordinates coincide with the center of the field of view. This process is repeated iteratively until the two coincide. At this point, the mechanical coordinates of the displacement device are recorded. These mechanical coordinates, together with the second pixel coordinates of the current calibration point, form a set of coordinate data for the current calibration pattern. Figure 15 This is a schematic diagram showing the alignment of the calibration point with the center of the field of view according to one embodiment of this disclosure. (See also...) Figure 15 After executing step S400 at least once, the coordinates of the field center and the calibration point are the same in the current field of view.

[0136] After obtaining a set of coordinate data for the current calibration pattern, the control displacement device is moved to a preset position C of the next calibration pattern (new calibration pattern), placing the calibration object in a new microscopic imaging position. This allows for the imaging of the new calibration pattern to obtain a new second image. Based on the pattern content in the new second image, the second pixel coordinates of the new calibration point are determined. The displacement device is then driven to align these new second pixel coordinates with the center of the new field of view, resulting in a set of coordinate data for the new calibration pattern. This process is repeated for other calibration patterns, obtaining at least five sets of coordinate data, which are then used to calculate the coordinate calibration matrix.

[0137] Figure 16 This is a schematic diagram of a coordinate calibration device for microscopic scanning, implemented in hardware using a processing system according to one embodiment of the present disclosure. (See also...) Figure 16 This disclosure also provides a coordinate calibration device 1000 for microscopic scanning. The coordinate calibration device 1000 of this embodiment may include a first pixel coordinate determination module 1002, a microscopic imaging module 1004, a second pixel coordinate determination module 1006, a mechanical coordinate acquisition module 1008, a calibration point iteration module 1010, and a coordinate relationship determination module 1012.

[0138] The first pixel coordinate determination module 1002 is used to determine the first pixel coordinates of the calibration points of multiple calibration patterns in the calibration image contained in the first captured image in the first captured image.

[0139] The first pixel coordinate determination module 1002 can determine the first pixel coordinate through the following 5 steps: (1) The driving displacement device moves the stage to the initial preset position, wherein a calibration object is set on the stage. When the stage is at the initial preset position, the calibration object is located at the first shooting position, which corresponds to the first camera. The calibration object at the first shooting position faces the first camera; (2) The first camera is controlled to take a picture of the calibration object to obtain the first shooting image, wherein the first shooting image includes the calibration image on the calibration object. The calibration image includes multiple calibration patterns, and the calibration point is located at the center of the calibration pattern. The above-mentioned multiple calibration patterns are at least 5 calibration patterns, the number of first pixel coordinates of the determined calibration points is at least 5, each calibration pattern includes only one calibration point, each calibration pattern has the same shape, each calibration pattern is divided into multiple sectors, the centers of the multiple sectors are the same and distributed around the center circumferentially; (3) threshold segmentation is performed on the first captured image to obtain multiple sub-regions; (4) for each sub-region, the calibration pattern region is identified from the sub-region according to the shape information of the calibration pattern; (5) the center coordinates of the calibration pattern region are determined and used as the first pixel coordinates of the calibration point in the calibration pattern in the first captured image.

[0140] The microscopic imaging module 1004 is used to perform a microscopic imaging step, which includes: taking a picture of the calibration object located at the current microscopic imaging position to obtain a second image, the second image being an image taken by a microscopic camera, and the microscopic imaging position being such that the second image includes at least a portion of the pattern content in the current target calibration pattern.

[0141] The microscopic imaging process can specifically include the following 7 steps: (1) driving the displacement device to move the calibration object to the current microscopic imaging position; (2) using the first step length as the preset step length to perform the focusing and imaging process, obtaining multiple first candidate images, wherein the focusing and imaging process includes: controlling the microscopic camera to move multiple times in the depth of field direction with a preset step length, and controlling the microscopic camera to take pictures of the calibration object after each movement, obtaining multiple candidate images; (3) performing a sharpness evaluation process on the above multiple first candidate images, obtaining multiple first sharpness values, wherein the sharpness evaluation process includes: using the candidate images as the images to be evaluated, converting the images to be evaluated to grayscale to obtain a grayscale image, determining the image gradient and variance of the grayscale image, and performing normalization calculation based on the variance and image gradient to obtain a sharpness value; (4) determining the first sharpness value corresponding to the optimal first sharpness value. (5) Within the range of movement including the first moving position, the second step length is used as the preset step length to perform the focusing and shooting step to obtain multiple second candidate shooting images, wherein the second step length is less than the first step length; (6) The above multiple second candidate shooting images are subjected to a sharpness evaluation step to obtain multiple second sharpness values; (7) The second shooting image is determined from the above multiple second sharpness values ​​through an image determination step, wherein the image determination step includes: determining the candidate shooting image with the best sharpness value and using it as the second shooting image, the microscopic shooting position is such that the second shooting image includes at least part of the pattern content in the current target calibration pattern, the above multiple calibration patterns include a first calibration pattern and multiple second calibration patterns, the size of the first calibration pattern is larger than the size of the second calibration pattern, and the current target calibration pattern included in the first second shooting image is the first calibration pattern.

[0142] The second pixel coordinate determination module 1006 is used to determine the second pixel coordinates of the calibration point of the current target calibration pattern in the coordinate system of the second captured image based on at least part of the pattern content.

[0143] The second pixel coordinate determination module 1006 can determine the second pixel coordinates in the following three steps: (1) perform line fitting on the above at least part of the pattern content to obtain multiple line segments, wherein the above at least part of the pattern content includes at least part of the region of multiple fan-shaped areas, and the line segments are the boundaries of the fan-shaped areas; (2) take two line segments as a group of lines, determine the intersection point of each group of lines in different groups of lines, and obtain multiple intersection points; (3) calculate the average coordinate value of multiple intersection points, and use the obtained average coordinate value as the second pixel coordinate of the calibration point of the current target calibration pattern, wherein the coordinates of the intersection point are the coordinates in the coordinate system of the second captured image.

[0144] The mechanical coordinate acquisition module 1008 is used to drive the displacement device to move the calibration object closer to the center of the field of view of the microscope in the horizontal direction until the current calibration point coincides with the center of the field of view, and to acquire the mechanical coordinates of the displacement device when they coincide.

[0145] The mechanical coordinate acquisition module 1008 can acquire mechanical coordinates in the following two steps: (1) Drive the displacement device to move the calibration object toward the center of the field of view of the microscope camera. When moving toward the center of the field of view of the microscope camera, the direction of movement can be obtained based on the relative position relationship between the second pixel coordinate and the center of the field of view in the field of view coordinate system. The distance of movement can be estimated based on the physical distance corresponding to the current pixel distance between the second pixel coordinate and the center of the field of view. (2) When the first movement step is completed each time, the new microscope imaging position formed by the movement is substituted into the microscope imaging step as the current microscope imaging position to obtain the new second pixel coordinate, until the new second pixel coordinate coincides with the center of the field of view, and the mechanical coordinate of the displacement device when it coincides is obtained.

[0146] The calibration point iteration module 1010 is used to execute the calibration point iteration steps, which include: driving the displacement device to move the calibration object to the microscopic imaging position of the next calibration pattern, and substituting the microscopic imaging position of the next calibration pattern as the current microscopic imaging position into the microscopic imaging step to obtain the mechanical coordinates of the displacement device corresponding to the calibration point of the next calibration pattern, until the mechanical coordinates of the displacement device of the calibration points of multiple calibration patterns are obtained.

[0147] The coordinate relationship determination module 1012 is used to obtain the coordinate transformation relationship based on the pixel coordinates of the calibration points of multiple calibration patterns and the mechanical coordinates of the corresponding displacement devices.

[0148] It should be noted that details not disclosed in the coordinate calibration device 1000 for microscopic scanning in this embodiment can be found in the details disclosed in the coordinate calibration method M10 for microscopic scanning in the above-described embodiment of this disclosure, and will not be repeated here.

[0149] The device 1000 may include corresponding modules that perform one or more steps in the flowchart described above. Therefore, each or more steps in the flowchart can be performed by a corresponding module, and the device may include one or more of these modules. A module may be one or more hardware modules specifically configured to perform a corresponding step, or implemented by a processor configured to perform a corresponding step, or stored in a computer-readable medium for implementation by a processor, or implemented through some combination thereof.

[0150] This hardware architecture can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application and overall design constraints of the hardware. Bus 1100 connects various circuits, including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400, such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0151] Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one connection line is used in this diagram, but this does not imply that there is only one bus or only one type of bus.

[0152] Figure 17 This is a schematic diagram of a microscopic scanning system according to one embodiment of the present disclosure. (See also...) Figure 17 This disclosure also provides a microscopic scanning system 2000, which includes a microscopic scanning coordinate calibration device 1000, an imaging system 2010, and a displacement device 2020.

[0153] Coordinate calibration device 1000 Figure 16 The coordinate calibration device shown may include a first pixel coordinate determination module, a microscopic imaging module, a second pixel coordinate determination module, a mechanical coordinate acquisition module, a calibration point iteration module, and a coordinate relationship determination module.

[0154] The imaging system 2010 includes a microscope camera, which is used to photograph the object to be photographed under the control of a coordinate calibration device to obtain an image. The imaging system may also include a navigation camera, a light source, an objective lens, and a stage. The navigation camera is used to capture the first image. The stage is used to hold the object to be photographed, such as a calibration object. The stage moves synchronously with the displacement device.

[0155] The displacement device 2020 is used to move under the control of a coordinate calibration device to synchronously move the object to be photographed. The displacement device may include an electric linear slide, a motor, and a driver to achieve linear displacement.

[0156] Any process or method description in the flowcharts or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain. The processor performs the various methods and processes described above. For example, the method embodiments of this disclosure may be implemented as software programs tangibly contained in a machine-readable medium, such as memory. In some embodiments, part or all of the software program may be loaded and / or installed via memory and / or a communication interface. When the software program is loaded into memory and executed by the processor, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the processor may be configured to perform one of the methods described above by any other suitable means (e.g., by means of firmware).

[0157] The logic and / or steps represented in the flowchart or otherwise described herein may be specifically implemented in any readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0158] It should be understood that various parts of this disclosure can be implemented in hardware, software, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0159] Those skilled in the art will understand that all or part of the steps of the methods described above can be implemented by a program instructing related hardware. This program can be stored in a readable storage medium, and when executed, it includes one or a combination of the steps of the method implementation. The storage medium can be volatile or non-volatile.

[0160] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a single processing module, or each unit can exist physically separately, or two or more units can be integrated into a single module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a readable storage medium. The storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0161] This disclosure also provides a readable storage medium storing executable instructions, which, when executed by a processor, are used to implement the coordinate calibration method for microscopic scanning according to any of the above embodiments.

[0162] For the purposes of this specification, a "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM). Furthermore, a readable storage medium can even be paper or other suitable media on which a program can be printed, since a program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in memory.

[0163] This disclosure also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the coordinate calibration method for microscopic scanning according to any of the above embodiments.

[0164] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment / mode or example is included in at least one embodiment / mode or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0165] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0166] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. A method of coordinate calibration for microscopic scanning, characterized by, include: The calibration points of multiple calibration patterns in the calibration image contained in the first captured image are determined to be at the first pixel coordinates in the first captured image; wherein, the first captured image is obtained by a first camera, and the first camera is a navigation camera; After obtaining the first image, the control displacement device moves the calibration object to another preset position so that the calibration object enters the field of view of the microscope camera; The microscopic imaging step involves taking a picture of the calibration object located at the current microscopic imaging position to obtain a second image. The second image is an image taken by a microscopic camera. The microscopic imaging position is such that the second image includes at least a portion of the pattern content in the current target calibration pattern. The second pixel coordinates of the calibration point of the current target calibration pattern in the coordinate system of the second captured image are determined based on at least part of the pattern content. The driving displacement device moves the calibration object horizontally toward the center of the field of view of the microscope until the second pixel coordinate of the current calibration point coincides with the center of the field of view, and the mechanical coordinates of the displacement device are obtained when they coincide. The calibration point iteration step involves driving the displacement device to move the calibration object to the microscopic imaging position of the next calibration pattern, and substituting the microscopic imaging position of the next calibration pattern as the current microscopic imaging position into the microscopic imaging step to obtain the mechanical coordinates of the displacement device corresponding to the calibration point of the next calibration pattern, until the mechanical coordinates of the displacement device of the calibration points of the multiple calibration patterns are obtained; and The coordinate transformation relationship is obtained based on the first pixel coordinates of the calibration points of the multiple calibration patterns and the corresponding mechanical coordinates of the displacement device; Each of the calibration patterns has the same shape, and each of the calibration patterns is divided into multiple sectors, with the centers of the multiple sectors being the same and distributed circumferentially around the centers; The method of determining the second pixel coordinates of the calibration point of the current target calibration pattern in the coordinate system of the second captured image based on the at least part of the pattern content includes: performing line fitting on the at least part of the pattern content to obtain multiple line segments, wherein the at least part of the pattern content includes at least a portion of a multiple sector, and the line segments are the boundaries of the sector; and determining the second pixel coordinates of the calibration point of the current target calibration pattern in the coordinate system of the second captured image based on the multiple line segments. The method of determining the second pixel coordinates of the calibration point of the current target calibration pattern in the coordinate system of the second captured image based on the multiple line segments includes: taking two line segments as a group of lines, determining the intersection point of each group of lines in different groups of lines to obtain multiple intersection points; calculating the average coordinate value of the multiple intersection points, and using the obtained average coordinate value as the second pixel coordinates of the calibration point of the current target calibration pattern, wherein the coordinates of the intersection points are the coordinates in the coordinate system of the second captured image.

2. The method of claim 1, wherein, The plurality of calibration patterns are at least 5 calibration patterns, and the number of first pixel coordinates of the determined calibration points is at least 5.

3. The method of claim 1, wherein, Each calibration pattern includes only one calibration point.

4. The method of claim 1, wherein, The plurality of calibration patterns include a first calibration pattern and a plurality of second calibration patterns. The size of the first calibration pattern is larger than the size of the second calibration pattern. The current target calibration pattern included in the first acquired second captured image is the first calibration pattern.

5. The method according to any one of claims 1-4, characterized in that, Determining the first pixel coordinates of the calibration points of multiple calibration patterns in the calibration image included in the first captured image, comprising: A first image is obtained by photographing a calibration object located at a first shooting position. The first image includes a calibration image on the calibration object, and the calibration image includes multiple calibration patterns. Determine the first pixel coordinates of the calibration points in the plurality of calibration patterns in the first captured image.

6. The method of claim 5, wherein, A first image is obtained by taking a picture of the marker located at the first shooting position, including: The driving displacement device moves the calibration object to a first shooting position, which corresponds to the first camera; and The first camera is controlled to take a picture of the calibration object to obtain a first captured image.

7. The method of claim 6, wherein, The driving displacement device moves the calibration object to the first shooting position, including: The driving displacement device moves the stage to an initial preset position. A calibration object is provided on the stage. When the stage is at the initial preset position, the calibration object is at the first shooting position.

8. The method according to claim 6 or 7, characterized in that, The calibrator located at the first shooting position is directly facing the first camera.

9. The method of claim 5, wherein, The calibration point is located at the center of the calibration pattern. Determining the first pixel coordinates of the calibration point in the first captured image in the plurality of calibration patterns includes: The first captured image is segmented using a threshold to obtain multiple sub-regions; For each sub-region, the calibration pattern region is identified from the sub-region based on the shape information of the calibration pattern; and The center coordinates of the calibration pattern area are determined and used as the first pixel coordinates of the calibration point in the calibration pattern in the first captured image.

10. The method of any one of claims 1-4, wherein, A second image is obtained by photographing the calibration object located at the current microscopic imaging position, including: The drive displacement device moves the calibration object to the current microscopic imaging position; and A second image is obtained by controlling a microscope camera to photograph the calibration object.

11. The method of claim 10, wherein, Controlling a microscope camera to capture images of the calibration object to obtain a second image includes: The focusing and shooting step involves controlling the microscope camera to take pictures of the calibration object at multiple different focal lengths to obtain multiple candidate images; The sharpness evaluation step involves evaluating the sharpness of each of the multiple candidate images obtained, and acquiring corresponding sharpness values; and The image determination step identifies the candidate image with the best sharpness value and uses it as the second image.

12. The method of claim 11, wherein, Controlling a microscope camera to photograph the calibration object at multiple different focal lengths includes: The microscope camera is controlled to move multiple times in the depth of field with a preset step size, and after each movement, the microscope camera is controlled to take a picture of the calibration object.

13. The method of claim 12, wherein, Controlling a microscope camera to capture images of the calibration object to obtain a second image includes: The focusing and shooting steps are performed using the first step length as a preset step length to obtain multiple first candidate images. The sharpness evaluation step is performed on the plurality of first candidate captured images to obtain a plurality of first sharpness values; Determine the first movement position corresponding to the optimal first resolution value; Within the movement range including the first moving position, the focusing and shooting steps are performed with the second step length as the preset step length to obtain multiple second candidate shooting images, wherein the second step length is smaller than the first step length; The sharpness evaluation step is performed on the plurality of second candidate images to obtain a plurality of second sharpness values; and The second captured image is determined from the plurality of second sharpness values ​​through the image determination step.

14. The method according to any one of claims 11-13, characterized in that, The sharpness evaluation steps include: The candidate captured images are used as images to be evaluated, and the images to be evaluated are converted to grayscale to obtain grayscale images; Determine the image gradient and variance of the grayscale image; and The sharpness value is obtained by normalizing the variance and the image gradient.

15. The method of any one of claims 1-4, wherein, The driving displacement device moves the calibration object horizontally toward the center of the field of view of the microscope camera until the second pixel coordinate of the current calibration point coincides with the center of the field of view, including: In the first moving step, the driving displacement device moves the calibration object towards the center of the field of view of the microscope camera; and In the first iteration step, each time the first movement step is completed, the new microscopic imaging position formed by the movement is substituted into the current microscopic imaging position to obtain a new second pixel coordinate, until the new second pixel coordinate coincides with the center of the field of view.

16. The method of claim 15, wherein, In the first moving step, when moving towards the center of the field of view of the microscope camera, the direction of movement is obtained based on the relative position relationship between the second pixel coordinate and the center of the field of view in the field of view coordinate system, and the distance of movement is obtained based on the current pixel distance between the second pixel coordinate and the center of the field of view.

17. The method of claim 16, wherein, The distance moved is obtained based on the estimated physical distance corresponding to the current pixel distance.

18. A readable storage medium, characterized by, The readable storage medium stores execution instructions, which, when executed by a processor, are used to implement the coordinate calibration method as described in any one of claims 1 to 17.

Citation Information

Patent Citations

  • Calibration method and device, calibration equipment and storage medium

    CN112634376A

  • Method for calibrating a stage-camera system, and a stage-camera system

    US20100066838A1

  • Reference sample for a microscope, methods and uses

    US20230057044A1