Depth information guided multi-focal-plane microscopic imaging method and device and medium

Abstract

The invention provides a depth information guided multi-focal-plane microscopic imaging method and device and a medium, and the method comprises the steps: carrying out the region division of a plurality of camera fields of view of a target sample, and obtaining a plurality of field of view regions of each camera field of view; collecting area depth values of the plurality of view field areas; according to the area depth values of the plurality of field-of-view areas, obtaining a field-of-view depth range of the corresponding camera field-of-view; acquiring a plurality of view field focal plane depths of each camera view field according to the view field depth range of each camera view field and a depth-of-field parameter of the microscopic equipment; performing image acquisition according to the plurality of view field focal plane depths of each camera view field to obtain a plurality of focal plane microscopic images of each camera view field; and according to the plurality of focal plane microscopic images of the plurality of camera fields of view, obtaining a panoramic depth fusion image corresponding to the target sample. The number of times of image acquisition can be effectively controlled, redundant image data is prevented from being generated, and the efficiency of image acquisition is improved.

Description

Technical Field

[0001] This application relates to the technical field of microscopic imaging, specifically to a depth-information-guided multifocal plane microscopic imaging method, apparatus, and medium. Background Technology

[0002] Extended depth-of-field microscopy is a method for obtaining high-resolution microscopic images through image data acquisition and fusion. However, the fundamental bottleneck in existing technologies lies in their "blind" acquisition strategy. Existing methods typically employ a "uniform axial scanning" mode, which mechanically acquires images of the entire target area at equal intervals and across multiple focal planes without knowing the surface morphology of the sample. This approach has significant drawbacks:

[0003] To ensure coverage of the entire depth of field for irregular samples, numerous focal planes must be set up, and redundant acquisition must be performed on empty focal planes in many sample-free areas. This not only significantly reduces imaging efficiency and increases unnecessary exposure, but also causes significant light damage or photobleaching to the samples. Secondly, this indiscriminate acquisition mode generates massive amounts of redundant image data, placing a huge burden on subsequent storage, transmission, and processing, making real-time or high-throughput imaging difficult to achieve. Summary of the Invention

[0004] The purpose of this application is to overcome the shortcomings and deficiencies in the prior art and to provide a depth information-guided multifocal plane microscopic imaging method, device and medium.

[0005] A first aspect of this application provides a depth-information-guided multifocal plane microscopy imaging method, comprising:

[0006] The target sample is divided into multiple camera fields of view to obtain several field of view regions included in each camera field of view;

[0007] Collect the regional depth values ​​of the aforementioned several field-of-view regions;

[0008] Based on the regional depth values ​​of the plurality of field of view regions, the corresponding field of view depth range of the camera field of view is obtained;

[0009] Based on the field of view depth range of each camera field of view and the depth of field parameters of the microscope, several field of view focal plane depths of each camera field of view are obtained.

[0010] Image acquisition is performed based on the depth of several focal planes of each camera's field of view to obtain multiple focal plane microscopic images of each camera's field of view.

[0011] A panoramic depth-fusion image corresponding to the target sample is obtained based on multiple focal plane microscopic images from the multiple camera fields of view.

[0012] As one implementation, the step of obtaining the field-of-view depth range of the corresponding camera field of view based on the regional depth values ​​of the plurality of field-of-view regions includes:

[0013] Obtain the maximum and minimum region depth values ​​corresponding to the field of view of each camera;

[0014] The field of view depth range of the corresponding camera field of view is obtained based on the maximum region depth value and the minimum region depth value.

[0015] As one implementation, the step of obtaining several focal plane depths of each camera's field of view based on the field of view depth range of each camera and the depth-of-field parameters of the microscope equipment includes:

[0016] Obtain the first focal plane depth within the field of view depth range;

[0017] If the focal depth range obtained based on the first focal depth and the depth of field parameters covers the field of view depth range, the first focal depth is determined as the field of view focal depth of the corresponding camera field of view.

[0018] If the focal depth range obtained based on the first focal depth and the depth of field parameter fails to cover the field of view depth range, the height of the first focal depth is adjusted according to a preset step distance to obtain a second focal depth; wherein, the preset step distance is less than or equal to the depth of field parameter.

[0019] The focal depth range is updated based on the latest second focal depth and the depth of field parameters. If the updated focal depth range covers the field of view depth range, the first focal depth and the obtained second focal depth are determined as the field of view focal depth of the corresponding camera field of view.

[0020] If the updated focal depth range fails to cover the field of view depth range, a new second focal depth is obtained based on the height of the second focal depth according to the preset step size, until the focal depth range updated based on the first focal depth, the obtained second focal depth, and the depth parameters covers the field of view depth range. The first focal depth and the obtained second focal depth are then determined as the field of view focal depth of the corresponding camera field of view.

[0021] As one implementation, the step of adjusting the height of the first focal plane depth according to a preset step size to obtain a second focal plane depth if the focal plane depth range obtained based on the first focal plane depth and the depth of field parameters fails to cover the field of view depth range includes:

[0022] If the focal depth range obtained based on the first focal depth and the depth of field parameters fails to cover the minimum area depth value of the field of view depth range, the height of the first focal depth is adjusted according to a preset step distance along the direction close to the minimum area depth value of the field of view depth range to obtain the second focal depth.

[0023] As one implementation, the step of adjusting the height of the first focal plane depth according to a preset step size to obtain a second focal plane depth if the focal plane depth range obtained based on the first focal plane depth and the depth of field parameters fails to cover the field of view depth range includes:

[0024] If the focal depth range obtained based on the first focal depth and the depth of field parameters fails to cover the maximum area depth value of the field of view depth range, the height of the first focal depth is adjusted according to a preset step distance along the direction close to the maximum area depth value of the field of view depth range to obtain the second focal depth.

[0025] As one implementation, the step of acquiring images based on the depth of several focal planes of each camera's field of view to obtain multiple focal plane microscopic images of each camera's field of view includes:

[0026] Based on the focal length of the microscope, obtain several objective lens heights corresponding to the several focal plane depths of the microscope.

[0027] Based on the field-of-view coordinates of the camera and the heights of the plurality of objective lenses, the microscope is driven to acquire images of the camera's field of view, thereby obtaining multiple focal plane microscopic images of the camera's field of view.

[0028] As one implementation, the step of obtaining a panoramic depth-fused image corresponding to the target sample based on multiple focal plane microscopic images from the multiple camera fields of view includes:

[0029] The multiple focal plane microscopic images of the multiple camera fields of view are subjected to grayscale conversion and filtering to obtain multiple focal plane microscopic grayscale images of the multiple camera fields of view.

[0030] Based on the pixel gradient values ​​of the multiple focal plane microscopic grayscale images, a focal plane index map of the multiple camera fields of view is generated; the focal plane index map records the pixel with the maximum pixel gradient value and the image index of the multiple focal plane microscopic images;

[0031] Based on the focal plane index map, pixel values ​​are extracted and fused from multiple focal plane microscopic images of the multiple camera fields of view to obtain the panoramic depth fused image.

[0032] In one implementation, the microscope equipment has several range sensors located next to the objective lens; the step of acquiring the regional depth values ​​of the several fields of view includes:

[0033] When acquiring images using the objective lens based on the depth of several focal planes of the current camera's field of view, the range sensor detects the regional depth values ​​of several fields of view regions of other cameras' fields of view.

[0034] Compared to related technologies, the depth-guided multi-focal plane microscopy imaging method of this application first divides the field of view of multiple cameras on the target sample into regions, obtaining several field of view regions included in each camera field of view. It then collects the region depth values ​​of these several field of view regions, and based on these region depth values, obtains the field of view depth range of the corresponding camera field of view. Next, based on the field of view depth range of each camera field of view and the depth-of-field parameters of the microscopic device, it obtains several focal plane depths of each camera field of view. Finally, it performs image acquisition based on these focal plane depths of each camera field of view, obtaining multiple focal plane microscopic images of each camera field of view. Based on these multiple focal plane microscopic images of the multiple camera field of view, it obtains a full-depth-of-view fused image of the target sample. This method combines the field of view depth range of multiple camera fields of view of the target sample with the depth-of-field parameters of the microscopic device for image acquisition, effectively controlling the number of image acquisitions, avoiding redundant image data, improving image acquisition efficiency, and preventing the influence of redundant image data on imaging, thus improving the image quality of the microscopic imaging.

[0035] A second aspect of this application provides a depth-information-guided multifocal plane microscopic imaging apparatus, comprising:

[0036] The field of view region division module is used to divide the multiple camera fields of view of the target sample into regions, and obtain several field of view regions included in each camera field of view.

[0037] The region depth value acquisition module is used to acquire the region depth values ​​of the several field-of-view regions;

[0038] The field of view depth range acquisition module is used to acquire the field of view depth range of the corresponding camera field of view based on the regional depth values ​​of the plurality of field of view regions.

[0039] The field-of-view focal plane depth acquisition module is used to acquire several field-of-view focal plane depths of each of the camera fields of view based on the field-of-view depth range of each camera field of view and the depth-of-field parameters of the microscope device.

[0040] The focal plane microscopic image acquisition module is used to acquire images based on several focal plane depths of each camera's field of view, thereby obtaining multiple focal plane microscopic images of each camera's field of view.

[0041] The panoramic depth fusion image acquisition module is used to obtain a panoramic depth fusion image corresponding to the target sample based on multiple focal plane microscopic images of the multiple camera fields of view.

[0042] A third aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the depth information-guided multifocal plane microscopy imaging method as described above.

[0043] To provide a clearer understanding of this application, the specific embodiments of this application will be described below in conjunction with the accompanying drawings. Attached Figure Description

[0044] Figure 1 This is a flowchart of a depth information-guided multifocal plane microscopy imaging method according to an embodiment of this application.

[0045] Figure 2 This is a schematic diagram of multiple camera fields of view in a depth information-guided multifocal plane microscopy imaging method according to an embodiment of this application.

[0046] Figure 3 This is a schematic diagram of the objective lens and range sensor of a depth information-guided multifocal plane microscopy imaging method according to an embodiment of this application.

[0047] Figure 4 This is a schematic diagram of the module connections of a depth information-guided multifocal plane microscopy imaging device according to an embodiment of this application.

[0048] 100. Multi-focal plane microscopic imaging device; 101. Field of view region division module; 102. Region depth value acquisition module; 103. Field of view depth range acquisition module; 104. Field of view focal plane depth acquisition module; 105. Focal plane microscopic image acquisition module; 106. Panoramic depth fusion image acquisition module. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0050] It should be understood that the described embodiments are merely some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of the embodiments of this application.

[0051] In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. In the description of this application, it should be understood that the terms "first," "second," "third," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances. The singular forms "a," "the," and "the" used in this application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. The word "if" as used herein can be interpreted as "when," "when," or "in response to determination."

[0052] Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0053] Please see Figure 1 This is a flowchart of a depth information-guided multifocal plane microscopy imaging method according to the first embodiment of this application, the method comprising:

[0054] S1: Divide the multiple camera fields of view of the target sample into regions to obtain several field of view regions included in each of the camera fields of view.

[0055] Please see Figure 2 , Figure 2 The diagram shows multiple camera fields of view for the target sample, arranged in a 5x4 grid. The number of field-of-view regions is set by the user, and each region is proportional to the corresponding camera field of view. For example, a cross division can be used to divide the camera field of view into four regions proportional to the camera field of view; or... Figure 2 The bottom row shows a camera field of view. Using a grid method, the camera field of view is divided into 9 field of view regions that are proportional to the camera field of view.

[0056] S2: Collect the regional depth values ​​of the several fields of view.

[0057] In step S2, a range sensor can be used to collect the regional depth values ​​of the plurality of field-of-view regions. For example, a range sensor can be used to traverse the plurality of field-of-view regions of each camera's field of view one by one to perform distance detection, thereby obtaining the regional depth values ​​of the plurality of field-of-view regions. Therefore, the regional depth values ​​are the results of the range sensor collecting distances to each field-of-view region.

[0058] In order to improve the efficiency of acquiring regional depth values, several ranging sensors with the same number of field-of-view regions as each camera can be used to simultaneously acquire regional depth values ​​of several field-of-view regions of the camera's field of view.

[0059] S3: Based on the regional depth values ​​of the plurality of field of view regions, obtain the field of view depth range of the corresponding camera field of view.

[0060] In this embodiment, by dividing the camera's field of view into several field of view regions for depth detection, the depth range of the camera's field of view can be obtained more accurately.

[0061] S4: Based on the field-of-view depth range of each of the camera fields of view and the depth-of-field parameters of the microscope, obtain several field-of-view focal plane depths of each of the camera fields of view.

[0062] Depth of field is a parameter that measures the range of distances in front of and behind the subject that an imaging device can capture in a clear image. Specifically, there is a certain length of space in front of the lens (in front of and behind the focal point or in front of and behind the focal plane). When the subject is located within this space, its image on the film is located within the same circle of confusion, and the image within this space is clear. The length of this space in which the subject is located is the depth of field parameter.

[0063] S5: Image acquisition is performed based on the depth of several focal planes of each camera's field of view to obtain multiple focal plane microscopic images of each camera's field of view.

[0064] In this embodiment, a region depth value acquisition and image acquisition for several focal plane depths can be performed using an XY displacement stage (e.g., LAB-XY-50-50-B, LAB Motion Systems) with translation and lifting functions. The XY displacement stage controls the movement of the target sample along the X or Y axis via translation, and controls the movement of the microscope objective along the Z axis via lifting (based on a stepper motor with step distances accurate to micrometers (µm), such as the PKP546N18A2, Oriental Motor). Figure 3As shown, the range sensor is fixed next to the objective lens, and the sensor's detection head is flush with the objective lens surface. Since the range sensor and objective lens are fixed together, by combining the spatial distance deviation between them and the coordinate data of several focal plane depths in the field of view, accurate image acquisition can be performed based on the focal plane depths of each camera's field of view. This avoids the human error caused by manually replacing the range sensor with the microscope objective lens after acquiring the area depth values ​​when the range sensor and objective lens are used independently. This improves the accuracy of acquiring multiple focal plane microscopic images of each camera's field of view.

[0065] S6: Based on the multiple focal plane microscopic images of the multiple camera fields of view, obtain a panoramic depth fusion image corresponding to the target sample.

[0066] The focal plane described in this application is a focal plane that passes through the focal point and is perpendicular to the principal optical axis.

[0067] Compared to related technologies, the depth-guided multi-focal plane microscopy imaging method of this application first divides the field of view of multiple cameras on the target sample into regions, obtaining several field of view regions included in each camera field of view. It then collects the region depth values ​​of these several field of view regions, and based on these region depth values, obtains the field of view depth range of the corresponding camera field of view. Next, based on the field of view depth range of each camera field of view and the depth-of-field parameters of the microscopic device, it obtains several focal plane depths of each camera field of view. Finally, it performs image acquisition based on these focal plane depths of each camera field of view, obtaining multiple focal plane microscopic images of each camera field of view. Based on these multiple focal plane microscopic images of the multiple camera field of view, it obtains a full-depth-of-view fused image of the target sample. This method combines the field of view depth range of multiple camera fields of view of the target sample with the depth-of-field parameters of the microscopic device for image acquisition, effectively controlling the number of image acquisitions, avoiding redundant image data, improving image acquisition efficiency, and preventing the influence of redundant image data on imaging, thus improving the image quality of the microscopic imaging.

[0068] In a feasible embodiment, step S3: obtaining the field-of-view depth range of the corresponding camera field of view based on the region depth values ​​of the plurality of field-of-view regions, includes:

[0069] S31: Obtain the maximum and minimum region depth values ​​corresponding to the field of view of each of the cameras.

[0070] Taking the camera's field of view as comprising 9 fields of view regions as an example, the depth values ​​of the 9 fields of view regions can be sorted from largest to smallest or from smallest to largest to obtain the first and last depth values ​​in the sorted order, thereby obtaining the maximum and minimum depth values. Alternatively, the maximum and minimum depth values ​​with the largest differences can be obtained based on the differences between the depth values ​​of the 9 fields of view regions.

[0071] S32: Based on the maximum region depth value and the minimum region depth value, obtain the corresponding field of view depth range of the camera's field of view.

[0072] The field of view depth range can be obtained by using the maximum and minimum region depth values ​​corresponding to the camera's field of view as the range boundaries.

[0073] In a feasible embodiment, step S4: obtaining several focal plane depths of each camera's field of view based on the field of view depth range of each camera's field of view and the depth-of-field parameters of the microscope equipment, includes:

[0074] S41: Obtain the first focal plane depth within the field of view depth range;

[0075] The first focal plane depth can be obtained by taking the median value of the field of view depth range as the first focal plane depth; or by taking the boundary of the field of view depth range as the first focal plane depth, for example, the maximum or minimum area depth value of the field of view depth range as the first focal plane depth; or by taking the depth of field parameter of the microscope device and the boundary of the field of view depth range as the first focal plane depth, for example, the maximum area depth value of the field of view depth range plus the depth of field parameter as the first focal plane depth, or the minimum area depth value of the field of view depth range minus the depth of field parameter as the first focal plane depth.

[0076] In other embodiments, the first focal plane depth can also be determined based on the depth value of a random region within the field of view depth range.

[0077] S42: If the focal depth range obtained based on the first focal depth and the depth of field parameters covers the field of view depth range, the first focal depth is determined as the field of view focal depth of the corresponding camera field of view.

[0078] Wherein, the sum of the first focal plane depth and the depth of field parameter is the minimum boundary value of the focal plane depth of field range, and the difference between the first focal plane depth and the depth of field parameter is the maximum boundary value of the focal plane depth of field range.

[0079] Covering the field of view depth range means completely covering the field of view depth range, not partially covering it. Completely covering the field of view depth range means that the focal plane depth of field range includes all values ​​of the field of view depth range.

[0080] If the focal depth range obtained based on the first focal plane depth and the depth of field parameters covers the field of view depth range, it means that only the first focal plane depth is needed to acquire images of the camera's field of view. The resulting image clearly records all the depth details of the camera's field of view. Therefore, the focal plane depth of the camera's field of view is only the first focal plane depth.

[0081] S43: If the focal depth range obtained based on the first focal depth and the depth of field parameter fails to cover the field of view depth range, the height of the first focal depth is adjusted according to a preset step distance to obtain a second focal depth; wherein the preset step distance is less than or equal to the depth of field parameter.

[0082] Wherein, the height of the first focal plane depth is the distance data parallel to the focal plane of the microscope.

[0083] Preferably, the preset step size is less than or equal to the depth-of-field parameter, and greater than half of the depth-of-field parameter, such as 90% or 80% of the depth-of-field parameter. If the focal plane depth-of-field range obtained based on the first focal plane depth and the depth-of-field parameter fails to cover the field of view depth range, it means that image acquisition of the camera's field of view is only performed based on the first focal plane depth. The resulting image can only clearly record some depth details of the camera's field of view, not all of them. Therefore, it is necessary to adjust the height of the first focal plane depth according to the preset step size to obtain a second focal plane depth, thereby increasing the number of images acquired based on the second focal plane depth, and thus obtaining more depth details of the camera's field of view.

[0084] S44: Update the focal plane depth range according to the latest second focal plane depth and the depth of field parameters. If the updated focal plane depth range covers the field of view depth range, determine the first focal plane depth and the obtained second focal plane depth as the corresponding field of view focal plane depth of the camera field of view.

[0085] After updating the focal depth range based on the latest second focal plane depth and the depth of field parameters, if the updated focal depth range covers the field of view depth range, it means that the image acquisition of the camera field of view based on the first focal plane depth and the obtained second focal plane depth clearly records all the depth details of the camera field of view. Therefore, the field of view focal plane depth of the camera field of view includes the first focal plane depth and the obtained second focal plane depth.

[0086] S45: If the updated focal depth range fails to cover the field of view depth range, a new second focal depth is obtained according to the height of the second focal depth based on the preset step size, until the focal depth range updated according to the first focal depth, the obtained second focal depth, and the depth parameters covers the field of view depth range, and the first focal depth and the obtained second focal depth are determined as the field of view focal depth of the corresponding camera field of view.

[0087] Specifically, along a direction away from the first focal plane, a new second focal plane depth is obtained based on the height of the second focal plane depth according to the preset step distance.

[0088] If the updated focal plane depth range fails to cover the field of view depth range, it means that the image acquired by the camera field of view based on the first focal plane depth and the obtained second focal plane depth can only clearly record some depth details of the camera field of view, not all depth details. Therefore, it is necessary to adjust the second focal plane depth according to the preset step size to obtain more second focal plane depth until the focal plane depth range updated based on the first focal plane depth, the obtained second focal plane depth, and the depth parameters covers the field of view depth range. At this time, the field of view focal plane depth of the camera field of view includes the first focal plane depth and all obtained second focal plane depths.

[0089] In this embodiment, based on the relationship between the focal plane depth range and the field of view depth range, it is determined whether it is necessary to continue acquiring new field of view focal plane depths. Several effective field of view focal plane depths can be accurately acquired for different field of view depth ranges, so as to accurately control the number of image acquisitions for each camera field of view.

[0090] In a feasible embodiment, step S43: if the focal depth range obtained based on the first focal depth and the depth-of-field parameters fails to cover the field of view depth range, adjusting the height of the first focal depth according to a preset step size to obtain the second focal depth includes:

[0091] S431: If the focal depth range obtained based on the first focal depth and the depth of field parameters fails to cover the minimum area depth value of the field of view depth range, the height of the first focal depth is adjusted according to a preset step distance along the direction close to the minimum area depth value of the field of view depth range to obtain the second focal depth.

[0092] In this embodiment, based on whether the focal depth range obtained from the first focal depth and the depth of field parameters covers the minimum area depth value of the field of view depth range, the height adjustment direction of the first focal depth can be determined to obtain a second focal depth that is closer to the minimum area depth value of the field of view depth range than the first focal depth. This allows the new focal depth range obtained based on the first focal depth, the obtained second focal depth, and the depth of field parameters to be accurately extended along the direction close to the minimum area depth value of the field of view depth range.

[0093] In a feasible embodiment, step S43: if the focal depth range obtained based on the first focal depth and the depth-of-field parameters fails to cover the field of view depth range, adjusting the height of the first focal depth according to a preset step size to obtain the second focal depth includes:

[0094] S432: If the focal depth range obtained based on the first focal depth and the depth of field parameters fails to cover the maximum area depth value of the field of view depth range, adjust the height of the first focal depth according to a preset step distance along the direction close to the maximum area depth value of the field of view depth range to obtain the second focal depth.

[0095] In this embodiment, based on whether the focal depth range obtained from the first focal depth and the depth parameters covers the maximum area depth value of the field of view depth range, the height adjustment direction of the first focal depth can be determined to obtain a second focal depth that is closer to the maximum area depth value of the field of view depth range than the first focal depth. This allows the new focal depth range obtained based on the first focal depth, the obtained second focal depth, and the depth parameters to be accurately extended along the direction close to the maximum area depth value of the field of view depth range.

[0096] In a feasible embodiment, step S5: acquiring images based on the focal plane depths of several fields of view of each camera to obtain multiple focal plane microscopic images of each camera's field of view includes:

[0097] S51: Based on the focal length of the microscope, obtain the heights of several objective lenses corresponding to the depths of several field focal planes of the microscope.

[0098] The objective lens height is defined as the sum of the focal plane depth and the focal length of the microscope. The objective lens height is the Z-axis parameter of the objective lens in the XYZ three-dimensional coordinate system of the microscope.

[0099] S52: Based on the field coordinates of the camera's field of view and the heights of the plurality of objective lenses, drive the microscope to acquire images of the camera's field of view, thereby obtaining multiple focal plane microscopic images of the camera's field of view.

[0100] In this embodiment, the microscope is driven to acquire images of the camera's field of view by using several objective lens heights obtained from the focal length of the microscope and the focal plane depths of the several fields of view, thereby accurately obtaining multiple focal plane microscopic images of the camera's field of view.

[0101] In a feasible embodiment, step S6: obtaining a panoramic depth-fused image corresponding to the target sample based on multiple focal plane microscopic images from the multiple camera fields of view includes:

[0102] S61: Perform grayscale and filtering processing on the multiple focal plane microscopic images of the multiple camera fields of view to obtain multiple focal plane microscopic grayscale images of the multiple camera fields of view.

[0103] This process involves performing a BGR to grayscale color space conversion on the current focal plane microscopic image, calculating the Sobel gradient magnitudes in the X and Y directions, and finally applying Gaussian filtering for smoothing to obtain the corresponding focal plane microscopic grayscale image.

[0104] Step S61 can be run via GPU to improve the efficiency of acquiring the focal plane microscopic grayscale image.

[0105] S62: Generate a focal plane index map of the multiple camera fields of view based on the pixel gradient values ​​of the multiple focal plane microscopic grayscale images; the focal plane index map records the pixel with the maximum pixel gradient value and the image index of the multiple focal plane microscopic images.

[0106] Step S62 can be implemented in parallel using a CPU. For example, OpenMP can be used to compare the gradient values ​​of all images at each pixel location in parallel and record the image index corresponding to the maximum gradient value to generate a pixel-level optimal focal plane index map. OpenMP is a framework for shared-memory parallel programming that supports C, C++, and Fortran languages ​​and achieves program parallelization through compiler instructions.

[0107] S63: Based on the focal plane index map, pixel values ​​are extracted and fused from multiple focal plane microscopic images of the multiple camera fields of view to obtain the panoramic depth fused image.

[0108] The index map records the original image number corresponding to the optimal sharpness of each pixel. During fusion, the pixel value is directly extracted from the corresponding image based on the index map, ensuring that each pixel of the final fused image comes from the sharpest focal plane micrograph.

[0109] In this embodiment, pixel values ​​are extracted and fused from multiple focal plane microscopic images of the multiple camera fields of view according to the focal plane index map. This ensures that each pixel of the final fused image comes from the clearest focal plane microscopic image, thereby obtaining a high-resolution panoramic depth-of-view fused image.

[0110] In one feasible embodiment, a plurality of ranging sensors are provided next to the objective lens of the microscope; step S2: acquiring the regional depth values ​​of the plurality of field-of-view areas includes:

[0111] When acquiring images using the objective lens based on the depth of several focal planes of the current camera's field of view, the range sensor detects the regional depth values ​​of several fields of view regions of other cameras' fields of view.

[0112] Other camera fields of view can be camera fields of view near the current camera field of view, such as the next camera field of view, or camera fields of view in another row.

[0113] Please see Figure 4 The second embodiment of this application provides a depth information-guided multifocal plane microscopic imaging device 100, comprising:

[0114] The field of view region division module 101 is used to divide the multiple camera fields of view of the target sample into regions, and obtain several field of view regions included in each of the camera fields of view.

[0115] The region depth value acquisition module 102 is used to acquire the region depth values ​​of the plurality of field-of-view regions;

[0116] The field of view depth range acquisition module 103 is used to acquire the field of view depth range of the corresponding camera field of view based on the regional depth values ​​of the plurality of field of view regions.

[0117] The field of view focal plane depth acquisition module 104 is used to acquire several field of view focal plane depths of each of the camera fields of view based on the field of view depth range of each camera field of view and the depth of field parameters of the microscope device.

[0118] The focal plane microscopic image acquisition module 105 is used to acquire images based on the focal plane depth of several fields of view of each of the camera fields of view, and obtain multiple focal plane microscopic images of each of the camera fields of view.

[0119] The panoramic depth fusion image acquisition module 106 is used to obtain a panoramic depth fusion image corresponding to the target sample based on multiple focal plane microscopic images of the multiple camera fields of view.

[0120] It should be noted that the depth information-guided multifocal plane microscopy imaging device 100 provided in the second embodiment of this application is only illustrated by the above-described division of functional modules when executing the depth information-guided multifocal plane microscopy imaging method. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the depth information-guided multifocal plane microscopy imaging device 100 provided in the second embodiment of this application and the depth information-guided multifocal plane microscopy imaging method of the first embodiment of this application belong to the same concept, and the implementation process is detailed in the method embodiment, which will not be repeated here.

[0121] A third aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the depth information-guided multifocal plane microscopy imaging method as described above.

[0122] The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0123] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0124] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function selected in one or more boxes.

[0125] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function selected in one or more boxes.

[0126] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0127] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0128] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0129] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0130] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A depth-information-guided multifocal plane microscopic imaging method, characterized in that, include: The target sample is divided into multiple camera fields of view to obtain several field of view regions included in each camera field of view; Collect the regional depth values ​​of the aforementioned several field-of-view regions; Based on the regional depth values ​​of the plurality of field of view regions, the field of view depth range of the corresponding camera field of view is obtained; Based on the field of view depth range of each camera field of view and the depth of field parameters of the microscope, several field of view focal plane depths of each camera field of view are obtained. Image acquisition is performed based on the depth of several focal planes of each camera's field of view to obtain multiple focal plane microscopic images of each camera's field of view. A panoramic depth-fusion image corresponding to the target sample is obtained based on multiple focal plane microscopic images from the multiple camera fields of view.

2. The depth information-guided multifocal plane microscopic imaging method according to claim 1, characterized in that, The step of obtaining the field-of-view depth range of the corresponding camera field of view based on the regional depth values ​​of the plurality of field-of-view regions includes: Obtain the maximum and minimum region depth values ​​corresponding to the field of view of each camera; The field of view depth range of the corresponding camera field of view is obtained based on the maximum region depth value and the minimum region depth value.

3. The depth information-guided multifocal plane microscopic imaging method according to claim 1, characterized in that, The step of obtaining several focal plane depths of each camera's field of view based on the field of view depth range of each camera and the depth-of-field parameters of the microscope equipment includes: Obtain the first focal plane depth within the field of view depth range; If the focal depth range obtained based on the first focal depth and the depth of field parameters covers the field of view depth range, the first focal depth is determined as the field of view focal depth of the corresponding camera field of view. If the focal depth range obtained based on the first focal depth and the depth of field parameter fails to cover the field of view depth range, the height of the first focal depth is adjusted according to a preset step distance to obtain a second focal depth; wherein, the preset step distance is less than or equal to the depth of field parameter. The focal depth range is updated based on the latest second focal depth and the depth of field parameters. If the updated focal depth range covers the field of view depth range, the first focal depth and the obtained second focal depth are determined as the field of view focal depth of the corresponding camera field of view. If the updated focal depth range fails to cover the field of view depth range, a new second focal depth is obtained based on the height of the second focal depth according to the preset step size, until the focal depth range updated based on the first focal depth, the obtained second focal depth, and the depth parameters covers the field of view depth range. The first focal depth and the obtained second focal depth are then determined as the field of view focal depth of the corresponding camera field of view.

4. The depth information-guided multifocal plane microscopic imaging method according to claim 3, characterized in that, The step of adjusting the height of the first focal plane depth according to a preset step size to obtain a second focal plane depth if the focal plane depth range obtained based on the first focal plane depth and the depth of field parameters fails to cover the field of view depth range includes: If the focal depth range obtained based on the first focal depth and the depth of field parameters fails to cover the minimum area depth value of the field of view depth range, the height of the first focal depth is adjusted according to a preset step distance along the direction close to the minimum area depth value of the field of view depth range to obtain the second focal depth.

5. The depth information-guided multifocal plane microscopic imaging method according to claim 3, characterized in that, The step of adjusting the height of the first focal plane depth according to a preset step size to obtain a second focal plane depth if the focal plane depth range obtained based on the first focal plane depth and the depth of field parameters fails to cover the field of view depth range includes: If the focal depth range obtained based on the first focal depth and the depth of field parameters fails to cover the maximum area depth value of the field of view depth range, the height of the first focal depth is adjusted according to a preset step distance along the direction close to the maximum area depth value of the field of view depth range to obtain the second focal depth.

6. The depth information-guided multifocal plane microscopic imaging method according to claim 1, characterized in that, The step of acquiring images based on the depth of several focal planes of each camera's field of view to obtain multiple focal plane microscopic images of each camera's field of view includes: Based on the focal length of the microscope, obtain several objective lens heights corresponding to the several focal plane depths of the microscope. Based on the field-of-view coordinates of the camera and the heights of the plurality of objective lenses, the microscope is driven to acquire images of the camera's field of view, thereby obtaining multiple focal plane microscopic images of the camera's field of view.

7. The depth information-guided multifocal plane microscopic imaging method according to claim 1, characterized in that, The step of obtaining a panoramic depth-fused image corresponding to the target sample based on multiple focal plane microscopic images from the multiple camera fields of view includes: The multiple focal plane microscopic images of the multiple camera fields of view are subjected to grayscale conversion and filtering to obtain multiple focal plane microscopic grayscale images of the multiple camera fields of view. Based on the pixel gradient values ​​of the multiple focal plane microscopic grayscale images, a focal plane index map of the multiple camera fields of view is generated; the focal plane index map records the pixel with the maximum pixel gradient value and the image index of the multiple focal plane microscopic images; Based on the focal plane index map, pixel values ​​are extracted and fused from multiple focal plane microscopic images of the multiple camera fields of view to obtain the panoramic depth fused image.

8. The depth information-guided multifocal plane microscopic imaging method according to claim 1, characterized in that, The microscope apparatus has several range sensors located next to its objective lens; the step of acquiring the regional depth values ​​of the several fields of view includes: When acquiring images using the objective lens based on the depth of several focal planes of the current camera's field of view, the range sensor detects the regional depth values ​​of several fields of view regions of other cameras' fields of view.

9. A depth-information-guided multifocal plane microscopic imaging device, characterized in that, include: The field of view region division module is used to divide the multiple camera fields of view of the target sample into regions, and obtain several field of view regions included in each camera field of view. The region depth value acquisition module is used to acquire the region depth values ​​of the several field-of-view regions; The field of view depth range acquisition module is used to acquire the field of view depth range of the corresponding camera field of view based on the regional depth values ​​of the plurality of field of view regions. The field-of-view focal plane depth acquisition module is used to acquire several field-of-view focal plane depths of each of the camera fields of view based on the field-of-view depth range of each camera field of view and the depth-of-field parameters of the microscope device. The focal plane microscopic image acquisition module is used to acquire images based on several focal plane depths of each camera's field of view, thereby obtaining multiple focal plane microscopic images of each camera's field of view. The panoramic depth fusion image acquisition module is used to obtain a panoramic depth fusion image corresponding to the target sample based on multiple focal plane microscopic images of the multiple camera fields of view.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, it implements the steps of the depth information-guided multifocal plane microscopy imaging method as described in any one of claims 1 to 8.

Images