A fusion method of dual-channel microscope camera and particle images

Abstract

The present invention discloses a method for fusing a dual-channel microscope camera and particle images. The method includes: upon detecting that a sample to be tested is placed at an imaging position, illuminating the sample to be tested with a dual-color correction light source to obtain a brightfield image and a fluorescent image; extracting the identification feature coordinates in the brightfield image and the fluorescent image respectively; and calculating the geometric transformation coefficient corresponding to the fluorescent image based on the identification feature coordinates. When it is detected that the sample to be tested includes particles, illuminating the sample to be tested with a dual-color correction light source and a fluorescent excitation light source to obtain a brightfield particle image and a fluorescent particle image. Correcting the fluorescent particle image based on the geometric transformation coefficient to obtain a fluorescent correction image, and fusing the fluorescent correction image with the brightfield particle image to generate a particle fusion image. This method achieves image correction and image fusion without the need for particles as reference objects, thereby improving the fusion accuracy and image quality of the particle fusion image.

Description

Technical Field

[0001] The present invention relates to the field of particle fluorescence microscopy technology, and in particular to a method for fusing a dual-channel microscope camera and particle images. Background Art

[0002] With the continuous advancement of science and technology, microscopic imaging is being used to qualitatively and quantitatively study particle morphology. Currently, the primary method is to acquire images through microscopic imaging and further process and analyze the image data. During the microscopic image capture and acquisition process, a variety of imaging methods are available, including fluorescence microscopy, darkfield microscopy, brightfield microscopy, phase contrast microscopy, and interference contrast microscopy.

[0003] In fluorescence imaging equipment, two imaging optical path channels need to be used simultaneously to collect bright field and fluorescence images of particles in order to obtain the morphological structure and material content information of the particles. Due to the assembly deviation in the two imaging optical path channels, the images obtained by the cameras in the two imaging optical path channels have deviations and cannot be directly fused and used. An image processing program is required to process and match the common features of the two images to calculate the corresponding deviation values, and then process the camera images of the two channels for fusion.

[0004] Existing fluorescence imaging equipment usually uses particles as the common feature for image matching. After the particles are excited, they become fluorescent, and the fluorescence image of the particles is obtained after being captured by a fluorescence camera. However, the particles may not always be static. Due to differences in their morphology and quantity, irregular movement, or merging of particles, the deviation values between the bright field image and the fluorescence image change, thereby reducing the accuracy of image fusion. Summary of the Invention

[0005] The present invention provides a method for fusing a dual-channel microscope camera and particle images, which solves the technical problem that when existing fluorescence imaging equipment fuses bright-field images and fluorescence images of particles, the deviation values of the two images change due to differences in their morphology and quantity, irregular movement, or merging between particles, resulting in a decrease in the accuracy of image fusion.

[0006] The present invention provides a dual-channel microscope camera, comprising an objective lens, a first beam splitter, a second beam splitter, a tube lens, a third beam splitter, a second light source filter, and a bright field camera, which are sequentially arranged along a first optical axis;

[0007] and a two-color correction light source arranged along a second optical axis, the second optical axis being perpendicular to the first optical axis and intersecting the first beam splitter;

[0008] and a fluorescence excitation light source arranged along a third optical axis, wherein the third optical axis is perpendicular to the first optical axis and intersects the second beam splitter;

[0009] and a first light source filter and a fluorescent camera sequentially arranged along a fourth optical axis, wherein the fourth optical axis is perpendicular to the first optical axis and intersects the third beam splitter.

[0010] Optionally, the light source assembly includes a two-color correction light source;

[0011] The two-color correction light source includes a first correction light source and a second correction light source.

[0012] Optionally, the light source assembly includes a first correction light source, a second correction light source and a fourth beam splitter;

[0013] The first correction light source and the fourth beam splitter are arranged along the second optical axis;

[0014] The second correction light source is arranged along a fifth optical axis, the fifth optical axis is perpendicular to the second optical axis and intersects the fourth beam splitter;

[0015] The wavelength range of the transmitted light of the fourth beam splitter is 620nm-680nm, and the wavelength range of the reflected light is 460nm-570nm.

[0016] Optionally, the wavelength range of the transmitted light of the first beam splitter is 400nm-650nm, and the wavelength range of the reflected light is 500nm-650nm;

[0017] The second beam splitter has a transmission light wavelength range of 510nm-750nm and a reflection light wavelength range of 400nm-500nm;

[0018] The wavelength range of the transmitted light of the third beam splitter is 620nm-680nm, and the wavelength range of the reflected light is 460nm-570nm.

[0019] Optionally, the transmittance wavelength range of the first light source filter is 525nm±10nm;

[0020] The light transmission wavelength range of the second light source filter is 625nm±15nm.

[0021] Optionally, the wavelength range of the first illumination light output by the first correction light source is 630nm±15nm;

[0022] The wavelength range of the second illumination light output by the second correction light source is 525nm±15nm;

[0023] The excitation light wavelength range output by the fluorescent excitation light source is 488nm±5nm.

[0024] The present invention also provides a particle image fusion method for a processor communicatively connected to any of the above-mentioned dual-channel microscope cameras, the method comprising:

[0025] When it is detected that a sample to be tested is placed at the imaging position, the light source component is called to illuminate the sample to be tested and obtain a bright field image and a fluorescence image;

[0026] extracting the coordinates of the identification features in the bright field image and the fluorescent image respectively;

[0027] Calculating the geometric transformation coefficient corresponding to the fluorescent image according to the identification feature coordinates;

[0028] When it is detected that the sample to be tested includes particles, calling the light source assembly and the fluorescent excitation light source to illuminate the sample to be tested to obtain a bright field particle image and a fluorescent particle image;

[0029] Correcting the fluorescent particle image according to the geometric transformation coefficient to obtain a fluorescence correction image;

[0030] The fluorescence correction image and the bright field particle image are fused to generate a particle fusion image.

[0031] Optionally, when it is detected that a sample to be tested is placed at the imaging position, the step of calling the light source assembly to illuminate the sample to be tested and acquiring a bright field image and a fluorescence image includes:

[0032] When it is detected that a sample to be measured is placed at the imaging position, the light source assembly is called to sequentially output the first illumination light and the second illumination light, which are reflected to the objective lens through the first beam splitter;

[0033] Focusing the first irradiation light and the second irradiation light respectively onto the sample to be tested through the objective lens;

[0034] When the sample to be tested is irradiated by the first irradiation light, a bright field image corresponding to the sample to be tested is acquired by a bright field camera;

[0035] When the sample to be tested is irradiated by the second irradiation light, a fluorescence image corresponding to the sample to be tested is acquired by a fluorescence camera.

[0036] Optionally, the step of respectively extracting the identification feature coordinates in the bright field image and the fluorescent image includes:

[0037] Locating identification features in the bright field image and the fluorescent image respectively;

[0038] Extracting edges of the identification features and performing shape fitting to obtain a fitting area;

[0039] The region center coordinates of the fitting region are extracted as identification feature coordinates.

[0040] Optionally, the identification feature coordinates include first feature coordinates and second feature coordinates, and the geometric transformation coefficients may include image scaling coefficients, image rotation coefficients, and image offset coefficients; and the step of calculating the geometric transformation coefficients corresponding to the fluorescence image based on the identification feature coordinates includes:

[0041] calculating a Euclidean distance between a second characteristic coordinate corresponding to the bright field image and a first characteristic coordinate corresponding to the bright field image to obtain a first characteristic distance;

[0042] calculating a Euclidean distance between a second characteristic coordinate corresponding to the fluorescent image and a first characteristic coordinate corresponding to the fluorescent image to obtain a second characteristic distance;

[0043] calculating a ratio between the first characteristic distance and the second characteristic distance to obtain an image scaling factor corresponding to the fluorescence image;

[0044] Determining an image rotation coefficient corresponding to the fluorescent image according to a preset dot product formula in combination with the first characteristic distance and the second characteristic distance;

[0045] performing an affine transformation on the first feature coordinates corresponding to the fluorescent image according to the image rotation coefficient and the image scaling coefficient to obtain updated identification feature coordinates;

[0046] The difference between the updated identification feature coordinates and the first feature coordinates corresponding to the fluorescent image is calculated to obtain an image shift coefficient.

[0047] Optionally, the step of determining the image rotation coefficient corresponding to the fluorescent image according to a preset dot product formula in combination with the first characteristic distance and the second characteristic distance includes:

[0048] Calculating a first difference between a second characteristic coordinate corresponding to the bright field image and a first characteristic coordinate corresponding to the bright field image to obtain a first vector;

[0049] Calculating a second difference between a second characteristic coordinate corresponding to the bright field image and a first characteristic coordinate corresponding to the bright field image to obtain a second vector;

[0050] Calculating a dot product of the first vector and the second vector;

[0051] Calculating a distance product between the first characteristic distance and the second characteristic distance;

[0052] An arc cosine value of a ratio between the dot product and the distance multiplication value is calculated to obtain an image rotation coefficient corresponding to the fluorescence image.

[0053] Optionally, the step of performing an affine transformation on the first feature coordinates corresponding to the fluorescent image according to the image rotation coefficient and the image scaling coefficient to obtain updated identification feature coordinates includes:

[0054] Constructing an affine transformation matrix using the image rotation coefficient and the image scaling coefficient;

[0055] The affine transformation matrix is used to perform affine transformation on the first characteristic coordinates corresponding to the fluorescent image to obtain updated identification characteristic coordinates.

[0056] It can be seen from the above technical solutions that the present invention has the following advantages:

[0057] The present invention provides a dual-channel microscope camera, comprising an objective lens, a first beamsplitter, a second beamsplitter, a tube lens, a third beamsplitter, a second light source filter, and a brightfield camera, arranged in sequence along a first optical axis; a light source assembly arranged along a second optical axis, the second optical axis being perpendicular to the first optical axis and intersecting the first beamsplitter; a fluorescence excitation light source arranged along a third optical axis, the third optical axis being perpendicular to the first optical axis and intersecting the second beamsplitter; and the first light source filter and the fluorescence camera, arranged in sequence along a fourth optical axis, the fourth optical axis being perpendicular to the first optical axis and intersecting the third beamsplitter. By using the light source assembly to illuminate a sample to be tested, both panoramic brightfield and fluorescence images of the sample to be tested can be acquired, regardless of whether it contains particles or not. Furthermore, the microscope camera structure is simplified, effectively reducing operating costs.

[0058] A particle image fusion method is also provided. When a sample to be tested is detected at the imaging position, the light source assembly is invoked to illuminate the sample to obtain a brightfield image and a fluorescence image. The coordinates of the identification features in the brightfield image and the fluorescence image are extracted respectively. The geometric transformation coefficient corresponding to the fluorescence image is calculated based on the identification feature coordinates. When the sample to be tested is detected to contain particles, the light source assembly and the fluorescence excitation light source are invoked to illuminate the sample to obtain a brightfield particle image and a fluorescent particle image. The fluorescent particle image is corrected based on the geometric transformation coefficient to obtain a corrected fluorescence image. The corrected fluorescence image and the brightfield particle image are fused to generate a fused particle image. This achieves image correction and image fusion without the need for particles as reference objects, thereby improving the fusion accuracy and image quality of the fused particle image. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0060] Figure 1 A schematic structural diagram of a dual-channel microscope camera provided in Example 1 of the present invention;

[0061] Figure 2 A schematic structural diagram of a dual-channel microscope camera equipped with a dual-color correction light source provided by an embodiment of the present invention;

[0062] Figure 3 Schematic diagram of the structure of a dual-color correction light source in an embodiment of the present invention;

[0063] Figure 4 A schematic structural diagram of a dual-channel microscope camera without a dual-color correction light source provided by an embodiment of the present invention;

[0064] Figure 5 A flowchart of a particle image fusion method provided in the second embodiment of the present invention;

[0065] Figure 6a Schematic diagram of extracting identification feature coordinates of an identification feature in a bright field image according to an embodiment of the present invention;

[0066] Figure 6b Schematic diagram of extracting identification feature coordinates of an identification feature in a fluorescence image according to an embodiment of the present invention;

[0067] Figure 7 A flowchart of the steps of a geometric transformation parameter determination process provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0068] An embodiment of the present invention provides a method for fusing a dual-channel microscope camera and particle images, which is used to solve the technical problem that when existing fluorescence imaging equipment fuses the bright field image and fluorescence image of particles, the deviation values of the two images will change due to differences in their morphology and quantity, irregular movement, or merging between particles, resulting in a decrease in the accuracy of image fusion.

[0069] In order to make the purpose, features, and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0070] See also Figure 1 , Figure 1 This is a structural schematic diagram of a dual-channel microscope camera provided in Example 1 of the present invention.

[0071] The present invention provides a dual-channel microscope camera, comprising an objective lens 200, a first beam splitter 500, a second beam splitter 600, a tube lens 800, a third beam splitter 700, a second light source filter 900, and a bright field camera 1100, which are sequentially arranged along a first optical axis.

[0072] and a light source assembly 30 disposed along a second optical axis, the second optical axis being perpendicular to the first optical axis and intersecting the first beam splitter 500;

[0073] and a fluorescence excitation light source 400 arranged along a third optical axis, the third optical axis being perpendicular to the first optical axis and intersecting the second beam splitter 600;

[0074] The first light source filter 1000 and the fluorescence camera 1200 are sequentially arranged along a fourth optical axis. The fourth optical axis is perpendicular to the first optical axis and intersects the third beam splitter 700 .

[0075] A beamsplitter is an optical component used to split incident light into two distinct beams at a specified ratio. Typically used at a 45° angle, with one side coated with a beam splitter coating and the other with an antireflection coating, it separates incident light into reflected and transmitted light. In this embodiment, it includes a first beamsplitter 500, a second beamsplitter 600, and a third beamsplitter 700, each configured to reflect or transmit light of different wavelengths.

[0076] The optical axis refers to the centerline of the light beam (light column) and is also the axis of symmetry of each channel in a dual-channel microscope camera. The light beam rotates around this axis without any change in optical properties. In this embodiment, it includes a first optical axis, a second optical axis, a third optical axis, and a fourth optical axis.

[0077] The sample 100 to be tested is mounted at the imaging position of the objective lens 200 and can include consumables and particles, or the consumable alone. The consumable is a container for the particles, does not contain fluorescent material, and has at least two specific shaped features, such as a plus sign and a circle, engraved on its surface. Particles are the subject of microscopic imaging and can emit fluorescence when illuminated by a fluorescent excitation light source, such as microparticles or cells.

[0078] The sample 100 to be tested may also include a cell suspension or a particle suspension. The particles may be insoluble objects, such as living cells, gel particles, oil droplets, particles filled with liquid, or solid particles. For example, the liquid may be a buffer suspension containing living cells.

[0079] The objective lens 200 is installed between the sample 100 to be tested and the first beam splitter 500 , and is used to converge the light beam to illuminate the sample 100 to be tested and image it on the fluorescence camera 1200 or the bright field camera 1100 .

[0080] The tube lens 800 is installed between the second beam splitter 600 and the third beam splitter 700 , and is used to cooperate with the objective lens 200 to image the sample 100 to be tested on the fluorescence camera 1200 or the bright field camera 1100 .

[0081] See also Figure 1 The dual-channel microscope camera starts with the sample 100 to be tested and is sequentially arranged along the first optical axis with an objective lens 200, a first beamsplitter 500, a second beamsplitter 600, a tube lens 800, a third beamsplitter 700, a second light source filter 900, and a brightfield camera 1100. The light source assembly 30 is arranged along the second optical axis and outputs a light beam that intersects the first optical axis at the first beamsplitter 500. The light beam is then reflected by the first beamsplitter 500 to the objective lens 200 for focusing. The fluorescence excitation light source 400 is arranged along the third optical axis and outputs a light beam that intersects the first optical axis at the second beamsplitter 600. Because the first beamsplitter 500 has different wavelength limits for transmission and reflection, the light beam reflected by the second beamsplitter 600 to the first beamsplitter 500 is transmitted to the objective lens 200 for focusing. In order to realize a dual-channel microscope camera, a fourth optical axis is divided at the third dichroic plate 700 , and a first light source filter 1000 and a fluorescence camera 1200 are sequentially arranged along the fourth optical axis.

[0082] The bright field camera 1100 is installed behind the second light source filter 900, receives the image formed by the sample to be tested 100, converts the sensing light signal into a digital circuit signal and outputs the image. The fluorescence camera 1200 is installed behind the first light source filter 1000, receives the image formed by the sample to be tested 100, converts the sensing light signal into a digital circuit signal and outputs the image.

[0083] See also Figure 2 and Figure 3 , the light source assembly 30 includes a two-color correction light source 300;

[0084] The dual-color correction light source 300 includes a first correction light source 301 and a second correction light source 302 .

[0085] In this embodiment of the present invention, after the dual-channel microscope camera detects that the sample 100 is placed at the imaging position, the first correction light source 301 in the dual-color correction light source 300 is activated to output a first illumination light with a wavelength range of 630nm±15nm. After being reflected by the first beam splitter 500 and then reflected by the objective lens 200, the objective lens 200 focuses the light onto the sample 100. At the same time, the brightfield camera 1100 is activated to scan and capture the sample 100. The optical signal obtained by the induction scanning is converted into a digital electrical signal and outputs a brightfield image to the processor for further image fusion.

[0086] Similarly, the second calibration light source 302 in the dual-color calibration light source 300 is activated to output a second illumination light with a wavelength range of 525 nm ± 15 nm. After being reflected by the first beam splitter 500 and then by the objective lens 200, the second illumination light is focused and illuminated onto the sample 100. The fluorescence camera 1200 is activated to scan and capture the sample 100. The optical signal obtained by the sensing scan is converted into a digital electrical signal and output as a fluorescence image to the processor for further image fusion.

[0087] See also Figure 3 , Figure 3 FIG. 3 is a schematic structural diagram of a light source assembly 30 in an embodiment of the present invention.

[0088] In an embodiment of the present invention, the light source assembly 30 may include a first calibration light source 301 and a second calibration light source 302 , and the switching illumination of the first illumination light and the second illumination light on the sample 100 to be tested is achieved by switching the light sources.

[0089] The main wavelength of the first correction light source may be 630 nm, and the main wavelength of the second correction light source may be 525 nm.

[0090] See also Figure 4 , Figure 4 Another structure of the light source assembly 30 of the present application is shown.

[0091] The light source assembly 30 includes a first correction light source 301, a second correction light source 302 and a fourth beam splitter 303;

[0092] The first calibration light source 301 and the fourth beam splitter 303 are arranged along the second optical axis;

[0093] The second correction light source 302 is disposed along a fifth optical axis. The fifth optical axis is perpendicular to the second optical axis and intersects the fourth beam splitter 303 .

[0094] In an embodiment of the present invention, the light source assembly 30 may include a first correction light source 301, a second correction light source 302, and a fourth beam splitter 303. By separating the two light sources, the influence between the light beams is reduced and the imaging quality is improved. Upon receiving the signal emitted by the light beam, the first correction light source 301 outputs a first illumination light with a wavelength range of 630nm±15nm. After transmitting through the fourth beam splitter 303, the first illumination light is reflected by the first beam splitter 500 to the objective lens 200, which is then focused by the objective lens 200 and irradiated onto the sample to be tested 100. At the same time, the bright field camera 1100 is activated to scan and photograph the sample to be tested 100, converting the optical signal obtained by the induction scanning into a digital electrical signal and outputting a bright field image to the processor, awaiting further image fusion.

[0095] Upon receiving the signal from the light beam emission, the second calibration light source 302 emits a second illumination light with a wavelength range of 525 nm ± 15 nm along the fifth optical axis. This light is reflected by the fourth beam splitter 303 and the first beam splitter 500 to the objective lens 200, where it is focused and irradiated onto the sample 100. The fluorescence camera 1200 is activated to scan and capture the sample 100, converting the optical signal from the sensing scan into a digital electrical signal and outputting the fluorescence image to the processor for further image fusion.

[0096] Optionally, the wavelength range of the transmitted light of the first beam splitter 500 is 400nm-650nm, and the wavelength range of the reflected light is 500nm-650nm;

[0097] The second beam splitter 600 transmits light in a wavelength range of 510nm-750nm, and reflects light in a wavelength range of 400nm-500nm;

[0098] The third beam splitter 700 transmits light in a wavelength range of 620nm-680nm, and reflects light in a wavelength range of 460nm-570nm;

[0099] The fourth beam splitter 303 transmits light in a wavelength range of 620 nm to 680 nm, and reflects light in a wavelength range of 460 nm to 570 nm.

[0100] In an embodiment of the present invention, the first beam splitter 500, the second beam splitter 600 and the third beam splitter 70 will exhibit different optical properties in different wavelength ranges. For example, the first beam splitter 500 will transmit the light beam in the wavelength range of 400nm-650nm and reflect the light beam in the wavelength range of 500nm-650nm. The second beam splitter or the third beam splitter is similar and will not be repeated here.

[0101] Optionally, the light transmission wavelength range of the first light source filter 1000 is 525 nm ± 10 nm;

[0102] The light transmission wavelength range of the second light source filter 900 is 625 nm ± 15 nm.

[0103] Optionally, the wavelength range of the first illumination light output by the first correction light source is 630 nm ± 15 nm;

[0104] The wavelength range of the second illumination light output by the second correction light source is 525nm±15nm;

[0105] The wavelength range of the excitation light output by the fluorescence excitation light source is 488nm±5nm.

[0106] In this embodiment, after detecting particles in the sample 100, the fluorescence excitation light source is activated, outputting excitation light with a main wavelength within the wavelength range of 488 nm ± 5 nm along the third optical axis. After being reflected by the second beam splitter 600, the excitation light is transmitted through the first beam splitter 500 and focused by the objective lens 200 onto the sample 100, thereby exciting the particles. The fluorescence camera 1200 then scans and captures the sample 100. The optical signals obtained by the sensing scan are converted into digital electrical signals and output to the processor, providing the data foundation for subsequent image fusion.

[0107] It should be noted that the first correction light source can be a red correction light source, the second correction light source can be a green correction light source, the first illumination light can be red light, and the second illumination light can be green light. The first light source filter is used to transmit the first illumination light, such as a red light filter. The second light source filter is used to transmit the second illumination light, such as a green light filter. The fluorescence excitation light source is used to output a specific excitation light that can excite the particles to emit fluorescence.

[0108] In an embodiment of the present invention, a dual-channel microscope camera is provided, comprising an objective lens, a first beamsplitter, a second beamsplitter, a tube lens, a third beamsplitter, a second light source filter, and a brightfield camera, arranged in sequence along a first optical axis; a light source assembly arranged along a second optical axis, the second optical axis being perpendicular to the first optical axis and intersecting the first beamsplitter; a fluorescence excitation light source arranged along a third optical axis, the third optical axis being perpendicular to the first optical axis and intersecting the second beamsplitter; and the first light source filter and a fluorescence camera, arranged in sequence along a fourth optical axis, the fourth optical axis being perpendicular to the first optical axis and intersecting the third beamsplitter. By using the light source assembly to illuminate a sample to be tested, both a panoramic brightfield image and a fluorescence image of the sample to be tested can be obtained, regardless of whether it contains particles or not. Furthermore, the microscope camera structure is simplified, effectively reducing its operating costs.

[0109] See also Figure 5 , Figure 5 This is a flowchart of the steps of a particle image fusion method provided in the second embodiment of the present invention.

[0110] An embodiment of the present invention provides a particle image fusion method, which is applied to a processor communicatively connected to a dual-channel microscope camera corresponding to the first embodiment. The method includes:

[0111] Step 401: When it is detected that a sample to be tested is placed at an imaging position, a light source component is called to illuminate the sample to be tested and obtain a bright field image and a fluorescence image.

[0112] In an embodiment of the present invention, since there may be many types of samples to be tested and errors may occur when acquiring a particle fusion image, when the processor detects that a sample to be tested is placed at the imaging position, the light source assembly may be called to illuminate the sample to be tested to provide a fluorescence scene and a bright field scene.

[0113] At the same time, a fluorescence image corresponding to the sample to be tested is obtained through a fluorescence camera, and a bright field image corresponding to the sample to be tested is obtained through a bright field camera to obtain the data basis for subsequent image calibration.

[0114] Optionally, step 401 may include the following sub-steps:

[0115] When it is detected that a sample to be measured is placed at the imaging position, the light source assembly is called to sequentially output the first illumination light and the second illumination light, which are reflected to the objective lens through the first beam splitter;

[0116] The first irradiation light and the second irradiation light are focused and irradiated onto the sample to be measured through the objective lens respectively;

[0117] When the sample to be tested is irradiated by the first irradiation light, a bright field image corresponding to the sample to be tested is acquired by a bright field camera;

[0118] When the sample to be tested is irradiated by the second irradiation light, a fluorescence image corresponding to the sample to be tested is acquired by the fluorescence camera.

[0119] In one embodiment of the present invention, when a sample to be tested is detected to be placed at the imaging position, the light source assembly can be called to sequentially output a first illumination light and a second illumination light, which are reflected by a first beam splitter to an objective lens. The objective lens then focuses the first illumination light and the second illumination light and irradiates the sample to be tested. When the sample to be tested is illuminated by the first illumination light, a brightfield camera captures the image of the sample to be tested on the objective lens and converts it into a digital circuit signal to obtain a corresponding brightfield image. When the sample to be tested is illuminated by the second illumination light, a fluorescence camera captures the image of the sample to be tested on the objective lens and converts it into a digital circuit signal to obtain a corresponding fluorescence image.

[0120] Among them, the first irradiation light can obtain a bright field image when irradiating the particles, and the second irradiation light can obtain a fluorescence image when irradiating the particles. The specific adjustment can be based on the type of particles. For example, when the particle type is cells, the first irradiation light can be red light, the second irradiation light can be green light, and the fluorescence excitation light source can be a blue laser light source.

[0121] It should be noted that a variety of methods can be used to detect whether a sample to be tested is placed at the imaging position. For example, a pressable protrusion can be provided at the imaging position. When a sample to be tested is placed at the imaging position, the pressable protrusion is depressed, thereby determining that the sample to be tested is placed at the imaging position. Alternatively, a photoelectric sensor can be used to measure the distance to the imaging aperture at the imaging position. If the imaging aperture is not detected, then the sample to be tested is determined to be placed at the imaging position. The specific implementation method is not limited by the embodiments of the present invention.

[0122] Step 402 : extract the coordinates of the identification features in the bright field image and the fluorescence image respectively.

[0123] Furthermore, step 402 may include the following sub-steps:

[0124] Locate the signature features in brightfield and fluorescence images respectively;

[0125] Extract the edges of the identification features and perform shape fitting to obtain the fitting area;

[0126] The center coordinates of the fitting area are extracted as the identification feature coordinates.

[0127] After acquiring the bright field image and the fluorescence image, the identification features in the bright field image and the fluorescence image can be located respectively, the edges of the representation features can be extracted, and shape fitting processing can be performed to obtain the fitting area, and then the center coordinates of the fitting area can be extracted as the identification feature coordinates.

[0128] like Figure 6a-6b As shown, Figure 6a FIG2 shows a schematic diagram of extracting the coordinates of a marking feature in a bright field image according to an embodiment of the present invention. Figure 6b A schematic diagram of extracting identification feature coordinates of identification features in a fluorescence image according to an embodiment of the present invention is shown.

[0129] Taking the plus sign feature 103 and the circular feature 104 set on the consumable 101 as an example, the plus sign feature 103 and the circular feature 104 can be edge extracted. The edge extraction algorithm is preferably canny. In order to improve the accuracy of the coordinate information, the extracted edges are subjected to circle fitting processing to obtain a circular fitting area. The center coordinates of the area are then extracted from the fitting area to determine as the identification feature coordinates, where the bright field image is Figure 6aThe plus sign feature coordinates W1(x1,y1); the circular feature coordinates W2(x2,y2), the fluorescence image is Figure 6b The plus sign feature FL1 (x1, y1); the circular feature FL2 (x2, y2).

[0130] The sample 100 to be tested may also include residual particles 102 .

[0131] In a specific implementation, there may be multiple types of identification feature coordinates in addition to the plus sign and the circle. After edge extraction of the identification feature, different graphics may be used for subsequent shape fitting, which is not limited in the embodiment of the present invention.

[0132] Step 403: Calculate the geometric transformation coefficients corresponding to the fluorescence image according to the identification feature coordinates.

[0133] See also Figure 7 The identified feature coordinates include the first feature coordinates and the second feature coordinates, and the geometric transformation coefficients may include an image scaling coefficient, an image rotation coefficient, and an image offset coefficient. Step 403 may include the following sub-steps S11-S16:

[0134] S11 . Calculate the Euclidean distance between the second characteristic coordinates corresponding to the bright field image and the first characteristic coordinates corresponding to the bright field image to obtain a first characteristic distance.

[0135] S12 . Calculate the Euclidean distance between the second characteristic coordinate corresponding to the fluorescent image and the first characteristic coordinate corresponding to the fluorescent image to obtain a second characteristic distance.

[0136] In the embodiment of the present invention, the first characteristic distance L1 between the second characteristic coordinate and the first characteristic coordinate in the bright field image and the second characteristic distance L2 between the second characteristic coordinate and the first characteristic coordinate in the fluorescence image can be calculated respectively:

[0137] L1=||W2-W1||

[0138] L2=||FL2-FL1||

[0139] S13. Calculate the ratio between the first characteristic distance and the second characteristic distance to obtain an image scaling factor corresponding to the fluorescence image.

[0140] In the embodiment of the present invention, the image scaling coefficient α corresponding to the fluorescence image is obtained by calculating the ratio between the first characteristic distance and the second characteristic distance:

[0141] α=L1 / L2

[0142] S14 , determining an image rotation coefficient corresponding to the fluorescent image according to a preset dot product formula in combination with the first characteristic distance and the second characteristic distance.

[0143] Furthermore, step S14 may include the following sub-steps:

[0144] Calculating a first difference between a second characteristic coordinate corresponding to the bright field image and a first characteristic coordinate corresponding to the bright field image to obtain a first vector;

[0145] Calculating a second difference between the second characteristic coordinate corresponding to the bright field image and the first characteristic coordinate corresponding to the bright field image to obtain a second vector;

[0146] Calculate the dot product of the first vector and the second vector;

[0147] Calculate the distance product between the first characteristic distance and the second characteristic distance;

[0148] The arc cosine value of the ratio between the dot product and the distance multiplication value is calculated to obtain the image rotation coefficient corresponding to the fluorescence image.

[0149] In the embodiment of the present invention, the first difference between the second characteristic coordinate corresponding to the bright field image and the first characteristic coordinate corresponding to the bright field image is calculated to obtain the first vector Calculate the second difference between the second characteristic coordinates corresponding to the bright field image and the first characteristic coordinates corresponding to the bright field image to obtain the second vector The image rotation coefficient corresponding to the fluorescence image is further calculated using the dot product formula:

[0150]

[0151] S15 . Performing an affine transformation on the first characteristic coordinates corresponding to the fluorescent image according to the image rotation coefficient and the image scaling coefficient to obtain updated identification characteristic coordinates.

[0152] In one example of the present invention, step S15 may include the following sub-steps:

[0153] Construct an affine transformation matrix using image rotation coefficients and image scaling coefficients;

[0154] An affine transformation matrix is used to perform affine transformation on the first characteristic coordinates corresponding to the fluorescent image to obtain updated identification characteristic coordinates.

[0155] In an embodiment of the present invention, after obtaining the image rotation coefficient and the image scaling coefficient, the image rotation coefficient and the image scaling coefficient may be used to construct an affine transformation matrix:

[0156]

[0157] The affine transformation matrix is further used to perform affine transformation on the first feature coordinate corresponding to the fluorescence image to obtain the updated identification feature coordinate FL1′(x′,y′):

[0158]

[0159] S16. Calculate the difference between the updated marker feature coordinates and the first feature coordinates corresponding to the fluorescent image to obtain an image shift coefficient.

[0160] After obtaining the updated identification feature coordinates and the first feature coordinates, the difference between the two is calculated to obtain the image offset coefficient Offset:

[0161] Offset=FL1′-W1

[0162] At this point, the image scaling factor, image rotation factor, and image offset factor are obtained, which can be used to correct the magnification deviation, angle deviation, and position deviation of the fluorescence image, respectively.

[0163] Step 404 : When it is detected that the sample to be tested includes particles, the light source assembly and the fluorescence excitation light source are called to illuminate the sample to be tested to obtain a bright field particle image and a fluorescent particle image.

[0164] In an embodiment of the present invention, if particles are detected in the sample to be tested, the first calibrated light source within the light source assembly can be used again to illuminate the sample to be tested, and a corresponding brightfield particle image can be obtained using the brightfield camera. A fluorescent excitation light source can also be used to illuminate the sample to be tested, and a corresponding fluorescent particle image can be obtained using the fluorescence camera.

[0165] Step 405: Correct the fluorescent particle image according to the geometric transformation coefficient to obtain a fluorescence corrected image.

[0166] Since all geometric transformation coefficients have been obtained in the aforementioned steps S11-S16, the geometric transformation coefficients can now be loaded through the image processing program, and the fluorescent particle image can be corrected to the same size as the bright field image to obtain a fluorescence correction image.

[0167] Step 406 : Fusing the fluorescence correction image and the bright field particle image to generate a particle fusion image.

[0168] In an embodiment of the present invention, the fluorescence correction image and the bright field particle image are matched and fused to generate a particle fusion image.

[0169] For example, image fusion is achieved by selecting fluorescence correction images or bright field particle images in channels of different colors to obtain a particle fusion image.

[0170] In traditional fluorescence imaging equipment, two imaging optical channels are required to simultaneously capture brightfield and fluorescence images of cells to obtain information on the cell's morphological structure and substance content. Due to assembly deviations between the two imaging optical channels, the images captured by the cameras in the two imaging optical channels are deviated and cannot be directly fused. An image processing program is required to match the common features of the two images and calculate the corresponding deviation values, and then process the camera images of the two channels for fusion. Existing fluorescence imaging equipment typically uses cells as the common feature for image matching. Cells are excited by blue light, which emits green light. After being captured by a fluorescence camera, a fluorescence image of the cells is obtained. However, the deviation values between the brightfield and fluorescence images of cells will change due to differences in cell morphology and number, which reduces the accuracy of image fusion.

[0171] To this end, the embodiment of the present application takes particles as cells as an example and provides a cell image fusion method. In this embodiment, after the geometric parameters are calculated through steps 401-403, if it is detected that the sample to be tested includes cells, the red light correction light source in the light source assembly can be called to output red light to illuminate the cells in the sample to be tested, so as to obtain the bright field cell image at this time. After obtaining the bright field cell image, a fluorescent excitation light source is further called, such as a blue laser light source to output blue light to illuminate the sample to be tested, to obtain a fluorescent cell image; the fluorescent cell image is corrected according to the geometric transformation coefficient to obtain a fluorescent correction image; the fluorescent correction image and the bright field cell image are fused to generate a cell fusion image. Since cells are not used in image correction, and image fusion is performed after dual-channel image acquisition is achieved through different light sources in the subsequent correction and fusion process, the accuracy of image fusion is improved.

[0172] In an embodiment of the present invention, when a sample to be tested is detected at the imaging position, the light source assembly is invoked to illuminate the sample to obtain a brightfield image and a fluorescence image. The coordinates of the identification features in the brightfield and fluorescence images are extracted, and the geometric transformation coefficients corresponding to the fluorescence image are calculated based on the identification feature coordinates. When the sample to be tested is detected to contain particles, the light source assembly and the fluorescence excitation light source are invoked to illuminate the sample to obtain a brightfield particle image and a fluorescent particle image. The fluorescent particle image is corrected based on the geometric transformation coefficients to obtain a corrected fluorescence image. The corrected fluorescence image and the brightfield particle image are then fused to generate a fused particle image. This achieves image correction and image fusion without the need for particles as reference objects, thereby improving the fusion accuracy and image quality of the fused particle image.

[0173] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0174] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A particle image fusion method, characterized in that: A processor for communication with a dual-channel microscope camera, wherein the dual-channel microscope camera is used to image a sample to be tested, wherein the sample to be tested includes at least a consumable material, wherein the consumable material is a container that does not contain a fluorescent substance; the dual-channel microscope camera includes an objective lens, a first beamsplitter, a second beamsplitter, a tube lens, a third beamsplitter, a second light source filter, and a bright field camera, which are sequentially arranged along a first optical axis; and a light source assembly arranged along a second optical axis, the second optical axis being perpendicular to the first optical axis and intersecting the first beam splitter; and a fluorescence excitation light source arranged along a third optical axis, wherein the third optical axis is perpendicular to the first optical axis and intersects the second beam splitter; and a first light source filter and a fluorescence camera sequentially arranged along a fourth optical axis, wherein the fourth optical axis is perpendicular to the first optical axis and intersects the third beam splitter; The light source assembly includes a two-color correction light source; The two-color correction light source includes a first correction light source and a second correction light source; Alternatively, the light source assembly includes a first correction light source, a second correction light source and a fourth beam splitter; The first correction light source and the fourth beam splitter are arranged along the second optical axis; The second correction light source is arranged along a fifth optical axis, the fifth optical axis is perpendicular to the second optical axis and intersects the fourth beam splitter; The method comprises: When it is detected that a sample to be tested is placed at the imaging position, the light source component is called to illuminate the sample to be tested and obtain a bright field image and a fluorescence image; Extracting the coordinates of the identification features in the bright field image and the fluorescent image respectively; the identification features to which the identification feature coordinates belong are located on a consumable in the sample to be tested, and the consumable is a container that does not contain a fluorescent substance; Calculating the geometric transformation coefficient corresponding to the fluorescent image according to the identification feature coordinates; When it is detected that the sample to be tested includes particles, calling the light source assembly and the fluorescent excitation light source to illuminate the sample to be tested to obtain a bright field particle image and a fluorescent particle image; Correcting the fluorescent particle image according to the geometric transformation coefficient to obtain a fluorescence correction image; The fluorescence correction image and the bright field particle image are fused to generate a particle fusion image.

2. The method according to claim 1, characterized in that The step of calling the light source assembly to illuminate the sample to be tested and obtain a bright field image and a fluorescence image when it is detected that a sample to be tested is placed at the imaging position includes: When it is detected that a sample to be measured is placed at the imaging position, the light source assembly is called to sequentially output the first illumination light and the second illumination light, which are reflected to the objective lens through the first beam splitter; Focusing the first irradiation light and the second irradiation light respectively onto the sample to be tested through the objective lens; When the sample to be tested is irradiated by the first irradiation light, a bright field image corresponding to the sample to be tested is acquired by a bright field camera; When the sample to be tested is irradiated by the second irradiation light, a fluorescence image corresponding to the sample to be tested is acquired by a fluorescence camera.

3. The method according to claim 1, characterized in that The step of respectively extracting the identification feature coordinates in the bright field image and the fluorescent image comprises: Locating identification features in the bright field image and the fluorescent image respectively; Extracting edges of the identification features and performing shape fitting to obtain a fitting area; The region center coordinates of the fitting region are extracted as identification feature coordinates.

4. The method according to claim 1, wherein The identification feature coordinates include a first feature coordinate and a second feature coordinate, and the geometric transformation coefficients include an image scaling coefficient, an image rotation coefficient, and an image offset coefficient; and the step of calculating the geometric transformation coefficients corresponding to the fluorescent image based on the identification feature coordinates includes: calculating a Euclidean distance between a second characteristic coordinate corresponding to the bright field image and a first characteristic coordinate corresponding to the bright field image to obtain a first characteristic distance; calculating a Euclidean distance between a second characteristic coordinate corresponding to the fluorescent image and a first characteristic coordinate corresponding to the fluorescent image to obtain a second characteristic distance; calculating a ratio between the first characteristic distance and the second characteristic distance to obtain an image scaling factor corresponding to the fluorescence image; Determining an image rotation coefficient corresponding to the fluorescent image according to a preset dot product formula in combination with the first characteristic distance and the second characteristic distance; performing an affine transformation on the first feature coordinates corresponding to the fluorescent image according to the image rotation coefficient and the image scaling coefficient to obtain updated identification feature coordinates; The difference between the updated identification feature coordinates and the first feature coordinates corresponding to the fluorescent image is calculated to obtain an image shift coefficient.

5. The method according to claim 4, characterized in that The step of determining the image rotation coefficient corresponding to the fluorescent image according to a preset dot product formula in combination with the first characteristic distance and the second characteristic distance includes: Calculating a first difference between a second characteristic coordinate corresponding to the bright field image and a first characteristic coordinate corresponding to the bright field image to obtain a first vector; Calculating a second difference between a second characteristic coordinate corresponding to the bright field image and a first characteristic coordinate corresponding to the bright field image to obtain a second vector; Calculating a dot product of the first vector and the second vector; Calculating a distance product between the first characteristic distance and the second characteristic distance; An arc cosine value of a ratio between the dot product and the distance multiplication value is calculated to obtain an image rotation coefficient corresponding to the fluorescence image.

6. The method according to claim 4, characterized in that The step of performing an affine transformation on the first feature coordinates corresponding to the fluorescent image according to the image rotation coefficient and the image scaling coefficient to obtain updated identification feature coordinates includes: Constructing an affine transformation matrix using the image rotation coefficient and the image scaling coefficient; The affine transformation matrix is used to perform affine transformation on the first characteristic coordinates corresponding to the fluorescent image to obtain updated identification characteristic coordinates.

Images