Image processing method and apparatus for flow cytometric microscopy imaging apparatus

Abstract

Image processing method and apparatus for a flow cytometry microscopy apparatus. The method comprises: in response to no particle passing through a flow channel of the flow cytometry microscopy apparatus, acquiring a first image not including a particle; in response to a particle passing through the flow channel of the flow cytometry microscopy apparatus, acquiring a second image; calculating a similarity of the first image and the second image to determine whether the particle is included in the second image; in response to the particle being included in the second image, determining a location of a particle region including the particle in the second image based on the first image and the second image; and segmenting a second region including the particle from the second image according to the determined location of the particle region, the second region having a size greater than a size of the particle region.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, and in particular to a method and device for segmenting cell images obtained based on flow cytometry microscopic imaging technology, an electronic device and a computer readable storage medium. BACKGROUND

[0002] Cell recognition and segmentation has always been a research focus in the fields of medical detection and judgment, drug development and research, and environmental detection. Flow cytometry has a high detection speed, up to thousands of cells per second, but the information dimension obtained by flow cytometry is small, and it cannot classify cells through high-dimensional information such as cell volume and shape, so the accuracy is not high, and it can only be used for quantitative analysis, and is commonly used for preliminary screening of cells. Microscopic examination can observe the image of the cell, but it needs to go through processes such as staining and film making, which is time-consuming, and the observation of cells generally relies on human recognition, which is extremely labor-intensive. Therefore, the observation method of staining and film making is generally used as the final observation method for distinguishing cells, and the detection sample is small, which cannot be detected on a large scale.

[0003] Cell segmentation is essentially an object detection algorithm. Current object detection algorithms include color-based recognition segmentation, contour-based recognition, and neural network-based target detection. Color-based recognition is not suitable for grayscale images; contour recognition is suitable for images with clear contours and obvious contrast between objects and backgrounds; and neural network-based target detection algorithms have complex models, consume a lot of computing resources, and have slow processing speed. Therefore, a special cell segmentation algorithm needs to be designed for flow cytometry microscopic images to achieve high-speed and efficient cell segmentation.

[0004] Therefore, it is desirable to provide a method that can quickly obtain images of individual cells and achieve high-speed and efficient cell segmentation. SUMMARY

[0005] The present disclosure is made in consideration of the above problems. One object of the present disclosure is to provide an image processing method and device for a flow cytometry microscopic imaging apparatus, an electronic device and a computer readable storage medium.

[0006] Embodiments of the present disclosure provide an image processing method for a flow cytometry imaging device, the method comprising: in response to no particle passing through a flow channel of the flow cytometry imaging device, acquiring a first image not including a particle; in response to a particle passing through the flow channel of the flow cytometry imaging device, acquiring a second image; calculating a similarity between the first image and the second image to determine whether the particle is included in the second image; in response to the particle being included in the second image, determining a position of a particle region including the particle in the second image based on the first image and the second image; and segmenting a second region including the particle from the second image according to the determined position of the particle region, the second region having a size larger than that of the particle region.

[0007] For example, a method according to embodiments of the present disclosure, wherein the flow cytometry imaging device comprises a light source, a condenser lens group, a flow channel, a microscope objective, a tube lens, and a camera, wherein the flow cytometry imaging device controls the particle in the liquid to be detected to flow stably through the center of the flow channel by controlling the flow of the liquid to be detected; wherein the light source is configured to emit a visible light beam; the condenser lens group is configured to condense the light beam and uniformly irradiate the light beam on the flow channel; the microscope objective is an infinite objective, focuses on the center of the flow channel, and images the flowing particle; the tube lens converges the outgoing light of the infinite objective to form an image, and adjusting the focal length of the tube lens can adjust the magnification of the image; and the camera is located at the back focal plane of the tube lens, takes the enlarged image to obtain the first image and the second image.

[0008] For example, a method according to embodiments of the present disclosure, wherein calculating the similarity between the first image and the second image comprises: respectively accumulating the gray value of each column of the first image and the second image into a row vector, obtaining a feature vector of the first image and a feature vector of the second image based on the accumulated row vectors; and calculating the Euclidean distance between the feature vector of the first image and the feature vector of the second image as the similarity between the first image and the second image.

[0009] For example, a method according to embodiments of the present disclosure, wherein determining whether the particle is included in the second image comprises: in response to the similarity being higher than a predetermined threshold, determining that the particle is not included in the second image; and in response to the similarity being lower than the predetermined threshold, determining that the particle is included in the second image.

[0010] For example, the method according to the embodiments of the present disclosure, wherein, in response to the second image including the microparticle, determining the position of the microparticle region including the microparticle in the second image based on the first image and the second image comprises: subtracting the second image from the first image pixel by pixel to obtain a residual image; performing a binarization process on the residual image to obtain a binarized residual image; determining the position of the microparticle region in the residual image based on the binarized residual image; and determining a pixel region in the second image corresponding to the microparticle region in the residual image as the position of the microparticle region in the second image.

[0011] For example, the method according to the embodiments of the present disclosure, wherein, in the binarization process on the residual image comprises: comparing each pixel in the residual image with a predefined binarization threshold, setting the corresponding pixel value in the residual image to 1 when the pixel value is greater than or equal to the binarization threshold, and setting the corresponding pixel value to 0 when the pixel value is less than the binarization threshold.

[0012] For example, the method according to the embodiments of the present disclosure, wherein, in the determination of the position of the microparticle region in the residual image based on the binarized residual image comprises: performing a morphological process on the binarized residual image, and determining a region with a pixel value of 1 in the morphologically processed residual image as the position of the microparticle region in the residual image.

[0013] For example, the method according to the embodiments of the present disclosure, wherein, in the segmentation of the second region including the microparticle from the second image according to the determined position of the microparticle region comprises one of: determining a minimum rectangular region surrounding the microparticle region in the second image based on the position of the microparticle region in the second image, and segmenting the minimum rectangular region from the second image to obtain the second region including the microparticle; extending the minimum rectangular region by a predetermined number of pixels in all directions, and segmenting the extended region from the second image to obtain the second region including the microparticle; and segmenting a fixed region with a fixed size larger than the microparticle region from the second image with the microparticle region as the center to obtain the second region including the microparticle.

[0014] For example, the method according to the embodiments of the present disclosure, wherein, the microparticle is a cell.

[0015] The embodiment of the present disclosure also provides an image processing device for a flow cytometry microscopy device, the device comprising: an image acquisition component configured to acquire a first image not including a micro-particle in response to the micro-particle not passing through a flow channel of the flow cytometry microscopy device, and acquire a second image in response to the micro-particle passing through the flow channel of the flow cytometry microscopy device; a micro-particle determination component configured to calculate a similarity between the first image and the second image to determine whether the micro-particle is included in the second image; a micro-particle positioning component configured to determine a position of a micro-particle region including the micro-particle in the second image based on the first image and the second image in response to the micro-particle being included in the second image; and a segmentation component configured to segment a second region including the micro-particle from the second image according to the determined position of the micro-particle region, the second region having a size greater than that of the micro-particle region.

[0016] The embodiment of the present disclosure also provides an electronic device comprising a memory and a processor, wherein the memory stores processor-readable program code, and the processor executes the program code to perform the method according to any one of the above methods.

[0017] The embodiment of the present disclosure also provides a computer-readable storage medium storing computer-executable instructions for performing the method according to any one of the above methods.

[0018] The image processing method for the flow cytometry microscopy device of the present disclosure can obtain an image of a single cell including rich image information, and can quickly pick and segment the obtained single cell image, so as to eliminate invalid images and invalid regions and reduce the amount of subsequent data storage and processing. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments of the present disclosure will be briefly introduced below. Obviously, the drawings described below only relate to some embodiments of the present disclosure, but not limit the present disclosure.

[0020] Figure 1 A schematic diagram of a flow cytometry microscopy device according to an embodiment of the present disclosure is shown;

[0021] Figure 2 A flow chart of an image processing method for a flow cytometry microscopy device according to an embodiment of the present disclosure is shown;

[0022] Figure 3 A more detailed step of step 204 is shown; Figure 2

[0023] Figure 4 ​An effect diagram obtained based on the image processing method for a flow cytometry microscopic imaging apparatus according to an embodiment of the disclosure is shown;

[0024] Figure 5 A schematic block diagram of an image processing apparatus for a flow particle microscopic imaging apparatus according to an embodiment of the disclosure is shown;

[0025] Figure 6 A schematic block diagram of an image processing apparatus for a flow particle microscopic imaging apparatus according to another embodiment of the disclosure is shown;

[0026] Figure 7 An architectural schematic diagram of an electronic device according to an embodiment of the disclosure is shown; and

[0027] Figure 8 A schematic diagram of a storage medium according to an embodiment of the disclosure is shown. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings, obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor also belong to the scope of protection of the present application.

[0029] The terms used in the specification are those general terms currently widely used in the art in consideration of the functions related to the present disclosure, but the terms can be changed according to the intention of those skilled in the art, precedents, or new technology in the art. In addition, specific terms can be selected by the applicant, and in this case, the detailed meaning thereof will be described in the detailed description of the present disclosure. Therefore, the terms used in the specification should not be understood as simple names, but based on the meaning of the terms and the overall description of the present disclosure.

[0030] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, however, any number of different modules can be used and run on the user terminal and / or server. The modules are only illustrative, and different aspects of the system and method can use different modules.

[0031] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed in sequence. On the contrary, various steps can be processed in reverse order or simultaneously as needed. Meanwhile, other operations can also be added to these processes, or one or more steps of operations can be removed from these processes.

[0032] The image processing method for the flow cytometry microscopy imaging device according to at least one embodiment of the present disclosure is described below by means of several examples or embodiments, which are not restrictive. As described below, different features in these specific examples or embodiments can be combined with each other without mutual contradiction, so as to obtain new examples or embodiments, which also belong to the scope of protection of the present disclosure.

[0033] An aspect of the present disclosure provides a flow cytometry microscopy imaging device for realizing high-speed cell imaging detection, which has the advantages of both rapid detection and rich image information. Another aspect of the present disclosure provides an image recognition and segmentation algorithm based on deep learning.

[0034] Figure 1 A flow cytometry microscopy imaging device 100 according to an embodiment of the present disclosure is shown. As shown in Figure 1 The device 100 can include a light source 101, a condenser lens group 102, a flow channel 103, a microscope objective 104, a tube lens 105, and a camera 106. As those skilled in the art should understand, in addition to the above components, the flow cytometry microscopy imaging device 100 can also include other components, which are not limited thereto.

[0035] As shown in Figure 1 The condenser lens group 102, the microscope objective 104, and the tube lens 105 can be located on the path of the light beam.

[0036] According to an embodiment of the present disclosure, the light source 101 can be configured to emit a visible light beam for providing illumination. The visible light refers to white light or monochromatic light in the visible light band, for example, light between 0.4 and 0.8 μm, including laser. The light source 101 can be a laser configured to emit pulsed laser light with adjustable pulse width and pulse frequency. The light source 101 can also include other components, such as a control component configured to determine the pulse width and pulse frequency of the laser, respectively, according to the flow rate of the flowing particles and the image acquisition frame rate of the image acquisition module (e.g., the camera), and to generate a control signal for driving the laser and a synchronization trigger signal corresponding to the control signal for the image acquisition module according to the determined pulse width and pulse frequency; a driving component configured to generate a driving current according to the control signal to drive the laser to emit pulsed laser light at the determined pulse width and pulse frequency.

[0037] According to an embodiment of the present disclosure, a sample flow containing microparticles can flow in the flow channel 103. The sheath fluid wrapping the microparticle sample flow enters the flow channel 103 as the sample flow, and the microparticles in the sample flow flow through the center of the flow channel 103 by controlling the flow of the sample flow in the flow channel 103.

[0038] The microscope objective 104 can employ an infinity objective, and can be configured to focus on the center of the flow channel 103 to image the micro-particles flowing through.

[0039] According to an embodiment of the present disclosure, the condenser lens group 102 can be configured to condense the light beam and uniformly irradiate the light beam on the flow channel 103. The condenser lens group 102 can constitute Kohler illumination with the light source 101. The Kohler illumination can change the concentric light beam emitted by the light source into a plurality of parallel light beams, which are uniformly irradiated on the target object (for example, the flow channel 103 here, and more specifically, the micro-particles in the flow channel 103), so that the illumination can be uniform and the light efficiency is high, and the imaging effect of the imaging system is better. The condenser lens group 102 can include two lenses, the first lens is arranged close to the light source 101, and the second lens is arranged after the first lens (i.e., farther away from the light source 101 than the first lens). The first lens condenses the light beam from the light source 101 on the front focal plane of the second lens. The condenser lens group 102 can also include more lenses, as long as the light beam from the light source 101 can be uniformly emitted.

[0040] According to an embodiment of the present disclosure, the microscope objective 104 can be configured to focus on the center of the flow channel 103 to collect light passing through the micro-particles in the flow channel 103, wherein the working distance of the microscope objective 104 is greater than the distance from the center of the flow channel 103 to the outer surface of the flow channel 103, and the numerical aperture (NA) of the microscope objective 104 is determined based on a predetermined resolution and a wavelength of the light beam.

[0041] For example, the microscope objective 104 can include one or more lenses. Alternatively, the microscope objective 104 can be an infinity objective, and the light passing through the infinity objective is emitted as a parallel light beam to infinity. Applying the infinity objective to the microscope objective 104 can weaken the influence of aberration on observation to a greater extent.

[0042] The tube lens 105 can be located between the microscope objective 104 and the image plane, and can be configured to, together with the microscope objective 104, magnify the image of the micro-particles and converge the magnified image of the micro-particles on the image plane. That is, the microscope objective 104 and the tube lens 105 together have a magnification function, and the magnification can be determined based on the focal lengths of the microscope objective 104 and the tube lens 105.

[0043] For example, the tube lens 105 can include one or more lenses. The tube lens 105 can converge the light emitted by the infinity objective and image on a finite image plane. Adjusting the focal length of the tube lens 105 can adjust the magnification of the imaging.

[0044] The camera 106 is arranged on the image plane and is configured to capture the image of the magnified micro-particles on the image plane.

[0045] As described above with reference to Figure 1 The flow cytometry imaging device according to the embodiments of the present disclosure described above utilizes a microscope system to take images of cells in a flow channel, and a camera to continuously take images of the liquid in the flow channel. Due to the different concentrations of cells, the images taken may appear empty or contain multiple cells, etc. At this time, an image processing algorithm is needed to select and segment the cells in the images obtained by the flow cytometry imaging device.

[0046] The present disclosure provides at least one embodiment of an image processing method for a flow cytometry imaging device. The method can be executed by a computer device integrated with the flow cytometry imaging device as shown in Figure 1 The computer device can be a single or multiple computer devices, each of which can have an architecture as shown in Figure 7 For example, the computer device obtains an image including or not including the microparticle through the camera 106 of the flow cytometry imaging device 100 in response to the microparticle not passing through the flow channel 103 of the flow cytometry imaging device 100 described above. For another example, the image processing method for the flow cytometry imaging device according to the embodiments of the present disclosure can be implemented by a remote computer system in communication with the flow cytometry imaging device. The computer system can be a server cluster or a single / multiple computer devices, and the image processing method for the flow cytometry imaging device can be executed by multiple servers distributedly arranged, or each computer device can execute the processing distributedly.

[0047] Figure 2 FIG. 2 is a flowchart illustrating an image processing method 200 for a flow cytometry imaging device according to an embodiment of the present disclosure. As described above with reference to Figure 2 The method 200 includes steps S201-S205.

[0048] In the present method, cells in a body fluid are taken as an example of microparticles. It should be understood that the microparticles can also be formed elements in river water based on the detection of river water quality, and the present disclosure does not limit this.

[0049] In step S201, in response to the microparticle not passing through the flow channel of the flow cytometry imaging device, a first image not including the microparticle is obtained.

[0050] For example, the flow cytometry imaging device herein is the flow cytometry imaging device 100 as described above with reference to Figure 1

[0051] ​According to one embodiment of the present disclosure, the sheath fluid of the unwrapped cell sample stream can be flowed into the flow channel as the sample stream, so as to keep the sample stream in the center of the flow channel. In response to no microparticle passing through the flow channel, the camera is triggered to take a picture, so as to obtain the picture of the cell-free sample stream flowing through the flow channel as the first image.

[0052] In addition, according to another embodiment of the present disclosure, the sheath fluid of the wrapped cell sample stream can also be directly flowed into the flow channel as the sample stream. When it is determined that no cell flows through the focal plane of the objective lens of the flow cytometry imaging device, the camera is triggered to capture an image of the cell-free as the first image. For example, the triggering here can be manual judgment, or triggering the camera to capture the image when no reflected light signal from the flow channel is detected.

[0053] Step S202, in response to the microparticle passing through the flow channel of the flow cytometry imaging device, a second image is obtained.

[0054] The sheath fluid wrapped cell sample stream is flowed into the channel as the sample stream, so as to keep the sample stream in the center of the channel, and then the camera is triggered to take a picture as the second image.

[0055] As described above, due to the different concentrations of cells, when the sample stream wrapped with cells flows through the flow channel of the flow cytometry imaging device, the obtained image can be empty or contain multiple cells. Therefore, after the second image is obtained based on the flow cytometry imaging device, it is necessary to first determine whether the second image contains cells. For the blank image without cells, it can be directly discarded, so as to reduce the amount of redundant data in the subsequent analysis of the cells in the cell image.

[0056] Step S203, the similarity between the first image and the second image is calculated to determine whether the microparticle is included in the second image. Specifically, in one embodiment of the step, the gray values of each column of the first image and the second image are respectively accumulated into row vectors, and the feature vectors of the first image and the second image are respectively obtained based on the accumulated row vectors. Taking the feature vector of the first image as an example, the gray values of each column of the first image obtained by the camera in step S201 are accumulated to obtain a row vector, and the feature vector of the first image is obtained based on the row vector. Similarly, the feature vector of the second image can also be obtained by using the same method.

[0057] For example, obtaining the feature vector of the image based on the row vector can include directly taking the accumulated row vector as the feature vector of the image. Of course, the row vectors can also be further normalized and taken as the feature vector of the image, and the present disclosure does not limit this.

[0058] Next, the Euclidean distance between the feature vector of the first image and the feature vector of the second image is calculated as the similarity of the first image and the second image. Specifically, the Euclidean distance is calculated by the following formula:

[0059]

[0060] where n is the dimension of the vector, x i is the feature vector of one dimension of the first image, y i is the feature vector of one dimension of the second image.

[0061] The smaller the value of the Euclidean distance is, the more similar the first image and the second image are. By setting a suitable predetermined threshold, it can be determined whether the second image contains the particle. Specifically, whether the particle is included in the second image is determined by the following steps: when the similarity of the first image and the second image is higher than the predetermined threshold, it is determined that the second image does not include the particle. When the similarity of the first image and the second image is lower than the predetermined threshold, it is determined that the second image includes the particle.

[0062] In addition, the algorithm for calculating the similarity of the first image and the second image can also use methods such as average value, statistical gray histogram distribution, correlation coefficient, etc., and the present disclosure does not limit this.

[0063] It should be understood that the above-mentioned predetermined threshold can be confirmed by a large number of orthogonal experiments, and the predetermined threshold can also be different when different particles are used. The value of the predetermined threshold is not specifically limited in the present application.

[0064] Step S204, in response to the second image including the particle, determining the position of the particle region including the particle in the second image based on the first image and the second image.

[0065] When it is confirmed that the second image contains the particle, if the particle is to be segmented, the particle needs to be located first to confirm the position of the particle in the second image. The specific steps of how to confirm the position of the particle in the second image can be described in detail in Figure 3 .

[0066] Step S205: According to the determined position of the particle region, a second region including the particle is segmented from the second image, and the size of the second region is greater than the size of the particle region.

[0067] After the position of the particle region is determined, the particle is segmented from the second image. The region including the particle segmented from the second image is defined as the second region in the present application, and it should be understood that in order not to damage the integrity of the particle, the second region segmented needs to be larger than the particle region.

[0068] For example, the second region can be a rectangular region, and in particular, based on the location of the particle region in the second image, a minimum rectangular region surrounding the particle region is determined in the second image, the minimum rectangular region having a vertical distance between the two longitudinal end points of the particle as one side length of the rectangle, and having a vertical distance between the two transverse end points of the particle as the second side length, so as to ensure that the minimum rectangular region can encompass the region where the particle is located, and the minimum rectangular region containing the particle is segmented from the second image, i.e., the second region defined above is obtained.

[0069] For example, to ensure the redundancy of the segmented region, the second region can also be a rectangular region slightly larger than the minimum rectangular region above. In particular, after obtaining the minimum rectangular region above, the four sides of the minimum rectangular region can be extended by a number of pixels (e.g., 5 pixels) in all directions, and the extended region is segmented from the second image as the second region.

[0070] For example, the second region can also be a pre-set region with a fixed side length (e.g., fixed length and width) that is larger than all particle regions. In particular, after determining the particle region, the particle region is extended to the fixed side length in all directions in the second image, and the extended region is segmented from the second image as the second region. It should be understood that all segmented images obtained in this way have a fixed size.

[0071] It should be understood by those skilled in the art that, in order to ensure the integrity of the segmented cell region, other regions of suitable size can also be set as the second region, which is not limited in the present disclosure.

[0072] Figure 3 A more detailed step of determining the location of the particle region containing the particle in the second image in step 204 is shown. Figure 2 As described above, step 204 includes sub-steps S301-S304. Figure 3

[0073] Step S301: Subtract the pixel values of the second image from the pixel values of the first image pixel by pixel to obtain a residual image.

[0074] Subtract the pixels of the picture as the second image from the pixels of the picture as the first image pixel by pixel, so as to obtain a new picture matrix as the residual image.

[0075] For example, since positive and negative values can occur after pixel subtraction here, the residual image here can be a picture matrix taking the absolute value of the pixel difference obtained after subtraction.

[0076] Step S302: Perform a binaryzation process on the residual image to obtain a binaryzation residual image.​

[0077] Specifically, the residual image is binarized, i.e., each pixel in the residual image is compared with a predefined binarization threshold, and if the pixel value is greater than or equal to the binarization threshold, the corresponding pixel value in the residual image is set to 1, and if the pixel value is less than the binarization threshold, the corresponding pixel value is set to 0. Similarly, the binarization threshold can be reasonably set based on actual conditions.

[0078] In this step, for example, if the cell-free region in the second image is basically consistent with the theoretical value of the corresponding region in the first image, the value of the cell-free region in the residual image after subtraction is the theoretical value 0, and if the cell-free region in the second image is significantly different from the corresponding region in the first image, there is a certain difference after subtraction. By setting a certain threshold, the residual image is binarized, so that the pixel value of the cell region contained therein is 1, and the pixel value of the cell-free region is 0.

[0079] Step S303: determining the position of the particle region in the residual image based on the binarized residual image.

[0080] Specifically, the binarized residual image is subjected to morphological processing, and the region with a pixel value of 1 in the morphologically processed residual image is determined as the position of the particle region in the residual image.

[0081] In this step, since the binarized residual image obtained after step S302 processing may have problems such as noise and inaccurate cell region, through morphological processing of the image, the noise in the background is filtered out, and then the cell region is filled, so that the cell region with accurate position in the residual image can be obtained.

[0082] For example, the morphological processing here is an erosion and dilation algorithm. Erosion performs a "shrink" or "thinning" operation on a binary image, and dilation performs a "lengthening" or "thickening" operation on a binary image. Thus, the background noise can be eliminated, and the holes in the target foreground can be filled.

[0083] Step S304: determining the pixel region in the second image corresponding to the particle region in the residual image as the position of the particle region in the second image.

[0084] As described above, after obtaining the position of the particle region in the second image, a second region containing the particle is segmented from the second image, wherein the second region is larger than the particle region.

[0085] Thus, by the above steps, the invalid images not containing cells in the images obtained by the flow cytometry microscopic imaging device are quickly selected, and the invalid regions in the images containing cells are quickly segmented, which can reduce the amount of data for subsequent storage and processing, so as to better perform feature extraction, classification and identification and other analysis tasks on the cell images, thereby assisting doctors in screening and diagnosing some diseases.

[0086] It should be noted that the embodiments of the present disclosure do not limit the execution order of each step, which can be adjusted according to actual conditions. For example, in some examples, the image processing method 200 for the flow cytometry microscopic imaging device provided by at least one embodiment of the present disclosure can selectively execute part of the steps S201-S205, or can execute some additional steps in addition to the steps S201-S205, and the embodiments of the present disclosure do not specifically limit this.

[0087] Figure 4 The effect diagram obtained based on the image processing method for the flow cytometry microscopic imaging device according to the embodiments of the present disclosure is shown.

[0088] Figure 4 (a) of FIG. 1 is a second image obtained based on the image processing method for the flow cytometry microscopic imaging device according to the embodiments of the present disclosure, Figure 4 (b) of FIG. 1 is a second region segmented from the second image by the image processing method for the flow cytometry microscopic imaging device, as shown, the second region includes a complete cell part.

[0089] The image processing method for the flow cytometry microscopic imaging device can obtain the image of a single cell including rich image information, and quickly select and segment the obtained single cell image, so that the subsequent feature extraction and classification and identification of the cell image can be better performed.

[0090] Based on the above embodiments, referring to Figure 5 FIG. 5 shows a schematic block diagram of the image processing device 500 for the flow particle microscopic imaging device according to the embodiments of the present disclosure. The device 500 at least includes an image acquisition component 501, a particle determination component 502, a particle positioning component 503 and a segmentation component 504. In the embodiments of the present disclosure, the image acquisition component 501, the particle determination component 502, the particle positioning component 503 and the segmentation component 504 can be integrated in the same flow cytometry microscopic imaging device, or can be divided into multiple devices, connected and communicated with each other, composed of a system for use, etc. For example, the image acquisition component 501 can be part of the flow cytometry microscopic imaging device, and the particle determination component 502, the particle positioning component 503 and the segmentation component 504 can be a computer device in communication with the flow cytometry microscopic imaging device, etc.

[0091] It should be understood that the image processing apparatus 500 for the flow micro-particle microscopic imaging apparatus provided by the embodiments of the present disclosure can implement the aforementioned image processing method 200 for the flow micro-particle microscopic imaging apparatus, and can also achieve similar technical effects to the aforementioned image processing method 200 for the flow micro-particle microscopic imaging apparatus.

[0092] Specifically, the image acquisition component 501 is configured to acquire a first image not including a micro-particle in response to the micro-particle not passing through a flow channel of the flow cytometric microscopic imaging apparatus, and acquire a second image in response to the micro-particle passing through the flow channel of the flow cytometric microscopic imaging apparatus. The micro-particle determination component 502 is used to calculate a similarity between the first image and the second image to determine whether the micro-particle is included in the second image. The micro-particle positioning component 503 is configured to determine a position of a micro-particle region including the micro-particle in the second image based on the first image and the second image in response to the micro-particle being included in the second image. The segmentation component 504 is configured to segment a second region including the micro-particle from the second image according to the determined position of the micro-particle region, and a size of the second region is greater than a size of the micro-particle region.

[0093] For example, the flow cytometric microscopic imaging apparatus includes a light source, a condenser lens group, a flow channel, a microscopic objective lens, a tube lens, and a camera, wherein the flow cytometric microscopic imaging apparatus controls a micro-particle in a liquid to be detected to stably flow through a center of the flow channel by controlling a flow of the liquid to be detected; the light source is configured to emit a visible light beam; the condenser lens group is configured to condense the light beam and uniformly irradiate the light beam on the flow channel; the microscopic objective lens is an infinite objective lens, focuses on the center of the flow channel, and images the flowing micro-particle; the tube lens converges and images the outgoing light of the infinite objective lens, and adjusting a focal length of the tube lens can adjust a magnification of the imaging; and the camera is located at a back focal plane of the tube lens, photographs the enlarged image, and obtains the first image and the second image.

[0094] For example, the micro-particle determination component 502 is further configured to: accumulate a gray value of each column of the first image and the second image into a row vector respectively, obtain a feature vector of the first image and a feature vector of the second image based on the accumulated row vectors respectively; and calculate a Euclidean distance between the feature vector of the first image and the feature vector of the second image as the similarity between the first image and the second image.

[0095] For example, the micro-particle determination component 502 is further configured to: determine that the micro-particle is not included in the second image in response to the similarity being higher than a predetermined threshold; and determine that the micro-particle is included in the second image in response to the similarity being lower than the predetermined threshold.

[0096] For example, the microparticle positioning component 503 is further configured to: subtract pixel values of the second image from the first image pixel by pixel to obtain a residual image; binarize the residual image to obtain a binarized residual image; determine the position of the microparticle region in the residual image based on the binarized residual image; and determine a pixel region in the second image corresponding to the microparticle region in the residual image as the position of the microparticle region in the second image.

[0097] For example, the microparticle positioning component 503 is further configured to: compare each pixel in the residual image with a predefined binarization threshold, and set a corresponding pixel value in the residual image to 1 when the pixel value is greater than or equal to the binarization threshold, and set the corresponding pixel value to 0 when the pixel value is less than the binarization threshold.

[0098] For example, the microparticle positioning component 503 is further configured to: perform morphological processing on the binarized residual image, and determine a region with a pixel value of 1 in the morphologically processed residual image as the position of the microparticle region in the residual image.

[0099] For example, the segmentation component 504 is further configured to perform one of the following steps: determine a minimum rectangular region surrounding the microparticle region in the second image based on the position of the microparticle region in the second image, and segment the minimum rectangular region from the second image to obtain a second region including the microparticle; extend the minimum rectangular region by a predetermined number of pixels in all directions, and segment the extended region from the second image to obtain a second region including the microparticle; and segment a fixed region with a fixed size larger than the microparticle region from the second image with the microparticle region as the center to obtain a second region including the microparticle.

[0100] For example, the microparticle is a cell.

[0101] Embodiments of the present disclosure also provide an image processing device for a flow cytometry microscopy imaging device, Figure 6 A schematic block diagram of an image processing device for a flow cytometry microscopy imaging device according to another embodiment of the present disclosure is shown. For example, as shown in FIG. 4, the image processing device 400 includes a microparticle positioning component 403 and a segmentation component 404. Figure 6As shown, the image processing apparatus 600 for the flow cytometric microscopy imaging apparatus can include one or more processors 601 and one or more memories 602. The one or more memories 602 have stored therein computer-executable instructions that, when executed by the one or more processors 601, can perform the image processing method for the flow cytometric microscopy imaging apparatus as described above. The one or more memories 602 and the one or more processors 601 can be interconnected through a bus system and / or other form of connection mechanism (not shown).

[0102] For example, the one or more memories 602 and the one or more processors 601 can be disposed in a single machine, can be disposed in a server, and can be disposed in a cloud for performing one or more steps of the image processing method for the flow cytometric microscopy imaging apparatus as described above.

[0103] For example, the one or more processors 601 can be a central processing unit (CPU), a digital signal processor (DSP), or other form of processing unit having data processing and / or program execution capabilities, such as a field-programmable gate array (FPGA), etc. For example, the central processing unit (CPU) can be of an X86 or ARM architecture, etc. The one or more processors 601 can be a general purpose processor or a special purpose processor, and can control other components in the image processing apparatus 500 for the flow cytometric microscopy imaging apparatus to perform desired functions.

[0104] For example, the one or more memories 602 can include any combination of one or more computer program products. The computer program product can include various forms of computer-readable storage media for storing information that is used in the implementation of the present application. For example, the one or more memories 602 can include one or more volatile memory devices, such as random access memory (RAM), and / or one or more non-volatile memory devices, such as read-only memory (ROM), hard disk drives, erasable programmable read-only memories (EPROMs), compact disk read-only memories (CD-ROMs), USB memory, flash memories, etc.

[0105] In addition, the method or apparatus according to the embodiments of the present application can also be implemented by means of Figure 7 The architecture of the electronic device 700 is shown. Figure 7 The architecture of the electronic device is schematically shown. As Figure 7As shown, the electronic device 700 can include a bus 701, one or more CPUs 702, a read-only memory (ROM) 703, a random access memory (RAM) 704, a communication port connected to a network 705, an input / output component 706, a hard disk 707, etc. The storage device in the electronic device 700, such as the ROM 703 or the hard disk 707, can store various data or files used in processing and / or communication of the method provided in the present application and program instructions executed by the CPU. The electronic device 700 can also include a user interface. Of course, Figure 7 The architecture shown is only exemplary, and in implementing different devices, some of the components shown can be omitted or added Figure 7 One or at least two components in the electronic device shown.

[0106] Embodiments of the present disclosure also provide a computer-readable storage medium. Figure 8 A schematic diagram 800 of a storage medium according to embodiments of the present disclosure is shown. As Figure 8 As shown, the computer executable instructions 801 are stored on the computer readable storage medium 802. When the computer executable instructions 801 are run by a processor, the image processing method for a flow cytometric microscopy device according to embodiments of the present disclosure described with reference to the above figures can be executed. The computer readable storage medium includes, but is not limited to, for example, a volatile memory and / or a non-volatile memory. The volatile memory may, for example, include a random access memory (RAM) and / or a cache memory, etc. The non-volatile memory may, for example, include a read-only memory (ROM), a hard disk, a flash memory, etc.

[0107] Embodiments of the present disclosure also provide a computer program product or computer program including computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the image processing method for a flow cytometric microscopy device according to embodiments of the present disclosure.

[0108] Those skilled in the art can understand that the content disclosed in the present disclosure can appear in various modifications and improvements. For example, the various devices or components described above can be implemented by hardware, or by software, firmware, or a combination of some or all of the three.

[0109] In addition, although the present disclosure makes various references to certain units in the system according to embodiments of the present disclosure, however, any number of different units can be used and run on the client and / or server. The units are only illustrative, and different aspects of the system and method can use different units.

[0110] Those skilled in the art can understand that all or part of the steps of the above-mentioned method can be instructed by a program to complete the relevant hardware, and the program can be stored in a computer readable storage medium, such as a read-only memory, a magnetic disk or an optical disk, etc. Alternatively, all or part of the steps of the above-mentioned embodiments can also be implemented using one or more integrated circuits. Accordingly, each module / unit in the above-mentioned embodiments can be implemented in the form of hardware or in the form of a software functional module. The present disclosure is not limited to any specific form of combination of hardware and software.

[0111] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0112] The above is a description of the present disclosure and should not be considered as a limitation. Although exemplary embodiments of the present disclosure are described, those skilled in the art will readily understand that many modifications can be made to the exemplary embodiments without departing from the novel teachings and advantages of the present disclosure. Accordingly, all such modifications are intended to be included within the scope of the present disclosure as defined in the claims. It should be understood that the above is a description of the present disclosure and should not be considered as a limitation. Although exemplary embodiments of the present disclosure are described, those skilled in the art will readily understand that many modifications can be made to the exemplary embodiments without departing from the novel teachings and advantages of the present disclosure. Accordingly, all such modifications are intended to be included within the scope of the present disclosure as defined in the claims. The present disclosure is limited by the claims and their equivalents.

Claims

1. An image processing method for a flow cytometry particle microscopy imaging device, the method comprising: In response to the fact that the particles did not pass through the flow channel of the flow cytometry particle microscopy device, a first image excluding the particles is acquired; A second image is acquired in response to the passage of microparticles through the flow channel of the flow cytometry microscopy device; Calculating the similarity between the first image and the second image to determine whether the second image contains particles, wherein calculating the similarity between the first image and the second image includes: accumulating the grayscale values ​​of each column of the first image and the second image into a row vector, respectively, and obtaining the feature vectors of the first image and the second image based on the accumulated row vectors; and calculating the Euclidean distance between the feature vectors of the first image and the second image as the similarity between the first image and the second image, and wherein determining whether the second image contains particles includes: determining that the second image does not contain particles in response to the similarity being higher than a predetermined threshold; and determining that the second image contains particles in response to the similarity being lower than a predetermined threshold; In response to the inclusion of particles in the second image, the pixel values ​​of the second image are subtracted pixel by pixel from the first image to obtain a residual image. Based on the residual image, the position of the particle region including the particles in the second image is determined. Determining the position of the particle region including the particles in the second image based on the residual image, in response to the inclusion of particles in the second image, includes: binarizing the residual image to obtain a binarized residual image; determining the position of the particle region in the residual image based on the binarized residual image; and determining the pixel region in the second image corresponding to the particle region in the residual image as the position of the particle region in the second image. Based on the determined location of the particle region, a second region including the particle is segmented from the second image, the size of the second region being larger than the size of the particle region.

2. The method according to claim 1, wherein, The flow cytometry microscopy device includes a light source, a condenser lens group, a flow cytometry tube, a microscope objective, a tube lens, and a camera. The flow cytometry microscopy device controls the flow of particles in the liquid to be detected to stably flow through the center of the flow cytometry pipe by controlling the flow of the liquid to be detected. The light source is configured to emit a visible light beam; The condenser lens group is configured to focus the light beam and uniformly illuminate the flow channel; The microscope objective is an infinity objective lens, which is focused on the center of the flow channel to image the flowing particles; The tube lens converges the light emitted from the infinity objective lens to form an image; adjusting the focal length of the tube lens adjusts the magnification of the image. The camera is located at the rear focal plane of the tube lens and captures the magnified image to obtain the first image and the second image.

3. The method according to claim 2, wherein, Binarizing the residual image includes: Each pixel in the residual image is compared with a predefined binarization threshold. When the pixel value is greater than or equal to the binarization threshold, the corresponding pixel value in the residual image is set to 1. When the pixel value is less than the binarization threshold, the corresponding pixel value is set to 0.

4. The method according to claim 3, wherein, Determining the position of the particle region in the residual image based on the binarized residual image includes: The binarized residual image is subjected to morphological processing, and the region with a pixel value of 1 in the morphologically processed residual image is determined as the position of the particle region in the residual image.

5. The method according to claim 1, wherein, Based on the determined location of the particle region, segmenting a second region including the particles from the second image includes one of the following: Based on the position of the particle region in the second image, a minimum rectangular region surrounding the particle region is determined in the second image, and the minimum rectangular region is segmented from the second image to obtain a second region including the particle; The minimum rectangular region is extended outward by a predetermined number of pixels, and the extended region is segmented from the second image to obtain a second region including the particles; as well as Centered on the particle region, a fixed region with a fixed size larger than the particle region is segmented from the second image to obtain a second region including the particle.

6. The method according to claim 1, wherein, The particles are cells.

7. An image processing apparatus for a flow cytometry microscopy imaging device, the apparatus comprising: The image acquisition unit acquires a first image excluding the particles in response to the particles not passing through the flow channel of the flow cytometry particle microscopy device, and acquires a second image in response to the particles passing through the flow channel of the flow cytometry particle microscopy device. A particle determination component calculates the similarity between a first image and a second image to determine whether the second image contains particles. The calculation of the similarity includes: accumulating the grayscale values ​​of each column of the first image and the second image into a row vector, respectively; obtaining feature vectors of the first image and the second image based on the accumulated row vectors; and calculating the Euclidean distance between the feature vectors of the first image and the second image as the similarity between the first image and the second image. Determining whether the second image contains particles includes: determining that the second image does not contain particles in response to the similarity being higher than a predetermined threshold; and determining that the second image contains particles in response to the similarity being lower than a predetermined threshold. A particle localization component, in response to the inclusion of particles in a second image, subtracts the pixel values ​​of the second image from those of a first image pixel by pixel to obtain a residual image. Based on the residual image, it determines the position of a particle region including particles in the second image. Determining the position of the particle region including particles in the second image based on the residual image, in response to the inclusion of particles in the second image, includes: binarizing the residual image to obtain a binarized residual image; determining the position of the particle region in the residual image based on the binarized residual image; and determining the pixel region in the second image corresponding to the particle region in the residual image as the position of the particle region in the second image. The segmentation component, based on the determined position of the particle region, segments a second region including the particle from the second image, the size of the second region being larger than the size of the particle region.

8. An electronic device comprising a memory and a processor, wherein, The memory stores processor-readable program code, which, when executed by the processor, performs the method according to any one of claims 1-6.

9. A computer-readable storage medium having stored thereon computer-executable instructions for performing the method of any one of claims 1-6.

Images