Sample analysis system, image analysis system and method of processing sample images thereof

Abstract

The application provides a sample analysis system, an image analysis system and a method for processing a sample image, the method comprising: detecting a sample to be tested to obtain various detection parameters of the sample to be tested; preparing a blood smear based on the sample to be tested; moving the blood smear relative to an imaging device so that the blood smear is located in a field of view of the imaging device; capturing a specific region of the blood smear in the field of view to obtain a sample image; and processing the sample image by using a trained neural network to obtain an enhanced target image for output to a user. The application uses a neural network to perform image enhancement, which can accurately identify various cells or other tangible substances and significantly improve the quality of the image.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of sample analysis, and more particularly, to a sample analysis system, an image analysis system and a method for processing a sample image thereof. BACKGROUND

[0002] In medical analysis, an image analysis system in a sample analysis system can take a sample image of a sample to be tested, and identify and extract various target components (such as red blood cells, white blood cells, etc.) therefrom to display to medical personnel, who can make a preliminary diagnosis by observing the images of the target components to find out whether the sample is abnormal. The system can help medical personnel save the work of manual microscopic observation, thereby saving manpower and resources.

[0003] However, in some environments or some shooting modes, the sample image taken by the image analysis system can have low quality, such as insufficient exposure, inaccurate focusing, low resolution (such as using a 40x objective lens), etc., resulting in that the finally output image does not meet the observation needs of users. The existing method to solve the above problems is to use a conventional image processing algorithm to process the taken image. However, using the above image processing method to process the image cannot achieve satisfactory cell recognition and image display effect. SUMMARY

[0004] The present application is proposed to solve the above problems. According to an aspect of the present application, a method for a sample analysis system to analyze a sample to be tested is provided, which comprises: transporting the sample to be tested to a blood analyzer in the sample analysis system, the blood analyzer detecting the sample to be tested to obtain various detection parameters of the sample to be tested; transporting the sample to be tested to a smear preparation device in the sample analysis system, the smear preparation device preparing a blood smear based on the sample to be tested; moving the blood smear relative to an imaging device in the sample analysis system by a smear holding device in the sample analysis system, so that the blood smear is located in a field of view of the imaging device; taking a specific area of the blood smear in the field of view by the imaging device to obtain a sample image; and processing the sample image by an image analysis device in the sample analysis system using a trained neural network to obtain an enhanced target image.

[0005] In one embodiment, the trained neural network is a deep neural network.

[0006] In one embodiment, the step of processing the sample image using the trained neural network to obtain an enhanced target image comprises: enhancing the entire sample image using the trained neural network to obtain an enhanced entire sample image.

[0007] In an embodiment, the step of enhancing the whole sample image using the trained neural network to obtain an enhanced whole sample image comprises: extracting image features from image data of the whole sample image using the trained neural network; performing nonlinear mapping processing on the extracted image features using the trained neural network; and performing image reconstruction on the processed image features using the trained neural network to obtain the enhanced whole sample image.

[0008] In an embodiment, the method further comprises: taking the enhanced whole sample image as an enhanced target image; or performing target detection on the enhanced whole sample image, and extracting a part of the sample image in a region where a target component is located as the enhanced target image.

[0009] In an embodiment, the step of processing the sample image using the trained neural network to obtain an enhanced target image comprises: performing target detection on a whole sample image, and extracting a part of the sample image in a region where a target component is located as a target image; and enhancing the target image using the trained neural network to obtain an enhanced target image.

[0010] In an embodiment, the step of enhancing the target image using the trained neural network to obtain an enhanced target image comprises: extracting image features from image data of the target image using the trained neural network; performing nonlinear mapping processing on the extracted image features using the trained neural network; and performing image reconstruction on the processed image features using the trained neural network to obtain the enhanced target image.

[0011] In an embodiment, the method further comprises: determining whether the sample to be tested needs to be taken for sample image shooting according to the detection parameter, and if so, performing the step of transporting the sample to be tested to the smear preparation device in the sample analysis system; and if not, not performing the step of transporting the sample to be tested to the smear preparation device in the sample analysis system.

[0012] In an embodiment, the sample image taken by the imaging device comprises a low-quality image; and the method further comprises: training a plurality of neural networks as neural networks for enhancing a low-quality image of a corresponding one of a plurality of low-quality types, respectively, for a plurality of low-quality images of the plurality of low-quality types.

[0013] In one embodiment, the step of processing the sample image using the trained neural network to obtain an enhanced target image further comprises: identifying a low-quality type of the low-quality image, and processing the sample image using a trained neural network for low-quality images of the low-quality type according to the low-quality type of the low-quality image.

[0014] According to another aspect of the present application, there is provided a method for processing a sample image by an image analysis system, the method comprising: moving a sample carrier carrying a sample to be tested relative to an imaging device in the image analysis system so as to position the sample carrier within a field of view of the imaging device; taking an image of a specific region on the sample carrier within the field of view by the imaging device to obtain a sample image; processing the sample image using a trained neural network by an image analysis device in the image analysis system to obtain an enhanced target image.

[0015] In one embodiment, the trained neural network is a deep neural network.

[0016] In one embodiment, the step of processing the sample image using the trained neural network to obtain an enhanced target image comprises: enhancing the entire sample image using the trained neural network to obtain an enhanced entire sample image.

[0017] In one embodiment, the step of enhancing the entire sample image using the trained neural network to obtain an enhanced entire sample image comprises: extracting image features from image data of the entire sample image using the trained neural network; performing non-linear mapping processing on the extracted image features using the trained neural network; and performing image reconstruction according to the processed image features using the trained neural network to obtain the enhanced entire sample image.

[0018] In one embodiment, the method further comprises: taking the enhanced entire sample image as the enhanced target image; or performing target detection on the enhanced entire sample image and extracting a portion of the sample image in a region where a target component is located as the enhanced target image.

[0019] In one embodiment, the step of processing the sample image using the trained neural network to obtain an enhanced target image comprises: performing target detection on the entire sample image and extracting a portion of the sample image in a region where a target component is located as a target image; enhancing the target image using the trained neural network to obtain an enhanced target image.

[0020] In an embodiment, the step of enhancing the target image by using the trained neural network comprises: extracting image features from the image data of the target image by using the trained neural network; performing nonlinear mapping processing on the extracted image features by using the trained neural network; and performing image reconstruction on the processed image features by using the trained neural network to obtain the enhanced target image.

[0021] In an embodiment, the sample image obtained by the imaging device comprises a low-quality image; the enhanced target image comprises a high-quality image; the low-quality type of the low-quality image comprises at least one of underexposure, image blur, and low resolution; and the high-quality image comprises an image with sufficient exposure, clear image, or high resolution.

[0022] In an embodiment, the method further comprises: training a plurality of neural networks as neural networks for enhancing low-quality images of a plurality of low-quality types, respectively.

[0023] In an embodiment, the step of enhancing the target image by using the trained neural network further comprises: identifying the low-quality type of the sample image, and processing the sample image by using the trained neural network for low-quality images of the low-quality type according to the low-quality type of the low-quality image.

[0024] In an embodiment, the sample to be tested comprises blood or urine, and the sample carrier comprises a sample smear or a counting chamber.

[0025] According to yet another aspect of the present application, there is provided a sample analysis system, comprising: a blood analyzer configured to detect a sample to be tested to obtain various detection parameters of the sample to be tested; a smear preparation device configured to prepare a blood smear based on the sample to be tested; and an image analysis system comprising an imaging device, a smear holding device, and an image analysis device, wherein the smear holding device is configured to relatively move the blood smear and the imaging device to enable the blood smear to be located in a field of view of the imaging device; the imaging device is configured to capture a specific region on the blood smear in the field of view to obtain a sample image; and the image analysis device is configured to perform the method as described above by using a trained neural network.

[0026] According to a further aspect of the present application, there is provided an image analysis system, characterized in that the image analysis system comprises an imaging device, a carrier holding device and an image analysis device, wherein the carrier holding device is configured to move a sample carrier carrying a sample to be measured relative to the imaging device so as to position the sample carrier within a field of view of the imaging device; the imaging device is configured to take an image of a specific area on the sample carrier within the field of view; and the image analysis device is configured to execute the method as described above using a trained neural network.

[0027] According to a further aspect of the present application, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by one or more processors, enables the method as described above to be performed.

[0028] According to a further aspect of the present application, there is provided a neural network trained to be used by an image analysis device to perform the method as described above.

[0029] The sample analysis system, the image analysis system and the method of processing a sample image according to the embodiments of the present application can accurately identify various cells or other tangible substances by using a neural network for image enhancement, and significantly improve the quality of the image. BRIEF DESCRIPTION OF DRAWINGS

[0030] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which like reference characters refer to like parts throughout the figures. The drawings provided are for illustrative purposes only and, therefore, should not be considered to be limiting of the present application. In the drawings:

[0031] Figure 1 A structural schematic diagram of a sample analysis system according to an embodiment of the present application is shown.

[0032] Figure 2 and Figure 3 A structural schematic diagram of a smear preparation device according to an embodiment of the present application is shown.

[0033] Figure 4 A schematic structural block diagram of an image analysis system in a sample analysis system according to an embodiment of the present application is shown.

[0034] Figure 5 A schematic structural block diagram of a control device in a sample analysis system according to an embodiment of the present application is shown.

[0035] Figure 6A schematic block diagram of an image analysis system according to another embodiment of the present application is shown.

[0036] Figure 7 A flow chart of steps of a method for analyzing a sample under test by a sample analysis system according to yet another embodiment of the present application is shown.

[0037] Figure 8 A flow chart of steps of a method for processing a sample image by an image analysis system according to yet another embodiment of the present application is shown.

[0038] Figure 9A An underexposed sample image taken by a 100x objective according to one embodiment of the present application is shown.

[0039] Figures 9B-9C An overexposed sample image taken by a 100x and 40x objective respectively according to one embodiment of the present application is shown.

[0040] Figure 10 A schematic diagram of image enhancement of an entire sample image using a trained neural network according to one embodiment of the present application is shown.

[0041] Figure 11 A schematic diagram of processing a sample image by an image analysis device according to another embodiment of the present application is shown. DETAILED DESCRIPTION

[0042] In order to make the objects, technical solutions, and advantages of the present application more apparent, the following will describe example embodiments of the present application in detail with reference to the accompanying drawings. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. It should be understood that the present application is not limited to the example embodiments described herein. Based on the embodiments of the present application described in the present application, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present application.

[0043] As described above, since the existing image processing method cannot achieve satisfactory cell recognition and image enhancement effect, in order to better perform cell recognition and image enhancement, the present application provides the following sample analysis system, image analysis system, and method for processing a sample image.

[0044] The following will describe the sample analysis system, image analysis system, and method for processing a sample image according to the present application in detail with reference to specific embodiments.

[0045] According to one embodiment of the present application, a sample analysis system is provided. With reference to Figure 1 , Figure 1A structural schematic diagram of a sample analysis system 100 according to an embodiment of the present application is shown.

[0046] As shown in Figure 1 The sample analysis system 100 comprises a blood analyzer 110, a smear preparation device 120, an image analysis system 130 and a control device 140.

[0047] The blood analyzer 110 is used for detecting (e.g. routine blood test) the sample to be tested, the smear preparation device 120 is used for preparing the blood smear based on the sample to be tested, the image analysis system 130 is used for image shooting and analysis of the cells in the smear, and the control device 140 is in communication connection with the blood analyzer 110, the smear preparation device 120 and the image analysis system 130.

[0048] The sample analysis system 100 further comprises a first conveying track 150 and a second conveying track 160, the first conveying track 150 is used for conveying the test tube rack 10 loaded with a plurality of test tubes 11 containing the sample to be tested from the blood analyzer 110 to the smear preparation device 120, and the second conveying track 160 is used for conveying the slide basket 20 loaded with a plurality of prepared blood smears 21 from the smear preparation device 120 to the image analysis system 130.

[0049] The control device 140 is in electrical connection with the first conveying track 150 and the second conveying track 160 and controls the actions thereof.

[0050] The sample analysis system 100 further comprises feeding mechanisms 170 and 180 respectively arranged corresponding to the blood analyzer 110 and the smear preparation device 120, each feeding mechanism 170 and 180 comprises a loading buffer area 171 and 181, a feeding detection area 172 and 182 and an unloading buffer area 173 and 183.

[0051] When the sample to be tested on the test tube rack 10 needs to be conveyed to the blood analyzer 110 for detection, the test tube rack 10 is first conveyed from the first conveying track 150 to the loading buffer area 171, then conveyed from the loading buffer area 171 to the feeding detection area 172 for detection by the blood analyzer 110, after the detection is completed, the test tube rack 10 is unloaded from the feeding detection area 172 to the unloading buffer area 173, and finally the test tube rack 10 is conveyed from the unloading buffer area 173 to the first conveying track 150.

[0052] Similarly, when the sample on the test tube rack 10 needs to be examined by microscope, the test tube rack 10 needs to be transported to the smear preparation device 120 to prepare a blood smear, the test tube rack 10 is first transported from the first transmission track 150 to the loading buffer area 181, then from the loading buffer area 181 to the feeding detection area 182 to prepare a blood smear by the smear preparation device 120, after the preparation of the blood smear is completed, the test tube rack 10 is unloaded from the feeding detection area 182 to the unloading buffer area 183, and finally the test tube rack 10 is transported from the unloading buffer area 183 to the first transmission track 150. The smear preparation device 120 stores the prepared blood smear in the slide basket 20, and the slide basket 20 containing the blood smear to be tested is transported to the image analysis system 130 through the second transmission track 160. The image analysis system 130 takes images of the cells in the sample on the blood smear to be tested and analyzes the images.

[0053] The sample analysis system 100 can further include a display device (not shown) for displaying the sample detection results, which can be provided on the blood analyzer 110, the smear preparation device 120, the image analysis system 130, or the control device 140, or provided separately.

[0054] The blood analyzer 110 detects the sample to be tested (e.g., whole blood, serum, peripheral blood, etc.) to obtain various detection parameters of the sample to be tested, such as white blood cells, red blood cells, hemoglobin, platelets, and other blood routine parameters, to preliminarily determine whether the sample to be tested is abnormal according to the various detection parameters.

[0055] Exemplarily, whether the sample to be tested needs to be imaged can be determined according to the detection parameters. Specifically, if yes, i.e., the detection parameters indicate that the blood cells of the sample to be tested are abnormal, such as low white blood cell count, platelet aggregation, lipemia, etc., the image of the cells needs to be taken, the sample to be tested is transported to the smear preparation device 120 to prepare a blood smear, and then the sample image is taken to further identify the cell morphology and thus make a more accurate diagnosis; if no, the step of transporting the sample to be tested to the smear preparation device 120 is not performed. This process can be performed by the smear preparation device 120, the image analysis system 130, and the control device 140 in cooperation.

[0056] The smear preparation device 120 is used to prepare a blood smear based on the sample to be tested (i.e., blood). Referring to Figure 2 and Figure 3 which shows a structural schematic diagram of the smear preparation device 120. The smear preparation device 120 can include a sampling mechanism 121 for extracting the sample, a slide loading mechanism 122 for moving the slide to the working line, a sample loading mechanism 123 for loading the sample to the slide, a slide pushing mechanism 124 for smearing the sample on the slide, a drying mechanism (not shown in the figure) for drying the blood film on the slide, and a staining mechanism 125 for staining the slide.

[0057] When the sampling mechanism 121 extracts the sample, the sample is first mixed, and then the sample is extracted by the sampling device (e.g., a sampling needle) in the sampling mechanism 121. Depending on the sample container, the sample can be extracted by puncture (the sample container has a cover, and the sampling device punctures the cover of the sample container), or the sample can be extracted by open extraction (the sample container is open, and the sampling device directly extracts the sample from the opening). If necessary, blood sample information detection can be performed to obtain information and compare the information. In some embodiments, the micro-sample loading mechanism 126 can move the test tube directly to the direction of the sampling device, or the sampling device can move to the direction of the test tube placed by the operator. In other implementations, the micro-sample loading mechanism 126 can also move the test tube directly to the direction of the sample loading mechanism 123, or the sample loading mechanism 123 can move to the direction of the test tube placed by the operator. After the blood sample is extracted by the sample loading mechanism 123 (e.g., a blood dropper), the sample loading is performed, which can reduce the demand for blood samples and achieve micro-sample and priority loading, because the blood does not need to be extracted by the sampling mechanism 121. After the sampling is completed, the blood is prepared to be dropped on the slide by the sample loading mechanism 123.

[0058] Correspondingly, the slide loading mechanism 122 extracts the slide and loads the slide to the corresponding position to facilitate the blood dropping operation. In some embodiments, after the slide extraction operation is completed, the slide left-right detection and slide cleaning operations can be performed, and then the slide is loaded. After the slide is loaded, the relevant information can be printed, and the slide front-back detection operation can be performed.

[0059] After the blood dropper of the sample loading mechanism 123 drops the sample on the slide, the slide pushing operation is performed, and the blood is pushed into a blood film shape on the slide by the slide pushing mechanism 124. Generally, after the slide pushing operation is completed, the blood film on the slide can be dried to stabilize the shape. In some embodiments, before the blood film is dried, the slide can be flipped to meet the corresponding requirements. In some embodiments, the dried blood smear can be subjected to drying detection to determine the drying effect of the blood film. In some embodiments, the dried blood smear can be subjected to blood film unfolding detection to determine whether the blood film is unfolded and whether the unfolded state meets the requirements. After the slide (blood smear) is pushed, the slide can be dyed (which can be achieved by the dyeing mechanism 125) or directly output (e.g., placed in the slide basket 20 for output).

[0060] As Figure 4As shown, the image analysis system 130 at least comprises an imaging device 131, a smear holding device 132 and an image analysis device 133. The smear holding device 132 is configured to move the smear relative to the imaging device 131 so that the blood smear is located in the field of view of the imaging device 131; the imaging device 131 comprises a camera 1312 and a lens set 1311, which are configured to capture a specific area on the blood smear in the field of view to obtain a sample image.

[0061] The lens set 1311 can comprise a first objective lens and a second objective lens. The first objective lens can be, for example, a 10x objective lens, and the second objective lens can be, for example, a 100x objective lens. The lens set 1311 can further comprise a third objective lens, which can be, for example, a 40x objective lens. The lens set 1311 can further comprise an ocular lens.

[0062] The image analysis system 130 further comprises an identification device 134, a slide clamping device 135 and a smear recycling device 136. The identification device 134 is configured to identify the identity information of the blood smear, the slide clamping device 135 is configured to clamp the blood smear from the identification device 134 to the smear holding device 132 for detection, and the smear recycling device 136 is configured to place the detected blood smear.

[0063] The image analysis system 130 further comprises a slide basket loading device 137 configured to load a slide basket containing blood smears to be detected, and the slide clamping device 135 is further configured to clamp the blood smears to be detected in the slide basket loaded by the slide basket loading device 137 to the identification device 134 for identity information identification. The slide basket loading device 137 is connected to the first transmission track 150, so that the blood smears prepared by the smear preparation device 120 can be transported to the image analysis system 130.

[0064] Due to various reasons, the sample image obtained by shooting can have various low-quality problems, such as underexposure, image blur, low resolution, etc., and thus image enhancement is needed for the sample image.

[0065] The image analysis device 133 can be configured to process the sample image obtained by shooting using a trained neural network to obtain an enhanced target image, which is output to the user.

[0066] Exemplarily, the neural network can be any deep neural network (DNN) known in the art, such as a generative adversarial network (GAN), a super-resolution convolutional neural network (SRCNN), etc., which is not limited in the present application. The deep neural network comprises an input layer, multiple hidden layers and an output layer, which automatically learns and discovers the distributed feature representation of the data. Compared with traditional machine learning algorithms, the deep neural network has the advantages of automatic learning and mining of data features without manual extraction of classification features, and can obtain higher recognition accuracy.

[0067] Specifically, the image analysis device 133 can determine that a sample image captured by the imaging device 131 is a low-quality image (see Figure 9A , which shows an underexposed sample image captured by a 100x objective), determine a capturing position corresponding to the low-quality image, and then adjust the capturing parameters of the imaging device 131 to capture a high-quality image at the capturing position (see Figure 9B and Figure 9C , which respectively show well-exposed sample images captured by a 100x and a 40x objective), and then train a neural network using the image data of the high-quality image to obtain a trained neural network, so that the trained neural network can enhance the low-quality image.

[0068] For example, whether an image is an underexposed image can be determined according to the exposure time of the camera, such as an exposure time less than 100 us being an underexposed sample image; whether a sample image is a low-resolution sample image can be determined according to the pixel resolution and physical size of the image, such as a distance represented by one pixel of the sample image being greater than a certain value (e.g., 0.05 um); and whether a sample image is a blurred image can be determined according to the gradient information of the image, such as the image gradient being less than a certain threshold (e.g., 100), or according to the distribution of high and low frequency components in the frequency domain, such as the proportion of low frequency components exceeding a certain threshold (e.g., 60%).

[0069] A deep neural network needs a large number of labeled samples for training, and the purpose is to optimize the weights of each neuron in the network to make the performance of the neural network optimal. The training of the neural network can use the gradient descent method, and the weights of all neurons are constantly adjusted according to the error (such as mean square error) between the predicted output of the current neural network and the target result, so that the error between the predicted output of the neural network and the target result is minimized.

[0070] Specifically, the training process can include: obtaining a low-quality image and a high-quality image of the same sample region captured by the imaging device 131, training the neural network according to the low-quality image and the high-quality image, and obtaining the parameters of the neural network.

[0071] For example, when the sample images captured by the imaging device include low-quality images of different low-quality types, for low-quality images of multiple low-quality types, multiple neural networks can be trained to enhance low-quality images of a corresponding low-quality type in the multiple low-quality types. For example, for low-quality images of multiple low-quality types, one neural network can be trained to enhance low-quality images of all low-quality types.

[0072] Specifically, a large number of underexposed images and corresponding overexposed images can be used to train the neural network in the present application, and then a trained neural network for underexposed images is obtained; or a large number of blurred images and corresponding clear images can be used to train the neural network, and then a trained neural network for blurred images is obtained; or a large number of low-resolution images and corresponding high-resolution images can be used to train the neural network, and then a trained neural network for low-resolution images is obtained; further, two or three types of low-quality images and corresponding high-quality images can be selected to train the neural network, and finally a neural network suitable for two or three types of low-quality images can be obtained.

[0073] Exemplarily, for underexposed images, the neural network can be DnCNN, LLCNN, LLNet, etc.; for blurred images, the neural network can be GAN, FCN, DnCNN, etc.; for low-resolution images, the neural network can be EDSN, SRCNN, RCAN, etc.

[0074] Exemplarily, a plurality of neural networks can be trained to enhance a corresponding target component in a plurality of target components in a low-quality image according to different target components contained in the low-quality image. Exemplarily, the target component is any one or more cells required to be detected or analyzed, such as red blood cells, white blood cells, platelets, etc.

[0075] Exemplarily, the image analysis device 133 can identify the low-quality type of the low-quality image, and according to the low-quality type of the low-quality image, the trained neural network for the low-quality image of the low-quality type is used to process the sample image.

[0076] Exemplarily, the low-quality type of the low-quality image includes at least one of underexposure, image blur, low resolution, etc., and the high-quality image includes overexposure, clear image, high-resolution image, etc., which are not limited by the present application.

[0077] In one embodiment, the image analysis device 133 can use the trained neural network to perform image enhancement on the entire sample image to obtain an enhanced entire sample image (for details, see the process schematic diagram of FIG. 13B). Figure 10 ) Exemplarily, the enhanced entire sample image can include a high-quality image.

[0078] Exemplarily, the image analysis device 133 uses the trained neural network to perform image enhancement on the entire sample image can include the following steps:

[0079] Step S1: extracting image features from image data of the entire sample image.

[0080] Step S2: performing nonlinear mapping processing on the extracted image features.

[0081] Step S3: performing image reconstruction according to the processed image features to obtain an enhanced whole sample image.

[0082] Specifically, taking an example of an SRCNN network containing three convolutional layers, which enhances a low-quality image, the following is described:

[0083] First, the low-quality image is input into the first convolutional layer, which extracts a plurality of image blocks from the low-quality image and obtains an N1 (such as 64) dimensional feature matrix through convolution operation; the formula is as follows, wherein F1(Y) represents the output of the first convolutional layer, W1 represents the convolution kernel of the first convolutional layer, and B1 represents the bias of the first convolutional layer.

[0084] F 1(Y)=max(0,W1*Y+B1)

[0085] Then, the second convolutional layer realizes nonlinear mapping through convolution, mapping the N1 dimensional feature matrix into an N2 (such as 32) dimensional feature matrix; the formula is as follows: wherein F2(Y) represents the output of the second convolutional layer, F1(Y) represents the output of the first convolutional layer, W2 represents the convolution kernel of the second convolutional layer, and B2 represents the bias of the second convolutional layer.

[0086] F2(Y)=max(0,W2*F1(Y)+B2)

[0087] Finally, the third convolutional layer performs convolution operation on the N2 dimensional feature matrix to realize image reconstruction; that is, the N2 dimensional feature matrix is restored to a high-quality image and output. The formula is as follows: wherein F3(Y) represents the output of the third convolutional layer, F2(Y) represents the output of the second convolutional layer, W3 represents the convolution kernel of the third convolutional layer, and B3 represents the bias of the third convolutional layer.

[0088] F3(Y)=W3*F2(Y)+B3

[0089] Exemplarily, according to the needs of the user, the image analysis device 133 can output the enhanced whole sample image as an enhanced target image to the user.

[0090] Exemplarily, the image analysis device 133 can also perform target detection on the enhanced whole sample image and extract a part of the sample image in the region where the target component is located as an enhanced target image for output to the user according to the user's needs. Specifically, any method known in the art can be used to perform target detection and image segmentation on the enhanced whole sample image, for example, target detection and image segmentation can be performed automatically using a neural network, target detection and image segmentation can also be performed using a conventional method, and image segmentation can also be performed manually by the user, which is not limited in the present application. For example, a detection and segmentation method based on edge detection can be used. The process of using the detection and segmentation method based on edge detection can be: first, performing grayscale processing on the sample image to obtain a grayscale image, then using an edge detection algorithm to identify the edges of various components in the image, and then segmenting various components (e.g., red blood cells, white blood cells, platelets, epithelial cells, crystals, casts, etc.) based on the detected edges, and obtaining the component of interest as the interest object according to the size, grayscale, etc. of various components. Exemplarily, the edge detection algorithm can include any edge detection algorithm known in the art, for example, a difference edge detection algorithm, a Roberts edge detection algorithm, a Sobel edge detection algorithm, a Prewitt edge detection algorithm, a Laplace edge detection algorithm, a LOG (Gaussian Laplace) edge detection algorithm, a Canny edge detection algorithm, etc., which is not limited in the present application.

[0091] In another embodiment, the image analysis device 133 can also perform target detection and image segmentation on the whole sample image obtained by shooting first, and extract a part of the sample image in the region where the target component is located as a target image, and then use the trained neural network to enhance the target image to obtain an enhanced target image (for the process schematic diagram, see Figure 11 ). Exemplarily, the enhanced target image includes a high-quality image.

[0092] Specifically, any method known in the art can be used to perform target detection and image segmentation on the whole sample image, for example, the detection and segmentation method based on edge detection described above, etc., which is not limited in the present application.

[0093] Exemplarily, the step of the image analysis device 133 enhancing the target image to obtain an enhanced target image can include:

[0094] Step S4: extracting image features from the image data of the target image using the trained neural network;

[0095] Step S5: performing nonlinear mapping processing on the extracted image features using the trained neural network; and

[0096] Step S6: image reconstruction is performed on the processed image features using the trained neural network to obtain an enhanced target image.

[0097] The control device 140 is in communication connection with the blood analyzer 110, the smear preparation device 120 and the image analysis system 130. As Figure 5 , the control device 140 at least includes a processing component 141, a RAM 142, a ROM 143, a communication interface 144, a storage 146 and an I / O interface 145, wherein the processing component 141, the RAM 142, the ROM 143, the communication interface 144, the storage 146 and the I / O interface 145 are in communication through a bus 147.

[0098] The processing component can be a CPU, a GPU or other chip with computing ability.

[0099] The storage 146 stores various computer programs and data required for executing the computer programs for the processing component 141 to execute, such as an operating system and application programs. In addition, data required to be stored locally during sample detection can also be stored in the storage 146.

[0100] The I / O interface 145 is composed of a serial interface such as USB, IEEE 1394 or RS-132C, a parallel interface such as SCSI, IDE or IEEE 1284, and an analog signal interface composed of a D / A converter and an A / D converter. The I / O interface 145 is connected with an input device composed of a keyboard, a mouse, a touch screen or other control buttons, and a user can directly input data to the control device 140 using the input device. In addition, the I / O interface 145 can also be connected with a display having a display function, such as a liquid crystal screen, a touch screen, an LED display screen, etc., and the control device 140 can output processed data in the form of image display data to the display for display, such as analysis data, instrument operating parameters, etc.

[0101] The communication interface 144 can be an interface of any communication protocol known at present. The communication interface 144 communicates with the outside through a network. The control device 140 can transmit data between any device connected through the network in a certain communication protocol through the communication interface 144.

[0102] According to another embodiment of the present application, an image analysis system is provided, referring to Figure 6 , Figure 6 A schematic structural block diagram of an image analysis system 600 according to an embodiment of the present application is shown.

[0103] As Figure 6As shown, the image analysis system 600 can include an imaging device 601, a carrier holding device 602, and an image analysis device 603.

[0104] The carrier holding device 602 is configured to move the sample carrier carrying the sample to be tested relative to the imaging device 601 so that the sample carrier is located in the field of view of the imaging device 601.

[0105] Exemplarily, the sample to be tested can be blood, urine, body fluid, bone marrow, and other excretions and secretions, etc., which are not limited by the present application. Exemplarily, the sample carrier can be a sample smear, a counting pool, etc. The blood, bone marrow, body fluid, and other excretions and secretions, etc. can be made into a sample smear (i.e. a slide) by a smear preparation device, and the urine can be carried by a counting pool.

[0106] Exemplarily, the carrier holding device 602 can be a driving device configured to drive the sample carrier or a lens group of the imaging device to move so that the sample carrier is located in the field of view of the imaging device 601. Exemplarily, the carrier holding device 602 can also be a grabbing device configured to grab the sample carrier and place it in the field of view of the imaging device 601.

[0107] The imaging device 601 includes a camera and a lens group, which can include a first objective lens and a second objective lens. Exemplarily, the first objective lens can be a 10x objective lens, and the second objective lens can be a 100x objective lens. The lens group can further include a third objective lens, which can be a 40x objective lens. The lens group can further include an ocular lens.

[0108] The imaging device 601 is configured to capture a specific area on the sample carrier in the field of view to obtain a sample image. Specifically, the imaging device 601 can capture the sample image using any one of the objective lenses.

[0109] Due to various reasons, the captured sample image can have various low-quality problems, such as underexposure, image blur, low resolution, etc., and thus needs to be image-enhanced.

[0110] The image analysis device 603 can be configured to process the captured sample image using a trained neural network to obtain an enhanced target image, which is output to the user.

[0111] Exemplarily, the neural network can be any deep neural network (DNN) known in the art, such as a generative adversarial network (GAN), a super-resolution convolutional neural network (SRCNN), etc., and the present application is not limited in this regard. A deep neural network comprises an input layer, multiple hidden layers, and an output layer, and automatically learns and discovers distributed feature representation of data by using a large number of samples. Compared with a traditional machine learning algorithm, the deep neural network has the advantages of automatic learning and mining of data features without manual extraction of classification features, and can obtain a higher recognition accuracy.

[0112] Specifically, the image analysis device 603 can determine that the sample image obtained by photographing is a low-quality image, determine a photographing position corresponding to the low-quality image, then adjust the photographing parameter of the imaging device 601, re-photograph at the photographing position to obtain a high-quality image of the photographing position, and then train the neural network by using the image data of the high-quality image to obtain a trained neural network, so that the trained neural network can enhance the low-quality image.

[0113] Specifically, the training process can comprise: obtaining a low-quality image and a high-quality image of the same sample region obtained by photographing by the imaging device 601, training the neural network according to the low-quality image and the high-quality image to obtain the parameters of the neural network.

[0114] Exemplarily, when the sample image obtained by photographing by the imaging device comprises low-quality images of different low-quality types, for low-quality images of multiple low-quality types, multiple neural networks can be trained to enhance low-quality images of a low-quality type corresponding to the multiple low-quality types, respectively. Exemplarily, for low-quality images of multiple low-quality types, one neural network can be trained to enhance low-quality images of all low-quality types.

[0115] Exemplarily, multiple neural networks can be trained to enhance a corresponding target component in multiple target components for different target components contained in low-quality images. Exemplarily, the target component is any one or more tangible elements that need to be detected or analyzed. For example, when the sample to be tested is blood or bone marrow, the target component can be red blood cells, white blood cells, platelets and other cells; when the sample to be tested is urine, the target component can be red blood cells, white blood cells, white blood cell clusters, bacteria, yeast-like fungi, epithelial cells, small round epithelial cells, crystals, transparent casts, non-transparent casts, mucus threads, etc.; when the sample to be tested is other excretions and secretions, the target component can be cellular components in samples such as feces, vaginal secretions, semen, prostatic fluid, sputum, etc.: common red blood cells, white blood cells, crystals, pathogenic microorganisms, etc. Organisms, epithelial cells, parasites, sperm, Trichomonas, prostate cholinergic bodies, prostate granular cells, alveolar macrophages, tumor cells, etc.; when the sample to be tested is body cavity fluid, the target components can be cellular components in cerebrospinal fluid, serous cavity effusion, joint cavity effusion, and amniotic fluid: common red blood cells, white blood cells, white blood cell clusters, bacteria, yeast-like fungi, epithelial cells, parasites, etc.; when the sample to be tested is desquamated cells, the target components can be epithelial cells, mesothelial cells, cancer cells, red blood cells, white blood cells, macrophages or tissue cells, necrotic matter (mucus, bacterial clusters, fungal clusters, plant cells, cotton wool and dye residues, etc.), parasites, etc.

[0116] Exemplarily, the image analyzing device 603 may identify the low-quality type of the low-quality image, and according to the low-quality type of the low-quality image, use a neural network trained for low-quality images of the low-quality type to process the sample image.

[0117] Exemplarily, the low-quality types of low-quality images include underexposure, blurred images, low resolution, etc., and the high-quality images include images with sufficient exposure, clear images, high resolution, etc., which are not limited in the present invention.

[0118] In one embodiment, the image analysis device 603 may use a trained neural network to perform image enhancement on the entire sample image to obtain an enhanced entire sample image. For example, the enhanced entire sample image may include a high-quality image.

[0119] Exemplarily, the image analysis device 603 may perform image enhancement on the entire sample image using a trained neural network, including the following steps:

[0120] Step S1: extracting image features from the image data of the entire sample image.

[0121] Step S2: performing nonlinear mapping processing on the extracted image features.

[0122] Step S3: image reconstruction according to the processed image features to obtain an enhanced whole sample image.

[0123] The specific processing procedures of the above steps S1-S3 can refer to the foregoing embodiments, which will not be described here again.

[0124] Exemplarily, according to the user's needs, the image analysis device 603 can output the enhanced whole sample image as an enhanced target image to the user.

[0125] Exemplarily, according to the user's needs, the image analysis device 603 can also perform target detection on the enhanced whole sample image, and extract a part of the sample image in the region where the target component is located as an enhanced target image for output to the user.

[0126] Specifically, any method known in the art can be used to perform target detection and image segmentation on the enhanced whole sample image. Exemplarily, the target detection and image segmentation can be performed automatically using a neural network, or can be performed using a conventional method, or can be performed manually by the user, which is not limited in the present application. For example, a detection and segmentation method based on edge detection can be used. The process of using the detection and segmentation method based on edge detection can be: first performing grayscale processing on the sample image to obtain a grayscale image, then using an edge detection algorithm to identify the edges of various components in the image, and then segmenting various components (such as red blood cells, white blood cells, platelets, epithelial cells, crystals, and casts) based on the detected edges, and obtaining the component of interest as the interest object according to the size and grayscale of various components. Exemplarily, the edge detection algorithm can include any edge detection algorithm known in the art, such as a difference edge detection algorithm, a Roberts edge detection algorithm, a Sobel edge detection algorithm, a Prewitt edge detection algorithm, a Laplace edge detection algorithm, a LOG (Gaussian Laplace) edge detection algorithm, a Canny edge detection algorithm, etc., which is not limited in the present application.

[0127] In another embodiment, the image analysis device 603 can also perform target detection on the whole sample image obtained by shooting, and extract a part of the sample image in the region where the target component is located as a target image by means such as image segmentation, and then use the trained neural network to enhance the target image to obtain an enhanced target image. Exemplarily, the enhanced target image includes a high-quality image.

[0128] Specifically, any method known in the art can be used to perform target detection and image segmentation on the whole sample image, such as the detection and segmentation method based on edge detection described above, which is not limited in the present application.

[0129] Exemplarily, the step of the image analysis device 603 enhancing the target image to obtain an enhanced target image can include:

[0130] Step S4: extracting image features from the image data of the target image by using the trained neural network;

[0131] Step S5: performing nonlinear mapping processing on the extracted image features by using the trained neural network; and

[0132] Step S6: performing image reconstruction on the processed image features by using the trained neural network to obtain the enhanced target image.

[0133] According to yet another embodiment of the present application, a method for a sample analysis system to analyze a sample under test is provided. Referring to Figure 7 , Figure 7 A step flowchart of a method 700 for a sample analysis system to analyze a sample under test according to an embodiment is shown.

[0134] As shown in Figure 7 , the method 700 can include the following steps:

[0135] Step S710: transporting the sample under test to a blood analyzer in the sample analysis system, and the blood analyzer detects the sample under test to obtain various detection parameters of the sample under test, so as to preliminarily determine whether the sample under test is abnormal according to the various detection parameters.

[0136] Exemplarily, the sample under test can include whole blood, serum, peripheral blood, etc. Exemplarily, the detection parameters can include blood routine parameters such as white blood cells, red blood cells, hemoglobin, platelets, etc., and can also include other blood parameters.

[0137] Exemplarily, the method 700 can further include determining whether the sample under test needs to be photographed according to the detection parameters. Specifically, if yes, i.e., the detection parameters indicate that the blood cells of the sample under test are abnormal, such as low white blood cell value, platelet aggregation, lipemia, etc., the image of the cells needs to be photographed, then the following steps are executed to further identify the cell morphology and thus make more accurate diagnosis; if no, the following steps are not executed.

[0138] Step S720: transporting the sample under test to a smear preparation device in the sample analysis system, and the smear preparation device prepares a blood smear based on the sample under test.

[0139] Specifically, after the sample to be tested is transported to the smear preparation device, the following steps can be used to prepare the blood smear: first, the sample is mixed, then the sample device (e.g., a sample needle) is used to draw the sample, the sample drawing mechanism is used to drop the drawn blood onto a slide, and the slide pushing mechanism is used to push the blood into a film shape on the slide. Generally, after the slide pushing operation is completed, the blood film on the slide can be dried to stabilize its shape.

[0140] Step S730: The smear holding device in the sample analysis system moves the blood smear relative to the imaging device in the sample analysis system so that the blood smear is located in the field of view of the imaging device.

[0141] Exemplarily, the smear holding device is used to place or hold the blood smear, and the driving device is used to drive the smear holding device or the imaging device to move so that the blood smear is located in the field of view of the imaging device. Exemplarily, the smear holding device can further include a grabbing device used to grab the blood smear and place it in the field of view of the imaging device.

[0142] Step S740: The imaging device captures a specific region of the blood smear in the field of view to obtain a sample image.

[0143] Exemplarily, the imaging device can use different objectives (e.g., objectives with magnifications of 40x, 100x, etc.) to capture the sample image of the specific region of the blood smear as needed. Due to various reasons, the captured sample image can have various low-quality problems, such as underexposure, image blur, low resolution, etc., and thus the sample image needs to be image-enhanced.

[0144] Step S750: The image analysis device in the sample analysis system processes the sample image using a trained neural network to obtain an enhanced target image.

[0145] After the target image is obtained, target classification can also be performed based on the target image, and the classification result and / or the target image can be output to the user.

[0146] Exemplarily, the neural network can be any deep neural network (DNN) known in the art, such as a generative adversarial network (GAN), a super-resolution convolutional neural network (SRCNN), etc., and the present application does not limit the neural network. The deep neural network includes an input layer, multiple hidden layers, and an output layer, and automatically learns and discovers distributed feature representations of data using a large number of samples. Compared with traditional machine learning algorithms, the deep neural network has the advantages of automatic learning and mining of data features without manual extraction of classification features, and can obtain higher recognition accuracy.

[0147] In one embodiment, the step can include: the image analysis device employs the trained neural network to perform image enhancement on the whole sample image to obtain an enhanced whole sample image. Illustratively, the enhanced whole sample image can include a high-quality image.

[0148] Illustratively, the image analysis device employing the trained neural network to perform image enhancement on the whole sample image can include the following steps:

[0149] Step S1: extracting image features from image data of the whole sample image.

[0150] Step S2: performing nonlinear mapping processing on the extracted image features.

[0151] Step S3: performing image reconstruction according to the processed image features to obtain an enhanced whole sample image.

[0152] The specific processing procedures of the above steps S1-S3 can refer to the foregoing embodiments, which will not be described here again.

[0153] Illustratively, according to user needs, the enhanced whole sample image can be used as an enhanced target image for output to the user.

[0154] Illustratively, according to user needs, the enhanced whole sample image can also be subjected to target detection, and a part of the sample image in a region where a target component is located can be extracted as an enhanced target image for output to the user.

[0155] Specifically, any method known in the art can be used for target detection and image segmentation on the enhanced whole sample image. For example, target detection and image segmentation can be performed automatically using a neural network, or target detection and image segmentation can be performed using a conventional method, or image segmentation can be performed manually by a user, and the present application does not limit the same. For example, an edge detection-based detection and segmentation method can be used. The process of using the edge detection-based detection and segmentation method can be as follows: first, the sample image is subjected to grayscale processing to obtain a grayscale image, then an edge detection algorithm is used to identify the edges of various components in the image, and then the various components (e.g., red blood cells, white blood cells, platelets, epithelial cells, crystals, casts, etc.) are segmented based on the detected edges, and the components of interest are obtained as the objects of interest according to the size, grayscale, etc. of the various components. For example, the edge detection algorithm can include any edge detection algorithm known in the art, such as a difference edge detection algorithm, a Roberts edge detection algorithm, a Sobel edge detection algorithm, a Prewitt edge detection algorithm, a Laplace edge detection algorithm, a LOG (Gaussian Laplace) edge detection algorithm, a Canny edge detection algorithm, etc., and the present application does not limit the same.

[0156] In another embodiment, the step can include: the image analysis device first performs target detection and image segmentation on the photographed whole sample image, and extracts a part of the sample image in the region of the target component as a target image, and then uses the trained neural network to enhance the target image to obtain an enhanced target image. For example, the enhanced target image includes a high-quality image.

[0157] Specifically, any method known in the art can be used for target detection and image segmentation on the whole sample image, such as the edge detection-based detection and segmentation method described above, and the present application does not limit the same.

[0158] For example, the step of the image analysis device enhancing the target image to obtain an enhanced target image can include:

[0159] Step S4: extracting image features from the image data of the target image using the trained neural network;

[0160] Step S5: performing nonlinear mapping processing on the extracted image features using the trained neural network; and

[0161] Step S6: performing image reconstruction on the processed image features using the trained neural network to obtain an enhanced target image.

[0162] Exemplarily, method 700 may also include: determining that the sample image obtained by capture is a low-quality image, obtaining the shooting position corresponding to the low-quality image, then adjusting the shooting parameters of the imaging device, re-shooting at the shooting position to obtain a high-quality image of the shooting position, and then training a neural network with the image data of the low-quality image and the high-quality image to obtain a trained neural network, so that the trained neural network can enhance the low-quality image.

[0163] Specifically, the training process may include: acquiring low-quality images and high-quality images taken by an imaging device on the same sample area, training a neural network based on the low-quality images and the high-quality images, and obtaining parameters of the neural network.

[0164] For example, when the sample images captured by the imaging device include low-quality images of different low-quality types, method 700 may further include: for low-quality images of multiple low-quality types, training multiple neural networks to enhance low-quality images corresponding to one of the multiple low-quality types. For example, for low-quality images of multiple low-quality types, training one neural network to enhance low-quality images of all low-quality types.

[0165] For example, method 700 may further include: training multiple neural networks to enhance corresponding target components from among the multiple target components, for different target components contained in the low-quality image. For example, the target component may be any one or more of the formed elements mentioned in the aforementioned embodiments, such as red blood cells, white blood cells, platelets, etc.

[0166] Exemplarily, the method 700 may further include: identifying a low-quality type of the sample image, and processing the sample image using a neural network trained for low-quality images of the low-quality type according to the low-quality type of the low-quality image.

[0167] Exemplarily, the low-quality types of low-quality images include underexposure, blurred images, low resolution, etc., and the high-quality images include images with sufficient exposure, clear images, high resolution, etc., which are not limited in the present invention.

[0168] According to another embodiment of the present invention, a method for processing a sample image in an image analysis system is provided. Figure 8 , Figure 8 A flowchart of a method 800 for an image analysis system to process a sample image according to one embodiment is shown.

[0169] like Figure 8 As shown, method 800 may include the following steps:

[0170] Step S810: moving the sample carrier carrying the sample to be tested relative to the imaging device in the image analysis system so that the sample carrier is located in the field of view of the imaging device.

[0171] Exemplarily, the sample to be tested can be blood, urine, body fluid, bone marrow, and other excretions and secretions, etc., which are not limited by the present application. Exemplarily, the sample carrier can be a sample slide, a counting pool, etc. Among them, blood, bone marrow, body fluid, and other excretions and secretions, etc. can be made into a sample slide (such as a blood smear), and urine can be carried by a counting pool.

[0172] Step S820: capturing a specific area on the sample carrier in the field of view by the imaging device to obtain a sample image.

[0173] Exemplarily, the imaging device can capture the sample image of the specific area of the blood smear according to the needs by using different objective lenses (for example, 40 times, 100 times, etc.). Due to various reasons, the sample image obtained by capturing may have various low-quality problems, such as underexposure, image blur, low resolution, etc., so that image enhancement is needed for the sample image.

[0174] Step S830: processing the sample image by the image analysis device in the image analysis system using the trained neural network to obtain an enhanced target image.

[0175] After obtaining the target image, target classification can be performed based on the target image, or the target image or the classification result obtained by target classification can be output to the user.

[0176] Exemplarily, the neural network can be any deep neural network (DNN) known in the art, such as a generative adversarial network (GAN), a super-resolution convolutional neural network (SRCNN), etc., which are not limited by the present application. The deep neural network includes an input layer, multiple hidden layers, and an output layer, which automatically learns and discovers the distributed feature representation of the data by using a large number of samples. Compared with traditional machine learning algorithms, the deep neural network has the advantages of not manually extracting classification features, automatically learning and mining data features, and obtaining higher recognition accuracy.

[0177] In one embodiment, the step can include: the image analysis device using the trained neural network to perform image enhancement on the entire sample image to obtain an enhanced entire sample image.

[0178] Exemplarily, the image analysis device using the trained neural network to perform image enhancement on the entire sample image can include the following steps:

[0179] Step S1: extracting image features from image data of the entire sample image.

[0180] Step S2: performing nonlinear mapping processing on the extracted image features.

[0181] Step S3: performing image reconstruction according to the processed image features to obtain an enhanced whole sample image. Exemplarily, the enhanced whole sample image can include a high-quality image.

[0182] The specific processing procedures of the above steps S1-S3 can refer to the foregoing embodiments, which will not be described here again.

[0183] Exemplarily, according to the user's needs, the enhanced whole sample image can be used as an enhanced target image for output to the user.

[0184] Exemplarily, according to the user's needs, the enhanced whole sample image can also be subjected to target detection, and a part of the sample image in the region where the target component is located can be extracted as an enhanced target image for output to the user.

[0185] Specifically, any method known in the art can be used to perform target detection and image segmentation on the enhanced whole sample image. Exemplarily, the target detection and image segmentation can be performed automatically using a neural network, or can be performed using a conventional method, or can be performed manually by the user, and the present application does not limit this. For example, an edge detection-based detection and segmentation method can be used. The process of using the edge detection-based detection and segmentation method can be as follows: first, performing grayscale processing on the sample image to obtain a grayscale image, then using an edge detection algorithm to identify the edges of various components in the image, and then segmenting various components (such as red blood cells, white blood cells, platelets, epithelial cells, crystals, and casts) based on the detected edges, and obtaining the component of interest as the interest object according to the size and grayscale of various components. Exemplarily, the edge detection algorithm can include any edge detection algorithm known in the art, such as a difference edge detection algorithm, a Roberts edge detection algorithm, a Sobel edge detection algorithm, a Prewitt edge detection algorithm, a Laplace edge detection algorithm, a LOG (Gaussian Laplace) edge detection algorithm, a Canny edge detection algorithm, etc., and the present application does not limit this.

[0186] In another embodiment, this step can include: the image analysis device performing target detection and image segmentation on the photographed whole sample image, and extracting a part of the sample image in the region where the target component is located as a target image, and then using the trained neural network to enhance the target image to obtain an enhanced target image. Exemplarily, the enhanced target image includes a high-quality image.

[0187] Specifically, any method known in the art can be employed to perform target detection and image segmentation on the whole sample image, such as the aforementioned edge detection-based detection and segmentation method, and the present application is not limited in this regard.

[0188] Exemplarily, the step of enhancing the target image to obtain an enhanced target image by the image analysis device can comprise:

[0189] Step S4: extracting image features from the image data of the target image by using the trained neural network;

[0190] Step S5: performing nonlinear mapping processing on the extracted image features by using the trained neural network; and

[0191] Step S6: performing image reconstruction on the processed image features by using the trained neural network to obtain an enhanced target image.

[0192] Exemplarily, the method 800 can further comprise: determining that a sample image obtained by shooting is a low-quality image, obtaining a shooting position corresponding to the low-quality image, then adjusting shooting parameters of the imaging device, re-shooting at the shooting position to obtain a high-quality image of the shooting position, and then training a neural network by using image data of the low-quality image and the high-quality image to obtain a trained neural network, so that the trained neural network can enhance the low-quality image.

[0193] Specifically, the training process can comprise: obtaining a low-quality image and a high-quality image obtained by the imaging device at the same shooting position, training the neural network according to the low-quality image and the high-quality image to obtain parameters of the neural network.

[0194] Exemplarily, the method 800 can further comprise: for a plurality of low-quality images of a plurality of low-quality types, a plurality of neural networks can be trained respectively as neural networks for enhancing low-quality images of a corresponding low-quality type in the plurality of low-quality types. Exemplarily, for a plurality of low-quality images of a plurality of low-quality types, one neural network can be trained as a neural network for enhancing low-quality images of all low-quality types.

[0195] Exemplarily, the method 800 can further comprise: for different target components contained in the low-quality image, a plurality of neural networks can be trained respectively as neural networks for enhancing a corresponding target component in a plurality of target components. Exemplarily, the target component can be any one or more of the aforementioned formed components, such as red blood cells, white blood cells, platelets, etc.

[0196] Exemplarily, the method 800 can further comprise: identifying a low-quality type of the low-quality image, and processing the sample image by using a trained neural network for the low-quality image of the low-quality type according to the low-quality type of the low-quality image.

[0197] Exemplarily, the low-quality type of the low-quality image comprises at least one of underexposure, image blur, low resolution, etc., and the high-quality image comprises a well-exposed, clear, high-resolution image, etc., which are not limited by the present disclosure.

[0198] In yet another embodiment, a computer readable medium is provided, having stored thereon a computer program, which, when being executed by a computer, performs a method as described in the above embodiments. Any tangible, non-transitory computer-readable medium can be used, including magnetic storage devices (hard disks, floppy disks, etc.), optical storage devices (CD-ROMs, DVDs, Blu-ray discs, etc.), flash memory, and / or the like. These computer program instructions can be loaded onto a general purpose computer, a special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions that execute on the computer or other programmable data processing apparatus create means for implementing the functions specified by the flowchart block or blocks. These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including an implementation to

[0199] In still another embodiment, a neural network is provided, which is trained to be used by an image analysis system to perform a method as described above.

[0200] Exemplarily, the neural network can be any deep neural network (DNN) known in the art, such as a generative adversarial network (GAN), a super-resolution convolutional neural network (SRCNN), etc., which are not limited by the present disclosure.

[0201] The main structure of a deep neural network is a combination of convolutional layers, activation layers, pooling layers, and fully connected layers. For example, a common SRCNN network extracts image-related features through a convolutional layer and an activation layer, then performs a non-linear mapping of the features through another convolutional layer and an activation layer, and finally reconstructs an enhanced image using a convolutional layer.

[0202] Exemplarily, one neural network can be trained to enhance sample images of all low-quality types (e.g., underexposure, image blur, low resolution, etc.).

[0203] Exemplarily, for different low-quality types of the photographed sample images, a plurality of neural networks can also be trained respectively to enhance sample images of a low-quality type in the different low-quality types.

[0204] Exemplarily, for different target components contained in the low-quality images, a plurality of neural networks can also be trained respectively to enhance a target component in the plurality of target components. Exemplarily, the target component can be any one or more of the aforementioned formed components, such as red blood cells, white blood cells, platelets, etc.

[0205] Technical effects of the present application:

[0206] The scheme of the present application uses neural networks for image enhancement, thereby being able to accurately identify various cells or other tangible substances and significantly improving the quality of images.

[0207] Although example embodiments have been described herein with reference to the accompanying drawings, it is to be understood that the above-described example embodiments are merely exemplary and are not intended to limit the scope of the present application thereto. Various changes and modifications can be made thereto by those having ordinary skill in the art without departing from the scope and spirit of the present application. All such changes and modifications are intended to be included within the scope of the present application as claimed in the appended claims.

[0208] In the specification provided herein, a large number of specific details are described. However, it can be understood that embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure the understanding of this specification.

[0209] Similarly, it is to be understood that, in order to simplify the present application and to help understand one or more of the various inventive aspects, various features of the present application are sometimes grouped into a single embodiment, figure or description of a related group. However, this grouping of features of the present application should not be interpreted as reflecting an intention that the claimed application requires more features than those explicitly recited in each claim. Rather, the inventive point is reflected in the corresponding claims, which can be solved with fewer features than all the features of a certain disclosed single embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into the detailed description, in which each claim itself serves as a separate embodiment of the present application.

[0210] Those skilled in the art will appreciate that all features described herein (including all companion claims, abstract and drawings) can be combined in any combination, except where such combinations are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) can be replaced by alternative features that serve the same, equivalent or similar purpose, unless expressly stated otherwise.

[0211] Furthermore, those skilled in the art will recognize that references in this specification to some embodiments include certain features and not others, unless expressly stated otherwise. The combination of features, whether described in the same or different embodiments, implies that the features so combined are within the scope of the application and form different embodiments of the application. For example, in the claims, any of the claimed embodiments can be used in any combination.

[0212] It is to be understood that the embodiments described above are merely illustrative of the application and do not pose a restriction on the scope of the application. Alternative embodiments can be devised without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The application can be implemented by means of both hardware and software, and any combination thereof. In a unitary claim, several devices or means can be listed, comprising means for performing a certain task. The use of the term "means" in a claim is intended to cover one or more elements that perform the specified function. The use of the terms "first", "second" and "third" etc. does not limit the number of these elements. These terms are only used as distinguishable names.

[0213] The above description is only specific embodiments of the present application or specific explanations of specific embodiments. The protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered by the protection scope of the present application. The protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for analyzing a sample to be tested by a sample analysis system, characterized in that: The method comprises: The sample to be tested is transported to the blood analyzer in the sample analysis system, and the blood analyzer tests the sample to be tested to obtain various test parameters of the sample to be tested; transporting the sample to be tested to a smear preparation device in the sample analysis system, wherein the smear preparation device prepares a blood smear based on the sample to be tested; The smear holding device in the sample analysis system moves the blood smear relative to the imaging device in the sample analysis system so that the blood smear is located within the field of view of the imaging device; photographing a specific area of ​​the blood smear within a field of view by the imaging device to obtain a sample image; The image analysis device in the sample analysis system processes the sample image using a trained neural network to obtain an enhanced target image; The method further comprises: for a plurality of different types of sample images, respectively training a plurality of neural networks to enhance a neural network corresponding to a type of sample image among the different types of sample images; The plurality of different types of sample images include sample images containing different target components; for the plurality of different types of sample images, respectively training the plurality of neural networks to enhance a neural network corresponding to a type of sample image among the different types of sample images includes: for the different target components contained in the sample images, respectively training the plurality of neural networks to enhance a neural network corresponding to a target component among the plurality of target components; The step of processing the sample image using the trained neural network to obtain an enhanced target image includes: The type of the sample image is identified, and according to the type of the sample image, a neural network trained for sample images of the type is used to process the sample image.

2. The method according to claim 1, wherein The sample image captured by the imaging device includes a low-quality image; the enhanced target image includes a high-quality image; the low-quality type of the low-quality image includes at least one of underexposure, image blur, and low resolution; the high-quality image includes an image with sufficient exposure, a clear image, or a high resolution; The plurality of sample images of different types include a plurality of low-quality images of different low-quality types; For a plurality of different types of sample images, respectively training a plurality of neural networks to enhance a neural network corresponding to a type of sample image among the different types of sample images includes: For low-quality images of multiple low-quality types, multiple neural networks are respectively trained to enhance the low-quality images corresponding to a low-quality type among the multiple low-quality types.

3. The method according to claim 2, wherein The method further comprises: For underexposed images, the neural network is DnCNN, LLCNN, or LLNet; and / or For blurred images, the neural network is GAN, FCN, or DnCNN; and / or For low-resolution images, the neural network is EDSN, SRCNN or RCAN.

4. The method according to claim 1, wherein The step of using the trained neural network to process the sample image to obtain an enhanced target image includes: The trained neural network is used to enhance the entire sample image to obtain an enhanced entire sample image.

5. The method according to claim 4, wherein The step of using the trained neural network to enhance the entire sample image to obtain an enhanced entire sample image includes: extracting image features from image data of the entire sample image using the trained neural network; Performing nonlinear mapping processing on the extracted image features using the trained neural network; and The trained neural network is used to perform image reconstruction based on the processed image features to obtain an enhanced entire sample image.

6. The method according to claim 4, wherein Also includes: Using the enhanced entire sample image as an enhanced target image; or, Target detection is performed on the enhanced entire sample image, and a portion of the sample image in the area where the target component is located is extracted as the enhanced target image.

7. The method according to claim 1, wherein The step of using the trained neural network to process the sample image to obtain an enhanced target image includes: Performing target detection on the entire sample image and extracting a portion of the sample image in the area where the target component is located as the target image; The trained neural network is used to enhance the target image to obtain an enhanced target image.

8. The method according to claim 7, wherein The step of using the trained neural network to enhance the target image to obtain an enhanced target image includes: extracting image features from image data of the target image using the trained neural network; Performing nonlinear mapping processing on the extracted image features using the trained neural network; and The trained neural network is used to perform image reconstruction on the processed image features to obtain the enhanced target image.

9. The method according to claim 1, wherein The method further includes: determining whether the sample to be tested needs to be photographed according to the detection parameters, If so, the step of transporting the sample to be tested to the smear preparation device in the sample analysis system is performed; If not, the step of transporting the sample to be tested to the smear preparation device in the sample analysis system is not performed.

10. The method according to any one of claims 1 to 9, wherein The trained neural network is a deep neural network; the sample images captured by the imaging device include low-quality images; The method further includes: for low-quality images of multiple low-quality types, training multiple neural networks respectively to be neural networks for enhancing low-quality images corresponding to one of the multiple low-quality types.

11. The method according to claim 10, wherein The step of using the trained neural network to process the sample image to obtain an enhanced target image further includes: The low-quality type of the sample image is identified, and according to the low-quality type of the low-quality image, a neural network trained for low-quality images of the low-quality type is used to process the sample image.

12. A method for processing a sample image in an image analysis system, characterized in that: The method comprises: Move a sample carrier carrying a sample to be tested relative to an imaging device in the image analysis system so that the sample carrier is located within a field of view of the imaging device; The imaging device captures a specific area on the sample carrier within a field of view to obtain a sample image; The image analysis device in the image analysis system processes the sample image using a trained neural network to obtain an enhanced target image; The method further comprises: for a plurality of different types of sample images, respectively training a plurality of neural networks to enhance a neural network corresponding to a type of sample image among the different types of sample images; The plurality of different types of sample images include sample images containing different target components; for the plurality of different types of sample images, respectively training the plurality of neural networks to enhance a neural network corresponding to a type of sample image among the different types of sample images includes: for the different target components contained in the sample images, respectively training the plurality of neural networks to enhance a neural network corresponding to a target component among the plurality of target components; The step of processing the sample image using the trained neural network to obtain an enhanced target image includes: The type of the sample image is identified, and according to the type of the sample image, a neural network trained for sample images of the type is used to process the sample image.

13. The method according to claim 12, wherein: The step of using the trained neural network to process the sample image to obtain an enhanced target image includes: The trained neural network is used to enhance the entire sample image to obtain an enhanced entire sample image.

14. The method according to claim 13, wherein The step of using the trained neural network to enhance the entire sample image to obtain an enhanced entire sample image includes: extracting image features from image data of the entire sample image using the trained neural network; Performing nonlinear mapping processing on the extracted image features using the trained neural network; and The trained neural network is used to perform image reconstruction based on the processed image features to obtain an enhanced entire sample image.

15. The method according to claim 13, wherein Also includes: Using the enhanced entire sample image as an enhanced target image; or, Target detection is performed on the enhanced entire sample image, and a portion of the sample image in the area where the target component is located is extracted as the enhanced target image.

16. The method according to claim 12, wherein The step of using the trained neural network to process the sample image to obtain an enhanced target image includes: Performing target detection on the entire sample image and extracting a portion of the sample image in the area where the target component is located as the target image; The trained neural network is used to enhance the target image to obtain an enhanced target image.

17. The method according to claim 16, wherein The step of using the trained neural network to enhance the target image to obtain an enhanced target image includes: extracting image features from image data of the target image using the trained neural network; Performing nonlinear mapping processing on the extracted image features using the trained neural network; and The trained neural network is used to perform image reconstruction on the processed image features to obtain the enhanced target image.

18. The method according to any one of claims 12 to 17, wherein The trained neural network is a deep neural network; the sample image captured by the imaging device includes a low-quality image; and the enhanced target image includes a high-quality image; The low-quality type of the low-quality image includes at least one of underexposure, blurred image and low resolution; the high-quality image includes an image with sufficient exposure, clear image or high resolution.

19. The method according to claim 18, wherein The plurality of sample images of different types include a plurality of low-quality images of different low-quality types; For a plurality of different types of sample images, respectively training a plurality of neural networks to enhance a neural network corresponding to a type of sample image among the different types of sample images includes: For low-quality images of multiple low-quality types, multiple neural networks are respectively trained to enhance the low-quality images corresponding to a low-quality type among the multiple low-quality types.

20. The method of claim 18, wherein: The method further comprises: For underexposed images, the neural network is DnCNN, LLCNN, or LLNet; and / or For blurred images, the neural network is GAN, FCN, or DnCNN; and / or For low-resolution images, the neural network is EDSN, SRCNN or RCAN.

21. The method according to claim 12, wherein The sample to be tested includes blood or urine; the sample carrier includes a sample smear or a counting pool.

22. A sample analysis system, characterized in that: The sample analysis system comprises: A blood analyzer is used to detect the sample to be tested to obtain various detection parameters of the sample to be tested; a smear preparation device, used for preparing a blood smear based on the sample to be tested; An image analysis system comprises an imaging device, a smear holding device and an image analysis device, wherein: The smear holding device is used to move the blood smear relative to the imaging device so that the blood smear is located within the field of view of the imaging device; The imaging device is used to capture a specific area on the blood smear within a visual field to obtain a sample image; The image analysis device is used to process the sample image using a trained neural network to obtain an enhanced target image; The image analysis device is further configured to: for a plurality of different types of sample images, respectively train a plurality of neural networks to enhance a neural network corresponding to a type of sample image among the different types of sample images; The image analysis device processes the sample image using the trained neural network to obtain an enhanced target image, comprising: The type of the sample image is identified, and according to the type of the sample image, a neural network trained for sample images of the type is used to process the sample image.

23. An image analysis system, characterized in that: The image analysis system includes an imaging device, a carrier holding device and an image analysis device, wherein: The carrier holding device is used to move the sample carrier carrying the sample to be tested relative to the imaging device so that the sample carrier is located within the field of view of the imaging device; The imaging device is used to capture a specific area on the sample carrier within a field of view to obtain a sample image; The image analysis device is used to process the sample image using a trained neural network to obtain an enhanced target image; The image analysis device is further configured to: for a plurality of different types of sample images, respectively train a plurality of neural networks to enhance a neural network corresponding to a type of sample image among the different types of sample images; The image analysis device processes the sample image using the trained neural network to obtain an enhanced target image, comprising: The type of the sample image is identified, and according to the type of the sample image, a neural network trained for sample images of the type is used to process the sample image.

24. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program, when executed by one or more processors, enables the performance of the method according to any one of claims 1 to 21.

25. A neural network, characterized in that The neural network is trained to be usable by an image analysis device to perform the method according to any one of claims 1 to 21.

Images