A deep learning-based parasitic worm assisted detection system

Abstract

The application discloses a kind of based on deep learning's parasitic auxiliary detection system, comprising: image acquisition module, for according to received detection instruction, obtains smear specimen image and carries out type mark;Image pre-processing module is used to carry out deblurring processing to smear specimen image to obtain the image to be measured smear image, and then it is input according to image type to thick blood smear detection module or thin blood smear detection module;Thick blood smear detection module is used to identify and remove white blood cell from the image to be measured smear image, and then carries out parasitic detection to obtain first detection result;Thin blood smear detection module is used to identify and remove white blood cell from the image to be measured smear image, and then carries out abnormal red blood cell detection to obtain second detection result;Interactive display module is used to display first detection result or second detection result.The application can realize more intelligent and efficient and reliable auxiliary detection to the parasitic condition in the blood smear of patient.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, in particular to a parasitic auxiliary detection system based on deep learning. BACKGROUND

[0002] At present, the dyed human blood smear sample is usually observed by artificial operation of a microscope to diagnose whether there is a parasite, but this operation mode also has the following disadvantages: first, the cost of training or hiring professional diagnostic personnel is high; second, artificial detection is time-consuming and low in efficiency; third, the pathological analysis process is tedious, low in automation and strong in subjectivity; fourth, in the environment of resource shortage, professional diagnostic personnel usually work independently, and there is no strict review system to ensure their diagnosis quality. Therefore, it is a problem to be solved by the present application to put forward a more intelligent and more efficient and reliable auxiliary detection system to replace the artificial detection operation mode. SUMMARY

[0003] The present application provides a parasitic auxiliary detection system based on deep learning to solve one or more technical problems existing in the prior art, at least to provide a beneficial choice or create conditions.

[0004] The present application provides a parasitic auxiliary detection system based on deep learning, which comprises an image acquisition module, an image preprocessing module, a thick blood smear detection module, a thin blood smear detection module and an interactive display module.

[0005] The image acquisition module is used to acquire a smear specimen image and mark its type according to the received detection instruction;

[0006] The image preprocessing module is used to perform deblurring processing on the smear specimen image to obtain a to-be-detected smear image, and then input it to the thick blood smear detection module or the thin blood smear detection module according to the image type;

[0007] The thick blood smear detection module is used to identify and remove white blood cells from the to-be-detected smear image, and then perform parasitic detection to obtain a first detection result, which includes a smear image marking the position of the parasite and the number of parasites;

[0008] The thin blood smear detection module is used to identify and remove white blood cells from the to-be-detected smear image, and then perform abnormal red blood cell detection to obtain a second detection result, which includes a smear image marking the position of the abnormal red blood cell, the total number of red blood cells and the number of abnormal red blood cells;

[0009] The interactive display module is used to display the first detection result or the second detection result.

[0010] Further, the image acquisition module is operated after the user places the prepared smear specimen on the microscope, comprising:

[0011] a first image acquisition unit, configured to acquire a smear specimen image fed back after an external camera device captures an image presented by a microscope objective according to a received thick blood smear detection instruction, and mark it as a thick blood smear type;

[0012] a second image acquisition unit, configured to acquire a smear specimen image fed back after an external camera device captures an image presented by a microscope objective according to a received thin blood smear detection instruction, and mark it as a thin blood smear type.

[0013] Further, the image preprocessing module comprises:

[0014] an acquisition unit, configured to convert the smear specimen image into a double-precision gray image, and then acquire a horizontal coordinate value of each pixel point to form a first data set, and acquire a vertical coordinate value of each pixel point to form a second data set;

[0015] a conversion unit, configured to convert the first data set into a first gradient set and the second data set into a second gradient set according to a gradient filter;

[0016] a parsing unit, configured to parse the first gradient set and the second gradient set by a fuzzy inference engine to obtain a first output set, and then perform binaryzation processing on each value in the first output set to obtain a second output set;

[0017] a first extraction unit, configured to create an initial mask image according to the second output set, and then extract a white pixel region therefrom;

[0018] an expansion unit, configured to determine a circle center in the white pixel region, expand the white pixel region by a maximum distance from the circle center to a boundary line of the white pixel region as a radius, and obtain a final mask image;

[0019] a second extraction unit, configured to extract a clear part from the smear specimen image by using the final mask image, and obtain a to-be-detected smear image.

[0020] Further, converting the first data set into a first gradient set according to a gradient filter comprises:

[0021] characterizing the gradient filter as a second row matrix, characterizing the first data set as a first row matrix, and performing convolution operation on the first row matrix and the second row matrix to obtain the first gradient set.

[0022] Further, the thick blood smear detection module comprises:

[0023] a first white blood cell recognition module, configured to recognize all white blood cells in the to-be-tested smear image and mark the positions of the white blood cells according to a K-means algorithm and an Otsu algorithm;

[0024] a parasite detection module, configured to remove all white blood cells from the to-be-tested smear image according to the position marking result, and perform parasite detection on the processed to-be-tested smear image by using a trained VGG19 neural network to obtain the first detection result.

[0025] Further, the first white blood cell recognition module comprises:

[0026] a generation unit, configured to convert the to-be-tested smear image into a first smear image in a LAB space, generate a first mask image corresponding to the first smear image by using the K-means algorithm, and generate a second mask image corresponding to the first smear image by using the Otsu algorithm;

[0027] a correction unit, configured to check and correct all connected domains included in the second mask image by using all connected domains included in the first mask image;

[0028] a first output unit, configured to finally output the to-be-tested smear image in which all white blood cells are marked after performing area eligibility screening on all connected domains included in the processed second mask image.

[0029] Further, the parasite detection module comprises:

[0030] a first preprocessing unit, configured to perform image contrast enhancement processing on the to-be-tested smear image after removing all white blood cells from the to-be-tested smear image according to the position marking result, and cut all candidate images from the processed to-be-tested smear image according to the appearance features of parasites;

[0031] a first detection unit, configured to perform parasite detection on the all candidate images by using the trained VGG19 neural network, and finally output the first detection result.

[0032] Further, the thin blood smear detection module comprises:

[0033] a second white blood cell recognition module, configured to screen all to-be-tested regions from the to-be-tested smear image by using the K-means algorithm, and recognize all white blood cells in the to-be-tested regions and mark the positions of the white blood cells by using the Otsu algorithm;

[0034] The abnormal red blood cell detection module is configured to remove all white blood cells from the smear image to be tested according to the position area marking result, extract all red blood cell images from the processed smear image to be tested by using a trained U-Net neural network and a Faster R-CNN neural network, and finally perform abnormal detection on all the red blood cell images by using a trained VGG19 neural network to obtain the second detection result.

[0035] Further, the second white blood cell recognition module comprises:

[0036] The third extraction unit is configured to convert the smear image to be tested into a second smear image in a LAB space, extract all optimal color areas from the second smear image by using a K-means algorithm, and obtain a third smear image.

[0037] The filling unit is configured to generate a third mask image corresponding to the third smear image by using an Otsu algorithm, and perform area screening and pixel filling on all connected domains contained in the third mask image.

[0038] The second output unit is configured to perform a morphological closing operation on all connected domains contained in the processed third mask image, perform area eligibility screening on all the connected domains, and finally output the smear image to be tested in which all white blood cells are marked.

[0039] Further, the abnormal red blood cell detection module comprises:

[0040] The second preprocessing unit is configured to remove all white blood cells from the smear image to be tested according to the position area marking result, generate a foreground mask image corresponding to the processed smear image to be tested by using a trained U-Net neural network, and extract all connected domains therefrom.

[0041] The positioning unit is configured to perform red blood cell recognition on all the extracted connected domains by using a trained Faster R-CNN neural network, and obtain coordinate positions of all red blood cells in the processed smear image to be tested.

[0042] The second detection unit is configured to extract all red blood cell images from the processed smear image to be tested according to the coordinate positions of all the red blood cells, perform parasitic infection detection on all the red blood cell images by using a trained VGG19 neural network, and finally output the second detection result.

[0043] The present application has at least the following beneficial effects: by introducing a blur inference device in the image preprocessing module to perform a de-blurring operation on the collected smear specimen image, the detection error caused by the mismatch of the focal length between the external camera device and the microscope can be compensated, and the relevant features of the smear specimen image can be more clearly reflected with the microscope objective. By setting the thick blood smear detection module and the thin blood smear detection module, the doctor can perform more comprehensive parasite detection operation on the completed smear specimen, thereby compensating for the detection error that may be caused by poor staining of the smear specimen. In any one of the above smear detection modules, the K-means algorithm and the Otsu algorithm are introduced to complete the initial white blood cell recognition task, and the relevant deep learning model is built to complete the final parasite detection task. Compared with the existing artificial microscope detection method, the doctor can obtain the auxiliary diagnosis result of the patient's blood smear in a short time, save the labor cost and time cost, and improve the detection reliability. BRIEF DESCRIPTION OF DRAWINGS

[0044] The accompanying drawings are used to provide a further understanding of the technical solutions of the present application, and constitute a part of the specification, and are used together with the embodiments of the present application to explain the technical solutions of the present application, and do not constitute a limitation on the technical solutions of the present application.

[0045] Figure 1 is a structural composition schematic diagram of a parasite auxiliary detection system based on deep learning in an embodiment of the present application;

[0046] Figure 2 is a schematic diagram of a smear specimen image in an embodiment of the present application;

[0047] Figure 3 is a schematic diagram of a to-be-detected smear image in an embodiment of the present application;

[0048] Figure 4 is a schematic diagram of an interactive interface for displaying a first detection result in an embodiment of the present application;

[0049] Figure 5 is a schematic diagram of an interactive interface for displaying a second detection result in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and do not limit the present application.

[0051] It is to be noted that although the functional modules are divided in the system schematic diagram, in some cases, the functions shown or described can be performed in a manner different from the module division in the system. The terms "first", "second", "third", "fourth" and the like in the specification of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific use order or sequence, and it should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a system, product or device including a series of units does not necessarily have to be limited to only those clearly listed units, but can include other units inherent to the product or device without being clearly listed.

[0052] First, the part of the nouns involved in the present application are explained as follows:

[0053] The fuzzy inference engine, also known as fuzzy inference system (abbreviated as FIS, full name Fuzzy Inference System), is a system based on fuzzy set theory and fuzzy reasoning and other technologies and has the ability to process fuzzy information, which can realize complex nonlinear mapping, and the system input and system output are both accurate numerical values, so it has broad application prospects.

[0054] The K-means algorithm is an iterative solution clustering algorithm, and the corresponding steps include: dividing the data into K groups and randomly selecting K samples as the initial clustering centers, then calculating the distance between each remaining sample and the current clustering center, assigning each sample to the nearest clustering center, at this time the clustering center and the assigned sample represent a cluster, and the clustering center is recalculated according to all the samples in the cluster each time a sample is assigned, this process will be repeated until a certain termination condition is met, which can be that no or few samples are reassigned to different clusters, or that no or few clustering centers change again, or that the local minimum of the error sum of squares is met.

[0055] The Otsu algorithm, also known as the maximum inter-class variance method or the Otsu algorithm, is mainly to divide the image into background and foreground two parts according to the gray level characteristics of the image (i.e. according to the threshold value), and the method of determining the best threshold value is to make the inter-class variance maximum, so as to make the image misclassification probability minimum.

[0056] Please refer to Figure 1 , Figure 1is a structural composition schematic diagram of a parasitic worm auxiliary detection system based on deep learning provided by the embodiment of the present application, and the system mainly comprises an image acquisition module, an image preprocessing module, a thin blood smear detection module, a thick blood smear detection module and an interactive display module; wherein the interactive display module is connected with the image acquisition module, the image acquisition module is connected with the image preprocessing module, the image preprocessing module is connected with the thin blood smear detection module, the thin blood smear detection module is connected with the interactive display module, the image preprocessing module is connected with the thick blood smear detection module, and the thick blood smear detection module is connected with the interactive display module.

[0057] In the specific implementation process of the present application, the image acquisition module starts to acquire smear specimen images and mark their types according to the detection instructions received from the interactive display module; the image preprocessing module performs deblurring operation on the acquired smear specimen images to obtain the to-be-detected smear images, and then transmits the to-be-detected smear images to the thin blood smear detection module or the thick blood smear detection module according to the image type markers carried by the to-be-detected smear images, that is, only one of the thin blood smear detection module and the thick blood smear detection module can be selected to assist in completing the detection task in each detection process.

[0058] After receiving the to-be-detected smear image, the thick blood smear detection module preferentially identifies and removes all white blood cells therefrom, and then performs parasitic worm detection operation on the processed to-be-detected smear image to obtain a first detection result, which includes but is not limited to the number of detected parasitic worms and the smear image in which all parasitic worm positions have been marked; the interactive display module performs interface intuitive display operation on the first detection result to provide a reference for the doctor's diagnosis process.

[0059] After receiving the to-be-detected smear image, the thin blood smear detection module preferentially identifies and removes all white blood cells therefrom, and then performs abnormal red blood cell detection operation on the processed to-be-detected smear image to obtain a second detection result, which includes but is not limited to the total number of detected red blood cells, the number of detected abnormal red blood cells and the smear image in which all abnormal red blood cell positions have been marked; the interactive display module performs interface intuitive display operation on the second detection result to provide a reference for the doctor's diagnosis process.

[0060] In the embodiment of the present application, when the doctor makes a smear specimen (which can be a thick blood smear specimen and / or a thin blood smear specimen) from the human blood sample of the to-be-detected person, and places it under the microscope by the doctor himself or other users while adjusting the microscope objective to the best observation position of the smear specimen, the image acquisition module is then put into use.

[0061] It should be noted that, in the field of parasitology, the thick blood smear specimen is required to lyse the red blood cells in the human blood sample to release the internal parasites, and the thin blood smear specimen is required to completely retain the red blood cells in the human blood sample to limit the internal parasites.

[0062] More specifically, the image acquisition module mainly includes a first image acquisition unit and a second image acquisition unit, and only one of them can be selected to run in each detection process;Wherein, the interactive display module is connected with the first image acquisition unit, the first image acquisition unit is connected with the external camera device, and the first image acquisition unit is connected with the image preprocessing module;The interactive display module is connected with the second image acquisition unit, the second image acquisition unit is connected with the external camera device, and the second image acquisition unit is connected with the image preprocessing module.

[0063] Need to be explained first, the interactive display module is provided with an interactive interface, which can be specifically seen from Figure 4 Or Figure 5 As shown, the interactive interface is provided with a first control representing the thick blood smear detection mode and a second control representing the thin blood smear detection mode.

[0064] In the specific implementation process of the application, when the doctor himself or other users triggers the first control on the interactive interface, the interactive display module responds to the triggering operation and generates a corresponding thick blood smear detection instruction, which is fed back to the first image acquisition unit;After receiving the thick blood smear detection instruction, the first image acquisition unit starts running, and after receiving the smear specimen image transmitted by the external camera device, it is marked as a thick blood smear type.

[0065] In another specific implementation process of the application, when the doctor himself or other users triggers the second control on the interactive interface, the interactive display module responds to the triggering operation and generates a corresponding thin blood smear detection instruction, which is fed back to the second image acquisition unit;After receiving the thin blood smear detection instruction, the second image acquisition unit starts running, and after receiving the smear specimen image transmitted by the external camera device, it is marked as a thin blood smear type.

[0066] Among them, the smear specimen image transmitted by the external camera device is actually obtained by the external camera device aiming at the microscope objective, that is, the smear specimen image refers to the image presented by the microscope objective, which can be specifically seen from Figure 2Need to explain, when the application provided by the system for auxiliary detection of the parasite is integrated into any mobile terminal device, the external camera device is the camera loaded on the mobile terminal device.

[0067] In the embodiment of the application, the image preprocessing module mainly comprises sequentially connected acquisition unit, conversion unit, analysis unit, first extraction unit, expansion unit, second extraction unit and third output unit; wherein the acquisition unit is connected with the first image acquisition unit, the acquisition unit is connected with the second image acquisition unit, the third output unit is connected with the thick blood smear detection module, and the third output unit is connected with the thin blood smear detection module; the functions of the above seven units are described as follows:

[0068] In the acquisition unit, since the smear specimen image is represented in the RGB color space, the smear specimen image is copied to obtain a smear specimen image copy, and then the smear specimen image copy is converted into a double-precision gray image, and then the horizontal coordinate value of each pixel point is extracted from the double-precision gray image in a predetermined order to construct a first data set, and the vertical coordinate value of each pixel point is extracted from the double-precision gray image in the predetermined order to construct a second data set; wherein the predetermined order includes but is not limited to the extraction order from left to right and then from top to bottom, or the extraction order from top to bottom and then from left to right, and the application sets the first data set and the second data set to be extracted and stored in the predetermined order, which can ensure that the two coordinate values at the same storage location in the two data sets belong to the same pixel point.

[0069] In the conversion unit, the first data set is converted by using a gradient filter constructed in advance to obtain a first gradient set, which is specifically represented as follows: when the double-precision gray image contains N pixel points, the first data set is regarded as a first row matrix of 1 row x N columns, since the gradient filter is represented by a second row matrix [-1, 1], the convolution operation is performed between the first row matrix and the second row matrix, and N+1 convolution results (also referred to as gradient values in the X-axis direction) are constructed into the first gradient set.

[0070] In the conversion unit, the second data set is further converted by using the gradient filter to obtain a second gradient set, which is specifically represented as follows: the second data set is regarded as a third row matrix of 1 row x N columns, at this time, the convolution operation is performed between the third row matrix and the second row matrix, and N+1 convolution results (also referred to as gradient values in the Y-axis direction) are constructed into the second gradient set.

[0071] In the analysis unit, the first gradient set and the second gradient set are subjected to fuzzy analysis by a fuzzy inference machine constructed in advance to obtain a first output set, which is characterized by:

[0072] First, a fuzzy inference machine with two input ends and one output end is constructed using an existing MAMFIS function, a zero-mean Gaussian membership function is assigned to each input end for pre-processing of input data, a triangular membership function is assigned to the output end for value limitation of output data, i.e., each value in the first output set is limited to the range of [0, 1], and a fuzzy inference rule is formulated for the fuzzy inference machine.

[0073] Second, two gradient values at the same storage location in the first gradient set and the second gradient set are sequentially input into the fuzzy inference machine, and the fuzzy inference rule is used to perform fuzzy analysis on the two gradient values to obtain an output value, thereby obtaining the first output set.

[0074] In the analysis unit, each value in the first output set is subjected to binary processing to obtain a second output set, which is characterized by: selecting a pixel threshold value (preferably 0.6) in the range of [0, 1], judging each value in the first output set against the pixel threshold value, changing all values greater than the pixel threshold value to 1 (representing black pixels), and changing all values less than or equal to the pixel threshold value to 0 (representing white pixels), thereby converting the first output set into the second output set.

[0075] In the first extraction unit, each pixel value contained in the second output set is used to generate an initial mask image that is adapted to the size of the smear specimen image, and a white pixel region composed of all white pixel values is extracted from the initial mask image, which is generally an irregular polygon.

[0076] In the expansion unit, the white pixel region should be a circular region due to the imaging rules of the microscope, and the irregular boundary line of the white pixel region is subjected to regularization processing, which is characterized by:

[0077] Firstly, the leftmost endpoint and the rightmost endpoint of the white pixel region are obtained along the X coordinate axis, the topmost endpoint and the bottommost endpoint of the white pixel region are obtained along the Y coordinate axis, a middle line perpendicular to the X coordinate axis is determined according to the leftmost endpoint and the rightmost endpoint, a middle line perpendicular to the Y coordinate axis is determined according to the topmost endpoint and the bottommost endpoint, and the intersection point of the two middle lines is defined as the center point of the white pixel region.

[0078] Secondly, the distances between the center point and each point on the irregular boundary line of the white pixel region are calculated, and the maximum distance is extracted therefrom.

[0079] Finally, a circle is constructed with the center point as the center and the maximum distance as the radius, and the closed region formed between the irregular boundary line of the white pixel region and the circle is filled with white pixels, so that the white pixel region is repaired into a circular region, thereby obtaining a final mask image adapted to the size of the smear specimen image.

[0080] In the second extraction unit, the final mask image is overlapped onto the smear specimen image, so that the image presented in the circular region is the clear part in the smear specimen image, thereby converting the smear specimen image into a test smear image, which can be specifically seen from Figure 3

[0081] In the third output unit, the test smear image is selectively output according to the image type label assigned to the smear specimen image, which is specifically manifested as follows: when the smear specimen image is assigned a label of a thick blood smear type, the test smear image is transmitted to the thick blood smear detection module; or when the smear specimen image is assigned a label of a thin blood smear type, the test smear image is transmitted to the thin blood smear detection module.

[0082] In the embodiment of the present application, the thick blood smear detection module comprises a first white blood cell recognition module and a parasite detection module; wherein the first white blood cell recognition module is mainly used for recognizing all white blood cells from the received test smear image and marking the position region thereof in combination with an Otsu algorithm and a K-means algorithm; and the parasite detection module is mainly used for removing all white blood cells from the received test smear image according to the position region marking result of all white blood cells, and then calling a pre-trained VGG19 neural network to perform a parasite detection operation on the current processed test smear image, so as to obtain the first detection result required finally.

[0083] ​More specifically, the first leukocyte recognition module mainly comprises a generating unit, a correction unit and a first output unit connected in sequence; wherein the generating unit is connected with the third output unit, and the first output unit is connected with the parasite detection module; the functions of the above three units are described as follows:

[0084] In the generating unit, since the received test smear image is represented in the RGB color space, the test smear image is first copied to obtain a test smear image copy, and the test smear image copy is converted into a first smear image represented in the LAB color space; secondly, the first smear image is copied to obtain a first smear image copy, the first mask image corresponding to the first smear image is generated by the K-means algorithm, and the second mask image corresponding to the first smear image copy is generated by the Otsu algorithm.

[0085] Wherein, the first mask image corresponding to the first smear image is generated by the K-means algorithm, which specifically represents that: the colors in the first smear image are automatically clustered into three categories by the K-means algorithm, only one cluster with the deepest color is retained, which represents the area where all leukocytes are located, and the other two clusters with weak color are removed, which represents the interference area (such as the area where parasites are located, the area where platelets are located and the area with high staining, etc.), and then the first mask image required is generated from the retained cluster according to the first smear image.

[0086] In the correction unit, since the position of the same leukocyte on the first mask image and the second mask image is the same, combined with all connected domains contained in the first mask image (hereinafter referred to as all first connected domains), the inspection and correction operation is performed on all connected domains contained in the second mask image (hereinafter referred to as all second connected domains), which specifically needs to perform the following steps:

[0087] (1) The ith first connected domain is obtained from the all first connected domains, and it is inquired whether the centroid positions of three second connected domains exist in the ith first connected domain; if yes, step (2) is executed; if not, step (3) is executed;

[0088] (2) in the interior of the i-th first connected domain, a triangle is drawn according to the positions of the three second connected domains, and the ratio of the area of the black pixels in the triangle to the area of the triangle is calculated, and then it is determined whether the ratio is less than a predetermined area threshold (preferably 0.7 in the present application); if yes, it means that the Otsu algorithm has an error in identifying the white blood cells associated with the i-th first connected domain, and at this time the i-th first connected domain is directly filled into the second mask image according to its centroid position to replace the three second connected domains, and then step (3) is continued; if no, it means that the Otsu algorithm has no error in identifying the white blood cells associated with the i-th first connected domain, and at this time step (3) is continued.

[0089] (3) it is determined whether i

[0090] In the first output unit, all qualified connected domains are screened from all connected domains contained in the processed second mask image by area threshold comparison, and then all white blood cells are marked and framed according to the centroid positions of the qualified connected domains on the smear image to be detected for output; wherein the area of each qualified connected domain is limited in the range of [900, 8000], so that the excessively adhered areas of white blood cells, the excessively dyed areas and the non-cell material areas can be excluded.

[0091] More specifically, the parasite detection module mainly includes a first preprocessing unit and a first detection unit connected in sequence, the first preprocessing unit is connected with the first output unit, and the first detection unit is connected with the interactive display module; the functions of the above two units are described as follows:

[0092] In the first preprocessing unit, since the parasites are in the non-white blood cell area, all white blood cells are removed from the received test smear image according to the position area marking result of all the recognized white blood cells, and then after performing the contrast enhancement processing operation on the current processed test smear image, all candidate images are directly intercepted from the enhanced test smear image in combination with the appearance features of the parasites (including morphological features, color features, and texture features, etc.), while recording the size and center point position of each candidate image, that is, the specific position of the center point of each candidate image in the enhanced test smear image; it should be noted that a template image can be constructed according to the appearance features of the parasites, and the template image is used to perform feature matching in the enhanced test smear image in the order of row by column or column by row to obtain all candidate images.

[0093] In the first detection unit, the VGG19 neural network trained in advance is input into the first detection unit to perform parasite detection operation, so as to detect all candidate images containing parasites (hereinafter referred to as all effective images) and their confidence scores, at this time, the specific number of all effective images is taken as the number of detected parasites, and the test smear image received by the generation unit is marked with a frame according to the size and center point position of each effective image, and the total accuracy is obtained by averaging all the confidence scores associated with all effective images, thereby forming a basic first detection result, which is transmitted to the interactive display module together with the number of all white blood cells removed by the first preprocessing unit for interface display. For details, see Figure 4

[0094] ​In the embodiment of the present application, the parasite detection module can be applied to detect malaria parasites, which belong to a class of parasites. When all white blood cells and part of the current smear image to be tested containing malaria parasites are removed, the relevant implementation process is as follows: in the first preprocessing unit, after performing contrast enhancement processing operation on the current smear image to be tested by using the limited contrast self-adaptive histogram equalization method, a first template image with a size of 64*64 is constructed according to the appearance characteristics of most malaria parasites measured by experiments, and all first candidate images are extracted from the current smear image to be tested by feature matching; in the first detection unit, a large number of thick blood smear images of infected malaria patients (only images of a few infected sites are intercepted) and a small number of thick blood smear images of non-infected persons (only images of a few healthy sites are intercepted) are collected to construct a thick blood smear database, and then the thick blood smear database is image augmented (data augmented) by adding noise (blurring), flipping and rotating, so that the number of two types of images is balanced, and then the updated thick blood smear database is used to learn and train the pre-built VGG19 neural network, and then the all first candidate images are input into the trained VGG19 neural network to perform parasite detection operation, so as to obtain the final detection result; wherein the large number of infected malaria patients are preferably 150 infected malaria patients, and the small number of non-infected persons are preferably 50 non-infected persons.

[0095] In the embodiment of the present application, the thin blood smear detection module includes a second white blood cell recognition module and an abnormal red blood cell detection module; wherein the second white blood cell recognition module is mainly used to extract all test regions in the received smear image by K-means algorithm, and then identify all white blood cells from the all test regions by Otsu algorithm and mark the position region of the white blood cells; the abnormal red blood cell detection module is mainly used to remove all white blood cells from the received smear image according to the position region marking result of all white blood cells, and then extract all red blood cell images from the current processed smear image by combining the pre-trained U-Net neural network and the pre-trained Faster R-CNN neural network, and finally input the all red blood cell images into the pre-trained VGG19 neural network to perform abnormal detection operation, so as to obtain the final second detection result required.

[0096] More specifically, the second white blood cell recognition module mainly includes a third extraction unit, a padding unit and a second output unit connected in sequence; wherein the third extraction unit is connected with the third output unit, and the second output unit is connected with the abnormal red blood cell detection module; the functions of the above three units are described as follows:

[0097] In the third extraction unit, since the received test smear image is represented in the RGB color space, the test smear image is first copied to obtain a test smear image copy, and the test smear image copy is converted into a second smear image represented in the LAB color space; the colors in the second smear image are automatically clustered into three categories by the K-means algorithm, only one cluster with the darkest color is retained, which represents the region where all white blood cells are located, and the other two clusters with weak colors are removed, which represent interference regions (such as regions where parasites are located, regions where platelets are located, and highly stained regions, etc.), all individual regions contained in the retained cluster are taken as all test regions, at this time the second smear image can be converted into a third smear image.

[0098] In the filling unit, the third mask image corresponding to the third smear image is first generated by the Otsu algorithm, and then area screening and pixel filling operations are performed on all connected domains contained in the third mask image. Specifically, all first qualified connected domains are screened from all connected domains contained in the third mask image by area threshold comparison, wherein the area size of each first qualified connected domain is limited in the range of [1500, ∞), and the interior of each first qualified connected domain is completely filled with black pixels.

[0099] In the second output unit, a disk-shaped structure element with a pixel size of 15 is first created to perform a morphological closing operation on all connected domains contained in the current processed third mask image, and then all second qualified connected domains are screened from all connected domains after the closing operation by area threshold comparison, and finally all white blood cells are marked with a frame selection on the test smear image according to the centroid positions of all second qualified connected domains, and then output; wherein the area size of each second qualified connected domain in all second qualified connected domains is limited in the range of [500, ∞).

[0100] More specifically, the abnormal red blood cell detection module mainly includes a second preprocessing unit, a positioning unit and a second detection unit connected in sequence, the second preprocessing unit is connected with the second output unit, and the second detection unit is connected with the interactive display module; the functions of the above three units are described as follows:

[0101] In the second preprocessing unit, since the parasites are in the non-white blood cell area, all white blood cells are removed from the received test smear image according to the position area marking result of all the recognized white blood cells, and then the current processed test smear image (hereinafter referred to as the first test smear image) is input into the pre-trained U-Net neural network to perform cell cluster and superpixel segmentation processing to generate a corresponding foreground mask image, and finally all connected domains are extracted from the foreground mask image.

[0102] In the positioning unit, the foreground mask image is input into the pre-trained Faster R-CNN neural network to perform red blood cell recognition operation on all the extracted connected domains, and then all the connected domains containing red blood cells (hereinafter referred to as all valid connected domains) are identified and recorded, and the centroid position of each valid connected domain is recorded.

[0103] In the second detection unit, all red blood cell images are extracted from the first test smear image according to the centroid position of each valid connected domain, and the size of each red blood cell image is preferably 224*224, and then all the red blood cell images are input into the pre-trained VGG19 neural network to perform parasite infection detection operation, so as to detect all the red blood cell images infected by parasites (hereinafter referred to as all valid red blood cell images) and their confidence scores, at this time, the specific number of all red blood cell images is taken as the total number of detected red blood cells, and the specific number of all valid red blood cell images is taken as the number of detected abnormal red blood cells, and the size and center point position of each valid red blood cell image (actually the centroid position of the corresponding valid connected domain) are marked in the test smear image received by the third extraction unit, and the average of all the confidence scores associated with all the valid red blood cell images is taken as the overall accuracy, thereby forming a basic second detection result, which is transmitted to the interactive display module together with the number of all white blood cells removed by the second preprocessing unit for interface display. For details, see Figure 5 .

[0104] It should be noted that the abnormal red blood cell detection module can be applied to detect red blood cells infected by Plasmodium, that is, when the first test smear image after removing all white blood cells may contain red blood cells infected by Plasmodium, a large number of thin blood smear images of malaria patients (only a few red blood cell images infected by Plasmodium are extracted) and a small number of thin blood smear images of non-infected persons (only a few red blood cell images not infected by Plasmodium are extracted) are collected to construct a thin blood smear database, and the VGG19 neural network mentioned in the second detection unit is trained after being built.

[0105] In the embodiment of the present application, by introducing a blur inference device in the image preprocessing module to perform a de-blurring operation on the collected smear specimen image, the detection error caused by the mismatch of the focal length between the external camera device and the microscope can be compensated, and the relevant features of the smear specimen image can be more clearly reflected with the microscope objective. By setting the thick blood smear detection module and the thin blood smear detection module, the doctor can perform more comprehensive parasite detection operation on the completed smear specimen, thereby compensating for the detection error that may be caused by poor staining of the smear specimen. In any one of the above smear detection modules, the K-means algorithm and the Otsu algorithm are introduced to complete the initial white blood cell recognition task, and then the relevant deep learning model is built to complete the final parasite detection task. Compared with the existing artificial microscope detection method, the doctor can obtain the auxiliary diagnosis result of the patient's blood smear in a short time, save the labor cost and time cost, and improve the detection reliability.

[0106] Although the description of the present application has been quite detailed and particularly described with respect to the described embodiments, it is not intended to be limited to any of these details or embodiments, but should be considered to effectively cover the intended scope of the present application by reference to the appended claims, taking into account the prior art to provide a broad interpretation of the claims. In addition, the present application is described above in embodiments that the inventors can foresee, the purpose of which is to provide a useful description, and non-essential modifications to the present application that have not yet been foreseen can still represent equivalent modifications of the present application.

Claims

1. A deep learning-based parasiticides assisted detection system, characterized in that, The system comprises an image acquisition module, an image preprocessing module, a thick blood smear detection module, a thin blood smear detection module and an interactive display module; The image acquisition module is configured to acquire a smear specimen image and mark the type of the smear specimen image according to a received detection instruction; The image preprocessing module is configured to perform deblurring processing on the smear specimen image to obtain a to-be-detected smear image, and input the to-be-detected smear image to the thick blood smear detection module or the thin blood smear detection module according to the type of the to-be-detected smear image; The thick blood smear detection module is configured to perform parasite detection after identifying and removing white blood cells from the to-be-detected smear image to obtain a first detection result, the first detection result comprising a smear image marking the positions of parasites and the number of parasites; The thin blood smear detection module is configured to perform abnormal red blood cell detection after identifying and removing white blood cells from the to-be-detected smear image to obtain a second detection result, the second detection result comprising a smear image marking the positions of abnormal red blood cells, the total number of red blood cells and the number of abnormal red blood cells; The interactive display module is configured to display the first detection result or the second detection result; The thick blood smear detection module comprises: A first white blood cell identification module configured to identify all white blood cells in the to-be-detected smear image and mark the positions of the white blood cells by combining a K-means algorithm and an Otsu algorithm; A parasite detection module configured to remove all white blood cells from the to-be-detected smear image according to the position marking result, and perform parasite detection on the processed to-be-detected smear image by using a trained VGG19 neural network to obtain the first detection result; The first white blood cell identification module comprises: A generation unit configured to convert the to-be-detected smear image into a first smear image in a LAB space, generate a first mask image corresponding to the first smear image by using the K-means algorithm, and generate a second mask image corresponding to the first smear image by using the Otsu algorithm; A correction unit configured to check and correct all connected domains contained in the second mask image by using all connected domains contained in the first mask image; A first output unit configured to finally output the to-be-detected smear image marking the positions of all white blood cells after performing area eligibility screening on all connected domains contained in the processed second mask image.

2. The deep learning-based parasiticides assisted detection system of claim 1, wherein, The image acquisition module is operated after a user places a prepared smear specimen under a microscope, and comprises: A first image acquisition unit configured to acquire a smear specimen image fed back after an image presented by an objective lens of the microscope is captured by an external camera device according to a received thick blood smear detection instruction, and mark the smear specimen image as a thick blood smear type; A second image acquisition unit configured to acquire a smear specimen image fed back after an image presented by an objective lens of the microscope is captured by an external camera device according to a received thin blood smear detection instruction, and mark the smear specimen image as a thin blood smear type.

3. The deep learning-based parasiticides assisted detection system of claim 1, wherein, The image preprocessing module comprises: The acquisition unit is configured to convert the smear specimen image into a double-precision gray image, acquire a horizontal coordinate value of each pixel point to form a first data set, and acquire a vertical coordinate value of each pixel point to form a second data set; The conversion unit is configured to convert the first data set into a first gradient set and the second data set into a second gradient set according to a gradient filter; The analysis unit is configured to analyze the first gradient set and the second gradient set by a fuzzy inference engine to obtain a first output set, and binarize each value in the first output set to obtain a second output set; The first extraction unit is configured to create an initial mask image according to the second output set, and extract a white pixel region from the initial mask image; The expansion unit is configured to determine a center of a circle in the white pixel region, expand the white pixel region by a maximum distance from the center of the circle to a boundary line of the white pixel region to obtain a final mask image; The second extraction unit is configured to extract a clear part from the smear specimen image by using the final mask image to obtain a test smear image.

4. The deep learning-based parasiticides assisted detection system of claim 3, wherein, The conversion of the first data set into the first gradient set according to the gradient filter includes: The first data set is represented as a first row matrix, the gradient filter is represented as a second row matrix, and the first row matrix and the second row matrix are convolved to obtain the first gradient set.

5. The deep learning-based parasiticides assisted detection system of claim 1, wherein, The parasitic detection module includes: The first preprocessing unit is configured to remove all white blood cells from the test smear image according to a position region marking result, perform image contrast enhancement processing, and then extract all candidate images from the processed test smear image according to the appearance characteristics of the parasites. The first detection unit is configured to use a trained VGG19 neural network to detect the parasites in the candidate images to finally output the first detection result.

6. The deep learning-based parasiticides assisted detection system of claim 1, wherein, The thin blood smear detection module includes: The second white blood cell recognition module is configured to use a K-means algorithm to screen all test regions from the test smear image, and use an Otsu algorithm to recognize all white blood cells in the test regions and mark positions. The abnormal red blood cell detection module is configured to remove all white blood cells from the test smear image according to a position region marking result, use a trained U-Net neural network and a Faster R-CNN neural network to extract all red blood cell images from the processed test smear image, and finally use a trained VGG19 neural network to detect abnormalities in the red blood cell images to obtain the second detection result.

7. The deep learning-based parasiticides assisted detection system of claim 6, wherein, The second white blood cell recognition module includes: The third extraction unit is configured to convert the test smear image into a second smear image in a LAB space, use a K-means algorithm to extract all test regions with optimal colors from the second smear image to obtain a third smear image. The filling unit is configured to use an Otsu algorithm to generate a third mask image corresponding to the third smear image, and perform area screening and pixel filling on all connected domains contained in the third mask image. The second output unit is configured to perform a morphological closing operation on all the connected domains contained in the processed third mask image, and finally output a test smear image in which all the positions of the white blood cells are marked after performing an area eligibility screening on all the connected domains.

8. The deep learning-based parasiticides assisted detection system of claim 6, wherein, The abnormal red blood cell detection module comprises: The second preprocessing unit is configured to remove all the white blood cells from the test smear image according to the position area marking result, generate a foreground mask image corresponding to the processed test smear image by using the trained U-Net neural network, and extract all the connected domains from the foreground mask image. The positioning unit is configured to perform red blood cell recognition on all the connected domains by using the trained Faster R-CNN neural network, and obtain the coordinate positions of all the red blood cells in the processed test smear image. The second detection unit is configured to extract all the red blood cell images from the processed test smear image according to the coordinate positions of all the red blood cells, and perform parasitic infection detection on all the red blood cell images by using the trained VGG19 neural network, and finally output the second detection result.

Images