Method, system and terminal for classifying formed elements in a sample under test

Abstract

This application provides a method, system, and terminal for classifying formed elements in a test sample. The method involves acquiring the original image of the test sample, extracting features from the original image to obtain first image features, locating formed elements in the test sample based on the first image features to obtain the positional information of the formed elements, obtaining a target image of the region where the formed elements are located from the original image based on the positional information of the formed elements, extracting features from the target image to obtain second image features, and identifying and classifying the formed elements based on the second image features and a preset neural network model to obtain descriptive information of the formed elements. The identification and classification process eliminates interference from other regions of the original image, thereby improving accuracy. Furthermore, after obtaining the descriptive information of the formed elements, the system outputs the descriptive information to introduce and explain the formed elements, facilitating understanding of each formed element in the test sample.

Description

Technical Field

[0001] This application belongs to the field of classification technology, and in particular relates to a method, system and terminal for classifying formed elements in a sample to be tested. Background Technology

[0002] In the field of blood sample analysis, identifying cells such as white blood cells, red blood cells, and platelets in blood samples has significant clinical value. Currently, blood sample analysis primarily involves the identification and classification of cells in the blood sample. The identification and classification process is as follows:

[0003] Image segmentation is performed on the blood sample image to obtain the cell regions in the sample image. Cell morphology recognition is then performed on the cell regions to determine the cell type. Although this method can identify cell type, its accuracy is relatively low. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a method, system, and terminal for classifying formed elements in a sample to be tested, and the technical solution is as follows:

[0005] On the one hand, this application provides a method for classifying formed elements in a sample to be tested, including:

[0006] Obtain the original image of the sample to be tested;

[0007] Feature extraction is performed on the original image to obtain the first image features;

[0008] Based on the first image features, the formed elements in the sample to be tested are located to obtain the location information of the formed elements;

[0009] Based on the location information of the formed elements, a target image of the region where the formed elements are located is obtained from the original image;

[0010] Feature extraction is performed on the target image to obtain the second image features;

[0011] Based on the second image features and the preset neural network model, the formed elements are identified and classified to obtain descriptive information of the formed elements. The descriptive information of the formed elements at least indicates the type of the formed elements.

[0012] Output the description information of the formed elements.

[0013] On the other hand, this application provides a terminal, including:

[0014] An image acquisition device configured to acquire the original image of the sample to be tested;

[0015] The processor is configured to execute the formed element classification method in the test sample to obtain descriptive information of the formed elements in the test sample.

[0016] Output interface, configured to output descriptive information of the formed elements.

[0017] Furthermore, this application provides a shaped component analysis system, comprising:

[0018] An imaging device is configured to capture images of a sample smear of a sample to be tested, thereby obtaining the original image of the sample to be tested.

[0019] A smear holding device configured to allow relative movement between the sample smear and the imaging device;

[0020] A control device configured to perform the formed element classification method in the above-mentioned test sample.

[0021] Furthermore, this application provides a method for analyzing formed elements, including:

[0022] The sample smear is moved relative to the imaging device by a smear holding device;

[0023] The original image of the sample to be tested is obtained by taking a picture of the sample smear using an imaging device.

[0024] The original image is processed by the control device as follows:

[0025] Feature extraction is performed on the original image to obtain the first image features;

[0026] Based on the first image features, the formed elements in the sample to be tested are located to obtain the location information of the formed elements;

[0027] Based on the location information of the formed elements, a target image of the region where the formed elements are located is obtained from the original image;

[0028] Feature extraction is performed on the target image to obtain the second image features;

[0029] Based on the second image features and the preset neural network model, the formed elements are identified and classified to obtain descriptive information of the formed elements. The descriptive information of the formed elements at least indicates the type of the formed elements.

[0030] Output the description information of the formed elements.

[0031] In another aspect, this application provides a storage medium storing computer program code, which, when executed, implements the formed element classification method in the test sample described above.

[0032] The formed element classification method described above for the test sample involves acquiring the original image of the test sample, extracting features from the original image to obtain first image features, locating the formed elements in the test sample based on the first image features to obtain the positional information of the formed elements, obtaining the target image of the region where the formed elements are located from the original image based on the positional information of the formed elements, extracting features from the target image to obtain second image features, and identifying and classifying the formed elements based on the second image features and a preset neural network model to obtain descriptive information of the formed elements. Because the second image features referenced in the identification and classification process are obtained from the target image of the region where the formed elements are located, interference from other regions of the original image is eliminated, thereby improving accuracy. Furthermore, after obtaining the descriptive information of the formed elements, the descriptive information is output to introduce and explain the formed elements, facilitating understanding of each formed element in the test sample. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This document illustrates a flowchart of a method for classifying formed elements in a test sample, as provided in an embodiment of this application.

[0035] Figure 2 A schematic diagram illustrating the multi-feature fusion method used to obtain a second image feature according to an embodiment of this application is shown;

[0036] Figure 3 This illustration shows a schematic diagram of the output description information provided in an embodiment of this application;

[0037] Figure 4 This illustration shows a schematic diagram of a method for classifying formed elements in a sample to be tested, provided in an embodiment of this application.

[0038] Figure 5 This illustration shows a schematic diagram of another method for classifying formed samples in a test sample provided in an embodiment of this application;

[0039] Figure 6 This document illustrates a flowchart of another method for classifying formed samples in an embodiment of this application.

[0040] Figure 7This diagram illustrates a method for obtaining descriptive information of formed elements by combining a classification heatmap with a pre-defined neural network model.

[0041] Figures 8 to 13 The diagrams show the classification results of the same formed element by six different preset neural network models.

[0042] Figure 14 This diagram illustrates another method for obtaining descriptive information of formed elements by combining classification heatmaps with a pre-defined neural network model.

[0043] Figure 15 This document illustrates a flowchart of another method for classifying formed elements in a test sample, as provided in an embodiment of this application.

[0044] Figure 16 This document illustrates a flowchart of another method for classifying formed elements in a test sample, as provided in an embodiment of this application.

[0045] Figure 17 A schematic diagram of the structure of a formed element analysis system provided in an embodiment of this application is shown. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0047] Please see Figure 1 This illustrates a method for classifying formed elements in a sample to be tested, provided by an embodiment of this application, which may include the following steps:

[0048] 101: Obtain the original image of the sample to be tested. The sample to be tested is the sample for formed element classification detection. Formed elements are substances in the sample that have a certain shape and can be identified by microscopic observation. By classifying and detecting the formed elements in the sample to be tested, such as determining the type of formed elements in the sample and whether the formed elements are normal or abnormal, these detection results for the formed elements in the sample can assist doctors in diagnosing diseases.

[0049] The sample to be tested can be any one of blood, bone marrow, urine, or body fluid. Of course, other types of samples can also be used, such as exfoliated cell samples, excretory samples other than urine, and secretion samples. The formed elements, the carrier of the sample, and the form of the sample will differ depending on the type of sample, which will be explained in detail below:

[0050] Blood samples, also known as peripheral blood, are transported on glass slides and take the form of blood films (a special form of sample smears). The formed elements include red blood cells, white blood cells, and platelets.

[0051] Bone marrow sample: The carrier is a glass slide; the form is a bone marrow smear; the formed elements include: various cell lines such as erythrocytes, granulocytes, lymphocytes, monocytes, plasma cells and other cells at different stages of maturity, as well as other cell types such as megakaryocytes, reticular cells, phagocytes, endothelial cells, and adipocytes.

[0052] Urine sample: The carrier is a counting chamber; the form is urine sediment (a special form of sample smear); the formed elements include: red blood cells, white blood cells, white blood cell clusters, bacteria, yeast-like bacteria, epithelial cells, small round epithelial cells, crystals, hyaline casts, non-hyaline casts, mucus threads, etc.

[0053] Body fluid samples: Body fluid samples may include, but are not limited to, cerebrospinal fluid, serous cavity effusion, joint cavity effusion, amniotic fluid, etc.; the carrier is a glass slide; the form is a sample smear; the formed elements include: red blood cells, white blood cells, white blood cell clusters, bacteria, yeast-like fungi, epithelial cells, parasites, etc.

[0054] Exfoliated cell samples: The carrier is a glass slide; the form is a sample smear; the formed elements include: epithelial cells, mesothelial cells, cancer cells, red blood cells, white blood cells, macrophages or tissue cells, necrotic material (mucus, bacterial clumps, fungal clumps, plant cells, cotton wool and dye residues, etc.), parasites, etc.

[0055] Other excretory and secretory samples besides urine: Other excretory and secretory samples may include, but are not limited to: feces, vaginal secretions, semen, prostatic fluid, sputum, etc.; the carrier is a glass slide; the form is a sample smear; formed elements include: red blood cells, white blood cells, crystals, pathogenic microorganisms, epithelial cells, parasites, sperm, trichomonas, prostatic cholinergic bodies, prostatic granular cells, alveolar macrophages, tumor cells, etc.

[0056] Correspondingly, the methods for obtaining the original image of the sample to be tested can be, but are not limited to, obtaining the original image of the sample on the carrier through a microscope, such as taking a picture of the carrier through a 100x microscope to obtain the original image of the sample. For example, for blood samples, blood is smeared on a glass slide, and a blood film is prepared on the slide as a blood sample using a slide pusher or similar means. The slide is then transferred to the imaging area of ​​the microscope, and the slide is photographed through a 100x microscope to obtain the original image of the sample.

[0057] 102: Extract features from the original image to obtain the first image features.

[0058] The first image feature is used to locate the formed elements in the sample to be tested. The formed elements in the sample to be tested have at least one feature among geometric shape, size and color. Through these features, the region of the formed elements in the sample to be tested can be located, thereby realizing the localization of the formed elements. Thus, the first image feature may include, but is not limited to, feature vectors that can indicate at least one of the geometric shape, size and color of the formed elements.

[0059] Feasible methods for obtaining the first image feature include, but are not limited to: after acquiring the original image, calling a feature extraction model (referred to as the first feature extraction model for distinction) to extract features from the original image, and obtaining the first image feature output by the first feature extraction model. The first feature extraction model is obtained by training historical original images and historical first image features, such as training a neural network model with historical original images and historical first image features to obtain the first feature extraction model. The neural network model includes, but is not limited to, CNN (Convolutional Neural Networks). The network structure of CNN includes, but is not limited to, ResNet50, GoogLeNet, AlexNet, DenseNet, InceptionV4, etc. The CNN outputs an N*1 dimensional numerical matrix as the first image feature, where N is a natural number. The value of N is not limited in this embodiment.

[0060] In this embodiment, multiple original images of the sample to be tested can be acquired. The shooting parameters of the multiple original images are different, such as the focus used in the process of shooting the multiple original images. Although the multiple original images correspond to the same sample to be tested, the different focus results in differences in the clarity of the multiple original images. Then, feature extraction is performed on each original image to obtain the image features corresponding to each original image. Based on the image features corresponding to each original image, the first image feature is obtained.

[0061] If the image features corresponding to each original image are fused to obtain the first image feature, the fusion method includes, but is not limited to, at least one of the following: stitching, dimensionality reduction, PCA (Principal Components Analysis), etc., which will not be described in detail in this embodiment.

[0062] 103: Based on the first image features, locate the formed elements in the test sample to obtain the positional information of the formed elements. The positional information of the formed elements is used to show the area occupied by the formed elements in the original image. The formed elements can be obtained from the area indicated by the positional information, thus realizing the localization of the formed elements in the test sample. For example, the positional information of the formed elements can be, but is not limited to, the coordinate representation of the formed elements in the original image, so as to separate the formed elements from the original image by coordinates.

[0063] In this embodiment, the location information of formed elements can be obtained through, but is not limited to, a localization model. The localization model is a pre-trained model capable of locating formed elements in the sample to be tested. After obtaining the first image features, the first image features are input into the localization model. The localization model locates the formed elements in the sample to be tested, and the location information of the formed elements output by the localization model is obtained. The process of obtaining the localization model is as follows:

[0064] The process involves acquiring historical original images of historical samples and historical location information of formed elements in historical samples; extracting features from historical original images to obtain historical original image features; and adjusting the model parameters of a preset network model based on historical original image features and historical location information to obtain a localization model.

[0065] The preset network model can be, but is not limited to, a BP (Back Propagation) neural network model. The network structure of the BP neural network is not limited. The BP neural network model is trained by using the features of the historical original images and historical location information to adjust its model parameters, thereby obtaining a localization model that can output the location information of formed elements.

[0066] Other methods for obtaining the location information of formed elements will not be described in detail in this embodiment.

[0067] 104: Based on the location information of formed elements, the target image of the region where the formed elements are located is obtained from the original image. Since the location information of formed elements is used to show the area occupied by the formed elements in the original image, the region occupied by the formed elements (i.e., the region where they are located) can be delineated in the original image using the location information of the formed elements, and the image of the region where they are located can be used as the target image.

[0068] 105: Extract features from the target image to obtain the second image features.

[0069] The second image feature is used to represent the descriptive information of the formed elements in the target image, and the descriptive information at least indicates the type of the formed elements. Different types of formed elements will have differences in at least one feature among geometry, size and color. Therefore, in this embodiment, the second image feature includes, but is not limited to, feature vectors that can indicate at least one of geometry, size and color of the formed elements.

[0070] Feasible methods for obtaining the second image features include, but are not limited to: after acquiring the target image, calling a feature extraction model (referred to as the second feature extraction model for distinction) to extract features from the target image, and obtaining the second image features output by the second feature extraction model. The second feature extraction model is obtained by training historical target images and historical second image features, such as training a neural network model with historical target images and historical second image features to obtain the second feature extraction model. The neural network model includes, but is not limited to, CNN. The network structure of CNN includes, but is not limited to, ResNet50, GoogLeNet, AlexNet, DenseNet, InceptionV4, etc. The CNN outputs an N*1 dimensional numerical matrix as the second image features, where N is a natural number. The value of N is not limited in this embodiment.

[0071] If at least two original images of the sample to be tested are obtained, and these original images are captured with different parameters, such as different focus points used during the capture of multiple original images, resulting in differences in sharpness among the original images corresponding to the same sample, then, based on the location information of the formed elements, target images of the region where the formed elements are located are obtained from the at least two original images, thereby obtaining at least two target images corresponding to the same formed element but with different sharpness.

[0072] After obtaining at least two target images, feature extraction is performed on each of the at least two target images to obtain image features for each target image; a second image feature is obtained based on the image features of each target image. Alternatively, the image features of each target image can be fused to obtain the second image feature. The fusion method includes, but is not limited to, at least one of the following: stitching, dimensionality reduction, PCA, etc., which will not be detailed in this embodiment.

[0073] Figure 2 This embodiment illustrates the process of obtaining the second image feature through multi-feature fusion, acquiring M target images of the same cell. Figure 2Taking 7 target images as an example, M target images are obtained from M original images based on cell location information. The M target images are respectively input into a second feature extraction model based on CNN to obtain the image features of each target image output by the second feature extraction model. The image features of each target image are then subjected to at least one fusion method such as stitching, dimensionality reduction, and PCA to obtain the second image features.

[0074] It should be noted here that although both the first image feature and the second image feature can be, but are not limited to, feature vectors that can indicate at least one of the geometric shape, size and color of formed elements, the first image feature is obtained based on the original image, while the second image feature is obtained based on the target image. The target image is an image of at least one formed element in the original image, and its size is significantly smaller than that of the original image. The amount of data in the target image is significantly smaller than that in the original image, thereby improving the efficiency of obtaining the second image feature.

[0075] 106: Based on the second image features and the preset neural network model, the formed elements are identified and classified to obtain the descriptive information of the formed elements. The descriptive information of the formed elements at least indicates the type of the formed elements.

[0076] The preset neural network model is a pre-trained model capable of recognizing and classifying formed elements in the test sample. After obtaining the second image features, these features are input into the preset neural network model to obtain the descriptive information of the formed elements output by the model. The training process of the preset neural network model is as follows:

[0077] The process involves acquiring historical descriptive information of formed elements from historical samples and corresponding historical images; extracting features from the historical images to obtain historical image features; and adjusting the model parameters of a preset neural network model based on the historical image features and historical descriptive information to obtain a preset neural network model for identifying and classifying formed elements. The preset neural network model may be, but is not limited to, a deep learning neural network model or a machine learning neural network model. The network structure of the deep learning neural network model or the machine learning neural network model is not limited in this embodiment.

[0078] Because the formed elements in different types of test samples differ, the focus of the corresponding preset neural network model will also differ when identifying and classifying the formed elements in any type of test sample. Therefore, if a preset neural network model is used to identify and classify all types of test samples, the preset neural network model needs to have the function of distinguishing the differences in formed elements in different types of test samples, so that a preset neural network model can identify and classify different types of test samples.

[0079] Of course, this embodiment can also train multiple preset neural network models, each with a different focus. For example, each preset neural network model corresponds to at least one type of sample. For instance, the preset neural network model can be trained using historical images and historical descriptions of samples such as bodily fluid samples, blood samples, and bone marrow samples to obtain a preset neural network model corresponding to any type of sample. In this way, during the identification and classification process using the preset neural network model, identification and classification can be performed based on at least one preset neural network model. The process is as follows:

[0080] If the number of preset neural network models is one, then the predicted information of formed elements output by the preset neural network model can be determined as the descriptive information of formed elements.

[0081] If there are at least two preset neural network models, formed elements can be identified and classified based on the second image features and at least two preset neural network models to obtain predicted information of formed elements; based on the predicted information of formed elements, descriptive information of formed elements can be obtained.

[0082] The predicted information of formed elements refers to the possible descriptive information of formed elements predicted by the preset neural network model. This predicted descriptive information can also indicate the type of formed element. The descriptive information predicted by any preset neural network model can serve as an intermediate result for the formed element. The descriptive information of the formed element is obtained by analyzing multiple intermediate results. If the number of preset neural network models is one or at least two, the corresponding methods for obtaining the descriptive information of the formed element include, but are not limited to, the following:

[0083] If there are at least two preset neural network models, the predicted information is weighted and fused based on the weight of each preset neural network model to obtain the descriptive information of formed components. This achieves the fusion of predicted descriptive information through multiple preset neural network models, thereby improving the accuracy of the descriptive information by combining multiple preset neural network models.

[0084] The weights of a pre-defined neural network model indicate the accuracy of its predictions. The reasonableness of the model's predictions of formed elements can serve as an objective standard for evaluating its accuracy. Lower reasonableness corresponds to lower accuracy and a smaller weight, while higher reasonableness corresponds to higher accuracy and a larger weight. The weights are thus determined using this objective standard of reasonableness. It's understandable that the weights can be preset values, such as those based on experience; or they can be derived from the confidence level of the predicted formed elements output by the model, for example, by setting a positive correlation between the weights and the confidence level. The confidence level is the probability value of the type output by the model, ranging from [0, 1], where 1 represents the highest confidence and 0 represents the lowest. The weights can be determined by setting the relationship between the weights and the confidence level.

[0085] The process of determining the rationality of the information output by the preset neural network model (which can be prediction or description) includes: calculating a classification heatmap of the target image based on the second image features, the classification heatmap being used to characterize the region of interest in the process of the preset neural network model recognizing the target image; and judging the rationality of the information output by the preset neural network model based on the classification heatmap.

[0086] For example, in the process of obtaining the second image features, the feature maps corresponding to each convolutional layer can be obtained through the second feature extraction model mentioned above, which are denoted as f1, f2, f3, ..., respectively. The weights corresponding to each convolutional map are w1, w2, w3, ..., and the heat map values ​​are obtained by multiplying the convolutional map with the corresponding weights and summing them: map = f1*w1 + f2*w2 + f3*w3 + ..., where map is the classification heat map of the target image, which indicates the region of interest and the heat value of the region of interest through the heat map value.

[0087] The lower the value of the classification heatmap, the more likely the preset neural network model is focusing on the wrong region during identification. Consequently, the reasonableness of the prediction information obtained based on the preset neural network model is also lower. Reasonableness can be used as a standard to judge the accuracy of the preset neural network model. The lower the reasonableness, the lower the accuracy, and the smaller the weight of the preset neural network model. On the other hand, the higher the value of the classification heatmap, the more reasonable the prediction information obtained based on the preset neural network model is, and the larger the weight of the preset neural network model is. This allows for the accurate setting of the weight of the preset neural network model based on the classification heatmap.

[0088] In this embodiment, one way to determine the weight value based on the classification heatmap is, but is not limited to, setting a correspondence between the classification heatmap and the weight, such as setting a correspondence between the heatmap value and the weight value. After determining the heatmap value of any preset neural network model, the weight can be determined based on the correspondence.

[0089] In addition to fusing multiple preset neural network models to obtain descriptive information of formed components, this embodiment can also employ other fusion methods, such as fusing to obtain second image features. One method for fusing to obtain second image features is as follows: Figure 2 As shown, another approach involves using at least two pre-defined neural network models to extract features from the target image, obtaining image features extracted by each model. These features are then fused to obtain a second image feature. For example, the image features extracted by each second feature extraction model can be dimensionality-reduced to form a single second image feature. Finally, at least one pre-defined neural network model is used to output descriptive information of the formed elements. In other words, the pre-defined neural network model possesses both feature extraction and recognition / classification capabilities. For details on the feature extraction and fusion process of the pre-defined neural network model, please refer to the description of the second feature extraction model above; it will not be elaborated upon here.

[0090] It's important to note here that as the living environment changes, the formed elements (such as cells) within the human body undergo certain changes. Correspondingly, the preset neural network model used to identify and classify these formed elements also needs to adapt. For example, if the reasonableness of the predicted information about formed elements output by the preset neural network model meets certain conditions, the model parameters of the preset neural network model that meets these conditions are adjusted so that it can change with the changes in formed elements. These specific conditions may include, but are not limited to, the reasonableness of the predicted information about formed elements output by the preset neural network model being less than a threshold. This preset condition triggers the adjustment of the model parameters of the preset neural network model. During the adjustment process, the historical samples used to train the preset neural network model can be changed; the specific process will not be described in this embodiment.

[0091] Besides predicting descriptive information through a preset neural network model, this embodiment can also employ other methods. One method is that the second image feature is used to characterize the pixel information of the target image. A feature vector is composed of the pixel information (such as pixel values) of each pixel in the target image, and the feature vector composed of pixel information is determined as the second image feature. Correspondingly, based on the second image feature, the formed elements are identified and classified to obtain the descriptive information of the formed elements, including: calculating at least one of the geometric shape, size, and color of the formed elements based on the pixel information of the target image; and identifying and classifying the formed elements based on at least one of the geometric shape, size, and color of the formed elements to obtain the descriptive information of the formed elements.

[0092] The above-mentioned descriptive information about formed elements is at least used to indicate the type of formed element. Correspondingly, the descriptive information may include: the type of formed element, the morphology of the formed element, and the structure of the formed element. The structure of the formed element is a supplementary description. The type of formed element indicates that it is one of the following: red blood cells, white blood cells, platelets, sediment, etc. Morphology and structure differ for different types of formed elements. The following explanation uses red blood cells, white blood cells, platelets, and sediment as examples:

[0093] Red blood cell morphology includes various forms defined by ICSH (International Committee for Standardization of Blood), such as normal, oval, teardrop, and fragmented. Red blood cell structure includes abnormal structures such as normal, basophilic stippling, and Papenheimer bodies.

[0094] The morphology of white blood cells includes: segmented neutrophils, band neutrophils, eosinophils, basophils, monocytes, lymphocytes, primitive cells, promyelocytes, metamyelocytes, metamyelocytes, pyknosis, reactive lymphocytes, immature monocytes, etc. The structure of white blood cells includes: abnormal nuclear morphology, vacuolation, abnormal granule distribution, etc.

[0095] Platelet morphology includes: large platelets, giant platelets, platelet aggregation, etc.

[0096] Sediment is an impurity that has no shape or structure.

[0097] 107: Output descriptive information of formed elements to facilitate understanding of the formed elements in the sample to be tested. In one embodiment, the descriptive information of the formed elements can be stored or displayed in association with the original image, such as displaying the type, morphology, and structure of the formed elements, or displaying statistical data on the type, morphology, and structure of the formed elements, such as the proportion of the number of each type of formed element to the total number of all types. In this embodiment, the output of descriptive information of formed elements includes, but is not limited to, the following methods:

[0098] One approach is to associate the descriptive information of the formed elements with the original image at least once; output the descriptive information of the formed elements during the output of the original image, so that the associated descriptive information of the formed elements can be output during the display of the original image. This not only allows viewing the original image, but also provides an intuitive understanding of the original image through the descriptive information of the formed elements.

[0099] The process of outputting descriptive information for formed elements during the output of the original image includes: based on the location information of the formed elements, marking the formed elements in the original image, and displaying descriptive information of the formed elements in the areas where they are located, such as... Figure 3 As shown, based on the location information of formed elements, the formed elements in the original image are marked with rectangular boxes, and the descriptive information of the formed elements enclosed in each rectangular box is displayed. This realizes the output of the descriptive information of formed elements in the original image, which makes it easier to view abnormal formed elements in the original image.

[0100] Another approach is to determine whether a formed element is normal or abnormal based on its descriptive information, and then output the descriptive information of the abnormal formed element. In this embodiment, the determination of whether a formed element is normal or abnormal can be made using the morphology and structure in the descriptive information. After identifying the abnormal formed element, the descriptive information of the abnormal formed element can be output directly, or the detection result obtained based on the descriptive information of the abnormal formed element can be output. For example, based on the descriptive information of the abnormal formed element, the proportion of the abnormal formed element in the sample to be tested can be determined, and the proportion of the abnormal formed element in the sample to be tested can be output.

[0101] This embodiment can also obtain the percentage of abnormal formed elements in the sample to be tested, determine the degree of abnormality of the sample, and output the degree of abnormality of the sample. For example, a pre-defined correspondence between percentage and degree of abnormality can be used. After determining the percentage of abnormal formed elements in the current sample to be tested, the degree of abnormality can be obtained based on the correspondence. The degree of abnormality can be represented by, but is not limited to, special symbols, such as "+". Different numbers of "+" represent different degrees of abnormality. For example, "+" / "++" / "++++" indicate that the degree of abnormality gradually increases.

[0102] The formed element classification method described above for the test sample involves acquiring the original image of the test sample, extracting features from the original image to obtain first image features, locating the formed elements in the test sample based on the first image features to obtain the positional information of the formed elements, obtaining the target image of the region where the formed elements are located from the original image based on the positional information of the formed elements, extracting features from the target image to obtain second image features, and identifying and classifying the formed elements based on the second image features and a preset neural network model to obtain descriptive information of the formed elements. Because the second image features referenced in the identification and classification process are obtained from the target image of the region where the formed elements are located, interference from other regions of the original image is eliminated, thereby improving accuracy. Furthermore, after obtaining the descriptive information of the formed elements, the descriptive information is output to introduce and explain the formed elements, facilitating understanding of each formed element in the test sample.

[0103] like Figure 4 As shown, after acquiring the original image of a test sample, feature extraction is performed on the original image. This can be achieved, for example, by using, but not limited to, the first feature extraction model described above, to extract features from the original image and obtain the first image features. The localization model is then called to process the first image features, locating the formed elements in the test sample. The localization model outputs the location information of the formed elements in the test sample. The target image is then obtained from the original image using the location information of the formed elements. For the target image, features can be extracted from the original image using, but not limited to, the second feature extraction model described above, to obtain the second image features. The process of extracting the second image features can refer to the above... Figure 2 The schematic diagram shown illustrates the fusion of image features corresponding to multiple target images to obtain a second image feature.

[0104] The second image features are input into a preset neural network model, which then identifies and classifies the formed elements pointed to by the second image features, obtaining descriptive information about the formed elements and outputting that descriptive information, such as... Figure 4 The diagram shows the output of descriptive information of formed elements in the original image and indicates the location of the formed elements in the original image.

[0105] This embodiment can also integrate the localization function of the localization model and the recognition and classification function of the preset neural network model into a single network model. For example, a network model capable of detecting both the location information and descriptive information of formed elements can be constructed based on network models with object detection functions, such as the YOLO network model and the SSD (Single Shot MultiBox Detector) network model. The execution process of the network model is as follows: Figure 5As shown, after inputting the original image, the network model obtains images of the original image at different scales. It then extracts features from these images at different scales, and based on the extracted features, performs localization, recognition, and classification, thereby outputting descriptive information of formed elements, such as... Figure 5 The image outputs descriptive information about formed elements from the original image. In the process of localization and classification using a network model, the model can perform identification and classification simultaneously with localization, resulting in faster searching of the areas containing formed elements and significantly reducing time consumption.

[0106] This embodiment does not limit the interrelationships between scale spaces or the number of images of different sizes in the images obtained by the network model.

[0107] Please see Figure 6 It illustrates a flowchart of another method for classifying formed samples in a test sample provided in an embodiment of this application, which may include the following steps:

[0108] 201: Obtain the original image of the sample to be tested.

[0109] 202: Extract features from the original image to obtain the first image features.

[0110] 203: Based on the first image features, locate the formed elements in the sample to be tested to obtain the position information of the formed elements.

[0111] 204: Based on the location information of formed elements, obtain the target image of the region where the formed elements are located from the original image.

[0112] 205: Extract features from the target image to obtain the second image features.

[0113] 206: Based on the second image features and the preset neural network model, the formed elements are identified and classified to obtain the descriptive information of the formed elements. The descriptive information of the formed elements at least indicates the type of the formed elements.

[0114] Steps 201 to 206 above: Please refer to steps 101 to 106 above, which will not be described again in this embodiment.

[0115] 207: Based on the second image features, determine the rationality of the descriptive information of the formed elements output by the preset neural network model. In this embodiment, the process of obtaining the rationality of the descriptive information output by the preset neural network model is as follows:

[0116] Based on the second image features, a classification heatmap of the target image is calculated. The classification heatmap is used to characterize the regions of interest in the process of the preset neural network model recognizing the target image. Based on the classification heatmap, the rationality of the descriptive information of the formed elements output by the preset neural network model is judged.

[0117] For example, in the process of obtaining the second image features, the feature maps corresponding to each convolutional layer can be obtained through the second feature extraction model mentioned above, which are denoted as f1, f2, f3, ..., respectively. The weights corresponding to each convolutional map are w1, w2, w3, ..., and the heat map values ​​are obtained by multiplying the convolutional map with the corresponding weights and summing them: map = f1*w1 + f2*w2 + f3*w3 + ..., where map is the classification heat map of the target image, which indicates the region of interest and the heat value of the region of interest through the heat map value.

[0118] Classification heatmaps can reveal the regions of interest that a pre-defined neural network model focuses on during the identification process. The descriptive information of formed elements output by the pre-defined neural network model is obtained based on these regions of interest. Therefore, classification heatmaps can serve as a reference for judging the rationality of the descriptive information of formed elements output by the pre-defined neural network model, in order to evaluate the rationality of the descriptive information of formed elements output by the pre-defined neural network model.

[0119] The lower the value of the classification heatmap, the more likely the preset neural network model is focusing on the wrong region during identification, thus indicating a lower degree of reasonableness in the descriptive information obtained by the model. Conversely, the higher the value of the classification heatmap, the more accurate the preset neural network model is in identifying the region, thus indicating a higher degree of reasonableness in the descriptive information obtained. Of course, other methods can also be used to determine reasonableness, which will not be elaborated upon in this embodiment.

[0120] In this embodiment, the rationality of the descriptive information output by the preset neural network model can be represented by, but is not limited to, numerical methods. For example, the larger the value, the higher the rationality, and vice versa. In this example, the method of determining the numerical value representing rationality can be, but is not limited to, setting a correspondence between the classification heatmap and the numerical value representing rationality. For example, setting a value correspondence between the heatmap value and the numerical value representing rationality. After determining the heatmap value of any preset neural network model, the numerical value representing rationality can be determined based on the value correspondence.

[0121] 208: If the reasonableness of the descriptive information of formed elements meets the preset conditions, then the descriptive information of formed elements is output. The preset conditions indicate that the reasonableness of the descriptive information of formed elements is relatively high. One manifestation of the preset conditions is that the reasonableness of the descriptive information output by the preset neural network model is greater than a preset level. For example, the preset level is represented by a numerical value representing reasonableness. The preset level can be a preset value. If the numerical value representing reasonableness is greater than the preset value, it is determined that the reasonableness of the descriptive information of formed elements is greater than the preset level, thus meeting the preset conditions. This means that the accuracy of the descriptive information of formed elements output by the preset neural network model is high, and the descriptive information of formed elements can be output.

[0122] 209: If the reasonableness of the description information of formed components does not meet the preset conditions, an alarm message will be output. If the reasonableness of the description information of formed components does not meet the preset conditions, it means that the accuracy of the description information of formed components output by the preset neural network model is low. At this time, an alarm message can be output to indicate that the currently output description information of formed components is unreasonable.

[0123] The formed element classification method in the above-mentioned test samples can calculate the classification heatmap of the target image based on the second image features; based on the classification heatmap, the rationality of the descriptive information of the formed elements output by the preset neural network model is judged, thereby realizing the rationality judgment of the descriptive information of the formed elements output by the preset neural network model through the classification heatmap, and outputting alarm information when the rationality does not meet the preset conditions, so as to indicate that the descriptive information of the formed elements currently output is unreasonable.

[0124] In this embodiment, the classification heatmap and the preset neural network model can also be combined in other ways to obtain descriptive information of formed elements. Figure 7 This paper demonstrates a feasible method for obtaining descriptive information of formed elements by combining classification heatmaps and a pre-defined neural network model, including the following steps:

[0125] 301: Based on the second graphic features and at least two preset neural network models, identify and classify formed elements to obtain the predicted information of formed elements output by each preset neural network model.

[0126] Among them, the predicted information of formed elements is the possible descriptive information of formed elements predicted by the preset neural network model. The possible descriptive information predicted by the model can also indicate the type of formed elements. The descriptive information predicted by any preset neural network model can be used as an intermediate result of formed elements. The descriptive information of formed elements can be obtained by analyzing multiple intermediate results. Thus, the descriptive information of formed elements can be obtained through multiple preset neural network models. In this way, even if the focus of a preset neural network model is wrong and the accuracy of its output prediction information is wrong, it can still be corrected by the prediction information output by other preset neural network models to improve the accuracy of the descriptive information.

[0127] 302: Based on the second image features, determine the rationality of the predicted information of the formed elements output by each preset neural network model.

[0128] One feasible approach is to calculate a classification heatmap for each preset neural network model based on the second image features. The classification heatmap is used to characterize the regions of interest that the preset neural network model focuses on during the identification of the target image. Based on the classification heatmap of each preset neural network model, the rationality of the predicted information of the formed elements output by each preset neural network model is judged.

[0129] The classification heatmap can reveal the regions of interest during the recognition process of the preset neural network model. The descriptive information of formed elements output by the preset neural network model is obtained based on these regions of interest. Therefore, the classification heatmap can serve as a reference for judging the rationality of the descriptive information of formed elements output by the preset neural network model, thereby evaluating its rationality. For details, please refer to the above embodiments. Other feasible methods for judging the rationality of the predicted information of formed elements output by the preset neural network model can be found in the descriptions of the above embodiments, and will not be repeated here.

[0130] 303: Based on the rationality of the prediction information of formed elements output by each preset neural network model and the prediction information of formed elements output by each preset neural network model, the descriptive information of formed elements is obtained.

[0131] One way to obtain the descriptive information of formed elements is to determine the weight of the preset neural network model based on the rationality of the predicted information of formed elements output by each preset neural network model, and obtain the descriptive information of formed elements based on the weight of each preset neural network model and the predicted information of formed elements output by each preset neural network model. For example, the predicted information can be weighted and fused using weights to obtain the descriptive information of formed elements. Please refer to the above embodiment for the process.

[0132] Another way to obtain descriptive information of formed components is to select one prediction information from all the prediction information output by the preset neural network models as descriptive information based on the reasonableness of the prediction information of the formed components output by each preset neural network model. For example, the prediction information output by the preset neural network model with the highest reasonableness can be selected as descriptive information. Alternatively, multiple preset neural network models (referred to as target model sets) that output the same prediction information can be determined from all preset neural network models. If multiple target model sets exist, the descriptive information is determined based on the reasonableness of the preset neural network models in the multiple target model sets. For example, the prediction information of the target model set with the highest reasonableness among the multiple target model sets can be used as descriptive information.

[0133] For example, 8 to Figure 13 The figures show the classification results of six different preset neural network models for the same formed element. Figures 8 to 13 The data includes three figures: Figure (a) shows the types of formed elements identified by the preset neural network model (the type of formed element is the one with the largest value); Figure (b) is a classification heatmap; and Figure (c) is a comparison diagram between the classification heatmap and the target image. Figures 8 to 13 The corresponding preset neural network models have the following network structures in order: resnet18, resnet50, densenet102, densenet201, Xception, and InceptionV4.

[0134] The formed elements identified by ResNet50, DenseNet102, and DenseNet201 are of the type "nucleated red," with thermal intensity values ​​of approximately 0.5. The formed elements identified by ResNet18, Xception, and InceptionV4 are of the type "lobed," with thermal intensity values ​​of approximately 0.1. Based on the thermal intensity values, the descriptive information output by ResNet50, DenseNet102, and DenseNet201 is more reasonable than that output by ResNet18, Xception, and InceptionV4. Furthermore, the descriptive information output by ResNet50, DenseNet102, and DenseNet201 is consistent. Therefore, "nucleated red" is selected as the descriptive information.

[0135] From the above Figures 8 to 13 As shown, any preset neural network model outputs two types of formed elements, namely kernel red and lobed, and shows the confidence level of each type of formed element. The confidence level is the probability value of the type output by the preset neural network model, and the value range is [0, 1], where 1 is the highest confidence level and 0 is the lowest confidence level. The type with the highest confidence level is determined as the type of formed element pointed to by the current target image.

[0136] Figure 14 This application illustrates another process for determining descriptive information by combining a preset neural network model and a heat map, which may include the following steps:

[0137] 401: Based on the second image features and the first preset neural network model, the formed elements are identified and classified to obtain the predicted information of the formed elements output by the first preset neural network model.

[0138] 402: Based on the second image features, determine the rationality of the predicted information of formed elements output by the first preset neural network model. For example, based on the second image features, calculate the classification heatmap of the first preset neural network model. The classification heatmap is used to characterize the region of interest of the first preset neural network model in the process of recognizing the target image. Based on the classification heatmap of the first preset neural network model, determine the rationality of the predicted information of formed elements output by the first preset neural network model. For details, please refer to the above embodiment, which will not be repeated here.

[0139] 403: If the reasonableness of the prediction information of formed elements output by the first preset neural network model is less than the preset threshold, then based on the second image features and the second preset neural network model, the formed elements are identified and classified to obtain the prediction information of formed elements output by the second preset neural network model.

[0140] The recognition speed of the second preset neural network model is slower than that of the first preset neural network model, but the accuracy of the second preset neural network model is greater than that of the first preset neural network model.

[0141] 404: Use the prediction information of formed elements output by the second preset neural network model as the description information of formed elements.

[0142] 405: If the reasonableness of the prediction information of the formed elements output by the first preset neural network model is greater than or equal to the preset threshold, then the prediction information of the formed elements output by the first preset neural network model shall be used as the description information of the formed elements.

[0143] The above Figure 14 The process for obtaining description information is shown below. Figure 7 The difference in the process of obtaining descriptive information shown is that: Figure 14The first and second preset neural network models called in the process have a certain relationship: the recognition speed of the second preset neural network model is slower than that of the first preset neural network model, but the accuracy of the second preset neural network model is greater than that of the first preset neural network model. Therefore, the first preset neural network model is called first to identify and classify the second image features. If the rationality of the first preset neural network model is determined to be greater than or equal to the preset threshold, the predicted information of the formed elements output by the first preset neural network model can be used as the descriptive information of the formed elements to improve the recognition efficiency of the descriptive information.

[0144] Based on the descriptive information of formed elements, the type of formed element is determined. This embodiment can also perform anomaly analysis for different types of formed elements. For example, if the descriptive information of the formed element indicates that the formed element is a red blood cell, a white blood cell, or a platelet, then at least one of the geometric shape, size, and color of the formed element is calculated; based on at least one of the geometric shape, size, and color of the formed element, it is determined whether the red blood cell is a normal or abnormal formed element. Further taking red blood cells as an example, such as... Figure 15 A flowchart of the classification method for formed elements in the test sample shows the abnormal analysis of red blood cells, relative to... Figure 1 The method shown may also include the following steps:

[0145] 108: If the descriptive information of the constituent elements indicates that the constituent elements are red blood cells, then calculate at least one of the following: the geometry, size, and color of the red blood cells.

[0146] 109: Determine whether a red blood cell is a normal red blood cell or an abnormal red blood cell based on at least one of the following: the geometry, size, and color of the red blood cell.

[0147] For normal red blood cells and abnormal red blood cells, there are differences in at least one aspect of geometry, size and color, and it is possible to determine whether a red blood cell is normal or abnormal based on at least one aspect of geometry, size and color. Thus, when the descriptive information of formed elements indicates that the formed elements are red blood cells, at least one of the geometry, size and color of the red blood cell can be calculated to determine whether it is a normal or abnormal red blood cell.

[0148] For example, for identified red blood cells, the size of the red blood cells can be calculated and classified into normal, large, and small cells to determine whether there are red blood cells of uneven size in the sample to be tested; or, for identified red blood cells, the color can be calculated and classified into normal, low pigment, and polychromatic to determine whether there are red blood cells with abnormal color in the sample to be tested.

[0149] The process of analyzing abnormal platelet counts, such as Figure 16The diagram illustrates a flowchart of another method for classifying formed elements in a test sample according to an embodiment of this application, which may include the following steps:

[0150] 501: Obtain the original image of the sample to be tested.

[0151] 502: Extract features from the original image to obtain the first image features.

[0152] 503: Based on the first image features, locate the formed elements in the sample to be tested to obtain the position information of the formed elements.

[0153] 504: Based on the location information of formed elements, obtain the target image of the region where the formed elements are located from the original image.

[0154] 505: Extract features from the target image to obtain the second image features.

[0155] 506: Based on the second image features and the preset neural network model, the formed elements are identified and classified to obtain the descriptive information of the formed elements. The descriptive information of the formed elements at least indicates the type of the formed elements.

[0156] Steps 201 to 206 above: Please refer to steps 101 to 106 above, which will not be described again in this embodiment.

[0157] 507: Output descriptive information for formed elements.

[0158] Steps 501 to 507 above: Please refer to steps 101 to 107 above, which will not be described again in this embodiment.

[0159] 508: If the descriptive information of the formed element indicates that the formed element is platelets, then based on the location information of the platelets in the test sample, determine whether platelets in the test sample are aggregated, such as whether the platelets in the test sample are aggregated in the tail.

[0160] One way to determine whether platelets aggregate in a sample is to calculate the number of platelets in the region pointed to by the platelet location information in the sample, and then determine whether platelet aggregation exists based on the number of platelets.

[0161] Based on the location information of platelets in the sample to be tested, the number of platelets in the same area pointed to by the location information can be calculated to determine whether platelet aggregation exists by counting. For example, if the number of platelets in the area pointed to by the location information reaches a certain number (such as a preset number), then platelet aggregation is determined.

[0162] The system calculates the number of platelets in the area pointed to by the platelet location information and outputs the number of platelets, thus displaying the number of platelets in the sample to be tested. This allows doctors to decide whether to retest the sample based on the number of platelets, etc.

[0163] If the description information of the formed element indicates that the formed element is platelets and there is aggregation, then a prompt message related to the test sample is output. The prompt message is used to instruct at least one of the following operations to be performed on the test sample: retesting, re-examination, and re-collection of the sample, so as to re-analyze the platelet abnormality in the test sample.

[0164] Furthermore, the descriptive information for formed elements can be platelet aggregation, indicating the presence of platelet aggregation in the test sample. Therefore, if the descriptive information for formed elements indicates platelet aggregation, it is determined that platelet aggregation exists in the test sample. Upon determining that platelet aggregation exists in the test sample, a prompt message related to the test sample can be output, instructing at least one of the following operations: retesting, re-examination, or re-collection of the sample.

[0165] For example, if platelet aggregation is found in the sample to be tested, a prompt message can be used to instruct the subject to have the sample re-collected to obtain a new sample for analysis.

[0166] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously, and steps in different embodiments can be combined with each other. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0167] Corresponding to the above method embodiments, this application also provides a terminal, including: an image acquisition device, a processor, and an output interface. The image acquisition device is configured to acquire an original image of a sample to be tested; the processor is configured to execute the formed element classification method in the sample to be tested to obtain descriptive information of the formed elements in the sample; and the output interface is configured to output the descriptive information of the formed elements.

[0168] One form of the terminal in this embodiment can be a cell image analysis device, and another form can be a computer or other device provided to doctors or monitoring personnel. After acquiring the original image of the sample to be tested, these devices can obtain descriptive information of each formed element in the sample to be tested based on the original image, such as the various types of cells included in the sample to be tested.

[0169] Figure 17 An optional structure of a formed element analysis system provided in an embodiment of this application is shown, which may include: an imaging device 10, a smear holding device (not shown), and a control device 30.

[0170] Imaging device 10 is configured to capture images of a sample smear of the sample to be tested, thereby obtaining an original image of the sample. Imaging device 10 may include a camera and a lens group and is used to capture images of cells in the sample smeared on the smear. Smear holding device is configured to move the sample smear relative to the imaging device so that the imaging device 10 can capture cell images of a specific area of ​​the sample smear. Control device 30 is configured to execute the formed element classification method in the sample to be tested, so as to obtain and output descriptive information of each formed element in the sample to be tested.

[0171] The smear holding device can be a stage or a robotic arm, configured to carry or hold the smear. The smear holding device or imaging device 10 can be connected to a drive device, and the smear holding device and imaging device 10 move relative to each other under the drive of the drive device.

[0172] Figure 17 The formed element analysis system shown may also include: a smear preparation device 40, a first transfer track 50, and a second transfer track 60.

[0173] The smear preparation device 40 is equipped with a sample smear for preparing the sample to be tested.

[0174] The first transport track 50 is configured to transport test tubes containing samples to be tested to the smear preparation device.

[0175] The second transfer track 60 is configured to transport the sample smear of the sample to be tested from the smear preparation device to the smear holding device.

[0176] The control device 30 is also configured to be electrically connected to the first and second transmission tracks and to control the movement of the first and second transmission tracks. Thus, the control device 30 controls the first and second transmission tracks to automatically transport the sample smears between the smear preparation device 40, the imaging device 10 and the control device 30, thereby completing the preparation and analysis of the sample smears.

[0177] The aforementioned formed element analysis system may further include: an identification device, a slide gripping device, and a sample smear recovery device. The identification device is used to identify the identity information of the sample smear, the slide gripping device is used to grip the sample smear from the identification device onto the sample smear holding device for testing, and the sample smear recovery device is used to place the tested sample smear.

[0178] The aforementioned formed element analysis system may further include: a slide basket loading device for loading a slide basket containing a sample smear to be tested; and a slide clamping device for clamping the sample smear to be tested from the slide basket loaded on the slide basket loading device and placing it into an identification device for identification information. The slide basket loading device is connected to a first transport track so that the sample smear prepared by the sample smear preparation device can be transported to the smear holding device. In one embodiment, the cell image analysis device may include an imaging device and a smear holding device.

[0179] In addition to the formed element analysis system, a formed element analysis method is also provided, which may include the following steps:

[0180] 1) The sample smear is moved relative to the imaging device by means of a smear holding device.

[0181] 2) The original image of the sample to be tested is obtained by taking a picture of the sample smear using an imaging device.

[0182] 3) The original image is processed as follows using the control device:

[0183] 31) Extract features from the original image to obtain the first image features.

[0184] Based on the first image features, the formed elements in the sample to be tested are located to obtain the positional information of the formed elements.

[0185] 32) Based on the location information of formed elements, obtain the target image of the region where the formed elements are located from the original image.

[0186] 33) Extract features from the target image to obtain the second image features.

[0187] 34) Based on the second image features and the preset neural network model, the formed elements are identified and classified to obtain the descriptive information of the formed elements. The descriptive information of the formed elements at least indicates the type of the formed elements.

[0188] 35) Output descriptive information of formed elements.

[0189] The process by which the control device processes the original image is not described in this embodiment.

[0190] The above-mentioned classification method may also include the following steps:

[0191] 4) The test tube containing the sample to be tested is transported to the smear preparation device using the first transfer track.

[0192] 5) The sample smear prepared by the smear preparation device is transported to the cell image analysis device using the second transport track to obtain the original image captured by the imaging device.

[0193] This application embodiment also provides a storage medium storing computer program code, which, when executed, implements the formed element classification method in the test sample described above.

[0194] It should be noted that the various embodiments in this specification can be described in a progressive manner, and the features described in the various embodiments can be substituted for or combined with each other. Each embodiment focuses on describing the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to mutually. For system-type embodiments, terminal-type embodiments, and method embodiments corresponding to system-type embodiments, since they are basically similar to the previous method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the previous method embodiments.

[0195] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0196] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0197] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for classifying formed elements in a sample to be tested, comprising: Obtain the original image of the sample to be tested; Feature extraction is performed on the original image to obtain the first image features; Based on the first image features, the formed elements in the sample to be tested are located to obtain the location information of the formed elements; Based on the location information of the formed elements, target images of the regions where the formed elements are located are obtained from at least two original images. The shooting parameters corresponding to each of the at least two original images are different, including: the focus used in the process of shooting the at least two original images is different, which makes the sharpness of the at least two original images different. Feature extraction is performed on at least two target images to obtain image features for each target image; a second image feature is obtained based on the image features of each target image; Based on the second image features and the preset neural network model, the formed elements are identified and classified to obtain descriptive information of the formed elements. The descriptive information of the formed elements at least indicates the type of the formed elements. Output the description information of the formed elements.

2. The method according to claim 1, wherein, Also includes: Based on the second image features, the rationality of the descriptive information of the formed elements output by the preset neural network model is determined; The step of outputting the description information of the formed element includes: if the rationality of the description information of the formed element meets the preset conditions, then the description information of the formed element is output. If the reasonableness of the description information of the formed elements does not meet the preset conditions, an alarm message will be output.

3. The method according to claim 1, wherein, The step of identifying and classifying the formed elements based on the second image features and a preset neural network model to obtain descriptive information about the formed elements includes: Based on the second image features and at least two preset neural network models, the formed elements are identified and classified to obtain the predicted information of the formed elements output by each preset neural network model; Based on the predicted information of the formed elements output by each preset neural network model, the descriptive information of the formed elements is obtained.

4. The method according to claim 3, wherein, The description information of the formed elements obtained based on the prediction information of each preset neural network model includes: The predicted information based on the weight output of each preset neural network model is weighted and fused to obtain the descriptive information of the formed elements.

5. The method according to claim 4, wherein, The weights of the preset neural network model are obtained based on the reasonableness of the prediction information of the formed elements output by the preset neural network model, and / or based on the confidence level of the prediction information of the formed elements output by the preset neural network model.

6. The method according to claim 1, wherein, The step of identifying and classifying the formed elements based on the second image features and a preset neural network model to obtain descriptive information about the formed elements includes: Based on the second image features and at least two preset neural network models, the formed elements are identified and classified to obtain the predicted information of the formed elements output by each preset neural network model; Based on the second image features, the rationality of the prediction information of the formed elements output by each preset neural network model is determined; Based on the rationality of the predicted information of the formed elements output by each preset neural network model and the predicted information of the formed elements output by each preset neural network model, the descriptive information of the formed elements is obtained.

7. The method according to any one of claims 2, 5, and 6, wherein, The process of determining the rationality of the information output by the preset neural network model includes: Based on the second image features, a classification heatmap of the target image is calculated, and the classification heatmap is used to characterize the region of interest in the process of the preset neural network model recognizing the target image; Based on the classification heatmap, the rationality of the information output by the preset neural network model is determined.

8. The method according to claim 1, wherein, The step of identifying and classifying the formed elements based on the second image features and a preset neural network model to obtain descriptive information about the formed elements includes: Based on the second image features and the first preset neural network model, the formed elements are identified and classified to obtain the predicted information of the formed elements output by the first preset neural network model. Based on the second image features, the rationality of the prediction information of the formed elements output by the first preset neural network model is determined; If the reasonableness of the predicted information of the formed elements output by the first preset neural network model is less than a preset threshold, then based on the second image features and the second preset neural network model, the formed elements are identified and classified to obtain the predicted information of the formed elements output by the second preset neural network model; the recognition speed of the second preset neural network model is less than that of the first preset neural network model, but the accuracy of the second preset neural network model is greater than that of the first preset neural network model. The prediction information of the formed elements output by the second preset neural network model is used as the description information of the formed elements; If the reasonableness of the prediction information of the formed element output by the first preset neural network model is greater than or equal to the preset threshold, then the prediction information of the formed element output by the first preset neural network model is used as the description information of the formed element.

9. The method according to claim 8, wherein, The step of determining the reasonableness of the predicted information of formed elements output by the first preset neural network model based on the second image features includes: Based on the second image features, a classification heatmap of the first preset neural network model is calculated. The classification heatmap is used to characterize the region of interest of the first preset neural network model in the process of recognizing the target image. Based on the classification heatmap of the first preset neural network model, the rationality of the prediction information of the formed elements output by the first preset neural network model is determined.

10. The method according to claim 1, wherein feature extraction is performed on at least two target images to obtain image features for each target image; obtaining second image features based on the image features of each target image includes: The target image is feature extracted using at least two of the preset neural network models to obtain the image features extracted by each preset neural network model; The image features extracted by each preset neural network model are fused to obtain the second image feature.

11. The method according to claim 1, wherein, The output of the descriptive information of the formed elements includes: At least the descriptive information of the formed elements is associated with the original image; During the output of the original image, the descriptive information of the formed elements is output.

12. The method according to claim 11, wherein, The step of outputting the descriptive information of the formed elements during the output of the original image includes: marking the formed elements in the original image based on the position information of the formed elements, and displaying the descriptive information of the formed elements in the area where the formed elements are located.

13. The method according to claim 11, wherein, The output of the descriptive information of the formed elements includes: Based on the description information of the formed element, determine whether the formed element is a normal formed element or an abnormal formed element, and output the description information of the abnormal formed element.

14. The method according to claim 13, wherein, Also includes: If the description information of the formed element indicates that the formed element is a red blood cell, a white blood cell, or a platelet, then at least one of the geometry, size, and color of the formed element is calculated; Based on at least one of the geometry, size, and color of the formed element, determine whether the formed element is a normal formed element or an abnormal formed element.

15. The method according to claim 13, wherein, Also includes: If the description information of the formed elements indicates that the formed elements are platelets, then based on the location information of the platelets in the sample to be tested, it is determined whether platelet aggregation exists in the sample to be tested; or, If the description information of the formed elements indicates that the formed elements are platelet aggregation, then it is determined that platelet aggregation exists in the sample to be tested.

16. The method according to claim 15, wherein, The determination of whether platelet aggregation exists in the test sample based on the platelet location information in the test sample includes: Based on the location information of platelets in the sample to be tested, the number of platelets in the region pointed to by the location information of the platelets is calculated; Based on the number of platelets, it is determined whether platelet aggregation exists.

17. The method according to claim 16, wherein, Also includes: Output the number of platelets.

18. The method according to claim 17, wherein, Also includes: If the description information of the formed element indicates that the formed element is platelets and there is aggregation, or if the description information of the formed element indicates that the formed element is platelet aggregation, then output a prompt message related to the sample to be tested; The prompt message is used to instruct at least one of the following operations: retesting, re-examining, and re-collecting the sample.

19. The method according to claim 1, wherein, The acquisition of the original image of the sample to be tested includes: acquiring the original image of the sample to be tested on the carrier using a microscope.

20. The method according to claim 18, wherein, The descriptive information of the formed element includes: the type of the formed element, the morphology of the formed element, and the structure of the formed element, wherein the structure of the formed element is a supplementary description of the formed element.

21. The method according to claim 1, wherein, The sample to be tested can be any one of blood, bone marrow, urine, or body fluid.

22. The method according to claim 1, wherein, The step of locating the formed elements in the sample to be tested based on the first image features to obtain the location information of the formed elements includes: inputting the first image features into a localization model, locating the formed elements in the sample to be tested through the localization model, and obtaining the location information of the formed elements output by the localization model.

23. A terminal, comprising: An image acquisition device configured to acquire the original image of the sample to be tested; A processor configured to execute a formed element classification method in a test sample as described in any one of claims 1 to 22, to obtain descriptive information of formed elements in the test sample; Output interface, configured to output descriptive information of the formed elements.

24. A shaped component analysis system, comprising: An imaging device is configured to capture images of a sample smear of a sample to be tested, thereby obtaining the original image of the sample to be tested. A smear holding device configured to allow relative movement between the sample smear and the imaging device; A control device configured to perform a morphological classification method for a sample to be tested as described in any one of claims 1 to 22.

25. The system according to claim 24, wherein, Also includes: A smear preparation apparatus, configured for preparing sample smears of the sample to be tested; A first transport track is configured to transport test tubes containing samples to be tested to the smear preparation apparatus; The second transport track is configured to transport the sample smear of the sample to be tested from the smear preparation device to the imaging device; The control device is also configured to be electrically connected to the first and second transmission tracks and to control the movement of the first and second transmission tracks.

26. A method for analyzing formed elements, comprising: The sample smear is moved relative to the imaging device by a smear holding device; The original image of the sample to be tested is obtained by taking a picture of the sample smear using an imaging device. The original image is processed by the control device as follows: Feature extraction is performed on the original image to obtain the first image features; Based on the first image features, the formed elements in the sample to be tested are located to obtain the location information of the formed elements; Based on the location information of the formed elements, target images of the region where the formed elements are located are obtained from at least two original images respectively. The shooting parameters corresponding to each of the at least two original images are different, including: the focus used in the process of shooting the at least two original images is different, so that the sharpness of the at least two original images is different. Feature extraction is performed on at least two target images to obtain image features for each target image; a second image feature is obtained based on the image features of each target image; Based on the second image features and the preset neural network model, the formed elements are identified and classified to obtain descriptive information of the formed elements. The descriptive information of the formed elements at least indicates the type of the formed elements. Output the description information of the formed elements.

27. The method according to claim 26, wherein, Also includes: The test tube containing the sample to be tested is transported to the smear preparation device using the first transfer track; The sample smear prepared by the smear preparation device is transported to the smear holding device using the second transport track to obtain the original image captured by the imaging device.

28. A storage medium storing computer program code, which, when executed, implements the morphological classification method for a sample to be tested as described in any one of claims 1 to 22.

Images