Cell slice image recognition method and system based on multi-model collaboration

Abstract

The present invention discloses a cell slice image recognition method and system based on multi-model collaboration, the method comprising: obtaining a cell digital slice image corresponding to a target biological tissue, inputting it into a plurality of trained special staining conversion models to obtain a plurality of special staining slice images of different staining forms corresponding to the cell digital slice image; inputting each of the special staining slice images into a trained nuclear morphology segmentation model to obtain a nuclear morphology segmentation image corresponding to each of the special staining slice images, and determining a nuclear morphology image corresponding to the cell digital slice image; inputting the nuclear morphology image into a trained risk prediction model to obtain a pathological risk prediction probability corresponding to the cell digital slice image. It can be seen that the present invention can realize fully automated pathological prediction of cell slices, and can use different staining conversions to improve the accuracy of cell nucleus recognition, so as to effectively assist in the prediction of pathological risks.

Description

Technical Field

[0001] The present invention relates to the technical field of biological image data processing, and in particular to a cell slice image recognition method and system based on multi-model collaboration. Background Art

[0002] In the study of intelligent imaging, pathological identification of cell sections based on image processing is an important research direction. Many popular image recognition algorithms have also shined in this field with high accuracy. However, when solving the task of cell section image recognition, the existing image processing technology does not consider using neural network models to realize the conversion of different staining images, and further using the characteristics of different staining images to improve the efficiency and effect of nuclear morphology segmentation and recognition. It can be seen that the existing technology has defects and needs to be solved urgently. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a cell slice image recognition method and system based on multi-model collaboration, which can realize fully automated pathological prediction of cell slices and can use different staining conversions to improve the accuracy of cell nucleus recognition, so as to effectively assist the prediction of pathological risks.

[0004] In order to solve the above technical problems, the first aspect of the present invention discloses a cell slice image recognition method based on multi-model collaboration, the method comprising:

[0005] Acquire digital cell slice images corresponding to target biological tissues;

[0006] Inputting the cell digital slice image into a plurality of trained special staining conversion models to obtain a plurality of special staining slice images of different staining forms corresponding to the cell digital slice image;

[0007] Inputting each of the special stained slice images into a trained nuclear morphology segmentation model to obtain a nuclear morphology segmentation image corresponding to each of the special stained slice images;

[0008] Determining the nuclear morphology image corresponding to the cell digital slice image according to the nuclear morphology segmentation images corresponding to all the special staining slice images;

[0009] The nuclear morphology image is input into the trained risk prediction model to obtain the pathological risk prediction probability corresponding to the cell digital slice image.

[0010] As an optional embodiment, in the first aspect of the present invention, the special staining conversion model includes at least one of an H&E staining conversion model, a PAS staining conversion model, a GMS staining conversion model, and a Gomori staining conversion model; and / or, the staining form of the special staining section image includes at least one of H&E staining, PAS staining, GMS staining, and Gomori staining.

[0011] As an optional embodiment, in the first aspect of the present invention, the special staining conversion model is a neural network model, and the special staining conversion model is trained by a training data set including a plurality of training cell slice images and staining images of corresponding staining forms; and / or, the nuclear morphology segmentation model is a HoVer-Net structure neural network model corresponding to the staining form of the input special staining slice image, and the nuclear morphology segmentation model is trained by a training data set including a plurality of training staining slice images corresponding to the staining form and corresponding nuclear morphology area annotations.

[0012] As an optional implementation, in the first aspect of the present invention, determining the nuclear morphology image corresponding to the target biological tissue according to the nuclear morphology segmentation images corresponding to all the special stained section images includes:

[0013] For each of the special stained slice images, determining an importance parameter corresponding to the special stained slice image;

[0014] Determining the objective function and constraint conditions of a dynamic programming algorithm according to the importance parameters of all the special stained slice images;

[0015] Establishing a dynamic programming algorithm model according to the objective function and the constraint conditions;

[0016] All of the specially stained section images are input into the dynamic programming algorithm model for iterative calculation until an optimal nuclear morphology image that satisfies the objective function and the constraint conditions is obtained, so as to obtain a nuclear morphology image corresponding to the target biological tissue; the nuclear morphology image includes image information of all of the nuclear morphology segmentation images within the image envelope of the nuclear morphology image.

[0017] As an optional implementation, in the first aspect of the present invention, the importance parameter includes a model weight and a clarity weight; the model weight is proportional to the model performance parameter of the training phase of the special staining conversion model corresponding to the staining form corresponding to the special staining slice image; the clarity weight is calculated by the following steps:

[0018] Based on the sharpness calculation formula, respectively calculating a first sharpness parameter corresponding to the nuclear morphology segmentation image of the special stained section image and a second sharpness parameter corresponding to the image area where the nuclear morphology segmentation image is not located;

[0019] Calculate the weighted sum average of the first sharpness parameter and the second sharpness parameter to obtain the clarity weight corresponding to the special stained section image; the first weight corresponding to the first sharpness parameter is greater than the second weight corresponding to the second sharpness parameter; the weight ratio of the first weight and the second weight is proportional to the image size ratio; the image size ratio is the ratio of the image size of the nuclear morphology segmentation image to the image size of the image area where the non-nuclear morphology segmentation image is located.

[0020] As an optional implementation, in the first aspect of the present invention, the objective function includes minimizing the image size of the nuclear morphology image; and the constraint condition includes:

[0021] The image envelope of the nuclear morphology image cannot exceed the maximum envelope corresponding to all the nuclear morphology segmentation images;

[0022] The edge point distance between any point on the image edge of the nuclear morphology image and at least two nearby image edge points of the nuclear morphology segmentation images satisfies: the edge point distance corresponding to the nuclear morphology segmentation image with a higher importance parameter is smaller.

[0023] As an optional implementation, in the first aspect of the present invention, the step of acquiring a digital cell slice image corresponding to a target biological tissue comprises:

[0024] Acquire the whole cell digital slice image corresponding to the target biological tissue;

[0025] According to the model input image size of the special staining conversion model and the nuclear morphology segmentation model, the whole cell digital slice image is segmented to obtain a plurality of cell digital slice images corresponding to the target biological tissue; and the method further comprises:

[0026] Determine the nuclear quantity weight and importance weight corresponding to each of the cell digital slice images; the nuclear quantity weight is proportional to the number of cell nuclei identified corresponding to the nuclear morphology image; the importance weight is proportional to the average of the importance parameters of all the special staining slice images corresponding to the cell digital slice image;

[0027] According to the nuclear quantity weight and the importance weight, a weighted sum average of the pathological risk prediction probabilities of all the cell digital slice images corresponding to the target biological tissue is calculated to obtain the pathological risk prediction probability corresponding to the target biological tissue.

[0028] As an optional embodiment, in the first aspect of the present invention, the special staining conversion model and the nuclear morphology segmentation model are trained by the following steps:

[0029] For the special staining conversion model and the nuclear morphology segmentation model corresponding to the same staining form, a comprehensive training data set including a plurality of cell digital slice images for training and the corresponding staining slice images of the staining form and corresponding nuclear morphology region annotations is prepared;

[0030] For the special staining conversion model or the nuclear morphology segmentation model, the comprehensive training data set is input into the model for training until convergence, the converged model is copied into the first model and the second model, the comprehensive training data set is subjected to the first data enhancement to obtain the data set input into the first model to output the prediction result of the first model, the comprehensive training data set is subjected to the second data enhancement to obtain the enhanced data set, the enhanced data set and the corresponding prediction result are input into the second model for training until convergence, and the converged second model is determined as the trained special staining conversion model or the nuclear morphology segmentation model; wherein the feature extraction network structure of the special staining conversion model and the nuclear morphology segmentation model is the same, and the feature recombination network and the loss calculation function are different.

[0031] The second aspect of the present invention discloses a cell slice image recognition system based on multi-model collaboration, the system comprising:

[0032] An acquisition module, used for acquiring digital slice images of cells corresponding to target biological tissues;

[0033] A conversion module, used for inputting the cell digital slice image into a plurality of trained special staining conversion models to obtain a plurality of special staining slice images of different staining forms corresponding to the cell digital slice image;

[0034] A segmentation module, used for inputting each of the special stained slice images into a trained nuclear morphology segmentation model to obtain a nuclear morphology segmentation image corresponding to each of the special stained slice images;

[0035] A determination module, used to determine the nuclear morphology image corresponding to the cell digital slice image according to the nuclear morphology segmentation images corresponding to all the special staining slice images;

[0036] The prediction module is used to input the nuclear morphology image into a trained risk prediction model to obtain the pathological risk prediction probability corresponding to the cell digital slice image.

[0037] As an optional embodiment, in the second aspect of the present invention, the special staining conversion model includes at least one of an H&E staining conversion model, a PAS staining conversion model, a GMS staining conversion model, and a Gomori staining conversion model; and / or, the staining form of the special staining section image includes at least one of H&E staining, PAS staining, GMS staining, and Gomori staining.

[0038] As an optional embodiment, in the second aspect of the present invention, the special staining conversion model is a neural network model, and the special staining conversion model is trained by a training data set including a plurality of training cell slice images and staining images of corresponding staining forms; and / or, the nuclear morphology segmentation model is a HoVer-Net structure neural network model corresponding to the staining form of the input special staining slice image, and the nuclear morphology segmentation model is trained by a training data set including a plurality of training staining slice images corresponding to the staining form and corresponding nuclear morphology area annotations.

[0039] As an optional implementation, in the second aspect of the present invention, the specific manner in which the determination module determines the nuclear morphology image corresponding to the target biological tissue according to the nuclear morphology segmentation images corresponding to all the special stained section images includes:

[0040] For each of the special stained slice images, determining an importance parameter corresponding to the special stained slice image;

[0041] Determining the objective function and constraint conditions of a dynamic programming algorithm according to the importance parameters of all the special stained slice images;

[0042] Establishing a dynamic programming algorithm model according to the objective function and the constraint conditions;

[0043] All of the specially stained section images are input into the dynamic programming algorithm model for iterative calculation until an optimal nuclear morphology image that satisfies the objective function and the constraint conditions is obtained, so as to obtain a nuclear morphology image corresponding to the target biological tissue; the nuclear morphology image includes image information of all of the nuclear morphology segmentation images within the image envelope of the nuclear morphology image.

[0044] As an optional implementation, in the second aspect of the present invention, the importance parameter includes a model weight and a clarity weight; the model weight is proportional to the model performance parameter of the training phase of the special staining conversion model corresponding to the staining form corresponding to the special staining slice image; the clarity weight is calculated by the following steps:

[0045] Based on the sharpness calculation formula, respectively calculating a first sharpness parameter corresponding to the nuclear morphology segmentation image of the special stained section image and a second sharpness parameter corresponding to the image area where the nuclear morphology segmentation image is not located;

[0046] Calculate the weighted sum average of the first sharpness parameter and the second sharpness parameter to obtain the clarity weight corresponding to the special stained section image; the first weight corresponding to the first sharpness parameter is greater than the second weight corresponding to the second sharpness parameter; the weight ratio of the first weight and the second weight is proportional to the image size ratio; the image size ratio is the ratio of the image size of the nuclear morphology segmentation image to the image size of the image area where the non-nuclear morphology segmentation image is located.

[0047] As an optional implementation, in the second aspect of the present invention, the objective function includes minimizing the image size of the nuclear morphology image; and the constraint condition includes:

[0048] The image envelope of the nuclear morphology image cannot exceed the maximum envelope corresponding to all the nuclear morphology segmentation images;

[0049] The edge point distance between any point on the image edge of the nuclear morphology image and at least two nearby image edge points of the nuclear morphology segmentation images satisfies: the edge point distance corresponding to the nuclear morphology segmentation image with a higher importance parameter is smaller.

[0050] As an optional implementation, in the second aspect of the present invention, the specific manner in which the acquisition module acquires the cell digital slice image corresponding to the target biological tissue includes:

[0051] Acquire the whole cell digital slice image corresponding to the target biological tissue;

[0052] According to the model input image size of the special staining conversion model and the nuclear morphology segmentation model, the whole cell digital slice image is segmented to obtain a plurality of cell digital slice images corresponding to the target biological tissue; and the prediction module is further used to perform the following steps:

[0053] Determine the nuclear quantity weight and importance weight corresponding to each of the cell digital slice images; the nuclear quantity weight is proportional to the number of cell nuclei identified corresponding to the nuclear morphology image; the importance weight is proportional to the average of the importance parameters of all the special staining slice images corresponding to the cell digital slice image;

[0054] According to the nuclear quantity weight and the importance weight, a weighted sum average of the pathological risk prediction probabilities of all the cell digital slice images corresponding to the target biological tissue is calculated to obtain the pathological risk prediction probability corresponding to the target biological tissue.

[0055] As an optional embodiment, in the second aspect of the present invention, the special staining conversion model and the nuclear morphology segmentation model are trained by the following steps:

[0056] For the special staining conversion model and the nuclear morphology segmentation model corresponding to the same staining form, a comprehensive training data set including a plurality of cell digital slice images for training and the corresponding staining slice images of the staining form and corresponding nuclear morphology region annotations is prepared;

[0057] For the special staining conversion model or the nuclear morphology segmentation model, the comprehensive training data set is input into the model for training until convergence, the converged model is copied into the first model and the second model, the comprehensive training data set is subjected to the first data enhancement to obtain the data set input into the first model to output the prediction result of the first model, the comprehensive training data set is subjected to the second data enhancement to obtain the enhanced data set, the enhanced data set and the corresponding prediction result are input into the second model for training until convergence, and the converged second model is determined as the trained special staining conversion model or the nuclear morphology segmentation model; wherein the feature extraction network structure of the special staining conversion model and the nuclear morphology segmentation model is the same, and the feature recombination network and the loss calculation function are different.

[0058] The third aspect of the present invention discloses another cell slice image recognition system based on multi-model collaboration, the system comprising:

[0059] A memory storing executable program code;

[0060] a processor coupled to the memory;

[0061] The processor calls the executable program code stored in the memory to execute part or all of the steps in the cell slice image recognition method based on multi-model collaboration disclosed in the first aspect of the present invention.

[0062] Compared with the prior art, the present invention has the following beneficial effects:

[0063] The present invention can use multiple different conversion models to convert cell slices into slices with different staining forms, and based on the slices with different staining forms, implement the segmentation and processing of cell nucleus morphology images, and predict pathological risks, thereby realizing fully automated pathological prediction of cell slices, and can utilize different staining conversions to improve the accuracy of cell nucleus recognition, so as to effectively assist in the prediction of pathological risks. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0065] Figure 1 It is a schematic flow chart of a cell slice image recognition method based on multi-model collaboration disclosed in an embodiment of the present invention;

[0066] Figure 2 It is a structural schematic diagram of a cell slice image recognition system based on multi-model collaboration disclosed in an embodiment of the present invention;

[0067] Figure 3 It is a schematic diagram of the structure of another cell slice image recognition system based on multi-model collaboration disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0068] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or terminal including a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products or terminals.

[0070] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0071] The present invention discloses a cell slice image recognition method and system based on multi-model collaboration, which can use multiple different conversion models to convert cell slices into slices with different staining forms, and segment and process cell nucleus morphology images based on slices with different staining forms, and predict pathological risks, thereby realizing fully automated pathological prediction of cell slices, and can use different staining conversions to improve the accuracy of cell nucleus recognition, so as to effectively assist in the prediction of pathological risks. The following are detailed descriptions.

[0072] Embodiment 1

[0073] See also Figure 1 , Figure 1 : is a flow chart of a cell slice image recognition method based on multi-model collaboration disclosed in an embodiment of the present invention. Figure 1 The described method can be applied to a corresponding image data processing device, an image data processing terminal, and an image data processing server, and the server can be a local server or a cloud server, and the embodiment of the present invention is not limited to the following. Figure 1 As shown, the cell slice image recognition method based on multi-model collaboration may include the following operations:

[0074] 101. Obtain a digital cell slice image corresponding to the target biological tissue.

[0075] Optionally, the digital slice image of the present invention can be a full-view digital slice. Specifically, the processing process is: the quantitative sample solution extracted from the culture medium is fixed on a glass slide, and then undergoes some necessary processing. These operations can be automatically completed by the digital slice system. After the above operations are completed, a microscope can be used to observe and make full-view digital slices. Specifically, the full-view digital slice scanning is to obtain a high-resolution digital image by scanning and collecting with a fully automatic microscope or optical magnification system, and the high-resolution digital image obtained is automatically stitched and processed with high precision in multiple fields of view by a computer to obtain high-quality visualization data (i.e., digital slices) for application in various fields of pathology.

[0076] 102. Input the cell digital slice image into a plurality of trained special staining conversion models to obtain a plurality of special staining slice images of different staining forms corresponding to the cell digital slice image.

[0077] As an optional embodiment, the special staining conversion model includes at least one of the H&E (hematoxylin-eosin staining) staining conversion model, the PAS (Periodic Acid-Schiff) staining conversion model, the GMS (Grocott Methenamine silver) staining conversion model, and the Gomori (Gomori's silver hydroxide ammonia staining) staining conversion model.

[0078] As an optional embodiment, the staining form of the special staining section image includes at least one of H&E staining, PAS staining, GMS staining, and Gomori staining.

[0079] As an optional embodiment, the special staining conversion model is a neural network model, and the special staining conversion model is trained by a training data set including a plurality of training cell slice images and staining images of corresponding staining forms.

[0080] 103. Input each special stained slice image into the trained nuclear morphology segmentation model to obtain a nuclear morphology segmentation image corresponding to each special stained slice image.

[0081] As an optional embodiment, the nuclear morphology segmentation model is a HoVer-Net structure neural network model corresponding to the staining form of the input special staining slice image, and the nuclear morphology segmentation model is obtained by training a training data set including a plurality of corresponding staining forms and corresponding nuclear morphology region annotations. In another case, the nuclear morphology segmentation model can be a U-Net structure neural network model. U-Net is a deformation of a convolutional neural network. The entire U-Net neural network mainly consists of two parts: a contracting path and an expanding path. The search path is mainly used to capture context information in the image, and the expansion path is to accurately locate the part that needs to be segmented out of the image. U-Net can train some relatively small sample data using data augmentation, especially medical-related data, so the emergence of U-Net is very helpful for deep learning of medical images with fewer samples.

[0082] 104. Determine the nuclear morphology image corresponding to the cell digital slice image according to the nuclear morphology segmentation images corresponding to all the special staining slice images.

[0083] 105. Input the nuclear morphology image into the trained risk prediction model to obtain the pathological risk prediction probability corresponding to the cell digital slice image.

[0084] Optionally, the pathology prediction probability may be a cancer prediction probability. Optionally, the risk prediction model may also be a neural network model, and may be obtained by training a training data set including a plurality of training nuclear morphology images and corresponding pathology risk annotations.

[0085] It can be seen that the method described in the embodiment of the present invention can use multiple different conversion models to convert cell slices into slices with different staining forms, and based on the slices with different staining forms, segment and process the cell nucleus morphology images, and predict the pathological risk, thereby realizing fully automated pathological prediction of cell slices, and can use different staining conversions to improve the accuracy of cell nucleus recognition, so as to effectively assist the prediction of pathological risk.

[0086] As an optional embodiment, determining the nuclear morphology image corresponding to the target biological tissue according to the nuclear morphology segmentation images corresponding to all the special staining slice images includes:

[0087] For each special stained slice image, determining the importance parameter corresponding to the special stained slice image;

[0088] According to the importance parameters of all special stained slice images, the objective function and constraint conditions of the dynamic programming algorithm are determined;

[0089] According to the objective function and constraints, a dynamic programming algorithm model is established;

[0090] All special staining slice images are input into the dynamic programming algorithm model for iterative calculation until the optimal nuclear morphology image that meets the objective function and constraint conditions is obtained, so as to obtain the nuclear morphology image corresponding to the target biological tissue; the nuclear morphology image includes the image information of all nuclear morphology segmentation images within the image envelope of the nuclear morphology image.

[0091] Optionally, the dynamic programming algorithm model may be a particle swarm optimization algorithm (PSO), which has the advantages of fast convergence speed, few parameters, and simple and easy algorithm implementation.

[0092] It can be seen that through the above embodiment, all special staining section images can be input into the dynamic programming algorithm model for iterative calculation until the optimal nuclear morphology image that meets the objective function and constraint conditions is obtained, so that the dynamic programming algorithm can be used to more accurately determine the nuclear morphology image corresponding to the target biological tissue, and the accuracy of cell nucleus identification can be improved to effectively assist in the prediction of pathological risks.

[0093] As an optional embodiment, the importance parameter includes a model weight and a clarity weight; the model weight is proportional to the model performance parameter of the training phase of the special staining conversion model corresponding to the staining form corresponding to the special staining slice image; the clarity weight is calculated by the following steps:

[0094] Based on the sharpness calculation formula, respectively calculating the first sharpness parameter corresponding to the nuclear morphology segmentation image of the special stained section image and the second sharpness parameter corresponding to the image area where the non-nuclear morphology segmentation image is located;

[0095] The weighted sum average of the first sharpness parameter and the second sharpness parameter is calculated to obtain the clarity weight corresponding to the special stained slice image; the first weight corresponding to the first sharpness parameter is greater than the second weight corresponding to the second sharpness parameter; the weight ratio of the first weight and the second weight is proportional to the image size ratio; the image size ratio is the ratio of the image size of the nuclear morphology segmentation image to the image size of the image area where the non-nuclear morphology segmentation image is located.

[0096] Optionally, the model performance parameter may be the prediction accuracy of the model or the average of the loss function values ​​of multiple training runs of the model.

[0097] Optionally, the sharpness calculation formula may be a rise distance technique algorithm, a modulation transfer function algorithm, a special sum frequency domain algorithm or a slant edge algorithm.

[0098] It can be seen that through the above embodiments, the importance parameter corresponding to the special staining section image can be accurately determined according to the model performance parameters and sharpness calculation of the corresponding special staining conversion model in the training phase, so that the nuclear morphology image corresponding to the target biological tissue can be more accurately determined based on this importance parameter in the subsequent dynamic programming algorithm, which can improve the accuracy of cell nucleus recognition and effectively assist in the prediction of pathological risks.

[0099] As an optional embodiment, the objective function includes minimizing the image size of the nuclear morphology image; and the constraints include:

[0100] The image envelope of the nuclear morphology image cannot exceed the maximum envelope corresponding to all nuclear morphology segmentation images;

[0101] The edge point distance between any point of the image edge of the nuclear morphology image and the image edge points of at least two nearby nuclear morphology segmentation images satisfies: the edge point distance corresponding to the nuclear morphology segmentation image with a higher importance parameter is smaller.

[0102] It can be seen that through the above embodiments, the objective function and constraints can be determined, so that the dynamic programming algorithm can be used in the subsequent process to more accurately determine the nuclear morphology image corresponding to the target biological tissue according to the objective function and constraints, thereby improving the accuracy of cell nucleus recognition and effectively assisting the prediction of pathological risks.

[0103] As an optional embodiment, obtaining a digital cell slice image corresponding to a target biological tissue includes:

[0104] Acquire the whole cell digital slice image corresponding to the target biological tissue;

[0105] According to the model input image size of the special staining conversion model and the nuclear morphology segmentation model, the whole cell digital slice image is segmented to obtain a plurality of cell digital slice images corresponding to the target biological tissue.

[0106] And, the method further comprises:

[0107] Determine the nuclear quantity weight and importance weight corresponding to each cell digital slice image; the nuclear quantity weight is proportional to the number of cell nuclei identified corresponding to the nuclear morphology image; the importance weight is proportional to the average of the importance parameters of all special staining slice images corresponding to the cell digital slice image;

[0108] According to the nuclear quantity weight and the importance weight, the weighted sum average of the pathological risk prediction probabilities of all cell digital slice images corresponding to the target biological tissue is calculated to obtain the pathological risk prediction probability corresponding to the target biological tissue.

[0109] It can be seen that through the above embodiments, the different weights of each cell digital slice image can be taken into account to calculate the weighted sum average of the pathological risk prediction probabilities of all cell digital slice images corresponding to the target biological tissue, so as to obtain the pathological risk prediction probability corresponding to the target biological tissue, which can improve the accuracy of cell pathological risk identification and effectively assist the study of pathological risk.

[0110] As an optional embodiment, the special staining conversion model and the nuclear morphology segmentation model are trained by the following steps:

[0111] For a special staining conversion model and a nuclear morphology segmentation model corresponding to the same staining form, a comprehensive training data set including a plurality of training cell digital slice images and corresponding staining slice images of the staining form and corresponding nuclear morphology region annotations is prepared;

[0112] For the special staining conversion model or the nuclear morphology segmentation model, the comprehensive training data set is input into the model for training until convergence, the converged model is copied as the first model and the second model, the comprehensive training data set is subjected to the first data enhancement to obtain the input into the first model to output the prediction result of the first model, the comprehensive training data set is subjected to the second data enhancement to obtain the enhanced data set, the enhanced data set and the corresponding prediction result are input into the second model for training until convergence, and the converged second model is determined as the trained special staining conversion model or nuclear morphology segmentation model; wherein the feature extraction network structure of the special staining conversion model and the nuclear morphology segmentation model is the same, and the feature recombination network and the loss calculation function are different.

[0113] Optionally, the first data enhancement method includes simple data enhancement methods such as rotation processing and mirror inversion processing. Optionally, the second data enhancement method includes more complex data enhancement methods such as dimension exchange processing, gamma transformation processing, logarithmic transformation processing and random erasure processing.

[0114] Through the above steps, first, the feature extraction networks of the two models can be jointly trained through the same set of comprehensive training data sets, so that the feature extraction networks of the two models can learn relevant parameter features and can effectively reduce the preparation cost of training. Second, the first model and the second model can be used to further improve the model training effect when the training data set is small. It can be seen that through the above embodiment, a special staining conversion model and a nuclear morphology segmentation model with better data processing effect can be obtained through low-cost training.

[0115] Embodiment 2

[0116] See also Figure 2 , Figure 2 : is a schematic diagram of a cell slice image recognition system based on multi-model collaboration disclosed in an embodiment of the present invention. Figure 2 The described system can be applied to corresponding image data processing devices, image data processing terminals, and image data processing servers, and the server can be a local server or a cloud server, which is not limited in the embodiments of the present invention. Figure 2 As shown, the system may include:

[0117] The acquisition module 201 is used to acquire a digital cell slice image corresponding to a target biological tissue.

[0118] Optionally, the digital slice image of the present invention can be a full-view digital slice. Specifically, the processing process is: the quantitative sample solution extracted from the culture medium is fixed on a glass slide, and then undergoes some necessary processing. These operations can be automatically completed by the digital slice system. After the above operations are completed, a microscope can be used to observe and make full-view digital slices. Specifically, the full-view digital slice scanning is to obtain a high-resolution digital image by scanning and collecting with a fully automatic microscope or optical magnification system, and the high-resolution digital image obtained is automatically stitched and processed with high precision in multiple fields of view by a computer to obtain high-quality visualization data (i.e., digital slices) for application in various fields of pathology.

[0119] The conversion module 202 is used to input the cell digital slice image into a plurality of trained special staining conversion models to obtain a plurality of special staining slice images of different staining forms corresponding to the cell digital slice image.

[0120] As an optional embodiment, the special staining conversion model includes at least one of the H&E (hematoxylin-eosin staining) staining conversion model, the PAS (Periodic Acid-Schiff) staining conversion model, the GMS (Grocott Methenamine silver) staining conversion model, and the Gomori (Gomori's silver hydroxide ammonia staining) staining conversion model.

[0121] As an optional embodiment, the staining form of the special staining section image includes at least one of H&E staining, PAS staining, GMS staining, and Gomori staining.

[0122] As an optional embodiment, the special staining conversion model is a neural network model, and the special staining conversion model is trained by a training data set including a plurality of training cell slice images and staining images of corresponding staining forms.

[0123] The segmentation module 203 is used to input each special stained slice image into the trained nuclear morphology segmentation model to obtain a nuclear morphology segmentation image corresponding to each special stained slice image.

[0124] As an optional embodiment, the nuclear morphology segmentation model is a HoVer-Net structure neural network model corresponding to the staining form of the input special staining slice image, and the nuclear morphology segmentation model is obtained by training a training data set including a plurality of corresponding staining forms and corresponding nuclear morphology region annotations. In another case, the nuclear morphology segmentation model can be a U-Net structure neural network model. U-Net is a deformation of a convolutional neural network. The entire U-Net neural network mainly consists of two parts: a contracting path and an expanding path. The search path is mainly used to capture context information in the image, and the expansion path is to accurately locate the part that needs to be segmented out of the image. U-Net can train some relatively small sample data using data augmentation, especially medical-related data, so the emergence of U-Net is very helpful for deep learning of medical images with fewer samples.

[0125] The determination module 204 is used to determine the nuclear morphology image corresponding to the cell digital slice image according to the nuclear morphology segmentation images corresponding to all the special staining slice images.

[0126] The prediction module 205 is used to input the nuclear morphology image into the trained risk prediction model to obtain the pathological risk prediction probability corresponding to the cell digital slice image.

[0127] Optionally, the pathology prediction probability may be a cancer prediction probability. Optionally, the risk prediction model may also be a neural network model, and may be obtained by training a training data set including a plurality of training nuclear morphology images and corresponding pathology risk annotations.

[0128] It can be seen that the device described in the embodiment of the present invention can use multiple different conversion models to convert cell slices into slices with different staining forms, and based on the slices with different staining forms, realize the segmentation and processing of cell nucleus morphology images, and predict pathological risks, thereby realizing fully automated pathological prediction of cell slices, and can use different staining conversions to improve the accuracy of cell nucleus identification, so as to effectively assist in the prediction of pathological risks.

[0129] As an optional embodiment, the specific manner in which the determination module 204 determines the nuclear morphology image corresponding to the target biological tissue according to the nuclear morphology segmentation images corresponding to all the special stained slice images includes:

[0130] For each special stained slice image, determining the importance parameter corresponding to the special stained slice image;

[0131] According to the importance parameters of all special stained slice images, the objective function and constraint conditions of the dynamic programming algorithm are determined;

[0132] According to the objective function and constraints, a dynamic programming algorithm model is established;

[0133] All special staining slice images are input into the dynamic programming algorithm model for iterative calculation until the optimal nuclear morphology image that meets the objective function and constraint conditions is obtained, so as to obtain the nuclear morphology image corresponding to the target biological tissue; the nuclear morphology image includes the image information of all nuclear morphology segmentation images within the image envelope of the nuclear morphology image.

[0134] Optionally, the dynamic programming algorithm model may be a particle swarm optimization algorithm (PSO), which has the advantages of fast convergence speed, few parameters, and simple and easy algorithm implementation.

[0135] It can be seen that through the above embodiment, all special staining section images can be input into the dynamic programming algorithm model for iterative calculation until the optimal nuclear morphology image that meets the objective function and constraint conditions is obtained, so that the dynamic programming algorithm can be used to more accurately determine the nuclear morphology image corresponding to the target biological tissue, and the accuracy of cell nucleus identification can be improved to effectively assist in the prediction of pathological risks.

[0136] As an optional embodiment, the importance parameter includes a model weight and a clarity weight; the model weight is proportional to the model performance parameter of the training phase of the special staining conversion model corresponding to the staining form corresponding to the special staining slice image; the clarity weight is calculated by the following steps:

[0137] Based on the sharpness calculation formula, respectively calculating the first sharpness parameter corresponding to the nuclear morphology segmentation image of the special stained section image and the second sharpness parameter corresponding to the image area where the non-nuclear morphology segmentation image is located;

[0138] The weighted sum average of the first sharpness parameter and the second sharpness parameter is calculated to obtain the clarity weight corresponding to the special stained slice image; the first weight corresponding to the first sharpness parameter is greater than the second weight corresponding to the second sharpness parameter; the weight ratio of the first weight and the second weight is proportional to the image size ratio; the image size ratio is the ratio of the image size of the nuclear morphology segmentation image to the image size of the image area where the non-nuclear morphology segmentation image is located.

[0139] Optionally, the model performance parameter may be the prediction accuracy of the model or the average of the loss function values ​​of multiple training runs of the model.

[0140] Optionally, the sharpness calculation formula may be a rise distance technique algorithm, a modulation transfer function algorithm, a special sum frequency domain algorithm or a slant edge algorithm.

[0141] It can be seen that through the above embodiments, the importance parameter corresponding to the special staining section image can be accurately determined according to the model performance parameters and sharpness calculation of the corresponding special staining conversion model in the training phase, so that the nuclear morphology image corresponding to the target biological tissue can be more accurately determined based on this importance parameter in the subsequent dynamic programming algorithm, which can improve the accuracy of cell nucleus recognition and effectively assist in the prediction of pathological risks.

[0142] As an optional embodiment, the objective function includes minimizing the image size of the nuclear morphology image; and the constraints include:

[0143] The image envelope of the nuclear morphology image cannot exceed the maximum envelope corresponding to all nuclear morphology segmentation images;

[0144] The edge point distance between any point of the image edge of the nuclear morphology image and the image edge points of at least two nearby nuclear morphology segmentation images satisfies: the edge point distance corresponding to the nuclear morphology segmentation image with a higher importance parameter is smaller.

[0145] It can be seen that through the above embodiments, the objective function and constraints can be determined, so that the dynamic programming algorithm can be used in the subsequent process to more accurately determine the nuclear morphology image corresponding to the target biological tissue according to the objective function and constraints, thereby improving the accuracy of cell nucleus recognition and effectively assisting the prediction of pathological risks.

[0146] As an optional embodiment, the specific manner in which the acquisition module 201 acquires the cell digital slice image corresponding to the target biological tissue includes:

[0147] Acquire the whole cell digital slice image corresponding to the target biological tissue;

[0148] According to the model input image size of the special staining conversion model and the nuclear morphology segmentation model, the whole cell digital slice image is segmented to obtain multiple cell digital slice images corresponding to the target biological tissue. And, the prediction module 205 is also used to perform the following steps:

[0149] Determine the nuclear quantity weight and importance weight corresponding to each cell digital slice image; the nuclear quantity weight is proportional to the number of cell nuclei identified corresponding to the nuclear morphology image; the importance weight is proportional to the average of the importance parameters of all special staining slice images corresponding to the cell digital slice image;

[0150] According to the nuclear quantity weight and the importance weight, the weighted sum average of the pathological risk prediction probabilities of all cell digital slice images corresponding to the target biological tissue is calculated to obtain the pathological risk prediction probability corresponding to the target biological tissue.

[0151] It can be seen that through the above embodiments, the different weights of each cell digital slice image can be taken into account to calculate the weighted sum average of the pathological risk prediction probabilities of all cell digital slice images corresponding to the target biological tissue, so as to obtain the pathological risk prediction probability corresponding to the target biological tissue, which can improve the accuracy of cell pathological risk identification and effectively assist the study of pathological risk.

[0152] As an optional embodiment, the special staining conversion model and the nuclear morphology segmentation model are trained by the following steps:

[0153] For a special staining conversion model and a nuclear morphology segmentation model corresponding to the same staining form, a comprehensive training data set including a plurality of training cell digital slice images and corresponding staining slice images of the staining form and corresponding nuclear morphology region annotations is prepared;

[0154] For the special staining conversion model or the nuclear morphology segmentation model, the comprehensive training data set is input into the model for training until convergence, the converged model is copied as the first model and the second model, the comprehensive training data set is subjected to the first data enhancement to obtain the input into the first model to output the prediction result of the first model, the comprehensive training data set is subjected to the second data enhancement to obtain the enhanced data set, the enhanced data set and the corresponding prediction result are input into the second model for training until convergence, and the converged second model is determined as the trained special staining conversion model or nuclear morphology segmentation model; wherein the feature extraction network structure of the special staining conversion model and the nuclear morphology segmentation model is the same, and the feature recombination network and the loss calculation function are different.

[0155] Optionally, the first data enhancement method includes simple data enhancement methods such as rotation processing and mirror inversion processing. Optionally, the second data enhancement method includes more complex data enhancement methods such as dimension exchange processing, gamma transformation processing, logarithmic transformation processing and random erasure processing.

[0156] Through the above steps, first, the feature extraction networks of the two models can be jointly trained through the same set of comprehensive training data sets, so that the feature extraction networks of the two models can learn relevant parameter features and can effectively reduce the preparation cost of training. Second, the first model and the second model can be used to further improve the model training effect when the training data set is small. It can be seen that through the above embodiment, a special staining conversion model and a nuclear morphology segmentation model with better data processing effect can be obtained through low-cost training.

[0157] Embodiment 3

[0158] See also Figure 3 , Figure 3 FIG. 1 is a schematic diagram of another cell slice image recognition system based on multi-model collaboration disclosed in an embodiment of the present invention. Figure 3 As shown, the system may include:

[0159] A memory 301 storing executable program codes;

[0160] a processor 302 coupled to the memory 301;

[0161] The processor 302 calls the executable program code stored in the memory 301 to execute part or all of the steps in the cell slice image recognition method based on multi-model collaboration disclosed in the first embodiment of the present invention.

[0162] Embodiment 4

[0163] An embodiment of the present invention discloses a computer storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute some or all steps in the cell slice image recognition method based on multi-model collaboration disclosed in Embodiment 1 of the present invention.

[0164] The system embodiment described above is only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without paying creative labor.

[0165] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on such an understanding, the above technical solution can be essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0166] Finally, it should be noted that the cell slice image recognition method and system based on multi-model collaboration disclosed in the embodiment of the present invention only discloses the preferred embodiment of the present invention, which is only used to illustrate the technical solution of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cell slice image recognition method based on multi-model collaboration, It is characterized in that The method comprises: Acquire digital cell slice images corresponding to target biological tissues; Inputting the cell digital slice image into a plurality of trained special staining conversion models to obtain a plurality of special staining slice images of different staining forms corresponding to the cell digital slice image; Input each of the special stained slice images into a trained nuclear morphology segmentation model to obtain a nuclear morphology segmentation image corresponding to each of the special stained slice images; the special staining conversion model is a neural network model, and the special staining conversion model is obtained by training a training data set including a plurality of training cell slice images and staining images of corresponding staining forms; the nuclear morphology segmentation model is a HoVer-Net structure neural network model corresponding to the staining form of the input special stained slice image, and the nuclear morphology segmentation model is obtained by training a training data set including a plurality of training staining slice images of corresponding staining forms and corresponding nuclear morphology region annotations; Determining the nuclear morphology image corresponding to the cell digital slice image according to the nuclear morphology segmentation images corresponding to all the special staining slice images includes: For each of the special stained slice images, an importance parameter corresponding to the special stained slice image is determined; the importance parameter includes a model weight and a clarity weight; the model weight is proportional to a model performance parameter of the training phase of the special staining conversion model corresponding to the staining form corresponding to the special stained slice image; the clarity weight is calculated by the following steps: Based on the sharpness calculation formula, respectively calculating a first sharpness parameter corresponding to the nuclear morphology segmentation image of the special stained section image and a second sharpness parameter corresponding to the image area where the nuclear morphology segmentation image is not located; Calculating a weighted sum average of the first sharpness parameter and the second sharpness parameter to obtain a clarity weight corresponding to the special stained slice image; a first weight corresponding to the first sharpness parameter is greater than a second weight corresponding to the second sharpness parameter; a weight ratio of the first weight and the second weight is proportional to an image size ratio; the image size ratio is a ratio of an image size of the nuclear morphology segmentation image to an image size of an image region where the non-nuclear morphology segmentation image is located; Determining the objective function and restriction conditions of a dynamic programming algorithm according to the importance parameters of all the special stained slice images; Establishing a dynamic programming algorithm model according to the objective function and the constraint conditions; Inputting all the special staining slice images into the dynamic programming algorithm model for iterative calculation until an optimal nuclear morphology image satisfying the objective function and the restriction condition is obtained, so as to obtain a nuclear morphology image corresponding to the target biological tissue; the nuclear morphology image includes image information of all the nuclear morphology segmentation images within the image envelope of the nuclear morphology image; The nuclear morphology image is input into the trained risk prediction model to obtain the pathological risk prediction probability corresponding to the cell digital slice image.

2. The cell slice image recognition method based on multi-model collaboration according to claim 1, It is characterized in that The special staining conversion model includes at least one of an H&E staining conversion model, a PAS staining conversion model, a GMS staining conversion model, and a Gomori staining conversion model; and / or, the staining form of the special staining section image includes at least one of H&E staining, PAS staining, GMS staining, and Gomori staining.

3. The cell slice image recognition method based on multi-model collaboration according to claim 1, It is characterized in that The objective function includes minimizing the image size of the nuclear morphology image; and the restriction conditions include: The image envelope of the nuclear morphology image cannot exceed the maximum envelope corresponding to all the nuclear morphology segmentation images; The edge point distance between any point on the image edge of the nuclear morphology image and at least two nearby image edge points of the nuclear morphology segmentation images satisfies: the edge point distance corresponding to the nuclear morphology segmentation image with a higher importance parameter is smaller.

4. The cell slice image recognition method based on multi-model collaboration according to claim 1, It is characterized in that The step of obtaining a digital cell slice image corresponding to the target biological tissue comprises: Acquire the whole cell digital slice image corresponding to the target biological tissue; According to the model input image size of the special staining conversion model and the nuclear morphology segmentation model, the whole cell digital slice image is segmented to obtain a plurality of cell digital slice images corresponding to the target biological tissue; and the method further comprises: Determine the nuclear quantity weight and importance weight corresponding to each of the cell digital slice images; the nuclear quantity weight is proportional to the number of cell nuclei identified corresponding to the nuclear morphology image; the importance weight is proportional to the average of the importance parameters of all the special staining slice images corresponding to the cell digital slice image; According to the nuclear quantity weight and the importance weight, a weighted sum average of the pathological risk prediction probabilities of all the cell digital slice images corresponding to the target biological tissue is calculated to obtain the pathological risk prediction probability corresponding to the target biological tissue.

5. The cell slice image recognition method based on multi-model collaboration according to claim 1, It is characterized in that The special staining conversion model and nuclear morphology segmentation model are trained by the following steps: For the special staining conversion model and the nuclear morphology segmentation model corresponding to the same staining form, a comprehensive training data set including a plurality of cell digital slice images for training and the corresponding staining slice images of the staining form and corresponding nuclear morphology region annotations is prepared; For the special staining conversion model or the nuclear morphology segmentation model, the comprehensive training data set is input into the model for training until convergence, the converged model is copied into a first model and a second model, the comprehensive training data set is subjected to a first data enhancement to obtain a data set input into the first model to output a prediction result of the first model, the comprehensive training data set is subjected to a second data enhancement to obtain an enhanced data set, the enhanced data set and the corresponding prediction result are input into the second model for training until convergence, and the converged second model is determined as the trained special staining conversion model or the nuclear morphology segmentation model; The special staining conversion model and the nuclear morphology segmentation model have the same feature extraction network structure, but different feature recombination networks and loss calculation functions.

6. A cell slice image recognition system based on multi-model collaboration, It is characterized in that The system comprises: An acquisition module, used for acquiring digital slice images of cells corresponding to target biological tissues; A conversion module, used for inputting the cell digital slice image into a plurality of trained special staining conversion models to obtain a plurality of special staining slice images of different staining forms corresponding to the cell digital slice image; A segmentation module, used for inputting each of the special stained slice images into a trained nuclear morphology segmentation model to obtain a nuclear morphology segmentation image corresponding to each of the special stained slice images; the special staining conversion model is a neural network model, which is trained by a training data set including a plurality of training cell slice images and staining images of corresponding staining forms; the nuclear morphology segmentation model is a HoVer-Net structure neural network model corresponding to the staining form of the input special stained slice image, which is trained by a training data set including a plurality of training staining slice images of corresponding staining forms and corresponding nuclear morphology region annotations; A determination module, used to determine the nuclear morphology image corresponding to the cell digital slice image according to the nuclear morphology segmentation images corresponding to all the special staining slice images, comprising: For each of the special stained slice images, an importance parameter corresponding to the special stained slice image is determined; the importance parameter includes a model weight and a clarity weight; the model weight is proportional to a model performance parameter of the training phase of the special staining conversion model corresponding to the staining form corresponding to the special stained slice image; the clarity weight is calculated by the following steps: Based on the sharpness calculation formula, respectively calculating a first sharpness parameter corresponding to the nuclear morphology segmentation image of the special stained section image and a second sharpness parameter corresponding to the image area where the nuclear morphology segmentation image is not located; Calculating a weighted sum average of the first sharpness parameter and the second sharpness parameter to obtain a clarity weight corresponding to the special stained slice image; a first weight corresponding to the first sharpness parameter is greater than a second weight corresponding to the second sharpness parameter; a weight ratio of the first weight and the second weight is proportional to an image size ratio; the image size ratio is a ratio of an image size of the nuclear morphology segmentation image to an image size of an image region where the non-nuclear morphology segmentation image is located; Determining the objective function and restriction conditions of a dynamic programming algorithm according to the importance parameters of all the special stained slice images; Establishing a dynamic programming algorithm model according to the objective function and the constraint conditions; Inputting all the special staining slice images into the dynamic programming algorithm model for iterative calculation until an optimal nuclear morphology image satisfying the objective function and the restriction condition is obtained, so as to obtain a nuclear morphology image corresponding to the target biological tissue; the nuclear morphology image includes image information of all the nuclear morphology segmentation images within the image envelope of the nuclear morphology image; The prediction module is used to input the nuclear morphology image into a trained risk prediction model to obtain the pathological risk prediction probability corresponding to the cell digital slice image.

7. A cell slice image recognition system based on multi-model collaboration, It is characterized in that The system comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the cell slice image recognition method based on multi-model collaboration as described in any one of claims 1-5.

Images