Cell image processing method, device, equipment and storage medium

Abstract

The present application provides a cell image processing method, device, equipment and storage medium, including: obtaining a composite detection network model formed by a plurality of pre-built sub-detection network models connected in parallel; the plurality of sub-detection network models correspond to images of sources of different sizes; intercepting the cell images to be processed according to the sources of different sizes to obtain the cell intercepted images of the corresponding source sizes; scaling the cell intercepted images corresponding to the sources of different sizes to cell scaled images corresponding to the target image size to obtain a plurality of cell scaled images; inputting the plurality of cell scaled images into the composite detection network model so that the composite detection network model inputs the plurality of cell scaled images into the respective sub-detection network models for forward reasoning, merging the forward reasoning results output by the respective sub-detection network models to obtain the cell detection results of the cell images to be processed, and obtaining the cell detection results obtained by the composite detection network model to improve the detection accuracy of the composite detection network model for cell images.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a cell image processing method, device, computer equipment and storage medium. Background Art

[0002] With the development of computer technology, computer technology can be used to detect cell types and cell positions in cell images. In traditional cell detection, cell samples can be made into cell smears, and the cell smears can be scanned with a high-resolution scanner to obtain the corresponding cell images; in the cell images, there are many types of cells, and each cell is in a different position in the cell image. How to process the cell images to obtain more accurate detection results is a problem that needs to be solved. Summary of the invention

[0003] Based on this, it is necessary to provide a cell image processing method, device, computer equipment and storage medium to address the above technical problems.

[0004] A cell image processing method, the method comprising:

[0005] Acquire a composite detection network model formed by combining a plurality of pre-built sub-detection network models in parallel; the plurality of sub-detection network models correspond to images of sources of different sizes;

[0006] Intercepting the cell images to be processed according to the different size sources to obtain cell intercepted images of the corresponding size sources;

[0007] Scaling the cell intercepted images corresponding to the sources of different sizes to cell scaled images corresponding to the target image size to obtain a plurality of cell scaled images;

[0008] Inputting the multiple cell zoom images into the composite detection network model, so that the composite detection network model inputs the multiple cell zoom images into each sub-detection network model for forward reasoning, and merging the forward reasoning results output by each sub-detection network model to obtain the cell detection result of the cell image to be processed;

[0009] The cell detection result obtained by the composite detection network model is obtained.

[0010] A cell image processing device, comprising:

[0011] A model acquisition module is used to acquire a composite detection network model formed by combining a plurality of pre-built sub-detection network models in parallel; the plurality of sub-detection network models correspond to images of sources of different sizes;

[0012] An image capture module, used to capture the cell images to be processed according to the different size sources to obtain cell capture images of the corresponding size sources;

[0013] An image scaling module, used for scaling the cell intercepted images corresponding to the sources of different sizes to cell scaled images corresponding to the target image size, to obtain a plurality of cell scaled images;

[0014] A forward reasoning module, used for inputting the multiple cell zoom images into the composite detection network model, so that the composite detection network model inputs the multiple cell zoom images into each sub-detection network model for forward reasoning, and combines the forward reasoning results output by each sub-detection network model to obtain the cell detection result of the cell image to be processed;

[0015] The detection result acquisition module is used to obtain the cell detection result obtained by the composite detection network model.

[0016] A computer device comprises a memory and a processor, wherein the memory stores a computer program and the processor executes the steps of the above method.

[0017] A computer-readable storage medium stores a computer program, wherein the computer program is used by a processor to execute the steps of the above method.

[0018] In the above-mentioned cell image processing method, device, computer equipment and storage medium, the computer equipment obtains a composite detection network model formed by a parallel combination of multiple pre-constructed sub-detection network models; the multiple sub-detection network models correspond to images of sources of different sizes; the cell images to be processed are intercepted according to the sources of different sizes to obtain cell intercepted images of the corresponding source sizes; the cell intercepted images corresponding to the sources of different sizes are scaled to cell scaled images corresponding to the target image size to obtain multiple cell scaled images; the multiple cell scaled images are input into the composite detection network model, so that the composite detection network model inputs the multiple cell scaled images into each sub-detection network model for forward reasoning, and the forward reasoning results output by each sub-detection network model are merged to obtain the cell detection results of the cell images to be processed; and the cell detection results obtained by the composite detection network model are obtained. The computer device is based on a composite detection network model composed of multiple sub-detection network models connected in parallel, and intercepts and scales the cell image to be processed to a uniform target image size according to the size source corresponding to each sub-detection network model, and then inputs the obtained cell scaled image into the corresponding sub-detection network model for forward reasoning and merges the various forward reasoning results to obtain the cell detection result of the cell image to be processed, thereby improving the detection accuracy of the composite detection network model for the cell image. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1is an internal structure diagram of a computer device in one embodiment;

[0020] Figure 2 is a schematic flow chart of a cell image processing method in one embodiment;

[0021] Figure 3 A schematic diagram of a process of a cell image processing method in another embodiment;

[0022] Figure 4 is a schematic diagram of sliding a window in one embodiment;

[0023] Figure 5 is a schematic flow chart of a cell image processing method in another embodiment;

[0024] Figure 6 FIG. 4 is a structural block diagram of a cell image processing device in one embodiment. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0026] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0027] The cell image processing method provided in this application can be applied to Figure 1 In the computer device shown in FIG. 1 , its internal structure diagram can be shown as follows: Figure 1As shown. The computer device includes a processor, a memory, a communication interface and an input device connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a cell image processing method is implemented. The input device of the computer device can be a button, a trackball or a touchpad set on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0028] Those skilled in the art will understand that Figure 1 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0029] In one embodiment, Figure 2 As shown, a cell image processing method is provided, and the method is applied to Figure 1 The computer device in the example is used to illustrate, including the following steps:

[0030] In step S201, a computer device obtains a composite detection network model formed by connecting a plurality of pre-built sub-detection network models in parallel; the plurality of sub-detection network models correspond to images of sources of different sizes.

[0031] After the training of the composite detection network model is completed, it can be used to perform cell detection on the image to be detected. The cell detection can include the cell category and the position of the cell in the image. The corresponding cell detection result can be the cell category in the image to be detected and the position of the cell on the image to be detected. The composite detection network model can be composed of multiple sub-detection network models (which can be RetinaNet models) connected in parallel, and each sub-detection network model corresponds to an input image of a source of different sizes. For example, Figure 5 It shows that the composite detection network model is composed of multiple RetinaNet models connected in parallel, and the size sources of the input images corresponding to RetinaNet models A, B and C are 512x512, 1024x1024, and 2048x2048, respectively.

[0032] Furthermore, in order to further improve the detection performance of the composite detection network model, the sub-detection network model may include a cascaded feature extraction network model and a feature detection network model; wherein the feature extraction network model is used to perform feature extraction processing on the cell zoom image to obtain the cell features of the cell zoom image at different scales; the feature detection network model is used to perform cell classification and orientation regression on the cell zoom image at each scale based on the cell features of the cell zoom image at each scale through each sub-feature detection network model to obtain forward reasoning results.

[0033] Exemplarily, if the sub-detection network model is a RetinaNet model, the feature extraction network model of the RetinaNet model can use but is not limited to ResNet (residual network, such as ResNet34), VGG, ResNeXt and other network structures. The feature extraction network model is subsequently combined with the feature detection network model FPN (feature pyramid net) to improve the detection performance of the model.

[0034] Taking the ResNet network structure as an example of a feature extraction network model, after the cell image is input into ResNet, ResNet can extract features of the cell image, obtain cell features at multiple different scales, and transmit the cell features to the feature detection network model; it can be understood that the cell image input into ResNet can be a scaled cell image (i.e., a scaled cell image).

[0035] It is understandable that those skilled in the art can modify the ResNet network structure according to actual conditions. Taking the modified ResNet34 as an example, when the input image size is 512x512, the structure table corresponding to the ResNet34 network structure is shown in Table 1:

[0036]

[0037] Table 1

[0038] The feature detection network model FPN can be composed of multiple class-box (category-orientation) sub-feature detection network models. The feature detection network model FPN performs cell classification and orientation regression on the cell image through the class-box sub-feature detection network model and based on the cell features at various scales extracted from the cell image by the feature network model.

[0039] Step S202: The computer device intercepts the cell images to be processed according to different size sources to obtain cell intercepted images of the corresponding size sources.

[0040] After obtaining the composite detection network model, the computer device can determine the preset size sources of each sub-detection network model in the composite detection network model, and perform interception processing (equivalent to image cropping processing) on ​​the cell image to be processed according to each size source, thereby obtaining cell interception images of multiple size sources. Exemplarily, the cell image to be processed can be an image collected by a magnifying scanner. When collecting, the magnifying scanner can use different microscopic magnifications, such as 20 times, 40 times, etc., and then the original image is cropped based on the size source corresponding to each sub-detection network model to obtain cell interception images of three size sources of 512x512, 1024x1024, and 2048x2048.

[0041] Furthermore, in order to accurately capture the cell image to be processed and avoid missing cells to ensure the detection performance of the model, the computer device may include the following steps when capturing the cell image: taking half of each size source as the sliding distance of the window; sliding the window horizontally and vertically along the cell image according to the sliding distance; and taking the image corresponding to the sliding window on the cell image as the cell captured image.

[0042] For example, Figure 5 As shown, the size source of RetinaNet model A is 512x512, so the sliding distances of the window in the horizontal and vertical directions of the cell image are 256 and 256 respectively (that is, the overlap is 256); the size source of RetinaNet model B is 1024x1024 (that is, the overlap is 1024), so the sliding distances of the window in the horizontal and vertical directions of the cell image are 512 and 512 respectively; the size source of RetinaNet model C is 2048x2048 (that is, the overlap is 2048), so the sliding distances of the window in the horizontal and vertical directions of the cell image are 1024 and 1024 respectively.

[0043] After sliding the window according to the sliding distance, the image corresponding to the sliding window on the cell image is used as the cell interception image; wherein, the image corresponding to the sliding window on the cell image can be understood as the overlapping part of the sliding window and the cell image, such as Figure 4 As shown, the overlap portion is determined by a transverse overlap dimension and a longitudinal overlap dimension.

[0044] Step S203: the computer device scales the cell intercepted images corresponding to the sources of different sizes to cell scaled images corresponding to the target image size, thereby obtaining a plurality of cell scaled images.

[0045] Among them, the target image size can be pre-set, which can be 512x512; illustratively, after obtaining the cell capture images corresponding to the three size sources of 512x512, 1024x1024, and 2048x2048, the computer device uniformly scales these cell capture images to the target image size of 512x512 to obtain multiple 512x512 cell scaled images; it can be understood that since the cell capture images correspond to different size sources, the cell scaled images obtained based on the cell capture images have a corresponding relationship with the different size sources.

[0046] In step S204, the computer device inputs the multiple cell zoom images into the composite detection network model, so that the composite detection network model inputs the multiple cell zoom images into each sub-detection network model for forward reasoning, and merges the forward reasoning results output by each sub-detection network model to obtain the cell detection results of the cell image to be processed.

[0047] Exemplarily, after obtaining a plurality of 512x512 cell scaled images, the computer device inputs the plurality of 512x512 cell scaled images into the composite detection network model, and then the composite detection network model inputs the plurality of 512x512 cell scaled images into the sub-detection network model of the corresponding size source, such as Figure 5 As shown, the cell scaled images corresponding to the size sources 512x512, 1024x1024, and 2048x2048 are input to RetinaNet models A, B, and C respectively; RetinaNet models A, B, and C output forward reasoning results of the complete scanned slice images (i.e., the cell images to be processed) corresponding to their respective size sources based on the input corresponding cell scaled images, and then the forward reasoning results of the respective RetinaNet models A, B, and C are merged, and the merged forward reasoning results are used as the result of the composite detection network model (i.e., Figure 5 The composite model result in FIG. 1 ), which can be understood as the cell detection result of the cell image to be processed.

[0048] Step S205: The computer device obtains the cell detection result obtained by the composite detection network model.

[0049] In the above-mentioned cell image processing method, the computer device obtains multiple pre-constructed; multiple sub-detection network models correspond to images of different size sources; intercepts the cell image to be processed according to the different size sources to obtain the cell intercepted image of the corresponding size source; scales the cell intercepted image corresponding to the different size sources to the cell scaled image corresponding to the target image size to obtain multiple cell scaled images; inputs the multiple cell scaled images into the composite detection network model, so that the composite detection network model inputs the multiple cell scaled images into each sub-detection network model for forward reasoning, merges the forward reasoning results output by each sub-detection network model, and obtains the cell detection result of the cell image to be processed; obtains the cell detection result obtained by the composite detection network model. The computer device is based on a composite detection network model composed of multiple sub-detection network models connected in parallel, and intercepts and scales the cell image to be processed according to the size source corresponding to each sub-detection network model to a unified target image size, and then inputs the obtained cell scaled image into the corresponding sub-detection network model for forward reasoning and merges the forward reasoning results to obtain the cell detection result of the cell image to be processed, thereby improving the detection accuracy of the composite detection network model for cell images.

[0050] In one embodiment, in order to further ensure the detection accuracy of the composite detection network model for cell images, before step S201, the computer device may also perform model training on each sub-detection network model, which may specifically include the following steps: when the computer device uses the training sample cell image to perform model training on each sub-detection network model, the loss value of the cell position in the training sample cell image is calculated using the minimization of the absolute error loss function, and the loss value of the cell category in the training sample cell image is calculated using the focus loss function.

[0051] In order to further ensure the detection accuracy of the composite detection network model for cell images, before step S201, the computer device can also perform the following steps: when the computer device performs model training on each sub-detection network model, it warms up the learning rate of each sub-detection network model and then uses cyclic cosine decrement to adjust the learning rate.

[0052] For example, when training the RetinaNet model, the L1 loss value is used to calculate the orientation loss of cells in the image (the L1 loss value is calculated as follows: ); For the calculation of the category loss value of cells in the image, the Focal Loss (focal loss function) is used (the calculation method of Focal Loss is: FL (p t )=-α t (1-p t ) γ log(p t)), because this calculation method can well solve the problems of complex samples and uneven sample classification.

[0053] When training a RetinaNet model, relevant (hyper)parameter choices may include:

[0054] (1) Model training uses stochastic gradient descent (SGD);

[0055] (2) Its learning rate is between 0.01 and 0.0001;

[0056] (3) Momentum parameter is between 0.9 and 0.99;

[0057] (4) Weight Decay is between 0.0001 and 0.00001;

[0058] (5) Use warmup learning rate adjustment during training, starting from 0 and linearly increasing to the initial learning rate;

[0059] (6) During training, after the warm-up learning rate is adjusted, the learning rate is adjusted using the cyclic cosine decreasing method to improve the training efficiency. The cycle period is between 10,000 and 100,000.

[0060] (7) The parameter Y in Focal Loss is 2 and α is 0.25.

[0061] In one embodiment, in order to further improve the accuracy of the cell detection results, after step S205, the computer device can also perform intra-class non-maximum suppression (NMS) on the cell candidate boxes in the cell image based on the cell detection results, which can include the following steps: determining, according to the cell detection results, the cell candidate boxes in the cell image used to select cells belonging to each cell category; obtaining the overlapping area between the cell candidate box with the maximum confidence in each cell category and other cell candidate boxes in the same cell category; and determining whether to retain other cell candidate boxes based on the overlapping area.

[0062] Exemplarily, when the computer device performs intra-class non-maximum suppression processing on the cell candidate boxes in the cell image, the following steps may be specifically included:

[0063] i. Sort the candidate boxes of each cell category from large to small according to their confidence;

[0064] ii. Starting from the cell candidate box with the maximum confidence, calculate the overlapping area between the cell candidate box with the maximum confidence and other cell candidate boxes under the same cell category, that is, calculate the overlapping area with the cell candidate box with a smaller confidence than it.

[0065] iii. Delete the cell candidate boxes with a smaller confidence level whose percentage of overlapping area is greater than a set threshold (0.3 to 0.5, for example, 0.5).

[0066] iv. Repeat the above steps until all the retained cell candidate boxes have undergone the above steps i-iii.

[0067] In one embodiment, in order to further improve the accuracy of the cell detection results, the computer device may also perform confidence filtering on the cell candidate frames in the cell image based on the cell detection results before performing intra-class non-maximum suppression processing on the cell candidate frames. That is, the cell candidate frames with confidences lower than a preset confidence threshold are deleted, and the cell candidate frames with confidences higher than (greater than or equal to) the preset confidence threshold are retained; the computer device performs intra-class non-maximum suppression processing on the retained cell candidate frames.

[0068] In one embodiment, the cell image processing method provided by the present application, when executed by a computer device, may include the following steps:

[0069] Step S301, when using the training sample cell image to perform model training on each sub-detection network model, the loss value of the cell position in the training sample cell image is calculated using the minimum absolute error loss function, and the loss value of the cell category in the training sample cell image is calculated using the focus loss function;

[0070] Step S302, when performing model training on each sub-detection network model, warm up the learning rate of each sub-detection network model and then adjust the learning rate using cyclic cosine decrement;

[0071] Step S303, obtaining a composite detection network model formed by connecting a plurality of pre-built sub-detection network models in parallel; the plurality of sub-detection network models correspond to images of sources of different sizes;

[0072] Step S304, taking half of each size source as the sliding distance of the window; sliding the window along the horizontal and vertical directions of the cell image according to the sliding distance; taking the image corresponding to the sliding window on the cell image as the cell intercepted image;

[0073] Step S305, scaling the cell intercepted images corresponding to the sources of different sizes to cell scaled images corresponding to the target image size, to obtain a plurality of cell scaled images;

[0074] Step S306, inputting a plurality of cell zoom images into the composite detection network model, so that the composite detection network model inputs the plurality of cell zoom images into each sub-detection network model for forward reasoning, and combines the forward reasoning results output by each sub-detection network model to obtain the cell detection result of the cell image to be processed;

[0075] Step S307, obtaining the cell detection result obtained by the composite detection network model;

[0076] Step S308, determining a cell candidate frame in the cell image for selecting cells belonging to each cell category according to the cell detection result; wherein the cell candidate frame has a confidence greater than or equal to a preset confidence threshold;

[0077] Step S309, obtaining the overlapping area between the cell candidate box with the maximum confidence in each cell category and other cell candidate boxes in the same cell category;

[0078] Step S310: Determine whether to retain other cell candidate frames according to the overlapping area.

[0079] In order to better understand the above method, an application example of the cell image processing method of the present application is described in detail below. In this application example, the cell image can be a cervical fluid-based thin layer cell image, and the cell image processing method of the present application can include data set construction, image augmentation processing of training data, pre-processing of normalized data, construction of a composite network detection model, and forward reasoning of the composite network detection model.

[0080] 1. Dataset construction:

[0081] a. Select a certain number of cervical liquid-based thin-layer cell images (scanned at 20x and / or 40x, etc.), annotate the target cells in the images with closed polygon outlines, and give the cell types. Store the annotation information as an annotation file corresponding to the corresponding liquid-based cell slices;

[0082] b. Divide the above data into a training set, a validation set, and a test set, and the ratio of the number of the training set, the validation set, and the test set is (7-8): (0.5-1): (1-2.5);

[0083] c. Preprocess the above image training data set and verification set:

[0084] c1. Obtain each coordinate position in the image, and randomly crop the image based on the label position, and ensure that the cropped image contains the polygon corresponding to the label, and the corresponding label polygon contained in the cropped image occupies more than 50% of the complete polygon area; the length and width of the cropped image size is between 256 and 2048, and the cropped image can be called a patch image;

[0085] c2. Process the points of the polygon corresponding to the label to generate a label box label. The label box is a square, determined by the upper left and lower right points. Therefore, by traversing the points of the polygon, the minimum and maximum values ​​of the horizontal axis and the minimum and maximum values ​​of the vertical axis can be found as the upper left and lower right points of the label box to generate the corresponding label box label.

[0086] 2. Perform image augmentation processing on the above training data:

[0087] a. Perform image orientation augmentation on the training set images:

[0088] a1. Randomly flip the training set images left and right;

[0089] a2. Perform random degree rotation (0 to 359 degrees) on the training set images (here the rotation is completed in c1 during cropping, that is, in addition to intercepting the original image, images with different rotation degrees are intercepted at the same time, and the corresponding rectangular annotation box is generated according to the label of the polygon annotation);

[0090] b. Perform image pixel augmentation on the training set images:

[0091] b1. Perform random pixel value changes based on brightness of the image, with the ratio change randomly ranging from -0.1 to 0.1;

[0092] b2. Perform random pixel value changes based on contrast on the image, with the ratio change randomly ranging from -0.1 to 0.1;

[0093] b3. Perform random pixel value changes on the image based on the r / g / b channels, with the ratio change randomly ranging from -0.2 to 0.2.

[0094] 3. Preprocessing of data for normalization

[0095] a. Normalize the image input to the model, including:

[0096] a1. Calculate the average pixel value and standard deviation of each channel of the RGB channel (color scale channel) of all slice images (ie, cervical liquid-based thin layer cell images) and obtain the result;

[0097] a2. Normalize the patch image entering the model. See b3 for the processing formula.

[0098] b. Formula:

[0099] b1. Average value formula:

[0100] b2. Standard deviation formula:

[0101] b3. Normalization formula:

[0102] Among them, p is the original image pixel value, and np is the normalized image pixel value.

[0103] 4. Construction of composite detection network model:

[0104] a. The composite detection network model consists of multiple RetinaNet models, which can be referenced Figure 5 Each RetinaNet model uses patch images of different sizes (length and width are 256 to 2048), and all patch images are uniformly scaled to images of fixed size (length and width are between 256 and 1024) as the input images of their corresponding RetinaNet models;

[0105] The RetinaNet model consists of two parts: feature extraction and detection. The feature extraction part can use but is not limited to ResNet, VGG, ResNeXt and other network structures, and will be combined with FPN in the future to improve accuracy. The detection part consists of several sub-networks, which correspond to different scale features in the feature extraction network. Its function is to regress the orientation of target cells of different scales and classify the cells at that position.

[0106] b. When training the RetinaNet model, the L1 loss value is used to calculate the orientation loss of cells in the image (the L1 loss value is calculated as: ); For the calculation of the category loss value of cells in the image, the Focal Loss calculation method is used (the calculation method of Focal Loss is: FL(p t )=-α t (1-p t ) γ log(p t )), because this calculation method can well solve the problems of complex samples and uneven sample classification;

[0107] c. When training the RetinaNet model, relevant (hyper) parameter selections may include:

[0108] (1) Model training uses stochastic gradient descent (SGD);

[0109] (2) Its learning rate is between 0.01 and 0.0001;

[0110] (3) Momentum parameter is between 0.9 and 0.99;

[0111] (4) Weight Decay is between 0.0001 and 0.00001;

[0112] (5) Use warmup learning rate adjustment during training, starting from 0 and linearly increasing to the initial learning rate;

[0113] (6) During training, after the warm-up learning rate is adjusted, the learning rate is adjusted using the cyclic cosine decreasing method to improve the training efficiency. The cycle period is between 10,000 and 100,000.

[0114] (7) The parameter Y in Focal Loss is 2 and α is 0.25.

[0115] 5. Forward reasoning of the composite detection network model:

[0116] a. After multiple RetinaNet models are trained, follow Figure 5 The structure of the composite detection network model is composed of the composite detection network model. The output of the composite detection network model and its RetinaNet model is the classification of the target cells and their corresponding positions. Each RetinaNet model corresponds to images of different source sizes. The example uses three capture sizes of 512x512, 1024x1024, and 2048x2048, and uniformly scales them to images of size 512x512. They are then input into the corresponding sub-models for forward reasoning, and the forward reasoning results of the complete slices corresponding to the respective sizes are obtained. All the results with a confidence level lower than 0.1 are filtered out and not retained, and the remaining cell candidate boxes are subjected to intra-class non-maximum suppression processing;

[0117] Among them, the non-maximum suppression is described as follows:

[0118] i. Sort the cell candidate boxes of each category from large to small according to their confidence.

[0119] ii. Starting from the cell candidate box with the maximum confidence, calculate the overlapping area with the cell candidate boxes with smaller confidence.

[0120] iii. Delete the cell candidate boxes with a smaller confidence level whose overlapping area percentage is greater than a set threshold (0.3 to 0.5, for example, 0.5).

[0121] iv. Repeat the above steps until all the retained cell candidate boxes have undergone the above steps i-iii.

[0122] b. Description of the image of the sliding window cropping patch in the forward process:

[0123] The image size is cropped according to the size required by the sub-model. To ensure accuracy and prevent problem cells from being missed, the overlap size during sliding the window is half of the crop size to ensure that the cells are fully covered. For example, when using 512x512, the horizontal overlap is 256 and the vertical overlap is 256.

[0124] In the above application example, the computer device is based on a composite detection network model composed of multiple sub-detection network models connected in parallel, and intercepts and scales the cell image to be processed to a unified target image size according to the size source corresponding to each sub-detection network model, and then inputs the obtained cell scaled image into the corresponding sub-detection network model for forward reasoning and merges the various forward reasoning results to obtain the cell detection result of the cell image to be processed, thereby improving the detection accuracy of the composite detection network model for the cell image.

[0125] It should be understood that although Figure 2-Figure 5 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-Figure 5 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0126] In one embodiment, Figure 6 As shown, a cell image processing device is provided, comprising:

[0127] The model acquisition module 601 is used to acquire a composite detection network model formed by combining a plurality of pre-built sub-detection network models in parallel; the plurality of sub-detection network models correspond to images of sources of different sizes;

[0128] An image capture module 602 is used to capture the cell images to be processed according to different size sources to obtain cell capture images of corresponding size sources;

[0129] An image scaling module 603 is used to scale cell intercepted images corresponding to sources of different sizes to cell scaled images corresponding to target image sizes, thereby obtaining multiple cell scaled images;

[0130] A forward reasoning module 604 is used to input multiple cell zoom images into the composite detection network model, so that the composite detection network model inputs the multiple cell zoom images into each sub-detection network model for forward reasoning, and merges the forward reasoning results output by each sub-detection network model to obtain the cell detection result of the cell image to be processed;

[0131] The detection result acquisition module 605 is used to obtain the cell detection result obtained by the composite detection network model.

[0132] In one embodiment, the sub-detection network model includes a cascaded feature extraction network model and a feature detection network model; wherein the feature extraction network model is used to perform feature extraction processing on the cell zoom image to obtain cell features of the cell zoom image at different scales; the feature detection network model is used to perform cell classification and orientation regression on the cell zoom image at each scale based on the cell features of the cell zoom image at each scale through each sub-feature detection network model to obtain a forward reasoning result.

[0133] In one embodiment, the above-mentioned device is also used to determine, based on the cell detection results, a cell candidate frame in the cell image for selecting cells belonging to each cell category; obtain the overlapping area between the cell candidate frame with the maximum confidence in each cell category and other cell candidate frames in the same cell category; and determine whether to retain other cell candidate frames based on the overlapping area.

[0134] In one embodiment, the cell candidate box has a confidence level greater than or equal to a preset confidence threshold.

[0135] In one embodiment, the image capture module 602 is further used to use half of each size source as the sliding distance of the window; slide the window horizontally and vertically along the cell image according to the sliding distance; and use the image corresponding to the sliding window on the cell image as the cell capture image.

[0136] In one embodiment, the above-mentioned device is also used to calculate the loss value of the cell position in the training sample cell image using the minimization of absolute error loss function when using the training sample cell image to perform model training on each sub-detection network model, and to calculate the loss value of the cell category in the training sample cell image using the focus loss function.

[0137] In one embodiment, the above-mentioned device is also used to adjust the learning rate of each sub-detection network model by using cyclic cosine decrement after warming up the learning rate of each sub-detection network model when performing model training on each sub-detection network model.

[0138] For the specific definition of the cell image processing device, please refer to the definition of the cell image processing method above, which will not be repeated here. Each module in the above-mentioned cell image processing device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0139] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above-mentioned various method embodiments when executing the computer program.

[0140] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0141] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0142] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0143] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A cell image processing method, characterized in that: include: Acquire a composite detection network model formed by combining a plurality of pre-built sub-detection network models in parallel; the plurality of sub-detection network models correspond to images of sources of different sizes; Intercepting the cell images to be processed according to the different size sources to obtain cell intercepted images of the corresponding size sources; Scaling the cell intercepted images corresponding to the sources of different sizes to cell scaled images corresponding to the target image size, to obtain a plurality of cell scaled images; the sizes of the plurality of cell scaled images are all the target image size; Inputting the multiple cell zoom images into the composite detection network model, so that the composite detection network model inputs the multiple cell zoom images into each sub-detection network model for forward reasoning, and merging the forward reasoning results output by each sub-detection network model to obtain the cell detection result of the cell image to be processed; The cell detection result obtained by the composite detection network model is obtained.

2. The method according to claim 1, characterized in that The sub-detection network model includes a cascaded feature extraction network model and a feature detection network model; wherein, The feature extraction network model is used to perform feature extraction processing on the cell zoom image to obtain cell features of the cell zoom image at different scales; The feature detection network model is used to perform cell classification and orientation regression on the cell zoom image at each scale based on the cell features of the cell zoom image at each scale through each sub-feature detection network model to obtain the forward reasoning result.

3. The method according to claim 1, characterized in that After acquiring the cell detection result obtained by the composite detection network model, the method further includes: Determining, according to the cell detection result, a cell candidate frame in the cell image for selecting cells belonging to each cell category; Obtaining the overlapping area between the cell candidate box with the maximum confidence in each cell category and other cell candidate boxes in the same cell category; Whether to retain the other cell candidate frames is determined according to the overlapping area.

4. The method according to claim 3, characterized in that The cell candidate box has a confidence level greater than or equal to a preset confidence threshold.

5. The method according to any one of claims 1 to 4, characterized in that: The method of intercepting the cell images to be processed according to the different size sources to obtain the cell intercepted images of the corresponding size sources includes: Take half of each size source as the sliding distance of the window; Sliding the window in the horizontal and vertical directions of the cell image according to the sliding distance; The image corresponding to the sliding window on the cell image is used as the cell captured image.

6. The method according to claim 1, characterized in that Before obtaining the composite detection network model formed by connecting a plurality of pre-built sub-detection network models in parallel, the method further includes: When using the training sample cell images to perform model training on each sub-detection network model, the loss value of the cell position in the training sample cell image is calculated using the minimization of absolute error loss function, and the loss value of the cell category in the training sample cell image is calculated using the focus loss function.

7. The method according to claim 1, characterized in that Before obtaining the composite detection network model formed by connecting a plurality of pre-built sub-detection network models in parallel, the method further includes: When performing model training on each sub-detection network model, each sub-detection network model is warmed up with a learning rate and then adjusted using cyclic cosine decrement.

8. A cell image processing device, characterized in that: The device comprises: A model acquisition module is used to acquire a composite detection network model formed by combining a plurality of pre-built sub-detection network models in parallel; the plurality of sub-detection network models correspond to images of sources of different sizes; An image capture module, used to capture the cell images to be processed according to the different size sources to obtain cell capture images of the corresponding size sources; An image scaling module, used for scaling the cell intercepted images corresponding to the sources of different sizes to cell scaled images corresponding to the target image size, to obtain a plurality of cell scaled images; the sizes of the plurality of cell scaled images are all the target image size; A forward reasoning module, used for inputting the multiple cell zoom images into the composite detection network model, so that the composite detection network model inputs the multiple cell zoom images into each sub-detection network model for forward reasoning, and combines the forward reasoning results output by each sub-detection network model to obtain the cell detection result of the cell image to be processed; The detection result acquisition module is used to obtain the cell detection result obtained by the composite detection network model.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images