A mitotic cell detection model training method, system, and readable storage medium

Abstract

The embodiment of the present application provides a mitotic cell detection model training method, system, and readable storage medium, the method comprising: expanding the mitotic cell region based on the initial image to be trained, and obtaining a corresponding target expanded image; inputting the target expanded image into the initial detection model for model training, and, during the training process, using the frame energy density to correct the loss function; wherein, when the training end condition is triggered, the target detection model is obtained. The implementation of this method can reduce the difficulty of mitotic cell detection.

Description

Technical Field

[0001] The present application relates to the technical field of pathological image processing, and in particular to a mitotic cell detection model training method, system, and readable storage medium. Background Art

[0002] Breast cancer is the second most common cancer in the world. In the early stage of the disease, it often manifests as breast lumps, nipple discharge, axillary lymph node enlargement and other symptoms. In the late stage, cancer cells may metastasize to distant organs and cause multi-organ lesions, which directly threaten the patient's life. With the improvement of medical level, breast cancer has become one of the most effective solid tumors. Therefore, early diagnosis of breast cancer helps to improve the survival rate of patients.

[0003] At present, the Nottingham grading system (NGS) has been adopted internationally to grade breast cancer. NGS consists of three morphological characteristics, including nuclear pleomorphism, tubule formation, and mitotic index. Among them, the statistics of the mitotic index require the detection and counting of mitotic cells, but because mitotic cells and non-mitotic cells are very similar in appearance, it is difficult to distinguish between the two, making analysis based on breast cancer microscope images a challenging task in clinical practice. On the other hand, a single slice generally contains hundreds of thousands of stained cells of various types, and mitotic cells are usually hidden among these dense cells, which further aggravates the difficulty of detecting mitotic cells. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a mitotic cell detection model training method, system, and readable storage medium, which can reduce the difficulty of detecting mitotic cells.

[0005] The present application also provides a method for training a mitotic cell detection model, comprising the following steps:

[0006] Based on the initial image to be trained, the mitotic cell region is expanded, and the corresponding target expanded image is obtained;

[0007] The target expanded image is input into the initial detection model for model training, and during the training process, the frame energy density is used to correct the loss function; wherein, when the training end condition is triggered, the target detection model is obtained.

[0008] In a second aspect, an embodiment of the present application further provides a mitotic cell detection model training system, the system comprising a region expansion module and a model training module, wherein:

[0009] The region expansion module is used to expand the mitotic cell region based on the initial image to be trained and obtain a corresponding target expanded image;

[0010] The model training module is used to input the target expanded image into the initial detection model for model training, and during the training process, the frame energy density is used to correct the loss function; wherein, when the training end condition is triggered, the target detection model is obtained.

[0011] In a third aspect, an embodiment of the present application further provides a readable storage medium, wherein the readable storage medium includes a mitotic cell detection model training method program. When the mitotic cell detection model training method program is executed by a processor, the steps of a mitotic cell detection model training method as described in any one of the above items are implemented.

[0012] As can be seen from the above, the mitotic cell detection model training method, system and readable storage medium provided in the embodiments of the present application expand the mitotic cell region before training the detection model, and avoid the reduction of model detection accuracy due to insufficient sampling information by expanding the detection range of the mitotic cell region. During the model training process, the frame energy density is used to correct the loss function, so that the model can be trained in the correct direction on the training set, reducing the difficulty of detecting mitotic cells.

[0013] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the embodiments of the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0015] Figure 1 A flow chart of a mitotic cell detection model training method provided in an embodiment of the present application.

[0016] Figure 2 Schematic diagram for expanding mitotic cells.

[0017] Figure 3 Schematic diagram of the density change of the proposed boxes during model training.

[0018] Figure 4 A structural schematic diagram of a mitotic cell detection model training system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0020] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0021] Please refer to Figure 1 , Figure 1 : is a flow chart of a mitotic cell detection model training method in some embodiments of the present application. The method comprises the following steps:

[0022] Step S100, based on the initial image to be trained, the mitotic cell region is expanded, and a corresponding target expanded image is obtained.

[0023] Specifically, since mitotic cells account for a small proportion and are also small in size, this will lead to insufficient sampling information. To solve the above problem, in the embodiment of the present application, mitotic cells are expanded by cell sample expansion technology; Figure 2 ,The cell sample expansion technology is mainly divided into three steps: (1) Copy the mitotic cell region from the original image. Figure 2 The area selected in the first picture on the left is the mitotic cell area mentioned above; (2) The cells are extracted using the region growing method, that is, the cell growth area is cut out from the extracted mitotic cell area. This area can be further referenced to the area located at Figure 2 (3) Copy the cut-out cell growth area to the area in the original image where there are no mitotic cells, and obtain the corresponding cell expansion data. It should be noted that the subsequent model training is mainly based on the cell expansion data.

[0024] Step S200, input the target expanded image into the initial detection model for model training, and during the training process, use the frame energy density to correct the loss function; wherein, when the training end condition is triggered, the target detection model is obtained.

[0025] Specifically, since the characteristics of mitotic cells are not obvious, in order to enable the detection model to fully learn the fine-grained characteristics of mitotic cells, the energy density training technology is used for model training in the current embodiment. The basis of the energy density training technology comes from experimental observations, that is, during the training period, the density difference between the predictions of mitotic cells and non-dividing cells becomes more and more significant (for details, please refer to Figure 3 ). Based on this experimental observation, in the current embodiment, it is proposed to use this density information to predict mitotic cells. Specifically, the more predictions for a cell, the more likely the cell is to be dividing, and these predictions should receive a smaller penalty. This can encourage the learning process of mitotic cell features.

[0026] As can be seen from the above, the mitotic cell detection model training method provided in the embodiment of the present application expands the mitotic cell region before training the detection model. By expanding the detection range of the mitotic cell region, the reduction in model detection accuracy due to insufficient sampling information is avoided. During the model training process, the frame energy density is used to correct the loss function, so that the model can be trained in the correct direction on the training set, reducing the difficulty of detecting mitotic cells.

[0027] In one embodiment, in step S100, based on the initial image to be trained, the mitotic cell region is expanded and a corresponding target expanded image is obtained, including: extracting the mitotic cell region from the initial image according to a preset extraction range; extracting the cell growth region from the mitotic cell region based on the region growing method; and copying the cell growth region to a target region in the initial image where no mitotic cells exist, so as to convert the initial image into a target expanded image.

[0028] Specifically, the extraction range includes a 30×30 pixel range; the preset neighborhood range includes an 8-domain range; and the growth criterion for determining the cell growth area includes that the difference between the grayscale value of the test point and the grayscale value of the regional growth point is less than 10.

[0029] It should be noted that when copying the cell growth area, the area can be randomly copied 0 to 10 times. Then, after each copy is rotated and transformed, the transformed copy is randomly pasted into the area where there are no mitotic cells in the initial image. In this way, the expansion based on mitotic cells can be avoided to avoid the reduction of model detection accuracy due to insufficient sampling information.

[0030] In one embodiment, step S102, based on the region growing method, extracts the cell growth region from the mitotic cell region, including: determining the regional growth point based on the image center of the initial image, and taking each pixel point within a preset neighborhood around the regional growth point as a point to be tested; based on the difference between the grayscale value of the point to be tested and the grayscale value of the regional growth point, determining and extracting the corresponding cell growth region.

[0031] Specifically, in the current embodiment, the image center of the initial image is taken as the regional growth point f(i, j), and each pixel point within the region of the regional growth point 8 is taken as a point to be tested. Then, based on the growth criterion determined by the difference between the gray value of the point to be tested and the gray value of the regional growth point, the cell growth region is determined.

[0032] It should be noted that the basic idea of ​​region growing is to group pixels with similar properties to form a region, and determine the cell growth region based on the region. Specifically, the method first finds a seed pixel for each region to be segmented, uses the seed pixel as the starting point for growth (i.e., the region growth point), and then merges the seed pixel and the pixels in the surrounding neighborhood that have the same or similar properties as the seed pixel (determined according to some predetermined growth or similarity criteria) into the region where the seed pixel is located. These new pixels are treated as new seeds and the above process is continued until no pixels that meet the conditions can be included. In this way, a region is grown.

[0033] In the above embodiment, the extraction of cell growth areas is performed based on the region growing method, and connected areas with the same characteristics can be segmented out. Moreover, the implementation concept of this method is simple, and only a few seed points are required to complete it. It can also provide good boundary information and segmentation results, thereby improving the extraction efficiency of cell growth areas.

[0034] In one embodiment, in step 200, during the training process, the loss function is corrected using the box energy density, including: determining a suggestion box for distinguishing the background from the detection object from the target expanded image; and calculating the average distance Density of a suggestion box by the following formula (1):

[0035]

[0036] In formula (1), D(b i ,b j ) indicates suggestion box b i With suggestion box j The Manhattan distance between the center points of , N represents the number of all suggestion boxes; based on the average distance Density of a suggestion box, the density energy Energy(b) of each suggestion box b is calculated by the following formula:

[0037]

[0038] Based on the density energy Energy(b), the loss function is corrected.

[0039] In the current embodiment, the average distance between each suggestion box is used as a measure of density, which shows that denser suggestion boxes have smaller average distances than independent suggestion boxes. Correcting the loss function based on the box energy density can train the network in the right direction on the training set. In fact, the correction of the loss function by means of box density energy is to mine the correct mitotic cell suggestion box from the image background, where it can be understood that objects surrounded by dense suggestion boxes in the image background are classified as mitotic cells.

[0040] In the current embodiment, box energy density is used to correct the loss function during model training. Because dense suggestion boxes are assigned a smaller weight when calculating the loss, these suggestion boxes have a greater chance of being predicted as areas containing mitotic cells, thereby further reducing the difficulty of detecting mitotic cells.

[0041] In one embodiment, based on the density energy Energy(b), the loss function is corrected, including: based on the density energy Energy(b), correcting the classification loss by the following formula:

[0042]

[0043] In formula (3), p is the prediction confidence, u is the annotation, k is the regression box index, and u k is the label of the kth regression box, 1 is the indicator function, If the label is background, its value is 1. If it is marked as foreground, it uses density energy, p k is the confidence of the kth regression box; based on the density energy Energy(b), the bounding box regression loss is corrected by the following formula:

[0044]

[0045] In formula (4), v k Mark the position of the kth regression box, To smooth the L1 loss, L1 is the L1 loss, b is the prediction confidence, and v is the label; combining formula (3) and formula (4), the total loss will be adjusted to:

[0046]

[0047] In formula (5), LBDE (p,u,b,v) is the total loss after correction.

[0048] In one embodiment, during the model training process, the method further includes: maintaining the variance of the gradients in all layers at the same level to avoid the loss gradient from disappearing or exploding due to improper initialization, wherein the activation of the target neuron in the first layer The gradient of is given by:

[0049]

[0050] In formula (6), the indicator function is the derivative from the rectified linear unit, ε represents the loss, represents a specific feature map, represents the convolution kernel weight, represents the convolution kernel, c represents the channel index, x represents the feature map, and q represents the group element index.

[0051] Please refer to Figure 4 As shown, the embodiment of the present application further provides a mitotic cell detection model training system 400, the system 400 includes a region expansion module 401 and a model training module 402, wherein:

[0052] The region expansion module 401 is used to expand the mitotic cell region based on the initial image to be trained, and obtain a corresponding target expanded image.

[0053] The model training module 402 is used to input the target expanded image into the initial detection model for model training, and during the training process, the frame energy density is used to correct the loss function; wherein, when the training end condition is triggered, the target detection model is obtained.

[0054] In one embodiment, the region expansion module 401 is further used to extract a mitotic cell region from the initial image according to a preset extraction range; extract a cell growth region from the mitotic cell region based on a region growing method; and copy the cell growth region to a target region in the initial image where no mitotic cells exist, so as to convert the initial image into a target expanded image.

[0055] In one embodiment, the region expansion module 401 is also used to determine the region growth point based on the image center of the initial image, and use each pixel point within a preset neighborhood around the region growth point as a test point; based on the difference between the grayscale value of the test point and the grayscale value of the region growth point, determine and extract the corresponding cell growth area.

[0056] In one embodiment, the extraction range includes a 30×30 pixel range; the preset neighborhood range includes an 8-domain range; and the growth criterion for determining the cell growth area includes that the difference between the grayscale value of the test point and the grayscale value of the regional growth point is less than 10.

[0057] In one embodiment, the model training module 402 is further used to determine a suggestion box for distinguishing the background from the detection object in the target expansion image; and calculate the average distance Density of a suggestion box by the following formula (1):

[0058]

[0059] In formula (1), D(b i ,b j ) indicates suggestion box b i With suggestion box j The Manhattan distance between the center points of , N represents the number of all suggestion boxes; based on the average distance Density of a suggestion box, the density energy Energy(b) of each suggestion box b is calculated by the following formula:

[0060]

[0061] Based on the density energy Energy(b), the loss function is corrected.

[0062] In one embodiment, the model training module 402 is further configured to correct the classification loss based on the density energy Energy(b) by the following formula:

[0063]

[0064] In formula (3), p is the prediction confidence, u is the annotation, k is the regression box index, and u k is the label of the kth regression box, 1 is the indicator function, If the label is background, its value is 1. If it is marked as foreground, it uses density energy, p k is the confidence of the kth regression box; based on the density energy Energy(b), the bounding box regression loss is corrected by the following formula:

[0065]

[0066] In formula (4), v k Mark the position of the kth regression box, To smooth the L1 loss, L1 is the L1 loss, b is the prediction confidence, and v is the label; combining formula (3) and formula (4), the total loss will be adjusted to:

[0067]

[0068] In formula (5), L BDE (p,u,b,v) is the total loss after correction.

[0069] In one embodiment, the model training module 402 is further configured to maintain the variance of the gradients in all layers at the same level to avoid the gradient vanishing or exploding due to improper initialization, wherein the activation of the target neuron in the first layer The gradient of is given by:

[0070]

[0071] In formula (6), the indicator function is the derivative from the rectified linear unit, ε represents the loss, represents a specific feature map, represents the convolution kernel weight, represents the convolution kernel, c represents the channel index, x represents the feature map, and q represents the group element index.

[0072] As can be seen from the above, the mitotic cell detection model training system provided in the embodiment of the present application expands the mitotic cell region before training the detection model. By expanding the detection range of the mitotic cell region, the reduction in model detection accuracy due to insufficient sampling information is avoided. During the model training process, the frame energy density is used to correct the loss function, so that the model can be trained in the correct direction on the training set, reducing the difficulty of detecting mitotic cells.

[0073] The embodiment of the present application provides a storage medium, and when the computer program is executed by the processor, the method in any optional implementation of the above embodiment is executed. Wherein, the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, referred to as EPROM), programmable read-only memory (Programmable Red-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic memory, flash memory, disk or optical disk.

[0074] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0075] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0076] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0077] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0078] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A mitotic cell detection model training method, characterized in that: The following steps are involved: Based on the initial image to be trained, the mitotic cell region is expanded, and the corresponding target expanded image is obtained; The target expanded image is input into the initial detection model for model training, and during the training process, the loss function is corrected using the frame energy density; wherein, when the training end condition is triggered, the target detection model is obtained; The method of expanding the mitotic cell region based on the initial image to be trained and obtaining a corresponding target expanded image includes: Extracting a mitotic cell region from the initial image according to a preset extraction range; Extracting a cell growth region from the mitotic cell region based on a region growing method; Copying the cell growth area to a target area in the initial image where no mitotic cells exist, so as to convert the initial image into a target expanded image; The method of extracting a cell growth region from the mitotic cell region based on the region growing method comprises: Determine a regional growth point based on the image center of the initial image, and use each pixel point within a preset neighborhood around the regional growth point as a point to be measured; Determine and extract the corresponding cell growth area based on the difference between the gray value of the point to be measured and the gray value of the regional growth point; The extraction range includes a 30×30 pixel range; The preset neighborhood range includes 8 domain ranges; The growth criterion for determining the cell growth region includes that the difference between the gray value of the test point and the gray value of the regional growth point is less than 10; During the training process, the frame energy density is used to correct the loss function, including: Determine a suggestion box for distinguishing a background from a detection object from the target expanded image; The average distance Density of a suggestion box is calculated using the following formula (1): In formula (1), D(b i ,b j ) indicates suggestion box b i With suggestion box j The Manhattan distance between the center points of , N represents the number of all suggestion boxes; Based on the average distance Density of the suggestion box, the density energy Energy(b) of each suggestion box b is calculated by the following formula: Based on the density energy Energy(b), correcting the loss function; The correction of the loss function based on the density energy Energy(b) includes: Based on the density energy Energy(b), the classification loss is corrected by the following formula: In formula (3), p is the prediction confidence, u is the annotation, k is the regression box index, and u k is the label of the kth regression box, 1 is the indicator function, If the label is background, its value is 1. If it is marked as foreground, it uses density energy, p k is the confidence of the kth regression box; Based on the density energy Energy(b), the bounding box regression loss is corrected by the following formula: In formula (4), v k Mark the position of the kth regression box, is the smoothed L1 loss, L1 is the L1 loss, b is the prediction confidence, and v is the annotation; Combining formula (3) and formula (4), the total loss will be adjusted to: In formula (5), L BDE (p,u,b,v) is the total loss after correction; During the model training process, the method further includes: The variance of the gradients in all layers is maintained at the same level to avoid the vanishing or exploding gradients of the loss due to improper initialization, where the activation of the target neuron in layer I is The gradient of is given by: In formula (6), the indicator function is the derivative from the rectified linear unit, ε represents the loss, represents a specific feature map, represents the convolution kernel weight, represents the convolution kernel, c represents the channel index, x represents the feature map, and q represents the group element index.

2. A mitotic cell detection model training system, characterized in that: The system includes a region expansion module and a model training module, wherein: The region expansion module is used to expand the mitotic cell region based on the initial image to be trained and obtain a corresponding target expanded image; The model training module is used to input the target expanded image into the initial detection model for model training, and during the training process, the frame energy density is used to correct the loss function; wherein, when the training end condition is triggered, the target detection model is obtained; The region expansion module is also used to extract a mitotic cell region from the initial image according to a preset extraction range; extract a cell growth region from the mitotic cell region based on a region growing method; copy the cell growth region to a target region in the initial image where no mitotic cells exist, so as to convert the initial image into a target expanded image; The method of extracting a cell growth region from the mitotic cell region based on the region growing method comprises: Determine a regional growth point based on the image center of the initial image, and use each pixel point within a preset neighborhood around the regional growth point as a point to be measured; Determine and extract the corresponding cell growth area based on the difference between the gray value of the point to be measured and the gray value of the regional growth point; The extraction range includes a 30×30 pixel range; The preset neighborhood range includes 8 domain ranges; The growth criterion for determining the cell growth region includes that the difference between the gray value of the test point and the gray value of the regional growth point is less than 10; During the training process, the frame energy density is used to correct the loss function, including: Determine a suggestion box for distinguishing a background from a detection object from the target expanded image; The average distance Density of a suggestion box is calculated using the following formula (1): In formula (1), D(b i ,b j ) indicates suggestion box b i With suggestion box j The Manhattan distance between the center points of , N represents the number of all suggestion boxes; Based on the average distance Density of the suggestion box, the density energy Energy(b) of each suggestion box b is calculated by the following formula: Based on the density energy Energy(b), correcting the loss function; The correction of the loss function based on the density energy Energy(b) includes: Based on the density energy Energy(b), the classification loss is corrected by the following formula: In formula (3), p is the prediction confidence, u is the annotation, k is the regression box index, and u k is the label of the kth regression box, 1 is the indicator function, If the label is background, its value is 1. If it is marked as foreground, it uses density energy, p k is the confidence of the kth regression box; Based on the density energy Energy(b), the bounding box regression loss is corrected by the following formula: In formula (4), v k Mark the position of the kth regression box, is the smoothed L1 loss, L1 is the L1 loss, b is the prediction confidence, and v is the annotation; Combining formula (3) and formula (4), the total loss will be adjusted to: In formula (5), L BDE (p,u,b,v) is the total loss after correction; The model training process also includes: The variance of the gradients in all layers is maintained at the same level to avoid the vanishing or exploding gradients of the loss due to improper initialization, where the activation of the target neuron in layer I is The gradient of is given by: In formula (6), the indicator function is the derivative from the rectified linear unit, ε represents the loss, represents a specific feature map, represents the convolution kernel weight, represents the convolution kernel, c represents the channel index, x represents the feature map, and q represents the group element index.

3. A readable storage medium, characterized in that: The readable storage medium includes a mitotic cell detection model training method program, and when the mitotic cell detection model training method program is executed by a processor, the steps of the method according to claim 1 are implemented.

Images