Three-level lymphatic structure identification method, device and equipment based on pathological section image

Abstract

The invention discloses a three-level lymphatic structure identification method and device based on a pathological section image, electronic equipment and a computer readable storage medium, and the method comprises the steps: obtaining and marking a pathological section image sample set which comprises pathological section image sample subsets of different development stages of a three-level lymphatic structure; constructing an initial three-level lymphatic structure recognition model, and training the recognition model by using the pathological section image sample set to obtain a three-level lymphatic structure recognition model; processing the to-be-recognized pathological section image by using the three-stage lymph structure recognition model to obtain a three-stage lymph structure recognition result; the three-level lymphatic structure identification result comprises a three-level lymphatic structure development stage and three-level lymphatic structure positioning information; and generating a three-level lymph structure mask pattern corresponding to the pathological section image to be recognized according to the three-level lymph structure recognition result, and outputting the three-level lymph structure mask pattern. According to the technical scheme, the development process of the three-stage lymph structure is comprehensively analyzed, so that accurate identification of different development stages of the three-stage lymph structure is realized.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a three-level lymphatic structure recognition method based on pathological section images, and also to a three-level lymphatic structure recognition device, electronic device and computer-readable storage medium based on pathological section images. Background Art

[0002] Among related technologies, identifying tertiary lymphoid structures using pathological slide images is a key component of pathological analysis and diagnosis. In-depth research has revealed that tertiary lymphoid structures (TLS) can be divided into early or immature TLS (eTLS), primary follicle-like TLS (pfl-TLS), and secondary follicle-like TLS (sfl-TLS). Sfl-TLS possess mature germinal centers and are generally considered to be typical mature TLS with immune function. However, current research on automated TLS identification has primarily focused on sfl-TLS, with less attention paid to identifying TLS at its complete developmental stage. However, early and primary follicle-like TLS also possess significant clinical significance.

[0003] Therefore, how to accurately identify the different developmental stages of the tertiary lymphoid structures is an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of this application is to provide a three-level lymphatic structure recognition method based on pathological section images, which realizes the accurate recognition of three-level lymphatic structures at different developmental stages; another purpose of this application is to provide a three-level lymphatic structure recognition device, electronic device and computer-readable storage medium based on pathological section images, all of which have the above-mentioned beneficial effects.

[0005] In a first aspect, the present application discloses a three-level lymphatic structure recognition method based on pathological slice images, comprising:

[0006] Acquire and annotate a pathology section image sample set; the pathology section image sample set includes a subset of pathology section image samples at different developmental stages of the tertiary lymphatic structure;

[0007] Constructing an initial three-level lymphatic structure recognition model, and training the initial three-level lymphatic structure recognition model using the pathological section image sample set to obtain a three-level lymphatic structure recognition model;

[0008] The pathological section image to be identified is processed using the three-level lymphatic structure recognition model to obtain a three-level lymphatic structure recognition result; the three-level lymphatic structure recognition result includes the three-level lymphatic structure development stage and the three-level lymphatic structure location information;

[0009] A three-level lymphatic structure mask image corresponding to the pathological section image to be identified is generated according to the three-level lymphatic structure recognition result, and the three-level lymphatic structure mask image is output.

[0010] Optionally, the initial three-level lymphatic structure recognition model is trained using the pathological section image sample set to obtain a three-level lymphatic structure recognition model, including:

[0011] For each pathological slice image in the pathological slice image sample set, determining pathological tissue annotation information in the pathological slice image;

[0012] Performing adaptive size cropping processing on the pathological section image according to the pathological tissue annotation information to obtain cropped images;

[0013] For each cropped image, determining pathological tissue coordinate information in the cropped image;

[0014] Generate an image file corresponding to the pathological slice image sample set according to each of the cropped images, and generate a coordinate file corresponding to the pathological slice image sample set according to each of the pathological tissue coordinate information;

[0015] The initial three-level lymphatic structure recognition model is trained using the image file and the coordinate file to obtain the three-level lymphatic structure recognition model.

[0016] Optionally, performing adaptive size cropping processing on the pathological section image according to the pathological tissue annotation information to obtain cropped images includes:

[0017] Determining a pathological tissue annotation point set in the pathological slice image according to the pathological tissue annotation information;

[0018] Determining the cutting size according to the pathological tissue annotation point set;

[0019] The pathological slice image is subjected to sliding segmentation according to the cropping size and a preset image overlap ratio to obtain each of the cropped images.

[0020] Optionally, determining the cropping size according to the pathological tissue annotation point set includes:

[0021] Determining the maximum pixel abscissa value and the minimum pixel abscissa value, and the maximum pixel ordinate value and the minimum pixel ordinate value in the pathological tissue annotation point set;

[0022] Determine a horizontal coordinate difference according to the maximum pixel horizontal coordinate value and the minimum pixel horizontal coordinate value;

[0023] Determine a vertical coordinate difference value according to the maximum pixel vertical coordinate value and the minimum pixel vertical coordinate value;

[0024] When the horizontal coordinate difference is greater than the vertical coordinate difference, determining a size greater than the horizontal coordinate difference as the cutting size;

[0025] When the horizontal coordinate difference is not greater than the vertical coordinate difference, a size greater than the vertical coordinate difference is determined as the cutting size.

[0026] Optionally, using the three-level lymphatic structure recognition model to process the pathological section image to be recognized to obtain a three-level lymphatic structure recognition result includes:

[0027] Segmenting the pathological section image to be identified to obtain segmented images;

[0028] For each segmented image, extract features of the segmented image to obtain target features;

[0029] Determining a target feature bounding box in the segmented image according to the target feature;

[0030] When the confidence level of the target feature bounding box exceeds a preset threshold, determining the target feature bounding box as a tertiary lymphatic structure bounding box;

[0031] Performing category probability prediction on the tertiary lymphatic structure bounding box to determine the developmental stage of the tertiary lymphatic structure corresponding to the segmented image;

[0032] Using the position information of the third-level lymphatic structure boundary box as the third-level lymphatic structure positioning information corresponding to the segmented image;

[0033] The tertiary lymphatic structure recognition result of the pathological section image to be recognized is determined according to the tertiary lymphatic structure development stage and tertiary lymphatic structure positioning information corresponding to each of the segmented images.

[0034] Optionally, after the pathological section image to be identified is processed using the three-level lymphatic structure identification model to obtain the three-level lymphatic structure identification result, the method further includes:

[0035] Determining the pixel area of ​​the tertiary lymphatic structure according to the tertiary lymphatic structure recognition result;

[0036] The actual tertiary lymphatic structure area of ​​the pathological tissue is calculated based on the tertiary lymphatic structure pixel area and the system pixel scale.

[0037] Optionally, the pathological slice image sample set further includes a tumor slice image sample subset of the pathological tissue.

[0038] In a second aspect, the present application further discloses a three-level lymphatic structure recognition device based on pathological slice images, comprising:

[0039] An acquisition module is used to acquire and annotate a pathological section image sample set; the pathological section image sample set includes a subset of pathological section image samples at different developmental stages of the tertiary lymphatic structure;

[0040] a training module, configured to construct an initial three-level lymphatic structure recognition model, and train the initial three-level lymphatic structure recognition model using the pathological section image sample set to obtain a three-level lymphatic structure recognition model;

[0041] a processing module, configured to process the pathological section image to be identified using the three-level lymphatic structure recognition model to obtain a three-level lymphatic structure recognition result; the three-level lymphatic structure recognition result includes the three-level lymphatic structure development stage and the three-level lymphatic structure location information;

[0042] A generating module is used to generate a three-level lymphatic structure mask image corresponding to the pathological section image to be identified according to the three-level lymphatic structure identification result, and output the three-level lymphatic structure mask image.

[0043] In a third aspect, the present application further discloses an electronic device, comprising:

[0044] memory for storing computer programs;

[0045] A processor is configured to implement the steps of any one of the above-mentioned three-level lymphatic structure recognition methods based on pathological section images when executing the computer program.

[0046] In a fourth aspect, the present application also discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the three-level lymphatic structure recognition methods based on pathological section images as described above are implemented.

[0047] The present application provides a three-level lymphatic structure recognition method based on pathological section images, comprising: obtaining and labeling a pathological section image sample set; the pathological section image sample set includes a subset of pathological section image samples at different developmental stages of the three-level lymphatic structure; constructing an initial three-level lymphatic structure recognition model, and using the pathological section image sample set to train the initial three-level lymphatic structure recognition model to obtain a three-level lymphatic structure recognition model; using the three-level lymphatic structure recognition model to process the pathological section image to be recognized to obtain a three-level lymphatic structure recognition result; the three-level lymphatic structure recognition result includes the three-level lymphatic structure developmental stage and the three-level lymphatic structure positioning information; generating a three-level lymphatic structure mask map corresponding to the pathological section image to be recognized based on the three-level lymphatic structure recognition result, and outputting the three-level lymphatic structure mask map.

[0048] By applying the technical solution provided in the present application, a fixed model training is performed using a pathological section image sample set including a subset of pathological section image samples at different developmental stages of the tertiary lymphatic structure to obtain a tertiary lymphatic structure recognition model, so that the tertiary lymphatic structure recognition model can realize the recognition of tertiary lymphatic structures at different developmental stages, obtain the tertiary lymphatic structure developmental stage and tertiary lymphatic structure positioning information of the pathological tissue, and generate a tertiary lymphatic structure mask map for medical personnel to perform pathological analysis. It can be seen that the present technical solution realizes the accurate recognition of tertiary lymphatic structures at different developmental stages by comprehensively analyzing the development process of the tertiary lymphatic structure.

[0049] The three-level lymphatic structure identification device, electronic device and computer-readable storage medium based on pathological section images provided in this application also have the above-mentioned technical effects, and this application will not elaborate on them here. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the prior art and the embodiments of the present application, the following is a brief introduction to the drawings required for describing the prior art and the embodiments of the present application. Of course, the drawings described below in connection with the embodiments of the present application are only part of the embodiments of the present application. For those skilled in the art, other drawings can be obtained based on the provided drawings without inventive effort, and the obtained other drawings also fall within the scope of protection of the present application.

[0051] Figure 1 A schematic diagram of the process of a three-level lymphatic structure recognition method based on pathological section images provided in this application;

[0052] Figure 2 A three-level lymphatic structure mask provided by this application;

[0053] Figure 3This is a schematic structural diagram of a three-level lymphatic structure recognition device based on pathological section images provided by this application;

[0054] Figure 4 This is a schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0055] The core of this application is to provide a three-level lymphatic structure recognition method based on pathological section images. This three-level lymphatic structure recognition method based on pathological section images realizes the accurate recognition of three-level lymphatic structures at different developmental stages; another core of this application is to provide a three-level lymphatic structure recognition device, electronic device and computer-readable storage medium based on pathological section images, all of which have the above-mentioned beneficial effects.

[0056] In order to describe the technical solutions in the embodiments of the present application more clearly and completely, the technical solutions in the embodiments of the present application will be introduced below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0057] The embodiment of the present application provides a three-level lymphatic structure recognition method based on pathological section images.

[0058] Please refer to Figure 1 , Figure 1 This is a flow chart of a three-level lymphatic structure recognition method based on pathological section images provided in the present application. The three-level lymphatic structure recognition method based on pathological section images may include the following S101 to S104.

[0059] S101: Acquire and annotate a pathology section image sample set; the pathology section image sample set includes a subset of pathology section image samples at different developmental stages of the tertiary lymphatic structure.

[0060] This step aims to achieve the acquisition of a pathological slice image sample set. It is understandable that the pathological slice image sample set is used to implement subsequent model training to train a three-level lymphatic structure recognition model (the three-level lymphatic structure recognition model described below), so that the three-level lymphatic structure recognition can be achieved based on the model. It should be pointed out that in order to achieve the recognition of three-level lymphatic structures at different developmental stages, the pathological slice image sample set can include a subset of pathological slice image samples at different developmental stages of the three-level lymphatic structure. In other words, the TLS slice image sample set can include an eTLS pathological slice image sample subset, a pfl-TLS pathological slice image sample subset, and an sfl-TLS pathological slice image sample subset.

[0061] In addition, the pathological slice image sample set may also include a tumor slice image sample subset of pathological tissue. Thus, the three-level lymphatic structure recognition model trained based on the pathological slice image sample set can simultaneously realize the recognition of three-level lymphatic structures at different developmental stages and tumor recognition.

[0062] Furthermore, each pathology slide image sample in the pathology slide image sample set may be obtained by preparing a standard HE (hematoxylin-eosin staining) stained section from a tissue specimen of a colorectal tumor patient, electronically scanning and saving it as a digital whole slide image (WSI), thereby obtaining a pathology slide image sample without tertiary lymphatic structure annotation. Furthermore, at least two pathologists annotate the WSI image with TLS and tumor regions at different developmental stages, maintaining consistent annotation information, to obtain an annotated pathology slide image sample. Thus, all pathology slide image samples are combined into a pathology slide image sample set.

[0063] In one embodiment of the present application, after obtaining the pathological slice image sample set, it may also include: for each pathological slice image in the pathological slice image sample set, performing an image enhancement operation on the pathological slice image; the image enhancement operation includes a combination of one or more of an image rotation operation, an image flip operation, an image smoothing operation, an image sharpening operation, and an image color conversion operation.

[0064] It is understandable that the image enhancement operation is intended to improve the visual effect of the image, which can not only improve the clarity of the image, but also highlight important features in the image, such as edges, contours, and contrast. Therefore, the image enhancement operation is more convenient for subsequent image processing and model training, which helps to improve the accuracy of model training and thus improve the application effect of the model. Among them, the specific image enhancement method used in the actual application scenario can be selected according to the actual situation, and this application does not limit this.

[0065] S102: constructing an initial three-level lymphatic structure recognition model, and training the initial three-level lymphatic structure recognition model using a pathological section image sample set to obtain a three-level lymphatic structure recognition model.

[0066] This step aims to implement model training based on the pathological section image sample set to obtain a three-level lymphatic structure recognition model. It is understandable that when the pathological section image sample set only includes a subset of pathological section image samples at different developmental stages of the three-level lymphatic structure, the three-level lymphatic structure recognition model trained based on the pathological section image sample set can be used to realize the recognition of the three-level lymphatic structure at different developmental stages; when the pathological section image sample set includes a subset of pathological section image samples at different developmental stages of the three-level lymphatic structure and a subset of tumor section image samples of pathological tissue, the three-level lymphatic structure recognition model trained based on the pathological section image sample set can be used to realize the recognition of the three-level lymphatic structure at different developmental stages and tumor recognition.

[0067] In one possible implementation, the initial tertiary lymphatic structure recognition model can be based on the YOLOv11-SEG (a segmentation network) architecture. Furthermore, the YOLOv11-SEG architecture can be used with the PyTorch framework to design a novel feature extraction network architecture that combines depthwise separable convolutions with residual connections. Depthwise separable convolutions significantly reduce computational effort by decomposing standard convolutions into depthwise and pointwise convolutions, while maintaining effective image feature extraction. Residual connections effectively address the vanishing gradient problem of deep neural networks during training, enabling the model to learn richer image features. Trained on a large amount of annotated data, the model can accurately identify the morphological features of tertiary lymphatic structures at different developmental stages and their boundaries, thereby achieving automated image segmentation. Furthermore, YOLOv11-SEG utilizes multi-scale feature fusion technology, enabling the model to maintain high-resolution segmentation of tertiary lymphatic structures of varying sizes. At the same time, the model can also adopt an attention mechanism to automatically focus on key areas, further improve the accuracy of segmentation, reduce the interference of background noise, and provide purer and more accurate image data for subsequent analysis.

[0068] During model training, the pathology slide image sample set can be divided into a training set: validation set: test set ratio of 7:3:1. The training set is used for model training, the validation set is used for model verification, and the test set is used to test the model's practical application effect. During training, default training parameters are pre-set and a blank model (the initial three-level lymphatic structure recognition model) is trained. By analyzing parameters and result graphs such as loss values, confusion matrices, F1 (the harmonic mean of precision and recall) curves, and PR curves (with recall on the horizontal axis and precision on the vertical axis), the model is continuously adjusted by adjusting the training parameters, adjusting the image ratio, and increasing the number of classified images, ultimately obtaining the optimal model (the three-level lymphatic structure recognition model).

[0069] In one embodiment of the present application, the initial three-level lymphatic structure recognition model is trained using a pathological section image sample set to obtain the three-level lymphatic structure recognition model, which may include:

[0070] For each pathological slice image in the pathological slice image sample set, determining pathological tissue annotation information in the pathological slice image;

[0071] Performing adaptive size cropping processing on the pathological section image according to the pathological tissue annotation information to obtain cropped images;

[0072] For each cropped image, determining the coordinate information of the pathological tissue in the cropped image;

[0073] Generate an image file corresponding to a pathological slice image sample set according to each cropped image, and generate a coordinate file corresponding to a pathological slice image sample set according to each pathological tissue coordinate information;

[0074] The initial three-level lymphatic structure recognition model is trained using the image file and the coordinate file to obtain the three-level lymphatic structure recognition model.

[0075] The embodiment of the present application provides a method for implementing a three-level lymphatic structure recognition model training based on a pathological section image sample set. It can be understood that a complete pathological section image contains a lot of complex feature information. In order to effectively improve the model training effect, the pathological section image can be cropped first, and the model training can be implemented with a cropped image of smaller specifications. Among them, for each cropped image, the pathological tissue coordinate information can be sampled and extracted in turn, and the image file and coordinate file can be generated respectively, so that the image file and coordinate file corresponding to the pathological section image sample set can be used to implement model training. It should be pointed out that digital pathological sections usually have a multi-resolution pyramid structure to adapt to different display and processing requirements. In order to ensure that the cropped image does not lose features, level 0 can be used for sampling. Here, level 0 refers to the highest resolution level of the digital pathological section image.

[0076] It should be pointed out that due to the different formats of pathological slice images, the corresponding methods of obtaining pathological tissue coordinate information will also be different. For example, for pathological slice images in the npdi file format, it is necessary to perform offset correction processing to achieve the acquisition of pathological tissue coordinate information; for pathological slice images in the Mrxs file format, it is not necessary to perform offset correction processing to directly achieve the acquisition of pathological tissue coordinate information. Furthermore, in order to facilitate subsequent data calculations and improve processing efficiency, the pathological tissue coordinate information in the coordinate file can also be normalized to convert it into a value between 0 and 1. The specific implementation method can be: the horizontal pixel coordinates in the pathological tissue coordinate information are divided by the horizontal pixel width; the vertical pixel coordinates in the pathological tissue coordinate information are divided by the vertical pixel height.

[0077] The pathological section image is cropped in an adaptive size according to the pathological tissue annotation information to obtain cropped images, which may include:

[0078] Determine a pathological tissue annotation point set in the pathological section image according to the pathological tissue annotation information;

[0079] Determine the cutting size based on the pathological tissue annotation point set;

[0080] The pathological slice image is segmented by sliding according to the cropping size and the preset image overlap ratio to obtain each cropped image.

[0081] The image cropping method provided in the embodiment of the present application can be implemented in combination with the sliding frame reasoning idea. First, the pathological tissue annotation point set in the pathological slice image, that is, the pathological tissue area in the pathological slice image, can be determined based on the pathological tissue annotation information; then, the cropping size can be determined based on the pathological tissue annotation point set, so that the pathological slice image can be slidably segmented in combination with the preset image overlap ratio to obtain various cropped images. It can be understood that in the embodiment of the present application, the sliding frame reasoning algorithm constructed based on the sliding frame reasoning idea can support adaptive size reasoning frame settings and overlap ratio settings, wherein the reasoning frame setting can modify the segmentation length of the digital pathological slice, and the overlap ratio is the overlap ratio of the reasoning frame image with the previous or previous image during the sliding process of the reasoning frame. Specifically, you can start sliding from the upper left corner of the image, first sliding horizontally, with each sliding step equal to "(1-overlapping rate) × segmentation length" to the right boundary of the image and stop. At this time, the inference box returns to x=0 (i.e., the upper left corner of the image), and then the inference box slides down "(1-overlapping rate) × segmentation length". At this time, the y value is fixed (i.e., the height is fixed before the next sliding action), and the translation and sliding process is repeated until the entire digital pathology slice is traversed. Among them, the model inference is performed each time the sliding box slides. Each time the target pixel coordinates in the cropped area are obtained, the coordinates are restored and converted into the coordinate dimension with the entire digital pathology slice as the coordinate system, and the data is saved. In addition, all the result data can be processed by the convex hull algorithm to merge repeated point sets to obtain accurate three-level lymphatic structure graded detection and segmentation results.

[0082] Furthermore, determining the cropping size based on the pathological tissue annotation point set may include:

[0083] In the pathological tissue annotation point set, determining the maximum pixel abscissa value and the minimum pixel abscissa value, the maximum pixel ordinate value and the minimum pixel ordinate value;

[0084] Determine the horizontal coordinate difference value according to the maximum pixel horizontal coordinate value and the minimum pixel horizontal coordinate value;

[0085] Determine the vertical coordinate difference value according to the maximum pixel vertical coordinate value and the minimum pixel vertical coordinate value;

[0086] When the horizontal coordinate difference is greater than the vertical coordinate difference, the size greater than the horizontal coordinate difference is determined as the cutting size;

[0087] When the difference in the horizontal coordinates is not greater than the difference in the vertical coordinates, a size greater than the difference in the vertical coordinates is determined as the cropping size.

[0088] Specifically, the cropping size can be: 1.2×max([max(points[0])-min(points[0])], [max(points[1])-min(points[1])]). Among them, max means taking the maximum value, min means taking the minimum value, points[0] represents the pixel horizontal coordinate value in the annotation point set, and points[1] represents the pixel vertical coordinate value in the annotation point set. It can be seen that max(points[0]) and min(points[0]) are the maximum pixel horizontal coordinate value and the minimum pixel horizontal coordinate value, max(points[1]) and min(points[1]) are the maximum pixel vertical coordinate value and the minimum pixel vertical coordinate value, respectively. [max(points[0])-min(points[0])] represents the horizontal coordinate difference between the maximum pixel horizontal coordinate value and the minimum pixel horizontal coordinate value, and [max(points[1])-min(points[1])] represents the vertical coordinate difference between the maximum pixel vertical coordinate value and the minimum pixel vertical coordinate value. In addition, in this embodiment, the adaptive size cropping method adopts 1.2 times sampling of the maximum bounding box, in order to prevent the samples in the training data from becoming small objects.

[0089] S103: Processing the pathological section image to be identified using the three-level lymphatic structure recognition model to obtain a three-level lymphatic structure recognition result; the three-level lymphatic structure recognition result includes the three-level lymphatic structure development stage and the three-level lymphatic structure location information.

[0090] This step aims to implement three-level lymphatic structure recognition processing based on the three-level lymphatic structure recognition model, and obtain the three-level lymphatic structure recognition results of the pathological slice image to be identified. Specifically, after obtaining the three-level lymphatic structure recognition model, it can be saved to a preset storage space. Upon receiving the pathological slice image to be identified, the three-level lymphatic structure recognition model can be retrieved from the preset storage space, and the pathological slice image to be identified can be input into the preset storage space for processing. The output of the three-level lymphatic structure recognition model is the three-level lymphatic structure recognition results of the pathological slice image to be identified. The three-level lymphatic structure recognition results can include the developmental stage of the three-level lymphatic structure (the developmental stage of the three-level lymphatic structure) and the positioning information of the three-level lymphatic structure (the actual location of the three-level lymphatic structure).

[0091] In one embodiment of the present application, the three-level lymphatic structure recognition model is used to process the pathological section image to be recognized to obtain the three-level lymphatic structure recognition result, which may include:

[0092] Segment the pathological section image to be identified to obtain segmented images;

[0093] For each segmented image, perform feature extraction on the segmented image to obtain the target features;

[0094] Determine the target feature bounding box in the segmented image based on the target feature;

[0095] When the confidence of the target feature bounding box exceeds a preset threshold, the target feature bounding box is determined as a third-level lymphatic structure bounding box;

[0096] Predict the class probability of the tertiary lymphatic structure bounding box and determine the developmental stage of the tertiary lymphatic structure corresponding to the segmented image;

[0097] The position information of the three-level lymphatic structure bounding box is used as the three-level lymphatic structure positioning information corresponding to the segmented image;

[0098] The tertiary lymphatic structure recognition result of the pathological section image to be identified is determined according to the tertiary lymphatic structure development stage and tertiary lymphatic structure positioning information corresponding to each segmented image.

[0099] The embodiment of the present application provides a method for processing a pathological section image to be identified based on a three-level lymphatic structure recognition model. It is understandable that the construction of the three-level lymphatic structure recognition model is based on cropped image training. Therefore, in the process of processing the pathological section image to be identified, it can also be segmented to obtain each segmented image, and the target features of each segmented image (including but not limited to low-level edge and texture features, high-level semantic features, etc.) are obtained by using a convolutional neural network for feature extraction, so that the target feature bounding box therein can be determined, that is, the bounding box that is considered to contain the three-level lymphatic structure feature; then, through confidence calculation and judgment, it can be determined whether each target feature bounding box actually contains the three-level lymphatic structure feature, so that it can be determined whether the target feature bounding box is a three-level lymphatic structure bounding box; further, through category probability prediction, the development stage of the three-level lymphatic structure within the three-level lymphatic structure bounding box can be determined, and the three-level lymphatic structure positioning information is obtained based on the position information of the three-level lymphatic structure bounding box, that is, the final three-level lymphatic structure recognition result is obtained.

[0100] Among them, the confidence level indicates the possibility that the target object (the target object in this application is the tertiary lymphatic structure) is contained in the corresponding target feature bounding box and the degree of match between the predicted target feature bounding box and the actual target object. It can be calculated by a specific activation function, such as the sigmoid function.

[0101] Among them, the category probability represents the possibility that the target object in the segmented image belongs to different categories (different developmental stages and / or tumors in the embodiment of the present application). The category corresponding to the highest category probability value is the final identification category.

[0102] The tertiary lymphatic structure positioning information may specifically be (x, y, w, h), where x and y represent the horizontal coordinate and vertical coordinate of the center point of the tertiary lymphatic structure bounding box, respectively, and w and h represent the width and height of the tertiary lymphatic structure bounding box, respectively.

[0103] S104: generating a three-level lymphatic structure mask image corresponding to the pathological slice image to be identified according to the three-level lymphatic structure recognition result, and outputting the three-level lymphatic structure mask image.

[0104] This step aims to realize the generation of the three-level lymphatic structure mask map. It is understandable that by generating the three-level lymphatic structure mask map for visual display, it helps medical staff to better observe the development of the three-level lymphatic structure, so as to provide the most appropriate and accurate diagnosis and treatment plan. Specifically, the coordinates under the digital pathology slice dimension can be scaled, and segmented, scaled and reassembled (that is, the digital pathology slice is segmented and scaled proportionally, and then reassembled together) to generate an image that is easy to show the target distribution as a whole, and then obtain a mask map, such as Figure 2 As shown, Figure 2 A three-level lymphatic structure mask provided by this application. Furthermore, the pixel coordinates of the test result data can be restored so that it is located in the coordinate dimension of the digital pathology slice viewing software (such as ndpviewer, slideviewer, etc.), and the coordinate file is obtained in conjunction with the digital pathology slice viewing software to directly view the results. In addition, in addition to the above-mentioned visualized three-level lymphatic structure mask, a readable npda file (a coordinate file that can be correctly parsed by ndpviewer) and a data report (which may include but is not limited to the classification of the three-level lymphatic structure development stage, area, position, tumor area, coverage position and other information) can be generated synchronously and connected to the upper digital platform for viewing and analysis.

[0105] In one embodiment of the present application, after processing the pathological section image to be identified using the three-level lymphatic structure recognition model to obtain the three-level lymphatic structure recognition result, the following steps may also be included:

[0106] Determine the pixel area of ​​the third-level lymphatic structure based on the third-level lymphatic structure identification results;

[0107] The actual tertiary lymphatic structure area of ​​the pathological tissue was calculated based on the tertiary lymphatic structure pixel area and the system pixel scale.

[0108] It can be understood that, in addition to the above-mentioned tertiary lymphatic structure recognition results including the tertiary lymphatic structure development stage and the tertiary lymphatic structure positioning information, the tertiary lymphatic structure area can be further calculated. Specifically, since the tertiary lymphatic structure recognition results include the tertiary lymphatic structure positioning information, the tertiary lymphatic structure pixel area can be calculated based on the tertiary lymphatic structure positioning information, that is, the tertiary lymphatic structure area in the pathological section image to be identified; further, combined with the system pixel scale, the actual tertiary lymphatic structure area of ​​the pathological tissue in the identified pathological section image can be calculated. Among them, the system pixel scale can be the pixel scale of the digital pathological section level 0 layer, denoted as X um / pixel, um is the unit of micrometer, pixel is the unit of pixel, and it is assumed that the pixel area of ​​the tertiary lymphatic structure is Y pixel 2 , then the actual area of ​​the tertiary lymphatic structure is: X 2 ×Y. It should be noted that the system pixel scale can usually be obtained directly in the digital pathology slide viewing software. If this is not possible, it can be calculated using the actual slice width (m um) and the slice pixel width (npixel) as X = m / n. Furthermore, the slice pixel area can be calculated using the slice pixel width (w pixel) and the slice pixel height (h pixel), that is, Y = w × h.

[0109] It can be seen that the three-level lymphatic structure recognition method based on pathological section images provided in the embodiment of the present application uses a pathological section image sample set including a subset of pathological section image samples at different developmental stages of the three-level lymphatic structure to perform fixed model training to obtain a three-level lymphatic structure recognition model, so that the three-level lymphatic structure recognition model can realize the recognition of the three-level lymphatic structure at different developmental stages, obtain the three-level lymphatic structure development stage and three-level lymphatic structure positioning information of the pathological tissue, and generate a three-level lymphatic structure mask map for medical personnel to realize pathological analysis. It can be seen that the present technical solution realizes the accurate recognition of the three-level lymphatic structure at different developmental stages by comprehensively analyzing the development process of the three-level lymphatic structure.

[0110] Finally, referring to the above-mentioned embodiments, taking TLS and tumor recognition as an example, please refer to Table 1, which is an information table of the recognition accuracy of TLS and tumor areas at different developmental stages provided by a three-level lymphatic structure recognition model in the embodiments of this application:

[0111] Table 1 Information on the recognition accuracy of the model for TLS and tumor regions at different developmental stages

[0112]

[0113] It can be seen that the three-level lymphatic structure recognition model provided in the embodiment of the present application can realize TLS recognition at different developmental stages and tumor area recognition with a high recognition accuracy.

[0114] The embodiment of the present application provides a three-level lymphatic structure recognition device based on pathological section images.

[0115] Please refer to Figure 3 , Figure 3 This is a schematic structural diagram of a three-level lymphatic structure recognition device based on pathological slice images provided by this application. The three-level lymphatic structure recognition device based on pathological slice images may include:

[0116] Acquisition module 1 is used to acquire and annotate a pathological section image sample set; the pathological section image sample set includes a subset of pathological section image samples at different developmental stages of the tertiary lymphatic structure;

[0117] Training module 2 is used to construct an initial three-level lymphatic structure recognition model and train the initial three-level lymphatic structure recognition model using a pathological section image sample set to obtain a three-level lymphatic structure recognition model;

[0118] Processing module 3, for processing the pathological section image to be identified using the three-level lymphatic structure recognition model to obtain a three-level lymphatic structure recognition result; the three-level lymphatic structure recognition result includes the three-level lymphatic structure development stage and the three-level lymphatic structure location information;

[0119] The generating module 4 is configured to generate a three-level lymphatic structure mask image corresponding to the pathological slice image to be identified according to the three-level lymphatic structure recognition result, and output the three-level lymphatic structure mask image.

[0120] It can be seen that the three-level lymphatic structure recognition device based on pathological section images provided in the embodiment of the present application uses a pathological section image sample set including a subset of pathological section image samples at different developmental stages of the three-level lymphatic structure to perform fixed model training to obtain a three-level lymphatic structure recognition model, so that the three-level lymphatic structure recognition model can realize the recognition of the three-level lymphatic structure at different developmental stages, obtain the three-level lymphatic structure development stage and the three-level lymphatic structure positioning information of the pathological tissue, and generate a three-level lymphatic structure mask map for medical personnel to realize pathological analysis. It can be seen that the present technical solution realizes the accurate recognition of the three-level lymphatic structure at different developmental stages by comprehensively analyzing the development process of the three-level lymphatic structure.

[0121] In one embodiment of the present application, the training module 2 may include:

[0122] A first determining unit is configured to determine pathological tissue annotation information in each pathological slice image in the pathological slice image sample set;

[0123] A cropping unit, configured to perform adaptive cropping processing on the pathological slice image according to the pathological tissue annotation information to obtain cropped images;

[0124] a second determining unit, configured to determine, for each cropped image, pathological tissue coordinate information in the cropped image;

[0125] A generating unit, configured to generate an image file corresponding to a pathological slice image sample set according to each cropped image, and to generate a coordinate file corresponding to the pathological slice image sample set according to coordinate information of each pathological tissue;

[0126] The training unit is used to train the initial three-level lymphatic structure recognition model using the image file and the coordinate file to obtain the three-level lymphatic structure recognition model.

[0127] In one embodiment of the present application, the cutting unit may include:

[0128] A first determining subunit is configured to determine a pathological tissue annotation point set in a pathological slice image according to the pathological tissue annotation information;

[0129] The second determining subunit is used to determine the cropping size according to the pathological tissue annotation point set;

[0130] The segmentation subunit is used to perform sliding segmentation on the pathological slice image according to the cropping size and the preset image overlap ratio to obtain each cropped image.

[0131] In one embodiment of the present application, the above-mentioned second determination subunit can be specifically used to determine the maximum pixel horizontal coordinate value and the minimum pixel horizontal coordinate value, the maximum pixel vertical coordinate value and the minimum pixel vertical coordinate value in the pathological tissue annotation point set; determine the horizontal coordinate difference based on the maximum pixel horizontal coordinate value and the minimum pixel horizontal coordinate value; determine the vertical coordinate difference based on the maximum pixel vertical coordinate value and the minimum pixel vertical coordinate value; when the horizontal coordinate difference is greater than the vertical coordinate difference, determine the size greater than the horizontal coordinate difference as the cropping size; when the horizontal coordinate difference is not greater than the vertical coordinate difference, determine the size greater than the vertical coordinate difference as the cropping size.

[0132] In one embodiment of the present application, the above-mentioned processing module 3 can be specifically used to segment the pathological section image to be identified to obtain each segmented picture; for each segmented picture, feature extraction is performed on the segmented picture to obtain target features; a target feature bounding box in the segmented picture is determined based on the target features; when the confidence of the target feature bounding box exceeds a preset threshold, the target feature bounding box is determined as a tertiary lymphatic structure bounding box; a category probability prediction is performed on the tertiary lymphatic structure bounding box to determine the tertiary lymphatic structure development stage corresponding to the segmented picture; the position information of the tertiary lymphatic structure bounding box is used as the tertiary lymphatic structure positioning information corresponding to the segmented picture; and the tertiary lymphatic structure recognition result of the pathological section image to be identified is determined based on the tertiary lymphatic structure development stage and the tertiary lymphatic structure positioning information corresponding to each segmented picture.

[0133] In one embodiment of the present application, the three-level lymphatic structure identification method based on pathological section images may also include a calculation module for processing the pathological section image to be identified using the three-level lymphatic structure identification model mentioned above to obtain the three-level lymphatic structure identification result, and then determining the three-level lymphatic structure pixel area according to the three-level lymphatic structure identification result; and calculating the actual three-level lymphatic structure area of ​​the pathological tissue based on the three-level lymphatic structure pixel area and the system pixel scale.

[0134] In one embodiment of the present application, the pathology slice image sample set may further include a tumor slice image sample subset of pathology tissue.

[0135] For an introduction to the apparatus provided in the embodiments of this application, please refer to the above method embodiments, which will not be elaborated in this application.

[0136] An embodiment of the present application provides an electronic device.

[0137] Please refer to Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in this application, which may include:

[0138] Memory 11, for storing computer programs;

[0139] The processor 10 can implement the steps of any one of the above-mentioned three-level lymphatic structure recognition methods based on pathological section images when executing the computer program.

[0140] like Figure 4 FIG. 1 is a schematic diagram of the structure of an electronic device, which may include a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, the memory 11, and the communication interface 12 communicate with each other via the communication bus 13.

[0141] In the embodiment of the present application, the processor 10 may be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field programmable gate array, or other programmable logic devices.

[0142] The processor 10 may call a program stored in the memory 11 . Specifically, the processor 10 may execute operations in an embodiment of a three-level lymphatic structure recognition method based on pathological section images.

[0143] The memory 11 is used to store one or more programs. The program may include program code, and the program code includes computer operating instructions. In the embodiment of the present application, the memory 11 stores at least a program for implementing the following functions:

[0144] Acquire and annotate a pathology section image sample set; the pathology section image sample set includes a subset of pathology section image samples at different developmental stages of the tertiary lymphatic structure;

[0145] An initial three-level lymphatic structure recognition model is constructed and trained using a sample set of pathological slice images to obtain a three-level lymphatic structure recognition model.

[0146] The pathological section images to be identified are processed using a three-level lymphatic structure recognition model to obtain three-level lymphatic structure recognition results. The three-level lymphatic structure recognition results include the three-level lymphatic structure development stage and the three-level lymphatic structure location information.

[0147] A three-level lymphatic structure mask image corresponding to the pathological slice image to be identified is generated according to the three-level lymphatic structure recognition result, and the three-level lymphatic structure mask image is output.

[0148] In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function, etc.; the data storage area may store data created during use.

[0149] In addition, the memory 11 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0150] The communication interface 12 may be an interface of a communication module, and is used to connect to other devices or systems.

[0151] Of course, it needs to be explained that Figure 4 The structure shown does not constitute a limitation on the electronic device in the embodiment of the present application. In actual applications, the electronic device may include Figure 4 More or fewer components than shown, or combinations of certain components.

[0152] An embodiment of the present application provides a computer-readable storage medium.

[0153] The computer-readable storage medium provided in the embodiment of the present application stores a computer program, and when the computer program is executed by a processor, it can implement the steps of any of the above-mentioned three-level lymphatic structure recognition methods based on pathological section images.

[0154] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., which can store program codes.

[0155] For an introduction to the computer-readable storage medium provided in the embodiments of the present application, please refer to the above method embodiments, and this application will not elaborate on them here.

[0156] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0157] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0158] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0159] The technical solution provided by the present application is described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only intended to help understand the method and core ideas of the present application. It should be noted that, for those skilled in the art, without departing from the principles of the present application, several improvements and modifications may be made to the present application, and these improvements and modifications also fall within the scope of protection of the present application.

Claims

1. A three-level lymphatic structure recognition method based on pathological section images, characterized in that: include: Acquire and annotate a sample set of pathology slide images; The pathological section image sample set includes a subset of pathological section image samples at different developmental stages of the tertiary lymphatic structure; Constructing an initial three-level lymphatic structure recognition model, and training the initial three-level lymphatic structure recognition model using the pathological section image sample set to obtain a three-level lymphatic structure recognition model; The pathological section image to be identified is processed using the three-level lymphatic structure recognition model to obtain a three-level lymphatic structure recognition result; The tertiary lymphatic structure identification result includes the tertiary lymphatic structure development stage and the tertiary lymphatic structure location information; A three-level lymphatic structure mask image corresponding to the pathological section image to be identified is generated according to the three-level lymphatic structure recognition result, and the three-level lymphatic structure mask image is output.

2. The three-level lymphatic structure recognition method based on pathological section images according to claim 1 is characterized in that: The initial three-level lymphatic structure recognition model is trained using the pathological section image sample set to obtain a three-level lymphatic structure recognition model, including: For each pathological slice image in the pathological slice image sample set, determining pathological tissue annotation information in the pathological slice image; Performing adaptive size cropping processing on the pathological section image according to the pathological tissue annotation information to obtain cropped images; For each cropped image, determining pathological tissue coordinate information in the cropped image; Generate an image file corresponding to the pathological slice image sample set according to each of the cropped images, and generate a coordinate file corresponding to the pathological slice image sample set according to each of the pathological tissue coordinate information; The initial three-level lymphatic structure recognition model is trained using the image file and the coordinate file to obtain the three-level lymphatic structure recognition model.

3. The three-level lymphatic structure recognition method based on pathological section images according to claim 2 is characterized in that: The pathological section image is subjected to adaptive size cropping processing according to the pathological tissue annotation information to obtain cropped images, including: Determining a pathological tissue annotation point set in the pathological slice image according to the pathological tissue annotation information; Determining the cutting size according to the pathological tissue annotation point set; The pathological slice image is subjected to sliding segmentation according to the cropping size and a preset image overlap ratio to obtain each of the cropped images.

4. The three-level lymphatic structure recognition method based on pathological section images according to claim 3 is characterized in that: Determining the cropping size according to the pathological tissue annotation point set includes: Determining the maximum pixel abscissa value and the minimum pixel abscissa value, and the maximum pixel ordinate value and the minimum pixel ordinate value in the pathological tissue annotation point set; Determine a horizontal coordinate difference according to the maximum pixel horizontal coordinate value and the minimum pixel horizontal coordinate value; Determine a vertical coordinate difference value according to the maximum pixel vertical coordinate value and the minimum pixel vertical coordinate value; When the horizontal coordinate difference is greater than the vertical coordinate difference, determining a size greater than the horizontal coordinate difference as the cutting size; When the horizontal coordinate difference is not greater than the vertical coordinate difference, a size greater than the vertical coordinate difference is determined as the cutting size.

5. The three-level lymphatic structure recognition method based on pathological section images according to claim 1 is characterized in that: The pathological section image to be identified is processed using the three-level lymphatic structure recognition model to obtain a three-level lymphatic structure recognition result, including: Segmenting the pathological section image to be identified to obtain segmented images; For each segmented image, extract features of the segmented image to obtain target features; Determining a target feature bounding box in the segmented image according to the target feature; When the confidence level of the target feature bounding box exceeds a preset threshold, determining the target feature bounding box as a tertiary lymphatic structure bounding box; Performing category probability prediction on the tertiary lymphatic structure bounding box to determine the developmental stage of the tertiary lymphatic structure corresponding to the segmented image; Using the position information of the third-level lymphatic structure boundary box as the third-level lymphatic structure positioning information corresponding to the segmented image; The tertiary lymphatic structure recognition result of the pathological section image to be recognized is determined according to the tertiary lymphatic structure development stage and tertiary lymphatic structure positioning information corresponding to each of the segmented images.

6. The three-level lymphatic structure recognition method based on pathological section images according to any one of claims 1 to 5, characterized in that: After the pathological section image to be identified is processed using the three-level lymphatic structure recognition model to obtain the three-level lymphatic structure recognition result, the method further includes: Determining the pixel area of ​​the tertiary lymphatic structure according to the tertiary lymphatic structure recognition result; The actual tertiary lymphatic structure area of ​​the pathological tissue is calculated based on the tertiary lymphatic structure pixel area and the system pixel scale.

7. The three-level lymphatic structure recognition method based on pathological section images according to claim 1 is characterized in that: The pathological slice image sample set also includes a tumor slice image sample subset of the pathological tissue.

8. A three-level lymphatic structure recognition device based on pathological slice images, characterized in that: include: An acquisition module is used to acquire and annotate a set of pathology slice image samples; The pathological section image sample set includes a subset of pathological section image samples at different developmental stages of the tertiary lymphatic structure; a training module, configured to construct an initial three-level lymphatic structure recognition model, and train the initial three-level lymphatic structure recognition model using the pathological section image sample set to obtain a three-level lymphatic structure recognition model; A processing module, configured to process the pathological section image to be identified using the three-level lymphatic structure recognition model to obtain a three-level lymphatic structure recognition result; The tertiary lymphatic structure identification result includes the tertiary lymphatic structure development stage and the tertiary lymphatic structure location information; A generating module is used to generate a three-level lymphatic structure mask image corresponding to the pathological section image to be identified according to the three-level lymphatic structure identification result, and output the three-level lymphatic structure mask image.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the three-level lymphatic structure recognition method based on pathological section images as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the three-level lymphatic structure recognition method based on pathological section images as described in any one of claims 1 to 7.

Images