A method, device, computer equipment and readable storage medium for identifying third-level lymphatic structures on pathological images

Through the adaptive patch size algorithm and boundary processing algorithm, the problem of low detection accuracy of tertiary lymphoid structures on pathological images is solved, more accurate detection and classification are achieved, and the coverage effect of detection field is improved.

CN118608914BActive Publication Date: 2024-11-29SHANGHAI DUNHUI MEDICAL TECH DEV CO LTD +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410506913.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-04-25
Publication Date
2024-11-29
Estimated Expiration
2044-04-25

AI Technical Summary

Technical Problem

The detection method of tertiary lymphatic structure on existing pathological images is not appropriate in the selection of the Patches scale, resulting in a decrease in detection accuracy. The tertiary lymphatic structure is easily destroyed when it is located at the Patches boundary, affecting the detection results.

Method used

The adaptive patch size algorithm is used to calculate the patch edge length, determine the appropriate Patches scale through clustering and edge length matching, and design an adaptive boundary processing algorithm to repair the tertiary lymphatic structure located on the boundary, and build a high-resolution cross-entropy loss function. Two-channel joint training is performed with the border regression loss function.

Benefits of technology

The detection accuracy of the tertiary lymph structure on pathological images is improved, the error impact caused by the special size is reduced, and the size of the tertiary lymph structure is ensured that the size of the tertiary lymph structure covers the area without manual prior setting, which enhances the detection field.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118608914B_ABST
    Figure CN118608914B_ABST
Patent Text Reader

Abstract

The present application relates to the technical field of pathological slice image detection, and in particular to a method, device, computer equipment and readable storage medium for identifying tertiary lymphatic structures on pathological images, the method comprising obtaining a target pathological image; using an improved adaptive patch size algorithm to calculate the patch side length to determine the patch side length of the target pathological image to be cut, and based on the patch side length, cutting the target pathological image into a number of target sub-images; identifying the target sub-image to obtain a detection result of whether the target sub-image has a tertiary lymphatic structure. The present application uses an adaptive patch size to cut the pathological image, and ensures that the size of the cut patch can cover more of the image area of ​​the tertiary lymphatic structure while ensuring high resolution as much as possible, thereby improving the detection accuracy of the tertiary lymphatic structure on the pathological image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of pathological slice image detection, and in particular to a method, device, computer equipment and readable storage medium for identifying third-level lymphatic structures on pathological images. Background Art

[0002] Pathological images are images with very high resolution, which can reach the cellular level. They are very important for medical diagnosis. Tertiary lymphoid structures (TLS) on pathological images are important pathological features that are important for predicting the patient's prognosis and can be detected at the cellular level. However, due to the ultra-high resolution of pathological images, image files are usually very large, and the source files occupy several GB of space. Deep learning modeling usually converts the source files into tensors, which increases the space consumption exponentially, and the video memory capacity of existing graphics cards cannot support it. Therefore, only two measures can be taken when performing deep learning modeling of pathological images: (1) significantly compress the original image and use the entire low-resolution image for training. This mode is required for projects that collect macroscopic features on pathological images; (2) retain the image resolution and divide the original image into multiple patches according to the grid for model learning. This mode is for projects that need to extract microscopic features from pathological images.

[0003] The scale of TLS is between macroscopic and microscopic, and is not suitable for large-scale resolution compression. Therefore, it is generally chosen to retain the image resolution and perform Patches segmentation before detection for deep learning modeling of pathological images. However, due to the particularity of TLS size, some tertiary lymphatic structures have a large size span, while others have a small size span, which can easily cause errors in the detection results of tertiary lymphatic structures.

[0004] With respect to the above-mentioned related technologies, the inventors have found that the existing method for detecting three-level lymphatic structures on pathological images has a problem of affecting the detection accuracy due to inappropriate selection of the scale of Patches. Summary of the invention

[0005] In order to improve the detection accuracy of the third-level lymphatic structure on the pathological image, the present application provides a method, an apparatus, a computer device and a readable storage medium for identifying the third-level lymphatic structure on the pathological image.

[0006] In a first aspect, the present application provides a method for identifying tertiary lymphatic structures in a pathological image.

[0007] This application is achieved through the following technical solutions:

[0008] A method for identifying three-level lymphatic structures on a pathological image comprises the following steps:

[0009] Acquire target pathological images;

[0010] Determine the side length of the patch to be cut of the target pathological image, and cut the target pathological image into a plurality of target sub-images based on the side length of the patch;

[0011] Identify the target sub-image to obtain a detection result of whether the target sub-image has a tertiary lymphatic structure;

[0012] Among them, the third-level lymphatic structure on the historical pathological image is obtained, and the size distribution data of the third-level lymphatic structure is counted;

[0013] Perform the first clustering on each third-level lymphatic structure, exclude outliers, and calculate the mean length and width of each third-level lymphatic structure except the outliers to obtain the first side length;

[0014] Perform a second clustering on each tertiary lymphatic structure, classify and count each tertiary lymphatic structure according to the preset category, and sort each type of tertiary lymphatic structure according to the size of the number. Select the first n types of tertiary lymphatic structures according to the sorting results to calculate the mean length and the mean width of all tertiary lymphatic structures in the first n types of tertiary lymphatic structures to obtain the second side length, where n is a positive integer and ≥2;

[0015] Based on the first side length and the second side length, selecting the maximum length mean and the maximum width mean to set the side length, to obtain a reference side length;

[0016] A candidate edge length queue is preset, and the patch edge length is automatically matched according to the candidate edge lengths in the candidate edge length queue until the reference edge length is greater than the candidate edge length, and the candidate edge length matched at this time is used as the patch edge length.

[0017] In a preferred example, the present application can be further configured as follows: when identifying the target sub-image, an improved three-level lymphatic structure detection model is used, and the training steps of the three-level lymphatic structure detection model include:

[0018] Acquire a historical pathological image, determine the side length of a patch to be cut from the historical pathological image, and cut the historical pathological image according to the side length of the patch to obtain a plurality of first sample images;

[0019] Inputting the first sample image into the deep learning model, training it in combination with the border regression loss function of the deep learning model, and outputting the detection result of whether there is a third-level lymphatic structure;

[0020] When the preset training conditions are met, the current deep learning model is used as an improved three-level lymphatic structure detection model.

[0021] In a preferred example, the present application can be further configured as follows: based on the first side length and the second side length, the maximum length average and the maximum width average are selected to set the side length, and when the reference side length is obtained, the reference side length is calculated using the following formula:

[0022]

[0023] In the formula, represents the length and width of the first side, Represents the length and width of the second side, a∈[1,3).

[0024] In a preferred example, the present application can be further configured as follows: after obtaining the detection result of whether the target sub-image has a third-level lymphatic structure, the following steps are also included:

[0025] When a tertiary lymphatic structure exists on any of the target sub-images, determining whether the position of the tertiary lymphatic structure is located on the boundary of the target sub-image;

[0026] If the position of the tertiary lymphatic structure is located on the boundary of the target sub-image, the target sub-image is repaired to obtain a repaired tertiary lymphatic structure, and then the tertiary lymphatic structure is classified;

[0027] If the position of the tertiary lymphatic structure is not on the boundary of the target sub-image, the tertiary lymphatic structure is classified.

[0028] In a preferred example, the present application may be further configured as follows: the step of determining whether the position of the third-level lymphatic structure is located on the boundary of the target sub-image includes:

[0029] Creating an aspect ratio dictionary, wherein the aspect ratio dictionary stores the aspect ratios of the borders of the three-level lymphatic structures on the target sub-image in real time;

[0030] Extracting a tertiary lymphatic structure whose center is located in a strip-shaped area whose width is shrunk inward by a preset number of pixels from the boundary of the target sub-image, and storing the aspect ratio of the border of the tertiary lymphatic structure in the aspect ratio dictionary;

[0031] Based on the preset outlier value, searching among the aspect ratios of all the tertiary lymphatic structures in the current aspect ratio dictionary whether there is a tertiary lymphatic structure whose aspect ratio product according to a preset multiple is greater than the outlier value;

[0032] If there is a tertiary lymphatic structure whose product of preset multiples of aspect ratio is greater than the outlier value, the tertiary lymphatic structure is regarded as an abnormal tertiary lymphatic structure. At this time, it is determined that the position of the tertiary lymphatic structure is located on the boundary of the target sub-image.

[0033] In a preferred example, the present application can be further configured as follows: when performing restoration processing on the target sub-image, an improved adaptive boundary entity processing algorithm is used, including the following steps:

[0034] Creating an attribution dictionary, wherein the attribution dictionary sequentially generates target sub-images corresponding to the third-level lymphatic structures in the aspect ratio dictionary according to the storage order of the aspect ratios of the borders of the third-level lymphatic structures in the aspect ratio dictionary;

[0035] According to the attribution dictionary, indexing the target sub-image corresponding to the abnormal tertiary lymphatic structure to calculate the circumscribed rectangular frame of the abnormal tertiary lymphatic structure in the target sub-image;

[0036] The average value of the length of the circumscribed rectangular frame and the length of the preset priori rectangular frame is solved, and the average value of the width of the circumscribed rectangular frame and the width of the priori rectangular frame is solved to obtain the inferred detection frame of the third-level lymphatic structure located on the boundary of the target sub-image.

[0037] In a preferred example, the present application can be further configured as follows: when typing the three-level lymphatic structure, an improved three-level lymphatic structure typing model is used, wherein the training steps of the three-level lymphatic structure typing model include:

[0038] Cutting out the original pathological image containing the third-level lymphatic structure from the historical pathological image to obtain a plurality of second sample images;

[0039] Optimizing based on the deep learning model architecture, adding a high-precision image classification branch at the input end to obtain a target model, wherein the high-precision image classification branch is used to receive the second sample image;

[0040] Design a cross entropy loss function for high-resolution three-level lymphatic structure images, combine it with the bounding box regression loss function of the deep learning model, and construct a target loss function;

[0041] Inputting the detection result of whether there is a tertiary lymphatic structure and the second sample image into the target model respectively, and performing dual-channel joint training and gradient back propagation optimization using the target loss function to output the typing result of the tertiary lymphatic structure;

[0042] When the preset training conditions are met, the current target model is used as an improved three-level lymphatic structure classification model.

[0043] In a preferred example, the present application can be further configured as follows: the cross entropy loss function of the high-resolution three-level lymphatic structure image is designed, and the border regression loss function of the deep learning model is combined to construct the target loss function, and the following formula is used:

[0044] Loss=Loss G +Loss2

[0045] Loss G =CrossEntropy_Loss

[0046] Loss2=DFL_Loss+CIOU_Loss

[0047] In the formula, Loss represents the target loss function, Loss G represents the cross entropy loss function, and the CrossEntropy_Loss function is selected. Loss2 represents the border regression loss function, DFL_Loss represents the distance field loss function, and CIOU_Loss represents the border loss function.

[0048] In a preferred example, the present application may be further configured as follows: before the step of inputting the detection result of whether there is a third-level lymphatic structure and the second sample image into the target model respectively, it also includes:

[0049] The detection result of whether there is a third-level lymphatic structure and the second sample image are respectively subjected to size standardization processing.

[0050] In a second aspect, the present application provides a device for identifying third-level lymphatic structures in pathological images.

[0051] This application is achieved through the following technical solutions:

[0052] A device for identifying three-level lymphatic structures on a pathological image, comprising:

[0053] A pathological image acquisition module, used for acquiring target pathological images;

[0054] A pathological image cutting module, used to determine the side length of a patch to be cut from a target pathological image, and cut the target pathological image into a plurality of target sub-images based on the side length of the patch;

[0055] A tertiary lymphatic structure detection module, used to identify the target sub-image and obtain a detection result of whether the target sub-image has a tertiary lymphatic structure;

[0056] The patch side length module is used to calculate the patch side length of the target pathological image to be cut using an improved adaptive patch size algorithm, including:

[0057] A statistical unit, used for obtaining the tertiary lymphatic structures on the historical pathological images and counting the size distribution data of the tertiary lymphatic structures;

[0058] A first clustering unit is used to perform a first clustering on each of the three-level lymphatic structures, exclude outliers, and calculate the mean length and the mean width of each of the three-level lymphatic structures except the outliers to obtain a first side length;

[0059] The second clustering unit is used to perform a second clustering on each tertiary lymphatic structure, classify and count each tertiary lymphatic structure according to a preset category, and sort each type of tertiary lymphatic structure according to the quantity, and select the first n types of tertiary lymphatic structures according to the sorting results to calculate the mean length and the mean width of all tertiary lymphatic structures in the first n types of tertiary lymphatic structures to obtain a second side length, wherein n is a positive integer and is ≥2;

[0060] A reference side length unit, used for selecting a maximum length mean and a maximum width mean to set the side length based on the first side length and the second side length, so as to obtain a reference side length;

[0061] The patch side length unit is used to preset a candidate side length queue, automatically match the patch side length according to the candidate side lengths in the candidate side length queue, until the reference side length is greater than the candidate side length, and use the candidate side length matched at this time as the patch side length.

[0062] In a preferred example, the present application can be further configured as follows: further comprising:

[0063] The three-level lymphatic structure detection model module is used to obtain historical pathological images, determine the side length of the patch to be cut of the historical pathological images, and cut the historical pathological images according to the patch side length to obtain a number of first sample images; the first sample images are input into the deep learning model, and the deep learning model is trained in combination with the border regression loss function to output the detection result of whether there is a three-level lymphatic structure; when the preset training conditions are met, the current deep learning model is used as an improved three-level lymphatic structure detection model.

[0064] In a preferred example, the present application can be further configured as follows: when the reference side length unit calculates the reference side length, the following formula is used:

[0065]

[0066] In the formula, represents the length and width of the first side, Represents the length and width of the second side, a∈[1,3).

[0067] In a preferred example, the present application can be further configured as follows: further comprising:

[0068] A judging module, configured to judge whether the position of the tertiary lymphatic structure is located on the boundary of the target sub-image when the tertiary lymphatic structure exists on any of the target sub-images;

[0069] A repair module, configured to repair the target sub-image if the position of the tertiary lymphatic structure is located on the boundary of the target sub-image;

[0070] The tertiary lymphatic structure classification module is used to classify the tertiary lymphatic structure.

[0071] In a preferred example, the present application can be further configured as follows: the judgment module includes:

[0072] An aspect ratio dictionary unit, used for creating an aspect ratio dictionary, wherein the aspect ratio dictionary stores the aspect ratios of the borders of the three-level lymphatic structures on the target sub-image in real time;

[0073] A reference unit, configured to extract a tertiary lymphatic structure whose center is located in a strip-shaped area whose width is shrunk inward by a preset number of pixels from the boundary of the target sub-image, and store the aspect ratio of the border of the tertiary lymphatic structure in the aspect ratio dictionary;

[0074] A search unit, configured to search, based on a preset outlier value, whether there is a third-level lymphatic structure whose aspect ratio product according to a preset multiple is greater than the outlier value among the aspect ratios of all the third-level lymphatic structures in the current aspect ratio dictionary;

[0075] The abnormal tertiary lymphatic structure identification unit is used to identify the tertiary lymphatic structure as an abnormal tertiary lymphatic structure if there is a tertiary lymphatic structure whose product of preset multiples of aspect ratio is greater than the outlier value, and at this time, determine that the position of the tertiary lymphatic structure is located on the boundary of the target sub-image.

[0076] In a preferred example, the present application can be further configured as follows: the repair module includes:

[0077] an attribution dictionary unit, configured to create an attribution dictionary, wherein the attribution dictionary sequentially generates target sub-images corresponding to the third-level lymphatic structures in the aspect ratio dictionary according to the storage order of the aspect ratios of the borders of the third-level lymphatic structures in the aspect ratio dictionary;

[0078] An indexing unit, configured to index the target sub-image corresponding to the abnormal tertiary lymphatic structure according to the attribution dictionary, so as to calculate a circumscribed rectangular frame of the abnormal tertiary lymphatic structure in the target sub-image;

[0079] The abnormal tertiary lymphatic structure positioning unit is used to solve the mean value of the length of the circumscribed rectangular frame and the length of the preset priori rectangular frame, and to solve the mean value of the width of the circumscribed rectangular frame and the width of the priori rectangular frame, so as to obtain the estimated detection frame of the tertiary lymphatic structure located on the boundary of the target sub-image.

[0080] In a preferred example, the present application can be further configured as follows: further comprising:

[0081] The three-level lymphatic structure classification model module is used to crop the original pathological images containing the three-level lymphatic structure from the historical pathological images to obtain a number of second sample images; based on the deep learning model architecture, optimization is performed, and a high-precision image classification branch is added to the input end to obtain a target model, wherein the high-precision image classification branch is used to receive the second sample image; a cross-entropy loss function of the high-resolution three-level lymphatic structure image is designed, and a target loss function is constructed in combination with the border regression loss function of the deep learning model; the detection result of whether there is a three-level lymphatic structure and the second sample image are respectively input into the target model, and the target loss function is used to perform dual-channel joint training and gradient back propagation optimization to output the classification results of the three-level lymphatic structure; when the preset training conditions are met, the current target model is used as an improved three-level lymphatic structure classification model.

[0082] In a preferred example, the present application can be further configured as follows: the expression of the objective loss function of the three-level lymphatic structure classification model module is as follows:

[0083] Loss=Loss G +Loss2

[0084] Loss G =CrossEntropy_Loss

[0085] Loss2=DFL_Loss+CIOU_Loss

[0086] In the formula, Loss represents the target loss function, Loss G represents the cross entropy loss function, and the CrossEntropy_Loss function is selected. Loss2 represents the border regression loss function, DFL_Loss represents the distance field loss function, and CIOU_Loss represents the border loss function.

[0087] In a preferred example, the present application can be further configured as follows: further comprising:

[0088] The tertiary lymphatic structure size standardization module is used to perform size standardization processing on the detection result of whether there is a tertiary lymphatic structure and the second sample image respectively, and then input the processed detection result of whether there is a tertiary lymphatic structure and the processed second sample image into the target model respectively.

[0089] In a third aspect, the present application provides a computer device.

[0090] This application is achieved through the following technical solutions:

[0091] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above-mentioned methods for identifying the third-level lymphatic structure on a pathological image when executing the computer program.

[0092] In a fourth aspect, the present application provides a computer-readable storage medium.

[0093] This application is achieved through the following technical solutions:

[0094] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned methods for identifying third-level lymphatic structures on a pathological image.

[0095] In summary, compared with the prior art, the technical solution provided by this application has at least the following beneficial effects:

[0096] The target pathological image is obtained, and the target pathological image is used to detect whether there is a tertiary lymphatic structure on the image; by retaining the resolution of the target pathological image, the improved adaptive patch size algorithm is used to calculate the patch side length, the patch side length of the target pathological image to be cut is determined, and the segmentation scale of Patches is adaptively calculated, and the original image is divided into multiple Patches according to the grid, and the target pathological image is cut into several target sub-images according to the patch side length, so as to ensure that the size of Patches can cover more image areas of the tertiary lymphatic structure while ensuring the resolution of Patches as much as possible; the target sub-image is identified to obtain the detection result of whether the target sub-image has a tertiary lymphatic structure; and then the side length of the patch to be cut is determined by constructing an adaptive Patches side length calculation algorithm, which reduces the error influence on the detection result caused by the particularity of the size of the tertiary lymphatic structure, and can obtain a more suitable tertiary lymphatic structure detection field of view without artificial prior setting, thereby improving the detection accuracy of the tertiary lymphatic structure on the pathological image. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] Figure 1 A schematic diagram of the overall process of a method for identifying three-level lymphatic structures in a pathological image provided as an exemplary embodiment of the present application.

[0098] Figure 2 A main flow chart of a method for identifying three-level lymphatic structures in a pathological image provided as another exemplary embodiment of the present application.

[0099] Figure 3 A schematic diagram of the positional relationship between an abnormal TLS and its circumscribed rectangular frame in a method for identifying a third-level lymphatic structure on a pathological image provided as another exemplary embodiment of the present application.

[0100] Figure 4 A schematic diagram of the positional relationship between an external rectangular frame, a priori rectangular frame and an inferred detection frame of an abnormal TLS in a method for identifying a third-level lymphatic structure on a pathological image provided as an exemplary embodiment of the present application.

[0101] Figure 5 A structural block diagram of a device for identifying three-level lymphatic structures in a pathological image provided as an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0102] This specific embodiment is merely an explanation of the present application and is not a limitation of the present application. After reading this specification, those skilled in the art may make modifications to the present embodiment without any creative contribution as needed, but such modifications are protected by the patent law as long as they are within the scope of the claims of the present application.

[0103] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0104] In addition, the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article, unless otherwise specified, generally means that the associated objects before and after are in an "or" relationship.

[0105] Due to the particularity of the size of the tertiary lymphatic structure, that is, some tertiary lymphatic structures have a larger size span, while some tertiary lymphatic structures have a smaller size span. How to select the scale of Patches will affect the detection accuracy of the tertiary lymphatic structure, so that the scale of selected Patches is limited. Therefore, the present application proposes a method for identifying the tertiary lymphatic structure on a pathological image by adaptively calculating the segmentation scale of Patches. By constructing an adaptive Patches edge length calculation algorithm, a more appropriate tertiary lymphatic structure detection field of view can be obtained without artificial prior setting, so as to achieve the effect of ensuring that the size of Patches can cover more of the tertiary lymphatic structure area while ensuring the resolution of Patches as much as possible (that is, the coverage range of Patches is reduced and the field of view is narrowed), thereby improving the detection accuracy of the tertiary lymphatic structure on the pathological image.

[0106] In addition, there is another problem in detecting the tertiary lymphatic structure after Patches segmentation. If the position of a TLS is exactly on the boundary of Patches segmentation, it will be divided into at least two parts, which will lead to the following consequences:

[0107] (1) The Patches segmentation boundary destroys the macroscopic structure of the tertiary lymphatic structure, resulting in a decrease in the sensitivity of detection. The tertiary lymphatic structure at the boundary line cannot be detected, which is easy to be missed and has a low detection rate;

[0108] (2) A large tertiary lymphoid structure is divided into multiple small incomplete tertiary lymphoid structures by the Patches boundary. Due to the partial loss, it will affect the further analysis of the tertiary lymphoid structure and interfere with the result classification of the tertiary lymphoid structure;

[0109] (3) One tertiary lymphatic structure was detected as multiple ones, which affected the quantitative statistical results.

[0110] Therefore, in order to address the situation where Patches segmentation will split the tertiary lymphatic structure that is exactly located on the dividing line in the original image and affect the detection accuracy of the model, an adaptive tertiary lymphatic structure processing algorithm on the Patches boundary, an adaptive incomplete tertiary lymphatic structure recognition algorithm and an adaptive tertiary lymphatic structure repair algorithm are innovatively designed to automatically fuse the fragments of the tertiary lymphatic structure on the Patches boundary and repair the damaged tertiary lymphatic structure on the Patches boundary, thereby improving the detection accuracy of the tertiary lymphatic structure on the pathological image.

[0111] The embodiments of the present application are further described in detail below in conjunction with the drawings in the specification.

[0112] Reference Figure 1 , an embodiment of the present application provides a method for identifying the third-level lymphatic structure in a pathological image, and the main steps of the method are described as follows.

[0113] S1: Acquire target pathological image;

[0114] S2: determining the side length of the patch to be cut of the target pathological image, and cutting the target pathological image into a plurality of target sub-images based on the side length of the patch;

[0115] S3: Identify the target sub-image to obtain a detection result of whether the target sub-image has a tertiary lymphatic structure;

[0116] Among them, the third-level lymphatic structure on the historical pathological image is obtained, and the size distribution data of the third-level lymphatic structure is counted;

[0117] Perform the first clustering on each third-level lymphatic structure, exclude outliers, and calculate the mean length and width of each third-level lymphatic structure except the outliers to obtain the first side length;

[0118] Perform a second clustering on each tertiary lymphatic structure, classify and count each tertiary lymphatic structure according to the preset category, and sort each type of tertiary lymphatic structure according to the size of the number. Select the first n types of tertiary lymphatic structures according to the sorting results to calculate the mean length and the mean width of all tertiary lymphatic structures in the first n types of tertiary lymphatic structures to obtain the second side length, where n is a positive integer and ≥2;

[0119] Based on the first side length and the second side length, selecting the maximum length mean and the maximum width mean to set the side length, to obtain a reference side length;

[0120] A candidate edge length queue is preset, and the patch edge length is automatically matched according to the candidate edge lengths in the candidate edge length queue until the reference edge length is greater than the candidate edge length, and the candidate edge length matched at this time is used as the patch edge length.

[0121] Specifically, a pathological image in which a tertiary lymphatic structure (TLS) may exist is acquired as a target pathological image.

[0122] Since TLS scales vary in size, it is necessary to calculate the size distribution of Patches, perform the first clustering of each tertiary lymphatic structure, exclude outliers, and perform the second clustering of each tertiary lymphatic structure. Each tertiary lymphatic structure is classified and counted according to the preset category, and each type of tertiary lymphatic structure is sorted by quantity. The top n types of tertiary lymphatic structures are selected according to the sorting results.

[0123] For example, in this embodiment, the DBSCAN clustering algorithm and the K-means clustering algorithm are used together to perform the first clustering and the second clustering on the TLS, respectively, and the length and width pixel sizes (X0, Y0), (X1, Y1), ..., (X n ,Y n ) as clustering data points for clustering, including:

[0124] DBSCAN clustering: Based on density clustering, the clustering results are obtained according to the density of cluster data points in the vector space. This clustering algorithm can exclude outliers, that is, cluster points with a small number. The cluster data points that are clustered into each category are all points in a space with a higher density. By using DBSCAN clustering, outlier TLS size outliers are excluded, and the average length and width of TLS excluding outliers is calculated. As the first side length, it is helpful to improve the quality of data used for subsequent model testing.

[0125] K-means clustering: cluster TLS data points into preset categories, such as 5 types of TLS, and sort them according to the number of clustered 5 types of TLS. Then, based on the sorted results, select the top 3 TLS, that is, get 3 types of TLS with similar categories and large numbers, and calculate the length and width averages of all TLS in these 3 types of TLS. As the second side length, TLS with similar category attributes and sample numbers that meet the model detection requirements are screened out to further improve the quality of data used for subsequent model detection.

[0126] Based on the first side length and the second side length, the maximum length mean and the maximum width mean are selected to set the side length, and a reference side length is obtained, which is saved for future use.

[0127] In one embodiment, based on the first side length and the second side length, the maximum length average and the maximum width average are selected to set the side length, and when the reference side length is obtained, the reference side length is calculated using the following formula:

[0128]

[0129] In the formula, represents the length and width of the first side, Represents the length and width of the second side, a∈[1,3).

[0130] For example, the reference side length is calculated as follows:

[0131]

[0132] Among them, according to prior experience, a is taken as 2. At this time, according to the calculated reference side length, a better patch side length can be matched, and then a more appropriate tertiary lymphatic structure detection field of view can be obtained by segmentation, so as to ensure that the size of Patches can cover more tertiary lymphatic structure areas while ensuring the resolution of Patches as much as possible, thereby improving the detection accuracy of tertiary lymphatic structures on pathological images.

[0133] a can also be selected according to business requirements.

[0134] A candidate edge length queue is preset, and the patch edge length is automatically matched according to the candidate edge lengths in the candidate edge length queue until the reference edge length is greater than the candidate edge length, and the candidate edge length matched at this time is used as the patch edge length.

[0135] For example, from the candidate queue in Table 1 below, automatically match the edge length of Patches until the reference edge length of Patches is greater than the candidate edge length for the first time:

[0136] Table 1

[0137] 40960 20480 10240 10660 5210 ↑ 2560 ↑

[0138] Take the candidate edge length 10240 as the Patches edge length.

[0139] Furthermore, during matching, the automatic matching of the patch edge lengths can be performed in the order of the candidate edge lengths from small to large, so as to further improve the matching efficiency of the Patches edge lengths.

[0140] After determining the patch side length of the target pathological image to be cut, the target pathological image is cut into several target sub-images based on the adaptive patch side length to retain the resolution of the pathological image. The pathological image is cut into multiple Patches according to the grid to obtain several target sub-images.

[0141] Finally, the target sub-image is identified to obtain a detection result of whether the target sub-image has a tertiary lymphatic structure, and whether a tertiary lymphatic structure exists on a number of Patches is automatically identified.

[0142] In one embodiment, when the target sub-image is recognized, an improved three-level lymphatic structure detection model is used, and the training steps of the three-level lymphatic structure detection model include:

[0143] Acquire a historical pathological image, determine the side length of a patch to be cut from the historical pathological image, and cut the historical pathological image according to the side length of the patch to obtain a plurality of first sample images;

[0144] Inputting the first sample image into the deep learning model, training it in combination with the border regression loss function of the deep learning model, and outputting the detection result of whether there is a third-level lymphatic structure;

[0145] When the preset training conditions are met, the current deep learning model is used as an improved three-level lymphatic structure detection model.

[0146] Specifically, since the YOLO-v8 target detection algorithm is an algorithm that takes into account both speed and accuracy in the field of target detection, the deep learning model of the present application can use the YOLO-v8 model to perform TLS detection on Patches, and by obtaining historical pathological images, adaptively determine the side length of the patch to be cut in the historical pathological images. The method is the same as the adaptive Patches side length calculation algorithm constructed above, and will not be repeated here.

[0147] The historical pathological image is cut according to the adaptive patch edge length to obtain a number of first sample images, which are input into the YOLO-v8 model, trained in combination with the border regression loss function of the model, and the detected TLS is output.

[0148] When the bounding box regression loss function converges or the function value of the bounding box regression loss function is less than a certain threshold, it is judged that the preset training conditions are met, and the current YOLO-v8 model is used as an improved three-level lymphatic structure detection model.

[0149] The detected TLS will be output as a txt file in the form of rectangular box center coordinates and vertex coordinates for further calls.

[0150] In one embodiment, a portion of the first sample images may be randomly extracted and input into a deep learning model for learning to improve the generalization ability of the model.

[0151] Reference Figure 2 , by constructing an adaptive Patches size calculation algorithm, determining the patch side length to perform patch cutting on the target pathological image, and obtaining several Patches for TLS detection. During detection, first determine whether TLS is on the Patches boundary; if TLS is not on the Patches boundary, then directly perform TLS typing; if TLS is on the Patches boundary, then give priority to using an adaptive TLS integration (fusion) algorithm to repair the TLS on the Patches boundary, and then perform TLS typing.

[0152] Reference Figure 1 In order to improve the situation that Patches segmentation will segment the third-level lymphatic structure located exactly on the boundary line in the original image and affect the accuracy of model detection, in one embodiment, after obtaining the detection result of whether the target sub-image has a third-level lymphatic structure, the following steps are also included:

[0153] S4: when there is a tertiary lymphatic structure on any of the target sub-images, determining whether the position of the tertiary lymphatic structure is located on the boundary of the target sub-image;

[0154] S51: If the position of the tertiary lymphatic structure is located on the boundary of the target sub-image, the target sub-image is repaired to obtain a repaired tertiary lymphatic structure, and then the tertiary lymphatic structure is classified;

[0155] S52: If the position of the tertiary lymphatic structure is not on the boundary of the target sub-image, the tertiary lymphatic structure is classified.

[0156] In one embodiment, the step of determining whether the position of the third-level lymphatic structure is located on the boundary of the target sub-image includes:

[0157] Creating an aspect ratio dictionary, wherein the aspect ratio dictionary stores the aspect ratios of the borders of the three-level lymphatic structures on the target sub-image in real time;

[0158] Extracting a tertiary lymphatic structure whose center is located in a strip-shaped area whose width is shrunk inward by a preset number of pixels from the boundary of the target sub-image, and storing the aspect ratio of the border of the tertiary lymphatic structure in the aspect ratio dictionary;

[0159] Based on the preset outlier value, searching among the aspect ratios of all the tertiary lymphatic structures in the current aspect ratio dictionary whether there is a tertiary lymphatic structure whose aspect ratio product according to a preset multiple is greater than the outlier value;

[0160] If there is a tertiary lymphatic structure whose product of preset multiples of aspect ratio is greater than the outlier value, the tertiary lymphatic structure is regarded as an abnormal tertiary lymphatic structure. At this time, it is determined that the position of the tertiary lymphatic structure is located on the boundary of the target sub-image.

[0161] In one embodiment, when the target sub-image is repaired, an improved adaptive boundary entity processing algorithm is used, including the following steps:

[0162] Creating an attribution dictionary, wherein the attribution dictionary sequentially generates target sub-images corresponding to the third-level lymphatic structures in the aspect ratio dictionary according to the storage order of the aspect ratios of the borders of the third-level lymphatic structures in the aspect ratio dictionary;

[0163] According to the attribution dictionary, indexing the target sub-image corresponding to the abnormal tertiary lymphatic structure to calculate the circumscribed rectangular frame of the abnormal tertiary lymphatic structure in the target sub-image;

[0164] The average value of the length of the circumscribed rectangular frame and the length of the preset priori rectangular frame is solved, and the average value of the width of the circumscribed rectangular frame and the width of the priori rectangular frame is solved to obtain the inferred detection frame of the third-level lymphatic structure located on the boundary of the target sub-image.

[0165] Specifically, the present application innovatively designs a set of adaptive three-level lymphatic structure processing algorithms on Patches boundaries to restore TLS segmented on Patches boundaries. The algorithm consists of two parts:

[0166] Part 1: Adaptive incomplete TLS identification algorithm:

[0167] 1) Build the TLS aspect ratio dictionary:

[0168]

[0169]

[0170] 2) As the model is trained, the aspect ratio of the TLS bounding box extracted on each patch is dynamically calculated and updated to the TLS aspect ratio dictionary;

[0171] 3) Extract the TLS whose center is located in the strip area with a width of 200 pixels contracted inward from the Patches partition boundary;

[0172] 4) Expand all values ​​in the TLS aspect ratio dictionary, put all the TLS aspect ratios detected in all Patches into a list, and generate a list of corresponding patches in order as follows:

[0173]

[0174] 5) Sort the aspect ratios of all TLS in the calculated patch and calculate the aspect ratios of all TLS based on the outlier calculation rule: for example, find TLS that is 1.5 times greater than the IQR (Interquartile Range, outlier) based on the aspect ratio values ​​of all remaining TLS, IQR is the interquartile range, Q1 = 25%, Median = 50%, Q3 = 75%, IQR = Q3-Q1, these are considered to be boundary TLS, because the TLS divided by the boundary usually has an abnormal aspect ratio due to its partial missing;

[0175] 6) According to the index, find the abnormal TLS cluster belonging to the current Patch in the "TLS belonging list" in 4), that is, the abnormal tertiary lymphatic structure.

[0176] Part II: Adaptive TLS Repair Algorithm:

[0177] (a) Find the length and width of the two TLS boxes obtained based on DBSCAN and K-means clustering in the previous article, and calculate the mean to obtain the TLS prior rectangular box as follows:

[0178]

[0179] (b) Calculate the bounding rectangle of the abnormal TLS cluster in 6) above, such as Figure 3 As shown, the dark blue solid rectangle is the abnormal TLS cluster, and the red dotted line is the circumscribed rectangle of the abnormal TLS cluster;

[0180] (c) Calculate the average of the length and width of the bounding rectangle of the abnormal TLS cluster and the length and width of the preset priori rectangular frame to obtain the estimated detection frame of the TLS cluster as follows: Figure 4 As shown, the red dotted line is the circumscribed rectangle of the abnormal TLS cluster, the green solid rectangular box is the TLS prior box, and the light blue solid rectangular box is the detection box of the TLS inferred by the algorithm.

[0181] Through the adaptive incomplete tertiary lymphatic structure recognition algorithm and the adaptive tertiary lymphatic structure repair algorithm, the fragments of the tertiary lymphatic structure on the Patches boundary are automatically fused, and the damaged tertiary lymphatic structure on the Patches boundary is repaired, thereby improving the detection accuracy of the tertiary lymphatic structure on the pathological image.

[0182] Furthermore, considering that the TLS structure appearing on the Patches boundary is inevitably destroyed during the Patches clipping process, the present application proposes a TLS detection typing model with higher accuracy when applied in actual scenarios.

[0183] In one embodiment, when the three-level lymphatic structure is classified, an improved three-level lymphatic structure classification model is used, wherein the training steps of the three-level lymphatic structure classification model include:

[0184] Cutting out the original pathological image containing the third-level lymphatic structure from the historical pathological image to obtain a plurality of second sample images;

[0185] Optimizing based on the deep learning model architecture, adding a high-precision image classification branch at the input end to obtain a target model, wherein the high-precision image classification branch is used to receive the second sample image;

[0186] Design a cross entropy loss function for high-resolution three-level lymphatic structure images, combine it with the bounding box regression loss function of the deep learning model, and construct a target loss function;

[0187] Inputting the detection result of whether there is a tertiary lymphatic structure and the second sample image into the target model respectively, and performing dual-channel joint training and gradient back propagation optimization using the target loss function to output the typing result of the tertiary lymphatic structure;

[0188] When the preset training conditions are met, the current target model is used as an improved three-level lymphatic structure classification model.

[0189] In one embodiment, the cross entropy loss function of the high-resolution three-level lymphatic structure image is designed, and the frame regression loss function of the deep learning model is combined to construct the target loss function, using the following formula:

[0190] Loss=Loss G +Loss2

[0191] Loss G =CrossEntropy_Loss

[0192] Loss2=DFL_Loss+CIOU_Loss

[0193] In the formula, Loss represents the target loss function, Loss G represents the cross entropy loss function, and the CrossEntropy_Loss function is selected. Loss2 represents the bounding box regression loss function. DFL_Loss represents the distance field loss function, that is, the difference between the distance field predicted by the target model and the actual distance field. The distance field is a coding method used to represent the relative positions of the center point and the four corner points of the bounding box. CIOU_Loss represents the bounding box loss function, which is used to measure the difference between the predicted box of the target model and the real box.

[0194] Among them, the expression of CrossEntropy_Loss function can be as follows:

[0195] CrossEntropy_Loss=-[y log y^+(1-y)log(1-y^)];

[0196] In the formula, y represents the sample label and y^ represents the target model prediction value.

[0197] The expression of the DFL_Loss function can be as follows:

[0198] DFL_Loss = ∑[d(p) - d_gt(p)] ^2 ;

[0199] Where d(p) represents the predicted value of point p in the distance field predicted by the target model, and d_gt(p) represents the value of point p in the true distance field.

[0200] The smaller the DFL_Loss, the smaller the difference between the distance field predicted by the target model and the true distance field, and the better the performance of the model.

[0201] The expression of CIOU_Loss function can be as follows:

[0202] CIOU_Loss = 1 – CIoU;

[0203] CIoU = IoU - α*v - β*C;

[0204] Where, CIoU (Complete IoU) is the IoU (Intersection over Union) between the predicted box and the real box of the target model; IoU represents the existing IoU, that is, the ratio of the intersection area to the union area between the predicted box and the real box of the target model; v represents the difference in the aspect ratio between the predicted box and the real box of the target model, which is measured by calculating the diagonal distance between the predicted box and the real box; C represents the normalized distance between the center point of the predicted box and the real box of the target model, which is measured by calculating the Euclidean distance between the center points of the predicted box and the real box; α and β are two adjustable parameters used to balance the contribution of v and C to CIoU, and their values ​​usually range from 0 to 2.

[0205] The smaller the CIOU_Loss, the smaller the difference between the predicted box of the target model and the true box, and the better the performance of the model.

[0206] In one embodiment, before the step of inputting the detection result of whether there is a third-level lymphatic structure and the second sample image into the target model respectively, the method further includes:

[0207] The detection result of whether there is a third-level lymphatic structure and the second sample image are respectively subjected to size standardization processing.

[0208] This application constructs a multi-task model Dual-YOLO-TLS-Grading model for joint TLS detection and high-precision image TLS classification by improving the YOLO-v8 model, that is, an improved three-level lymphatic structure classification model.

[0209] Specifically, based on the optimization of the YOLO-v8 model architecture, a high-precision image classification branch is added, which supports the extraction of the detected TLS high-resolution original image from the svs pathology image source file, where the high-precision image is the inherent attribute of the existing pathology image, and the image containing TLS directly cropped from the pathology image is the TLS high-resolution original image. The improved three-level lymphatic structure classification model consists of the following two branches:

[0210] First branch: YOLO-v8 target detection algorithm:

[0211] This branch is the YOLO-v8 target detection algorithm, which is used to detect TLS on Patches. The detected TLS will be output as a txt file in the form of the center coordinates and vertex coordinates of the rectangular box for further call. The TLS rectangular box related data detected in this step will be used as the input of the second branch. After the YOLO-v8 target detection branch is calculated, the loss function of this branch is temporarily stored in the video memory.

[0212] Second branch: High-resolution TLS typing algorithm:

[0213] This branch is a comprehensive module for further classification of TLS detected by the YOLO-v8 detection model, which includes the following sub-modules:

[0214] High-resolution original image cropping module: This module is used to crop the detected TLS from the high-resolution WSI (panoramic digital slice, usually stored in .svs or .tif format) original image as the input of the high-resolution image classification model;

[0215] Boundary TLS repair module: This module includes the adaptive incomplete TLS identification algorithm and the adaptive TLS repair algorithm mentioned above:

[0216] TLS high-definition image standardization module: The detected TLS usually have different sizes and shapes, and the images input to the deep neural network need to be of uniform size, so this step requires standardizing the size of each TLS, such as 512*512;

[0217] Joint training module integrating YOLO-v8 loss function:

[0218] The existing YOLO-v8 loss functions include:

[0219] (i) Category classification loss function: Classify the detected target and determine which specific target category it belongs to. The specific formula is as follows:

[0220] Loss1=BCE_Loss

[0221] (ii) Bounding box regression loss function:

[0222] Loss2=DFL_Loss+CIOU_Loss

[0223] Since this application is only used for the detection of one category of TLS, the above category classification loss function is meaningless in the application scenario of this application, and this application also needs to construct a classification modeling loss function for high-resolution TLS images, so the model architecture of YOLO-v8 is modified as follows:

[0224] The multi-label classification module of YOLO-v8 is changed to a multi-classification module based on Inception-v3. The multi-classification module based on Inception-v3 adopts the multi-classification cross entropy loss function torch.nn.CrossEntropyLoss based on the Pytorch library. The specific formula is as follows:

[0225] Loss G =CrossEntropy_Loss

[0226] After completing the above modifications, the loss function (target loss function) of the Dual-YOLO-TLS-Grading model of the improved three-level lymphatic structure classification model of the present application is:

[0227] Loss=Loss G +Loss2

[0228] Loss=CrossEntrop_Loss+DFL_Loss+CIOU_Loss

[0229] When the above-mentioned objective loss function converges or its function value is less than a certain threshold, it is judged that the preset training conditions are met, and the current target model is used as an improved three-level lymphatic structure classification model and output, which can effectively detect and accurately classify TLS, thereby improving the model's detection accuracy for TLS.

[0230] In summary, a method for identifying tertiary lymphatic structures on pathological images obtains a target pathological image to detect whether there is a tertiary lymphatic structure on the image; and by retaining the resolution of the pathological image, an improved adaptive patch size algorithm is used to calculate the patch side length, the patch side length of the pathological image to be cut is determined, and the segmentation scale of Patches is adaptively calculated, the original image is segmented into multiple Patches according to the grid, and the target pathological image is segmented into several target sub-images according to the patch side length, so as to ensure that the size of the Patches can cover more of the image area of ​​the tertiary lymphatic structure while ensuring the resolution of the Patches as much as possible; the target sub-image is identified to obtain a detection result of whether the target sub-image has a tertiary lymphatic structure; and then the side length of the patch to be cut is determined by constructing an adaptive Patches side length calculation algorithm, thereby reducing the error caused by the particularity of the size of the tertiary lymphatic structure to the detection result, and a more suitable tertiary lymphatic structure detection field of view can be obtained without artificial prior setting, thereby improving the detection accuracy of the tertiary lymphatic structure on the pathological image.

[0231] The present application adaptively selects the side length of the Patches cutting according to the shape of the prior tertiary lymphatic structure (TLS), reduces the interference of human subjective factors, and can obtain a relatively reasonable side length of the Patches cutting.

[0232] The present application can better solve the pain points of existing high-resolution image target detection by adaptively restoring the segmented TLS located on the patch boundary, that is, when patches are cut, the TLS detection objects at the boundary will be segmented, resulting in repeated detection of the same TLS, and the detection accuracy of pathological structures such as tertiary lymphatic structures that require overall structure judgment for classification will be destroyed, interfering with statistical data and TLS classification results.

[0233] Since the morphological characteristics of TLS are of great significance in its classification, this application performs dual-channel joint training and gradient back propagation optimization by integrating the loss function of the TLS detection model and the loss function of the TLS classification model. The detection and classification are based on two data sources with different resolutions: low-resolution Patches-level TLS detection and high-resolution TLS-level classification. This ensures that the two loss functions are related to each other and belong to different data sources. By performing gradient back propagation optimization on them at the same time, the two tasks can promote each other and achieve an overall better detection effect.

[0234] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0235] Reference Figure 5 The present application also provides a device for identifying the three-level lymphatic structure on a pathological image. The device for identifying the three-level lymphatic structure on a pathological image corresponds one-to-one to the method for identifying the three-level lymphatic structure on a pathological image in the above embodiment. The device for identifying the three-level lymphatic structure on a pathological image includes:

[0236] A pathological image acquisition module, used for acquiring target pathological images;

[0237] A pathological image cutting module, used to determine the side length of a patch to be cut from a target pathological image, and cut the target pathological image into a plurality of target sub-images based on the side length of the patch;

[0238] A tertiary lymphatic structure detection module, used to identify the target sub-image and obtain a detection result of whether the target sub-image has a tertiary lymphatic structure;

[0239] The patch side length module is used to calculate the patch side length of the target pathological image to be cut using an improved adaptive patch size algorithm, including:

[0240] A statistical unit, used for obtaining the tertiary lymphatic structures on the historical pathological images and counting the size distribution data of the tertiary lymphatic structures;

[0241] A first clustering unit is used to perform a first clustering on each of the three-level lymphatic structures, exclude outliers, and calculate the mean length and the mean width of each of the three-level lymphatic structures except the outliers to obtain a first side length;

[0242] The second clustering unit is used to perform a second clustering on each tertiary lymphatic structure, classify and count each tertiary lymphatic structure according to a preset category, and sort each type of tertiary lymphatic structure according to the quantity, and select the first n types of tertiary lymphatic structures according to the sorting results to calculate the mean length and the mean width of all tertiary lymphatic structures in the first n types of tertiary lymphatic structures to obtain a second side length, wherein n is a positive integer and is ≥2;

[0243] A reference side length unit, used for selecting a maximum length mean and a maximum width mean to set the side length based on the first side length and the second side length, so as to obtain a reference side length;

[0244] The patch side length unit is used to preset a candidate side length queue, automatically match the patch side length according to the candidate side lengths in the candidate side length queue, until the reference side length is greater than the candidate side length, and use the candidate side length matched at this time as the patch side length.

[0245] A device for identifying the third-level lymphatic structure on a pathological image also includes:

[0246] The three-level lymphatic structure detection model module is used to obtain historical pathological images, determine the side length of the patch to be cut of the historical pathological images, and cut the historical pathological images according to the patch side length to obtain a number of first sample images; the first sample images are input into the deep learning model, and the deep learning model is trained in combination with the border regression loss function to output the detection result of whether there is a three-level lymphatic structure; when the preset training conditions are met, the current deep learning model is used as an improved three-level lymphatic structure detection model.

[0247] In one embodiment, the reference side length unit calculates the reference side length using the following formula:

[0248]

[0249] In the formula, represents the length and width of the first side, Represents the length and width of the second side, a∈[1,3).

[0250] A device for identifying the third-level lymphatic structure on a pathological image also includes:

[0251] A judging module, configured to judge whether the position of the tertiary lymphatic structure is located on the boundary of the target sub-image when the tertiary lymphatic structure exists on any of the target sub-images;

[0252] A repair module, configured to repair the target sub-image if the position of the tertiary lymphatic structure is located on the boundary of the target sub-image;

[0253] The tertiary lymphatic structure classification module is used to classify the tertiary lymphatic structure.

[0254] In one embodiment, the determination module includes:

[0255] An aspect ratio dictionary unit, used for creating an aspect ratio dictionary, wherein the aspect ratio dictionary stores the aspect ratios of the borders of the three-level lymphatic structures on the target sub-image in real time;

[0256] A reference unit, configured to extract a tertiary lymphatic structure whose center is located in a strip-shaped area whose width is shrunk inward by a preset number of pixels from the boundary of the target sub-image, and store the aspect ratio of the border of the tertiary lymphatic structure in the aspect ratio dictionary;

[0257] A search unit, configured to search, based on a preset outlier value, whether there is a third-level lymphatic structure whose aspect ratio product according to a preset multiple is greater than the outlier value among the aspect ratios of all the third-level lymphatic structures in the current aspect ratio dictionary;

[0258] The abnormal tertiary lymphatic structure identification unit is used to identify the tertiary lymphatic structure as an abnormal tertiary lymphatic structure if there is a tertiary lymphatic structure whose product of preset multiples of aspect ratio is greater than the outlier value, and at this time, determine that the position of the tertiary lymphatic structure is located on the boundary of the target sub-image.

[0259] In one embodiment, the repair module includes:

[0260] an attribution dictionary unit, configured to create an attribution dictionary, wherein the attribution dictionary sequentially generates target sub-images corresponding to the third-level lymphatic structures in the aspect ratio dictionary according to the storage order of the aspect ratios of the borders of the third-level lymphatic structures in the aspect ratio dictionary;

[0261] An indexing unit, configured to index the target sub-image corresponding to the abnormal tertiary lymphatic structure according to the attribution dictionary, so as to calculate a circumscribed rectangular frame of the abnormal tertiary lymphatic structure in the target sub-image;

[0262] The abnormal tertiary lymphatic structure positioning unit is used to solve the mean value of the length of the circumscribed rectangular frame and the length of the preset priori rectangular frame, and to solve the mean value of the width of the circumscribed rectangular frame and the width of the priori rectangular frame, so as to obtain the estimated detection frame of the tertiary lymphatic structure located on the boundary of the target sub-image.

[0263] A device for identifying the third-level lymphatic structure on a pathological image also includes:

[0264] The three-level lymphatic structure classification model module is used to crop the original pathological images containing the three-level lymphatic structure from the historical pathological images to obtain a number of second sample images; based on the deep learning model architecture, optimization is performed, and a high-precision image classification branch is added to the input end to obtain a target model, wherein the high-precision image classification branch is used to receive the second sample image; a cross-entropy loss function of the high-resolution three-level lymphatic structure image is designed, and a target loss function is constructed in combination with the border regression loss function of the deep learning model; the detection result of whether there is a three-level lymphatic structure and the second sample image are respectively input into the target model, and the target loss function is used to perform dual-channel joint training and gradient back propagation optimization to output the classification results of the three-level lymphatic structure; when the preset training conditions are met, the current target model is used as an improved three-level lymphatic structure classification model.

[0265] In one embodiment, the expression of the objective loss function of the three-level lymphatic structure classification model module is as follows:

[0266] Loss=Loss G +Loss2

[0267] Loss G =CrossEntropy_Loss

[0268] Loss2=DFL_Loss+CIOU_Loss

[0269] In the formula, Loss represents the target loss function, Loss G represents the cross entropy loss function, and the CrossEntropy_Loss function is selected. Loss2 represents the border regression loss function, DFL_Loss represents the distance field loss function, and CIOU_Loss represents the border loss function.

[0270] A device for identifying the third-level lymphatic structure on a pathological image also includes:

[0271] The tertiary lymphatic structure size standardization module is used to perform size standardization processing on the detection result of whether there is a tertiary lymphatic structure and the second sample image respectively, and then input the processed detection result of whether there is a tertiary lymphatic structure and the processed second sample image into the target model respectively.

[0272] The specific definition of a device for identifying a third-level lymphatic structure on a pathological image can be found in the above definition of a method for identifying a third-level lymphatic structure on a pathological image, which will not be repeated here.

[0273] Each module in the above-mentioned device for identifying the third-level lymphatic structure on a pathological image can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0274] In one embodiment, a computer device is provided, which may be a server. The computer device includes a processor, a memory, a network interface, and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the method for identifying the third-level lymphatic structure on any one of the above-mentioned pathological images is implemented.

[0275] In one embodiment, a computer-readable storage medium is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, any one of the above-mentioned methods for identifying the third-level lymphatic structure on the pathological image is implemented.

[0276] In one embodiment, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the computer program implements any of the above-mentioned methods for identifying the third-level lymphatic structure on a pathological image.

[0277] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. When the computer program is executed, it may include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0278] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above.

Claims

1. A method for identifying the third-level lymphatic structure in a pathological image, characterized in that: The following steps are included: Obtain target pathological images; Determine the side length of the patch to be cut of the target pathological image, and cut the target pathological image into a plurality of target sub-images based on the side length of the patch; Identify the target sub-image to obtain a detection result of whether the target sub-image has a tertiary lymphatic structure; Among them, the third-level lymphatic structure on the historical pathological image is obtained, and the size distribution data of the third-level lymphatic structure is counted; Perform the first clustering on each third-level lymphatic structure, exclude outliers, and calculate the mean length and width of each third-level lymphatic structure except the outliers to obtain the first side length; Perform a second clustering on each tertiary lymphatic structure, classify and count each tertiary lymphatic structure according to the preset category, and sort each type of tertiary lymphatic structure according to the size of the number. Select the first n types of tertiary lymphatic structures according to the sorting results to calculate the mean length and the mean width of all tertiary lymphatic structures in the first n types of tertiary lymphatic structures to obtain the second side length, where n is a positive integer and ≥2; Based on the first side length and the second side length, selecting the maximum length mean and the maximum width mean to set the side length, to obtain a reference side length; A candidate edge length queue is preset, and the patch edge length is automatically matched according to the candidate edge lengths in the candidate edge length queue until the reference edge length is greater than the candidate edge length, and the candidate edge length matched at this time is used as the patch edge length.

2. The method for identifying the third-level lymphatic structure on the pathological image according to claim 1, characterized in that: When identifying the target sub-image, an improved three-level lymphatic structure detection model is used. The training steps of the three-level lymphatic structure detection model include: Acquire a historical pathological image, determine the side length of a patch to be cut from the historical pathological image, and cut the historical pathological image according to the side length of the patch to obtain a plurality of first sample images; Inputting the first sample image into the deep learning model, training it in combination with the border regression loss function of the deep learning model, and outputting the detection result of whether there is a third-level lymphatic structure; When the preset training conditions are met, the current deep learning model is used as an improved three-level lymphatic structure detection model.

3. The method for identifying the third-level lymphatic structure on the pathological image according to claim 1, characterized in that: Based on the first side length and the second side length, the maximum length average and the maximum width average are selected to set the side length. When the reference side length is obtained, the reference side length is calculated using the following formula: In the formula, Indicates the length and width of the first side length, Represents the length and width of the second side, a∈[1,3).

4. The method for identifying the third-level lymphatic structure on a pathological image according to any one of claims 1 to 3, characterized in that: After obtaining the detection result of whether the target sub-image has a third-level lymphatic structure, the following steps are also included: When a tertiary lymphatic structure exists on any of the target sub-images, determining whether the position of the tertiary lymphatic structure is located on the boundary of the target sub-image; If the position of the tertiary lymphatic structure is located on the boundary of the target sub-image, the target sub-image is repaired to obtain a repaired tertiary lymphatic structure, and then the tertiary lymphatic structure is classified; If the position of the tertiary lymphatic structure is not on the boundary of the target sub-image, the tertiary lymphatic structure is classified.

5. The method for identifying the third-level lymphatic structure on the pathological image according to claim 4, characterized in that: The step of determining whether the position of the tertiary lymphatic structure is located on the boundary of the target sub-image comprises: Creating an aspect ratio dictionary, wherein the aspect ratio dictionary stores the aspect ratios of the borders of the three-level lymphatic structures on the target sub-image in real time; Extracting a tertiary lymphatic structure whose center is located in a strip-shaped area whose width is shrunk inward by a preset number of pixels from the boundary of the target sub-image, and storing the aspect ratio of the border of the tertiary lymphatic structure in the aspect ratio dictionary; Based on the preset outlier value, searching among the aspect ratios of all the tertiary lymphatic structures in the current aspect ratio dictionary whether there is a tertiary lymphatic structure whose aspect ratio product according to a preset multiple is greater than the outlier value; If there is a tertiary lymphatic structure whose product of preset multiples of aspect ratio is greater than the outlier value, the tertiary lymphatic structure is regarded as an abnormal tertiary lymphatic structure. At this time, it is determined that the position of the tertiary lymphatic structure is located on the boundary of the target sub-image.

6. The method for identifying the third-level lymphatic structure on the pathological image according to claim 5, characterized in that: When the target sub-image is repaired, an improved adaptive boundary entity processing algorithm is used, which includes the following steps: Creating an attribution dictionary, wherein the attribution dictionary sequentially generates target sub-images corresponding to the third-level lymphatic structures in the aspect ratio dictionary according to the storage order of the aspect ratios of the borders of the third-level lymphatic structures in the aspect ratio dictionary; According to the attribution dictionary, indexing the target sub-image corresponding to the abnormal tertiary lymphatic structure to calculate the circumscribed rectangular frame of the abnormal tertiary lymphatic structure in the target sub-image; The average value of the length of the circumscribed rectangular frame and the length of the preset priori rectangular frame is solved, and the average value of the width of the circumscribed rectangular frame and the width of the priori rectangular frame is solved to obtain the inferred detection frame of the third-level lymphatic structure located on the boundary of the target sub-image.

7. The method for identifying the third-level lymphatic structure on the pathological image according to claim 4, characterized in that: When the three-level lymphatic structure is classified, an improved three-level lymphatic structure classification model is used, wherein the training steps of the three-level lymphatic structure classification model include: Cutting out the original pathological image containing the third-level lymphatic structure from the historical pathological image to obtain a plurality of second sample images; Optimizing based on the deep learning model architecture, adding a high-precision image classification branch at the input end to obtain a target model, wherein the high-precision image classification branch is used to receive the second sample image; Design a cross entropy loss function for high-resolution three-level lymphatic structure images, combine it with the bounding box regression loss function of the deep learning model, and construct a target loss function; Inputting the detection result of whether there is a tertiary lymphatic structure and the second sample image into the target model respectively, and performing dual-channel joint training and gradient back propagation optimization using the target loss function to output the typing result of the tertiary lymphatic structure; When the preset training conditions are met, the current target model is used as an improved three-level lymphatic structure classification model.

8. The method for identifying the third-level lymphatic structure on the pathological image according to claim 7, characterized in that: The cross entropy loss function of the high-resolution three-level lymphatic structure image is designed, and the frame regression loss function of the deep learning model is combined to construct the target loss function, using the following formula: Loss=Loss G +Loss2 Loss G =CrossEntropy_Loss Loss2=DFL_Loss+CIOU_Loss Where Loss represents the target loss function, Loss G represents the cross entropy loss function, and the CrossEntropy_Loss function is selected. Loss2 represents the border regression loss function, DFL_Loss represents the distance field loss function, and CIOU_Loss represents the border loss function.

9. The method for identifying the third-level lymphatic structure on the pathological image according to claim 7, characterized in that: Before the step of inputting the detection result of whether there is a third-level lymphatic structure and the second sample image into the target model respectively, the method further includes: The detection result of whether there is a third-level lymphatic structure and the second sample image are respectively subjected to size standardization processing.

10. A device for identifying the third-level lymphatic structure in a pathological image, characterized in that: include, A pathological image acquisition module, used for acquiring target pathological images; A pathological image cutting module, used to determine the side length of a patch to be cut from a target pathological image, and cut the target pathological image into a plurality of target sub-images based on the side length of the patch; A tertiary lymphatic structure detection module, used to identify the target sub-image and obtain a detection result of whether the target sub-image has a tertiary lymphatic structure; The patch side length module is used to calculate the patch side length of the target pathological image to be cut using an improved adaptive patch size algorithm, including: A statistical unit, used for obtaining the third-level lymphatic structure on the historical pathological image and counting the size distribution data of the third-level lymphatic structure; A first clustering unit is used to perform a first clustering on each of the three-level lymphatic structures, exclude outliers, and calculate the mean length and the mean width of each of the three-level lymphatic structures except the outliers to obtain a first side length; The second clustering unit is used to perform a second clustering on each tertiary lymphatic structure, classify and count each tertiary lymphatic structure according to a preset category, and sort each type of tertiary lymphatic structure according to the quantity, and select the first n types of tertiary lymphatic structures according to the sorting results to calculate the mean length and the mean width of all tertiary lymphatic structures in the first n types of tertiary lymphatic structures to obtain a second side length, wherein n is a positive integer and is ≥2; A reference side length unit, used for selecting a maximum length mean and a maximum width mean to set the side length based on the first side length and the second side length, so as to obtain a reference side length; The patch side length unit is used to preset a candidate side length queue, automatically match the patch side length according to the candidate side lengths in the candidate side length queue, until the reference side length is greater than the candidate side length, and use the candidate side length matched at this time as the patch side length.

11. The device for identifying the third-level lymphatic structure on the pathological image according to claim 10, characterized in that: Also includes, The three-level lymphatic structure detection model module is used to obtain a historical pathological image, determine the side length of a patch to be cut from the historical pathological image, and cut the historical pathological image according to the side length of the patch to obtain a plurality of first sample images; The first sample image is input into the deep learning model, and training is performed in combination with the border regression loss function of the deep learning model to output the detection result of whether there is a third-level lymphatic structure; when the preset training conditions are met, the current deep learning model is used as an improved third-level lymphatic structure detection model.

12. The device for identifying the third-level lymphatic structure on the pathological image according to claim 10, characterized in that: The reference side length unit uses the following formula to calculate the reference side length: In the formula, Indicates the length and width of the first side length, Represents the length and width of the second side, a∈[1,3).

13. The device for identifying the third-level lymphatic structure on a pathological image according to any one of claims 10 to 12, characterized in that: Also includes, A judging module, configured to judge whether the position of the tertiary lymphatic structure is located on the boundary of the target sub-image when the tertiary lymphatic structure exists on any of the target sub-images; A repair module, configured to repair the target sub-image if the position of the tertiary lymphatic structure is located on the boundary of the target sub-image; The tertiary lymphatic structure classification module is used to classify the tertiary lymphatic structure.

14. The device for identifying the third-level lymphatic structure on the pathological image according to claim 13, characterized in that: The determination module includes, An aspect ratio dictionary unit, used for creating an aspect ratio dictionary, wherein the aspect ratio dictionary stores the aspect ratios of the borders of the three-level lymphatic structures on the target sub-image in real time; A reference unit, configured to extract a tertiary lymphatic structure whose center is located in a strip-shaped area whose width is shrunk inward by a preset number of pixels from the boundary of the target sub-image, and store the aspect ratio of the border of the tertiary lymphatic structure in the aspect ratio dictionary; A search unit, configured to search, based on a preset outlier value, whether there is a third-level lymphatic structure whose aspect ratio product according to a preset multiple is greater than the outlier value among the aspect ratios of all the third-level lymphatic structures in the current aspect ratio dictionary; The abnormal tertiary lymphatic structure identification unit is used to identify the tertiary lymphatic structure as an abnormal tertiary lymphatic structure if there is a tertiary lymphatic structure whose product of preset multiples of aspect ratio is greater than the outlier value, and at this time, determine that the position of the tertiary lymphatic structure is located on the boundary of the target sub-image.

15. The device for identifying the third-level lymphatic structure on the pathological image according to claim 14, characterized in that: The repair module includes, an attribution dictionary unit, configured to create an attribution dictionary, wherein the attribution dictionary sequentially generates target sub-images corresponding to the third-level lymphatic structures in the aspect ratio dictionary according to the storage order of the aspect ratios of the borders of the third-level lymphatic structures in the aspect ratio dictionary; An indexing unit, configured to index the target sub-image corresponding to the abnormal tertiary lymphatic structure according to the attribution dictionary, so as to calculate a circumscribed rectangular frame of the abnormal tertiary lymphatic structure in the target sub-image; The abnormal tertiary lymphatic structure positioning unit is used to solve the mean value of the length of the circumscribed rectangular frame and the length of the preset priori rectangular frame, and to solve the mean value of the width of the circumscribed rectangular frame and the width of the priori rectangular frame, so as to obtain the estimated detection frame of the tertiary lymphatic structure located on the boundary of the target sub-image.

16. The device for identifying the third-level lymphatic structure on the pathological image according to claim 13, characterized in that: Also includes, The three-level lymphatic structure classification model module is used to crop the original pathological images containing the three-level lymphatic structure from the historical pathological images to obtain a number of second sample images; based on the deep learning model architecture, a high-precision image classification branch is added to the input end to obtain a target model, wherein the high-precision image classification branch is used to receive the second sample image; a cross-entropy loss function of the high-resolution three-level lymphatic structure image is designed, and a target loss function is constructed in combination with the border regression loss function of the deep learning model; the detection result of whether there is a three-level lymphatic structure and the second sample image are respectively input into the target model, and the target loss function is used to perform dual-channel joint training and gradient back propagation optimization to output the classification results of the three-level lymphatic structure; when the preset training conditions are met, the current target model is used as an improved three-level lymphatic structure classification model.

17. The device for identifying the third-level lymphatic structure on the pathological image according to claim 16, characterized in that: The expression of the objective loss function of the three-level lymphatic structure classification model module is as follows: Loss=Loss G +Loss2 Loss G =CrossEntropy_Loss Loss2=DFL_Loss+CIOU_Loss Where Loss represents the target loss function, Loss G represents the cross entropy loss function, and the CrossEntropy_Loss function is selected. Loss2 represents the border regression loss function, DFL_Loss represents the distance field loss function, and CIOU_Loss represents the border loss function.

18. The device for identifying the third-level lymphatic structure on the pathological image according to claim 16, characterized in that: Also includes, The tertiary lymphatic structure size standardization module is used to perform size standardization processing on the detection result of whether there is a tertiary lymphatic structure and the second sample image respectively, and then input the processed detection result of whether there is a tertiary lymphatic structure and the processed second sample image into the target model respectively.

19. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Citation Information

Patent Citations

  • Pathological image processing method and device, computer equipment and storage medium

    CN113077440A

  • Three-level lymph structure identification method, device and equipment

    CN115063403A