Abnormal region detection and positioning system and method based on deep learning

Abstract

The invention relates to an abnormal region detection and positioning system and method based on deep learning, and the system comprises an obtaining module which is used for obtaining a to-be-detected digital pathological image group; the to-be-detected digital pathological image group comprises a to-be-detected digital pathological image and a depth image; the training module is used for training a deep learning detection model; the determination module is used for inputting the to-be-detected digital pathological image group into the deep learning detection model and determining a detection result and a positioning result of an abnormal region in the to-be-detected digital pathological image; by adopting the system and the method, a large amount of pathological image data can be efficiently processed, and the position information and the detection result of the abnormal area can be accurately positioned in the image, so that the accurate detection of the abnormal area is realized.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a system and method for detecting and locating abnormal regions based on deep learning. Background Art

[0002] In the field of digital pathology images, abnormal area detection and location is an important technical task. There may be various abnormal areas in pathology images, such as tumors, inflammation, cell mutations, etc. Accurately detecting, locating and displaying these abnormal areas is of great significance for pathological diagnosis and treatment.

[0003] Traditional abnormal region detection methods for pathological images usually rely on manual annotation or rule-based feature extraction methods. However, manual annotation requires a lot of manpower and time, and there are problems of subjectivity and poor consistency. Rule-based feature extraction methods often have poor effects on different types of abnormal regions and cannot adapt to complex and changeable pathological images. In addition, many products cannot accurately detect and locate abnormal regions, and there is an urgent need for a system and method that can simultaneously detect and locate abnormal regions. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems in the above-mentioned technology to a certain extent. To this end, the present invention aims to automatically detect and locate abnormal areas and accurately locate abnormal areas in digital pathological images.

[0005] To achieve the above objectives, the embodiment of the present invention proposes an abnormal area detection and positioning system based on deep learning, including:

[0006] An acquisition module, used for acquiring a group of digital pathology images to be detected; the group of digital pathology images to be detected includes a digital pathology image to be detected and a depth image;

[0007] Training module, used to train deep learning detection models;

[0008] The determination module is used to input the digital pathology image group to be detected into the deep learning detection model to determine the detection results and positioning results of the abnormal areas in the digital pathology images to be detected.

[0009] Preferably, the training module includes:

[0010] A data acquisition submodule is used to acquire a sample digital pathology image data set; the sample digital pathology image data set includes a sample digital pathology image, a sample depth image, an abnormal region positioning label and an abnormality degree classification label corresponding to the sample digital pathology image data set;

[0011] A feature extraction submodule is used to extract features from a sample digital pathology image data set to obtain feature point information;

[0012] The training submodule is used to input the extracted feature point information, the abnormal area positioning labels and the abnormal degree classification labels corresponding to the sample digital pathology image data group into the deep learning convolutional neural network for training to obtain a deep learning detection model.

[0013] Preferably, the feature extraction submodule includes:

[0014] A first feature extraction unit is used to obtain the gray value of each pixel in the sample digital pathology image to obtain a first feature point;

[0015] A second feature extraction unit is used to obtain gradient information of each pixel point in the sample digital pathology image to obtain second feature point information;

[0016] A third feature extraction unit is used to obtain the normalized depth information of each pixel in the sample depth image to obtain third feature point information;

[0017] The determining unit is used to determine the feature point information in the sample data group based on the first feature point information, the second feature point information and the third feature point information.

[0018] Preferably, the training submodule includes:

[0019] An abnormality localization unit is used to input the extracted feature point information and the abnormal area localization label corresponding to the sample digital pathology image data group into a deep learning convolutional neural network for training to obtain an abnormal area localization sub-model;

[0020] The abnormality degree unit is used to input the extracted feature point information and the abnormality degree classification labels corresponding to the sample digital pathology image data group into the deep learning convolutional neural network for training to obtain an abnormality degree sub-model;

[0021] A generation unit is used to obtain a deep learning detection model based on the abnormal area positioning sub-model and the abnormal degree sub-model.

[0022] Preferably, the generating unit further includes:

[0023] A first loss calculation subunit, used to calculate a first loss corresponding to the abnormal area positioning submodel;

[0024] A second loss calculation subunit, used to calculate a second loss corresponding to the abnormality degree submodel;

[0025] Generate subunits for:

[0026] Adjusting the model parameters of the abnormal region positioning sub-model based on the first loss, and obtaining the abnormal region positioning sub-model when it is determined that the abnormal region positioning sub-model converges;

[0027] Adjusting the model parameters of the abnormality degree sub-model based on the second loss, and obtaining the abnormality degree sub-model when it is determined that the abnormality degree sub-model converges;

[0028] A deep learning detection model is obtained based on the abnormal area positioning sub-model and the abnormality degree sub-model.

[0029] Preferably, the first loss is calculated as follows:

[0030]

[0031] Among them, L A represents the first loss corresponding to the training of the A-th target sample digital pathology image for calculating the abnormal area localization sub-model; IoU is the intersection over union ratio, b p 、b gt The prediction frame is obtained from the training target sample digital pathology image, and the coordinates of the center point of the real target frame are obtained from the target sample digital pathology image; w c h represents the width of the minimum bounding box between the predicted box obtained by the training target sample digital pathology image and the real target box obtained by the target sample digital pathology image; c It represents the height of the minimum bounding box between the predicted box obtained by the training target sample digital pathology image and the real target box obtained by the target sample digital pathology image; d(b p , b gt ) represents the Euclidean distance between the center point of the predicted box obtained from the training target sample digital pathology image and the center point of the real target box obtained from the target sample digital pathology image; w p represents the width of the prediction box obtained from the training target sample digital pathology image; h p represents the height of the prediction box obtained from the training target sample digital pathology image; w gt represents the width of the real box obtained from the target sample digital pathology image; h gt The height of the true box obtained by representing the target sample digital pathology image.

[0032] Preferably, the second loss is calculated as follows:

[0033]

[0034] Among them, L B represents the second loss obtained by the B-th target sample digital pathology image, and M represents the total number of pixels included in the B-th target sample digital pathology image; ∑ xy () represents traversing every pixel in the target sample digital pathology image; T xy represents the loss weight of the pixel at the coordinate (x, y); c represents the total number of abnormality classifications; ∑ c () represents the sum of the losses of each abnormality classification in the target sample digital pathology image; Yxy Z represents the probability that the pixel at (x, y) belongs to a certain abnormality level, determined according to the abnormality level classification label annotated by the sample digital pathology image data set; xy It represents the probability that the pixel point at (x, y) belongs to a category of abnormality, determined according to the abnormality classification label predicted by the sample digital pathology image data set.

[0035] Preferably, it further comprises: a noise reduction module, which is used to perform noise reduction on the digital pathology image group to be detected before the determination module inputs the digital pathology image group to be detected into the deep learning detection model;

[0036] The noise reduction module comprises:

[0037] Noise detection unit, used to:

[0038] Grayscale the digital pathology images to be detected in the group of digital pathology images to be detected to obtain a grayscale image, and obtain the grayscale value of each pixel in the grayscale image;

[0039] Randomly select a pixel point in the grayscale image to obtain a first pixel point; determine a first target area with the first pixel point as the center and a first preset distance as the radius; determine a second target area with the first pixel point as the center and a second preset distance as the radius; the second preset distance is greater than the first preset distance;

[0040] Take any pixel point in the second target area after removing the first target area to obtain a second pixel point; determine a second target sub-area with the second pixel point as the center and the first preset distance as the radius; the second target sub-area has no intersection with the first target area;

[0041] Obtaining gray level co-occurrence matrices corresponding to the first target region and the second target sub-region respectively; determining a first texture feature vector corresponding to the first target region based on the gray level co-occurrence matrix corresponding to the first target region; determining a second texture feature vector corresponding to the second target sub-region based on the gray level co-occurrence matrix corresponding to the second target sub-region;

[0042] Determining texture similarity between the first target region and the second target sub-region based on the first texture feature vector and the second texture feature vector;

[0043] Traversing each pixel point in the second target sub-region to obtain a number of texture similarities;

[0044] When it is determined that all texture similarities are less than a preset texture similarity threshold, the first pixel point is a noise point;

[0045] Traversing each pixel point in the grayscale image, determining all noise points in the grayscale image; based on all noise points in the grayscale image, determining noise points in the digital pathology image to be detected;

[0046] Noise reduction unit for:

[0047] Randomly select a noise point in the digital pathology image to be detected to obtain a noise point to be processed;

[0048] Taking the noise point to be processed as the center, determine a region to be processed of size M×N; wherein M and N are both odd numbers and both M and N are greater than 1;

[0049] Calculate the similarity distance between the noise point to be processed and other pixel points in the area to be processed; determine the texture intensity value of the noise point to be processed based on the maximum similarity distance;

[0050] Preset the maximum and minimum values ​​of noise reduction;

[0051] Determine the noise reduction weight of each pixel in the area to be processed based on the texture intensity value and the preset maximum and minimum noise reduction amplitudes;

[0052] Determine the grayscale value of the noise point to be processed after denoising based on the denoising weight of each pixel point in the area to be processed;

[0053] All noise points in the digital pathology image to be detected are traversed to obtain a denoised image of the digital pathology image to be detected.

[0054] Preferably, the third feature extraction unit comprises:

[0055] The feature point extraction subunit is used to perform corner point detection on each pixel in the sample depth image using the Harris algorithm to obtain a set of feature points of the sample depth image;

[0056] The depth value determination subunit is used to obtain the depth value of each sample depth image feature point in the sample depth image feature point set; determine the maximum depth value and the minimum depth value based on the depth value of each sample depth image feature point;

[0057] The normalization subunit is used to determine the normalized depth feature value according to the depth value, the maximum depth value and the minimum depth value of each sample depth image feature point, and determine the third feature point information based on the depth feature value.

[0058] Preferably, a method for detecting and locating abnormal areas based on deep learning comprises:

[0059] Acquire a group of digital pathology images to be detected; the group of digital pathology images to be detected includes a digital pathology image to be detected and a depth image;

[0060] Training deep learning detection models;

[0061] The digital pathology image group to be detected is input into the deep learning detection model to determine the detection results and positioning results of the abnormal areas in the digital pathology images to be detected.

[0062] The beneficial effects of the present invention are as follows: through the training of the deep learning model, the system can automatically learn and extract the features of abnormal areas in digital pathology images, thereby realizing efficient and accurate detection and positioning of the digital pathology images to be detected; the deep learning model can effectively utilize the depth information of the pathology image and comprehensively consider the global context information, thereby capturing more comprehensive abnormal area features; since the deep learning model has good generalization ability, the system can process digital pathology images of various types and complexities and has good robustness; through the training of the deep learning model and the automated detection process, the system can quickly and reliably detect and locate abnormal areas in pathology images without human intervention; it can effectively improve the analysis efficiency and accuracy of digital pathology images, and has important application value for the automatic diagnosis and treatment of diseases in the field of medical imaging.

[0063] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0064] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0066] Figure 1 is a block diagram of an abnormal area detection and positioning system based on deep learning according to an embodiment of the present invention;

[0067] Figure 2 is a block diagram of a training module according to one embodiment of the present invention;

[0068] Figure 3 is a block diagram of a feature extraction submodule according to an embodiment of the present invention. DETAILED DESCRIPTION

[0069] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0070] Example 1

[0071] like Figure 1 As shown, a deep learning-based abnormal area detection and positioning system includes:

[0072] An acquisition module, used for acquiring a group of digital pathology images to be detected; the group of digital pathology images to be detected includes a digital pathology image to be detected and a depth image;

[0073] Training module, used to train deep learning detection models;

[0074] The determination module is used to input the digital pathology image group to be detected into the deep learning detection model to determine the detection results and positioning results of the abnormal areas in the digital pathology images to be detected.

[0075] The beneficial effects of the above technical solution are as follows: through the training of the deep learning model, the system can automatically learn and extract the features of abnormal areas in digital pathology images, thereby realizing efficient and accurate detection and positioning of the digital pathology images to be detected; the deep learning model can effectively utilize the depth information of the pathology image and comprehensively consider the global context information to capture more comprehensive abnormal area features; because the deep learning model has good generalization ability, the system can process digital pathology images of various types and complexities and has good robustness; through the training of the deep learning model and the automated detection process, the system can quickly and reliably detect and locate abnormal areas in pathology images without human intervention; it can effectively improve the analysis efficiency and accuracy of digital pathology images, and has important application value for the automatic diagnosis and treatment of diseases in the field of medical imaging.

[0076] Example 2

[0077] like Figure 2 As shown, the training module includes:

[0078] A data acquisition submodule is used to acquire a sample digital pathology image data set; the sample digital pathology image data set includes a sample digital pathology image, a sample depth image, an abnormal region positioning label and an abnormality degree classification label corresponding to the sample digital pathology image data set;

[0079] A feature extraction submodule is used to extract features from a sample digital pathology image data set to obtain feature point information;

[0080] The training submodule is used to input the extracted feature point information, the abnormal area positioning labels and the abnormal degree classification labels corresponding to the sample digital pathology image data group into the deep learning convolutional neural network for training to obtain a deep learning detection model.

[0081] The beneficial effects of the above technical solution are as follows: the data acquisition submodule can acquire a sample digital pathology image data set, including digital pathology images, depth images, abnormal area positioning labels and abnormal degree classification labels; the information of digital pathology images and depth images is comprehensively utilized to make the training data more comprehensive and rich; the feature extraction submodule can extract feature point information from the sample digital pathology image data set; the deep learning network can extract more representative features based on the characteristics of digital pathology images and depth images, and improve the accuracy of abnormal area detection and positioning; the training submodule can input the extracted feature point information, abnormal area positioning labels and abnormal degree classification labels into the deep learning convolutional neural network. Through training in the network; through training, the deep learning detection model can learn the characteristics and positioning methods of abnormal areas in digital pathology images, and realize accurate detection and positioning of abnormal areas; through the deep learning training process, the system can automatically learn and optimize features based on a large amount of sample data, thereby improving the accuracy and robustness of abnormal area detection and positioning; it can handle the changes and abnormal characteristics of different pathological images and adapt to diverse disease conditions; through multi-source data acquisition, comprehensive feature extraction and deep learning training, it can effectively improve the accuracy and reliability of abnormal area detection and positioning of digital pathology images, which has important application significance for assisting doctors in diagnosing and treating diseases.

[0082] Example 3

[0083] like Figure 3 As shown, the feature extraction submodule includes:

[0084] A first feature extraction unit is used to obtain the gray value of each pixel in the sample digital pathology image to obtain first feature point information;

[0085] A second feature extraction unit is used to obtain gradient information of each pixel point in the sample digital pathology image to obtain second feature point information;

[0086] A third feature extraction unit is used to obtain the normalized depth information of each pixel in the sample depth image to obtain third feature point information;

[0087] The determining unit is used to determine the feature point information in the sample data group based on the first feature point information, the second feature point information and the third feature point information.

[0088] In this embodiment, the specific implementation method for determining the first feature point information is that after obtaining the grayscale value, an edge detection algorithm (such as Canny edge detection) can be used to extract edge feature points in the image, and the first feature point information is determined based on the edge feature points. The first feature point is the edge point in the sample digital pathology image, the second feature point is the point where the edge gradient changes with respect to the pixel point in the sample digital pathology image, and the third feature point is the corner point of the depth change in the sample depth image.

[0089] In this embodiment, the specific implementation method of obtaining the gradient information of each pixel in the sample digital pathology image and obtaining the second feature point information is as follows: for example, for each pixel, convolution operation is performed on it with surrounding pixels to calculate the gradient. The commonly used convolution operator is the Sobel operator, which calculates the gradient in the horizontal direction and the vertical direction respectively;

[0090] Gradient calculation in the horizontal direction: Use the Sobel operator in the horizontal direction, usually:

[0091] gx=[[-1,0,1],

[0092] [-2,0,2],

[0093] [-1,0,1]]

[0094] The Sobel operator is convolved with the image to obtain the gradient value in the horizontal direction. gy=[[-1,-2,-1],

[0095] [0,0,0],

[0096] [1,2,1]]

[0097] The Sobel operator is convolved with the image to obtain the gradient value in the vertical direction.

[0098] According to the gradient values ​​in the horizontal and vertical directions, the gradient amplitude and direction of the pixel point can be calculated. Generally, the following formula is used for calculation:

[0099]

[0100] In this embodiment, the feature point information in the sample data set is composed based on the edge feature information of the first feature point, the gradient information of the second feature point and the depth information of the third feature point.

[0101] In this embodiment, the specific implementation method of the third feature point information is: using the Harris corner detection feature extraction method, a local detection window is designed on the image, and when the window moves slightly in all directions, the average grayscale change of the window is examined; when the average grayscale change exceeds the set threshold, the central pixel of the window is extracted as the corner point; in this way, a series of feature points can be extracted from the sample depth image, and their depth values ​​can be further obtained; then the maximum and minimum depth values ​​are determined based on these depth values, and finally normalized according to these depth values ​​and their maximum and minimum values ​​to obtain normalized depth feature values; for the detection The measured feature points need to be described for subsequent matching and recognition; some feature description algorithms, such as SIFT feature description, SURF feature description, ORB feature description, etc., can be used to calculate the local feature vectors around the feature points to describe the appearance and context information of the feature points; the feature points in images of different depths are matched to find the corresponding feature point pairs; distance measurement, similarity measurement and other methods can be used for matching; the matched feature point pairs are screened and optimized to remove unreliable matching points and improve the matching accuracy; the third feature point information can be obtained by screening and optimizing according to distance, similarity, geometric constraints, consistency and other standards.

[0102] The beneficial effects of the above technical solution are as follows: the first feature extraction unit can obtain the grayscale value of each pixel in the sample digital pathology image, which is very important for describing the brightness and contrast information of the image; the second feature extraction unit can obtain the gradient information of each pixel in the sample digital pathology image, which is very important for describing the edge, texture and other detail information of the image; the third feature extraction unit can obtain the normalized depth information of each pixel in the sample depth image, which is very important for describing the depth change of the image; the final feature point information is determined based on the first feature point information, the second feature point information and the third feature point information, and different types of features can be comprehensively utilized to improve the robustness and identification ability of the features. By combining multiple features, the features of the image can be described more accurately, and the accuracy and reliability of subsequent tasks can be improved;

[0103] Example 4

[0104] The training submodule comprises:

[0105] An abnormality localization unit is used to input the extracted feature point information and the abnormal area localization label corresponding to the sample digital pathology image data group into a deep learning convolutional neural network for training to obtain an abnormal area localization sub-model;

[0106] The abnormality degree unit is used to input the extracted feature point information and the abnormality degree classification labels corresponding to the sample digital pathology image data group into the deep learning convolutional neural network for training to obtain an abnormality degree sub-model;

[0107] A generation unit is used to obtain a deep learning detection model based on the abnormal area positioning sub-model and the abnormal degree sub-model.

[0108] In this embodiment, the extracted feature point information and the abnormal area positioning labels corresponding to the sample digital pathology image data group are input into the deep learning convolutional neural network for training. One implementation method may be: the feature point information extracted from the digital pathology image and the abnormal area positioning labels are matched one by one, and for each sample digital pathology image, the abnormal area positioning label is associated with it; the feature point information and the abnormal area positioning label are organized into a data set; the data can be organized into a tuple or a dictionary, where each sample contains image data, feature point information and the corresponding abnormal area positioning label; the data set is loaded into a deep learning convolutional neural network (CNN) model based on the YOLO algorithm for iterative training.

[0109] In this embodiment, the extracted feature point information and the abnormality degree classification labels corresponding to the sample digital pathology image data group are input into the deep learning convolutional neural network for training. One implementation method may be: the feature point information, digital pathology image data and abnormality degree classification labels are organized into a training set. Ensure that the feature point information of each sample corresponds one-to-one with the corresponding image data and classification labels. Divide the training set into training data and verification data. The training data can usually be divided at a ratio of 70-80%, and the rest is used to verify the network performance; the digital pathology image data is passed into the CNN model as input; and the training set is used to iteratively train the deep learning convolutional neural network (CNN) model based on the YOLO algorithm.

[0110] The beneficial effects of the above technical solution are as follows: through training, the abnormality localization sub-model can improve the ability to accurately locate abnormal areas, help to accurately locate abnormal parts in digital pathology images, and help doctors or diagnostic personnel quickly and accurately locate abnormal parts, providing important references for subsequent diagnosis and treatment; the normal degree sub-model can accurately judge the severity of the abnormality through training, and can provide more fine-grained abnormality degree classification results, which helps doctors or diagnostic personnel to conduct in-depth analysis of abnormal cases; the deep learning detection model can simultaneously consider the information of abnormal area positioning and abnormality degree classification, thereby improving the accuracy and efficiency of abnormality detection; comprehensively consider the information of abnormal area positioning and abnormality degree, thereby providing comprehensive abnormality evaluation and prediction results, which helps doctors or diagnostic personnel to formulate early intervention strategies.

[0111] Example 5

[0112] The generating unit further includes:

[0113] A first loss calculation subunit, used to calculate a first loss corresponding to the abnormal area positioning submodel;

[0114] A second loss calculation subunit, used to calculate a second loss corresponding to the abnormality degree submodel;

[0115] Generate subunits for:

[0116] Adjusting the model parameters of the abnormal region positioning sub-model based on the first loss, and obtaining the abnormal region positioning sub-model when it is determined that the abnormal region positioning sub-model converges;

[0117] Adjusting the model parameters of the abnormality degree sub-model based on the second loss, and obtaining the abnormality degree sub-model when it is determined that the abnormality degree sub-model converges;

[0118] A deep learning detection model is obtained based on the abnormal area positioning sub-model and the abnormality degree sub-model.

[0119] In this embodiment, the first loss is calculated based on the difference between the abnormal region positioning label and the model prediction output, and the purpose is to measure the error of the model in the abnormal region positioning task.

[0120] In this embodiment, the second loss is calculated based on the difference between the abnormality level classification label and the model prediction output, and is intended to measure the error of the model in the abnormality level classification task.

[0121] In this embodiment, the sub-unit adjusts the parameters of the abnormal region positioning sub-model and the abnormal degree sub-model according to the first loss and the second loss; and updates the parameters through back propagation so that the model gradually converges to the optimal state during the training process.

[0122] The above technical solution has the following beneficial effects: it comprehensively considers the information of abnormal area location and abnormal degree to provide more accurate and comprehensive abnormality detection results; the adjustment of model parameters can further improve the performance and robustness of the model; through iterative optimization of training and generation sub-units, a deep learning detection model with better performance is finally obtained.

[0123] Example 6

[0124] The first loss is calculated as:

[0125]

[0126] Among them, L A represents the first loss corresponding to the training of the A-th target sample digital pathology image to calculate the abnormal area localization sub-model, IoU is the intersection over union ratio, b p 、bgt The prediction frame is obtained from the training target sample digital pathology image, and the coordinates of the center point of the real target frame are obtained from the target sample digital pathology image; w c h represents the width of the minimum bounding box between the predicted box obtained by the training target sample digital pathology image and the real target box obtained by the target sample digital pathology image; c It represents the height of the minimum bounding box between the predicted box obtained by the training target sample digital pathology image and the real target box obtained by the target sample digital pathology image; d(b p , b gt ) represents the Euclidean distance between the center point of the predicted box obtained from the training target sample digital pathology image and the center point of the real target box obtained from the target sample digital pathology image; w p represents the width of the prediction box obtained from the training target sample digital pathology image; h p represents the height of the prediction box obtained from the training target sample digital pathology image; w gt represents the width of the real box obtained from the target sample digital pathology image; h gt The height of the true box obtained by representing the target sample digital pathology image.

[0127] In this embodiment, L1 is the center point loss between the predicted box obtained by the training target sample digital pathology image and the center point of the real target box obtained by the target sample digital pathology image. By normalizing the distance between the center points, a closer prediction box is assigned to the abnormal area to improve the positioning accuracy of the center point;

[0128] In this embodiment, L2 is the edge length loss between the predicted box obtained from the training target sample digital pathology image and the real target box obtained from the target sample digital pathology image; The width loss between the predicted box obtained for the training target sample digital pathology image and the real target box obtained for the target sample digital pathology image; There is a high loss between the predicted box obtained for the training target sample digital pathology image and the real target box obtained for the target sample digital pathology image.

[0129] The above technical solution has the following beneficial effects: the calculation of the first loss can provide an indicator to measure the performance of the model in the abnormal area positioning task. By monitoring and optimizing the first loss, the performance of the model can be continuously fed back to improve the positioning precision and accuracy; the first loss, as an objective function in the training process, can help adjust the model parameters to minimize the error of abnormal area positioning; by reasonably setting the weights of the loss function and the optimization algorithm, the training strategy can be improved to improve the convergence speed and stability of the model; by accurately calculating the first loss and optimizing it, the performance of the abnormal area positioning sub-model can be improved, so that it can more accurately locate abnormal areas in digital pathology images; this helps doctors or diagnosticians make more accurate diagnosis and treatment decisions.

[0130] Example 7

[0131] The second loss is calculated as:

[0132]

[0133] Among them, L B represents the second loss obtained by the B-th target sample digital pathology image, and M represents the total number of pixels included in the B-th target sample digital pathology image; ∑ xy () represents traversing every pixel in the target sample digital pathology image; T xy represents the loss weight of the pixel at the coordinate (x, y); c represents the total number of abnormality classifications; ∑ c () represents the sum of the losses of each abnormality classification in the target sample digital pathology image; Y xy Z represents the probability that the pixel at (x, y) belongs to a certain abnormality level, determined according to the abnormality level classification label annotated by the sample digital pathology image data set; xy It represents the probability that the pixel point at (x, y) belongs to a category of abnormality, determined according to the abnormality classification label predicted by the sample digital pathology image data set.

[0134] The above technical solution has the following beneficial effects: By comparing the difference between the model's prediction results and the true labels, the model can be helped to better understand and learn the different abnormalities in digital pathology images. By optimizing the second loss, the accuracy and robustness of the abnormality classification model can be effectively improved; the abnormality level is one of the important features of abnormal areas in digital pathology images, and accurately predicting the abnormality level is crucial for doctors or diagnosticians to make correct diagnosis and treatment decisions; by calculating the second loss and optimizing it, the prediction accuracy of the abnormality sub-model can be improved, so that it can more accurately predict the degree of abnormal areas in digital pathology images; the calculation and optimization of the second loss can enhance the performance and accuracy of the abnormality classification model, improve the accuracy of abnormality prediction, and also promote the generalization ability of the model, providing more accurate and reliable assistance for the analysis and diagnosis of digital pathology images.

[0135] Example 8

[0136] It also includes: a noise reduction module, which is used to reduce the noise of the digital pathology image group to be detected before the determination module inputs the digital pathology image group to be detected into the deep learning detection model;

[0137] The noise reduction module comprises:

[0138] Noise detection unit, used to:

[0139] Grayscale the digital pathology images to be detected in the group of digital pathology images to be detected to obtain a grayscale image, and obtain the grayscale value of each pixel in the grayscale image;

[0140] Randomly select a pixel point in the grayscale image to obtain a first pixel point; determine a first target area with the first pixel point as the center and a first preset distance as the radius; determine a second target area with the first pixel point as the center and a second preset distance as the radius; the second preset distance is greater than the first preset distance;

[0141] Take any pixel point in the second target area after removing the first target area to obtain a second pixel point; determine a second target sub-area with the second pixel point as the center and the first preset distance as the radius; the second target sub-area has no intersection with the first target area;

[0142] Obtaining gray level co-occurrence matrices corresponding to the first target region and the second target sub-region respectively; determining a first texture feature vector corresponding to the first target region based on the gray level co-occurrence matrix corresponding to the first target region; determining a second texture feature vector corresponding to the second target sub-region based on the gray level co-occurrence matrix corresponding to the second target sub-region;

[0143] Determining texture similarity between the first target region and the second target sub-region based on the first texture feature vector and the second texture feature vector;

[0144] Traversing each pixel point in the second target sub-region to obtain a number of texture similarities;

[0145] When it is determined that all texture similarities are less than a preset texture similarity threshold, the first pixel point is a noise point;

[0146] Traverse every pixel in the grayscale image and determine all the noise points in the grayscale image;

[0147] Based on all the noise points in the grayscale image, determining the noise points in the digital pathology image to be detected;

[0148] Noise reduction unit for:

[0149] Randomly select a noise point in the digital pathology image to be detected to obtain a noise point to be processed;

[0150] Taking the noise point to be processed as the center, determine a region to be processed of size M×N; wherein M and N are both odd numbers and both M and N are greater than 1;

[0151] Calculate the similarity distance between the noise point to be processed and other pixel points in the area to be processed; determine the texture intensity value of the noise point to be processed based on the maximum similarity distance;

[0152] Preset the maximum and minimum values ​​of noise reduction;

[0153] Determine the noise reduction weight of each pixel in the area to be processed based on the texture intensity value and the preset maximum and minimum noise reduction amplitudes;

[0154] Determine the grayscale value of the noise point to be processed after denoising based on the denoising weight of each pixel point in the area to be processed;

[0155] All noise points in the digital pathology image to be detected are traversed to obtain a denoised image of the digital pathology image to be detected.

[0156] In this embodiment, the digital pathology image to be detected contains color information of three channels: red, green, and blue, while the grayscale image contains only brightness information of one channel. By graying, the color information of the image can be simplified to a single brightness value, reducing the complexity of the information and facilitating a better understanding and analysis of the image; the pixel value in the grayscale image only represents the brightness information, and the color information is ignored; this can make the image contrast more prominent and the contours and details of the target object clearer and more definite; compared with color images, the amount of data in grayscale images is reduced by 3 times, which means that operations and calculations can be performed more quickly during image processing and calculation, reducing the complexity and time cost of processing.

[0157] In this embodiment, the second preset distance is set to 3 times the first preset distance.

[0158] In this embodiment, the second target sub-region has no intersection with the first target region, which greatly reduces the amount of calculation while ensuring the noise point judgment result, and has outstanding substantive characteristics and significant progress.

[0159] In this embodiment, the gray level co-occurrence matrix is ​​a common method for describing texture by studying the spatial correlation characteristics of gray levels. The gray level co-occurrence matrix is ​​obtained by statistically analyzing the conditions where two pixels at a certain distance on an image have a certain gray level.

[0160] For example, take any point (x, y) in the image (N×N) and another point (x+a, y+b) that deviates from it, and let the grayscale value of the point pair be (g1, g2). Let the point (x, y) move across the entire screen, and you will get various (g1, g2) values. Let the number of grayscale values ​​be k, and there are k square combinations of (g1, g2). For the entire screen, count the number of times each (g1, g2) value appears, and then arrange them into a square matrix. Then use the total number of times (g1, g2) appears to normalize them to the probability of occurrence P(g1, g2). Such a square matrix is ​​called a grayscale co-occurrence matrix. Different numerical combinations of distance difference values ​​(a, b) can get joint probability matrices under different circumstances. The value of (a, b) should be selected according to the characteristics of the texture periodic distribution. For finer textures, small difference values ​​such as (1, 0), (1, 1), and (2, 0) are selected. When a=1, b=0, the pixel pair is horizontal, i.e., 0 degree scanning; when a=0, b=1, the pixel pair is vertical, i.e., 90 degree scanning; when a=1, b=1, the pixel pair is right diagonal, i.e., 45 degree scanning; when a=-1, b=1, the pixel pair is left diagonal, i.e., 135 degree scanning; in this way, the probability of two pixel gray levels occurring at the same time converts the spatial coordinates of (x, y) into a description of a "grayscale pair" (g1, g2), forming a grayscale co-occurrence matrix.

[0161] The gray-level co-occurrence matrix can reflect the texture features of local areas in the image, such as the directionality, coarseness and uniformity of the texture; these texture features are of great significance for image analysis and recognition; each element in the gray-level co-occurrence matrix represents the probability of two gray levels appearing at the same time; this statistical information can be used for further image analysis and processing, such as texture classification, image recognition, etc.; the gray-level co-occurrence matrix can be used for contrast enhancement. If the values ​​in the gray-level co-occurrence matrix are concentrated in a certain area (for example, for continuous gray-value images, the values ​​are concentrated in the diagonal; for structured images, the values ​​are concentrated in the position deviating from the diagonal), the gray-level co-occurrence matrix has a larger value. If the values ​​in G are distributed more evenly (such as images with severe noise), ASM has a smaller value; the gray-level co-occurrence matrix reflects the position distribution characteristics between pixels with the same or close brightness. It not only reflects the distribution characteristics of brightness, but also reflects the spatial dependence of the image. This helps us understand and analyze images more comprehensively.

[0162] In this embodiment, the specific implementation method of determining the texture similarity between the first target region and the second target sub-region is as follows:

[0163]

[0164] In this embodiment, T AB represents the texture similarity between the first target region and the second target sub-region; A(1,m) is the mth element in the texture feature vector corresponding to the first target region, B(1,p) is the pth element in the texture feature vector corresponding to the first target region, n represents the total number of elements in the texture feature vector; A represents the first target region; B represents the second target sub-region.

[0165] In this embodiment, the similarity distance between the noise point to be processed and other pixel points in the area to be processed is calculated, and the specific implementation method is Euclidean distance.

[0166] In this embodiment, the texture intensity value is calculated as follows:

[0167] Q(x,y)=m*D max +(1-m)*P max

[0168] Where Q(x, y) represents the texture intensity value at the noise point (x, y); m is a preset value, and the value range of m is (0, 1). max represents the maximum similarity distance between the noise point to be processed and other pixels in the area to be processed; P max =max{Q(x-1,y-1), Q(x-,y), Q(x,y-1), Q(x,y+1)}.

[0169] In this embodiment, the maximum and minimum values ​​of the noise reduction amplitude are preset to ensure that the noise reduction degrees of various regions of the entire image do not differ too much.

[0170] In this embodiment, the noise reduction weight of each pixel point in the area to be processed is determined based on the texture intensity value and the preset maximum and minimum values ​​of the noise reduction amplitude; the calculation method is:

[0171]

[0172] Among them, T represents the noise reduction weight of the noise point, Q max Indicates the maximum noise reduction amplitude; Q min Indicates the minimum value of noise reduction; q∈[Q min , Q max ]; d represents the similarity distance between the noise point to be processed and other pixel points in the area to be processed; e represents the natural constant.

[0173] The beneficial effects of the above technical scheme are: by performing denoising processing on the digital pathology image to be detected, the noise points in the image can be effectively removed, and the detection accuracy of the subsequent deep learning model can be improved; the texture similarity is calculated by using the grayscale co-occurrence matrix and texture feature vector, so that the noise points in the image can be accurately identified, and the misjudgment of non-noise points can be avoided; the denoising processing is performed on the area to be processed with a size of M×N, so as to ensure the uniformity and stability of the denoising effect; by presetting the maximum and minimum values ​​of the denoising amplitude to determine the denoising weights of each pixel point in the area to be processed, adaptive denoising processing of noise points of different intensities can be achieved.

[0174] Example 9

[0175] The third feature extraction unit comprises:

[0176] The feature point extraction subunit is used to perform corner point detection on each pixel in the sample depth image using the Harris algorithm to obtain a set of feature points of the sample depth image;

[0177] The depth value determination subunit is used to obtain the depth value of each sample depth image feature point in the sample depth image feature point set; determine the maximum depth value and the minimum depth value based on the depth value of each sample depth image feature point;

[0178] The normalization subunit is used to determine the normalized depth feature value according to the depth value, the maximum depth value and the minimum depth value of each sample depth image feature point, and determine the third feature point information based on the depth feature value.

[0179] In this embodiment, Harris corner detection is a practical and effective feature extraction method, which is mainly used in the field of image processing; the core idea of ​​the algorithm is to design a local detection window on the image, and when the window moves slightly in various directions, the average grayscale change of the window is examined; when the average grayscale change exceeds the set threshold, the central pixel of the window is extracted as a corner point. In this way, we can extract a series of feature points from the sample depth image and further obtain their depth values; then determine the maximum and minimum depth values ​​based on these depth values, and finally perform normalization processing based on these depth values ​​and their maximum and minimum values ​​to obtain normalized depth feature values; the detected feature points need to be described for subsequent matching and recognition; some feature description algorithms, such as SIFT feature description, SURF feature description, ORB feature description, etc., can be used to calculate the local feature vectors around the feature points to describe the appearance and context information of the feature points; match the feature points in different depth images to find the corresponding feature point pairs; distance metrics, similarity metrics and other methods can be used for matching; the matched feature point pairs are screened and optimized to remove unreliable matching points and improve the matching accuracy; they can be screened and optimized according to distance, similarity, geometric constraints, consistency and other criteria; such a processing method can effectively improve the robustness of the features while retaining more image information, thereby helping to improve the accuracy of subsequent image matching and recognition.

[0180] The beneficial effects of the above technical solution are as follows: by using the Harris algorithm to detect corner points of each pixel in the sample depth image, the corner point features in the image can be effectively extracted, thereby improving the accuracy and robustness of subsequent processing; by obtaining the depth values ​​of the feature points of the sample depth image and determining the maximum and minimum depth values, the depth features can be normalized to eliminate the scale differences between different images and improve the comparability and distinguishability of the features; by determining the third feature point based on the normalized depth feature value, the feature information of the image can be further enriched, thereby improving the accuracy and robustness of image matching and recognition.

[0181] Example 10

[0182] A method for detecting and locating abnormal areas based on deep learning, comprising:

[0183] Acquire a group of digital pathology images to be detected; the group of digital pathology images to be detected includes a digital pathology image to be detected and a depth image;

[0184] Training deep learning detection models;

[0185] The digital pathology image group to be detected is input into the deep learning detection model to determine the detection results and positioning results of the abnormal areas in the digital pathology images to be detected.

[0186] The beneficial effects of the above technical solution are as follows: through the training of the deep learning model, the system can automatically learn and extract the features of abnormal areas in digital pathology images, thereby realizing efficient and accurate detection and positioning of the digital pathology images to be detected; the deep learning model can effectively utilize the depth information of the pathology image and comprehensively consider the global context information to capture more comprehensive abnormal area features; because the deep learning model has good generalization ability, the system can process digital pathology images of various types and complexities and has good robustness; through the training of the deep learning model and the automated detection process, the system can quickly and reliably detect and locate abnormal areas in pathology images without human intervention; it can effectively improve the analysis efficiency and accuracy of digital pathology images, and has important application value for the automatic diagnosis and treatment of diseases in the field of medical imaging.

[0187] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0188] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0189] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0190] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0191] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. An abnormal area detection and positioning system based on deep learning, characterized in that: include: An acquisition module, used for acquiring a group of digital pathology images to be detected; the group of digital pathology images to be detected includes a digital pathology image to be detected and a depth image; Training module, used to train deep learning detection models; The determination module is used to input the digital pathology image group to be detected into the deep learning detection model to determine the detection results and positioning results of the abnormal areas in the digital pathology images to be detected.

2. The abnormal area detection and positioning system based on deep learning according to claim 1 is characterized in that: The training module comprises: A data acquisition submodule is used to acquire a sample digital pathology image data set; the sample digital pathology image data set includes a sample digital pathology image, a sample depth image, an abnormal region positioning label and an abnormality degree classification label corresponding to the sample digital pathology image data set; A feature extraction submodule is used to extract features from a sample digital pathology image data set to obtain feature point information; The training submodule is used to input the extracted feature point information, the abnormal area positioning labels and the abnormal degree classification labels corresponding to the sample digital pathology image data group into the deep learning convolutional neural network for training to obtain a deep learning detection model.

3. The abnormal area detection and positioning system based on deep learning according to claim 2 is characterized in that: The feature extraction submodule comprises: A first feature extraction unit is used to obtain the gray value of each pixel in the sample digital pathology image to obtain first feature point information; A second feature extraction unit is used to obtain gradient information of each pixel point in the sample digital pathology image to obtain second feature point information; A third feature extraction unit is used to obtain the normalized depth information of each pixel in the sample depth image to obtain third feature point information; The determining unit is used to determine the feature point information in the sample data group based on the first feature point information, the second feature point information and the third feature point information.

4. The abnormal area detection and positioning system based on deep learning according to claim 2 is characterized in that: The training submodule comprises: An abnormality localization unit is used to input the extracted feature point information and the abnormal area localization label corresponding to the sample digital pathology image data group into a deep learning convolutional neural network for training to obtain an abnormal area localization sub-model; The abnormality degree unit is used to input the extracted feature point information and the abnormality degree classification labels corresponding to the sample digital pathology image data group into the deep learning convolutional neural network for training to obtain an abnormality degree sub-model; A generation unit is used to obtain a deep learning detection model based on the abnormal area positioning sub-model and the abnormal degree sub-model.

5. The abnormal area detection and positioning system based on deep learning according to claim 4 is characterized in that: The generating unit further includes: A first loss calculation subunit, used to calculate a first loss corresponding to the abnormal area positioning submodel; A second loss calculation subunit, used to calculate a second loss corresponding to the abnormality degree submodel; Generate subunits for: Adjusting the model parameters of the abnormal region positioning sub-model based on the first loss, and obtaining the abnormal region positioning sub-model when it is determined that the abnormal region positioning sub-model converges; Adjusting the model parameters of the abnormality degree sub-model based on the second loss, and obtaining the abnormality degree sub-model when it is determined that the abnormality degree sub-model converges; A deep learning detection model is obtained based on the abnormal area positioning sub-model and the abnormality degree sub-model.

6. The abnormal area detection and positioning system based on deep learning according to claim 5, characterized in that: The first loss is calculated as: Among them, L A represents the first loss corresponding to the training of the A-th target sample digital pathology image for calculating the abnormal area localization sub-model, IoU is the intersection over union ratio, b p , b gt The prediction frame is obtained from the training target sample digital pathology image, and the coordinates of the center point of the real target frame are obtained from the target sample digital pathology image; w c h represents the width of the minimum bounding box between the predicted box obtained by the training target sample digital pathology image and the real target box obtained by the target sample digital pathology image, c It represents the height of the minimum bounding box between the predicted box obtained by the training target sample digital pathology image and the real target box obtained by the target sample digital pathology image; d(b p , b gt ) represents the Euclidean distance between the center point of the predicted box obtained from the training target sample digital pathology image and the center point of the real target box obtained from the target sample digital pathology image; w p represents the width of the prediction box obtained by the training target sample digital pathology image, h p represents the height of the prediction box obtained from the training target sample digital pathology image; w gt represents the width of the real box obtained from the target sample digital pathology image; h gt The height of the true box obtained by representing the target sample digital pathology image.

7. The abnormal area detection and positioning system based on deep learning according to claim 5, characterized in that: The second loss is calculated as: Among them, L B represents the second loss obtained by the B-th target sample digital pathology image, and M represents the total number of pixels included in the B-th target sample digital pathology image; ∑ xy () represents traversing every pixel in the target sample digital pathology image, T xy represents the loss weight of the pixel at the coordinate (x, y); c represents the total number of abnormality classifications, ∑ c () represents the sum of the losses of various abnormality classifications in the target sample digital pathology image, Y xy Z represents the probability that the pixel at (x, y) belongs to a certain abnormality level, determined according to the abnormality level classification label annotated by the sample digital pathology image data set. xy It represents the probability that the pixel point at (x, y) belongs to a category of abnormality, determined according to the abnormality classification label predicted by the sample digital pathology image data set.

8. The abnormal area detection and positioning system based on deep learning according to claim 1, characterized in that: Also includes: A denoising module, used for denoising the digital pathology image group to be detected before the determination module inputs the digital pathology image group to be detected into the deep learning detection model; The noise reduction module comprises: Noise detection unit, used to: Grayscale the digital pathology images to be detected in the group of digital pathology images to be detected to obtain a grayscale image, and obtain the grayscale value of each pixel in the grayscale image; Randomly select a pixel point in the grayscale image to obtain a first pixel point; determine a first target area with the first pixel point as the center and a first preset distance as the radius; determine a second target area with the first pixel point as the center and a second preset distance as the radius; the second preset distance is greater than the first preset distance; Take any pixel point in the second target area after removing the first target area to obtain a second pixel point; determine a second target sub-area with the second pixel point as the center and the first preset distance as the radius; the second target sub-area has no intersection with the first target area; Obtaining gray level co-occurrence matrices corresponding to the first target region and the second target sub-region respectively; determining a first texture feature vector corresponding to the first target region based on the gray level co-occurrence matrix corresponding to the first target region; determining a second texture feature vector corresponding to the second target sub-region based on the gray level co-occurrence matrix corresponding to the second target sub-region; Determining texture similarity between the first target region and the second target sub-region based on the first texture feature vector and the second texture feature vector; Traversing each pixel point in the second target sub-region to obtain a number of texture similarities; When it is determined that all texture similarities are less than a preset texture similarity threshold, the first pixel point is a noise point; Traverse every pixel in the grayscale image and determine all the noise points in the grayscale image; Based on all the noise points in the grayscale image, determining the noise points in the digital pathology image to be detected; Noise reduction unit for: Randomly select a noise point in the digital pathology image to be detected to obtain a noise point to be processed; Taking the noise point to be processed as the center, determine a region to be processed of size M×N; wherein M and N are both odd numbers and both M and N are greater than 1; Calculate the similarity distance between the noise point to be processed and other pixel points in the area to be processed; determine the texture intensity value of the noise point to be processed based on the maximum similarity distance; Preset the maximum and minimum values ​​of noise reduction; Determine the noise reduction weight of each pixel in the area to be processed based on the texture intensity value and the preset maximum and minimum noise reduction amplitudes; Determine the grayscale value of the noise point to be processed after denoising based on the denoising weight of each pixel point in the area to be processed; All noise points in the digital pathology image to be detected are traversed to obtain a denoised image of the digital pathology image to be detected.

9. The abnormal area detection and positioning system based on deep learning according to claim 3, characterized in that: The third feature extraction unit comprises: The feature point extraction subunit is used to perform corner point detection on each pixel in the sample depth image using the Harris algorithm to obtain a set of feature points of the sample depth image; The depth value determination subunit is used to obtain the depth value of each sample depth image feature point in the sample depth image feature point set; determine the maximum depth value and the minimum depth value based on the depth value of each sample depth image feature point; The normalization subunit is used to determine the normalized depth feature value according to the depth value, the maximum depth value and the minimum depth value of each sample depth image feature point, and determine the third feature point information based on the depth feature value.

10. A method for detecting and locating abnormal areas based on deep learning, characterized in that: include: Acquire a group of digital pathology images to be detected; the group of digital pathology images to be detected includes a digital pathology image to be detected and a depth image; Training deep learning detection models; The digital pathology image group to be detected is input into the deep learning detection model to determine the detection results and positioning results of the abnormal areas in the digital pathology images to be detected.

Images