Pathological index determination method, device, equipment and storage medium

Abstract

The present application discloses a method, device, equipment and storage medium for determining pathological indicators, which includes: obtaining a pathological image to be processed including a marked area; preprocessing the pathological image to be processed of the marked area to obtain a mask of the first area and a mask of the second area, and inputting the pathological image to be processed into a pre-trained deep learning segmentation model in parallel to obtain a segmentation result of the first area; fusing the mask of the first area, the mask of the second area and the segmentation result of the first area to determine the target area; and statistically analyzing the pixel saturation of the target area to determine the pathological indicator. The scheme can extract the layout details of the pathological image, thereby accurately obtaining the target area, and then accurately statistically analyzing the pixel saturation of the target area to determine the pathological indicator, thereby reducing the occurrence of unstable pathological indicators due to artificial subjective factors and improving the efficiency of pathological diagnosis.

Description

Technical Field

[0001] The present invention generally relates to the field of image processing technology, and specifically relates to a method, device, equipment and storage medium for determining pathological indicators. Background Art

[0002] With the continuous development of medical technology, analysis and screening based on pathological images are important means of disease diagnosis and treatment in modern medicine. For example, analysis of the tissue area of ​​the animal eye is generally carried out on the distribution / morphology of its blood vessels. By observing the diseased tissue, some pathological indicators can be obtained, thereby providing auxiliary means for drug trials, postoperative analysis, specifying medical plans, evaluating treatment effects, and so on.

[0003] At present, one method in the related technology is to manually determine the pathological indicators based on the pathological images, but this method only relies on manual experience and lacks objectivity, resulting in low accuracy in determining the pathological indicators; another method is to use traditional deep learning basic network structures such as VGGNet, ResNet, Inception and DenseNet, and extract the global semantic information of the original image through a series of downsampling. However, this solution does not pay enough attention to local details and global contrast information, and the use of these basic network structures requires the use of pre-trained models on ImageNet. If the distribution of the target data to be predicted is significantly different from the distribution of the ImageNet data set, the prediction accuracy of the network structure is poor. Summary of the invention

[0004] In view of the above-mentioned defects or deficiencies in the prior art, it is desirable to provide a method, device, equipment and storage medium for determining pathological indicators.

[0005] In a first aspect, the present invention provides a method for determining a pathological index, the method comprising:

[0006] Acquire a pathological image to be processed including the marked area;

[0007] Preprocessing the pathological image to be processed of the marked area to obtain a mask of the first area and a mask of the second area, and inputting the pathological image to be processed into a pre-trained deep learning segmentation model in parallel to obtain a segmentation result of the first area, wherein the deep learning segmentation model has feature extraction and image segmentation capabilities;

[0008] Fusing the mask of the first area, the mask of the second area, and the segmentation result of the first area to determine a target area;

[0009] The pixel saturation of the target area is statistically analyzed to determine the pathological index.

[0010] In one embodiment, preprocessing the pathological image to be processed of the marked area to obtain a mask of the first area and a mask of the second area includes:

[0011] Using a morphological filter to filter the pathological image to be processed to obtain a filtered pathological image;

[0012] Binarizing the filtered pathological image by a local adaptive threshold method to determine a binary image of the first region;

[0013] Performing contour screening processing based on the binary image of the first area to determine a mask of the first area;

[0014] Converting the pathological image to be processed into a Lab color gamut image through the RGB channel;

[0015] Thresholding the a channel in the Lab color gamut image to obtain a binary image of the second region;

[0016] Perform contour screening processing based on the binary image of the second area to determine a mask of the second area.

[0017] In one embodiment, the morphological filter includes a Sato filter, a Gaussian filter and a median filter, and the pathological image to be processed is filtered using the morphological filter to obtain a filtered pathological image, including:

[0018] Processing the pathological image to be processed through the Sato filter to filter out image interference information to obtain a first processed image;

[0019] Performing linear smoothing filtering on the first processed image through the Gaussian filter to obtain a second processed image;

[0020] The second processed image is subjected to noise information elimination processing by the median filter to obtain a filtered pathological image.

[0021] In one embodiment, thresholding the a channel in the Lab color gamut image to obtain a binary image of the second region includes:

[0022] Obtaining the pixel value of the pathological image to be processed in the Lab color gamut image in channel a;

[0023] Compare the pixel value of the a channel with a preset pixel threshold;

[0024] Binarization is performed on the region corresponding to the pixel value of the a channel being less than the preset pixel threshold to generate a binary image of the second region.

[0025] In one embodiment, the deep learning segmentation model includes U 2 Net network model, input the pathological image to be processed into the pre-trained U 2 Net network model, the segmentation results of the first region are obtained, including:

[0026] The pathological image to be processed is passed through the U 2 The encoder and decoder in the Net network model are processed to obtain a significant probability map;

[0027] Through the U 2 Net network model, upsampling the salient probability map to the same size as the pathological image to be processed, and determining the upsampling result;

[0028] The up-sampling results are fused by cascading operations to obtain fused results;

[0029] The fusion result is processed through a convolution operation and an activation function to obtain a segmentation result of the first region.

[0030] In one embodiment, the U 2 Net network model is constructed through the following steps:

[0031] Access to historical pathology images;

[0032] Preprocessing the historical pathological image to obtain a historical contour image of the first area;

[0033] Based on the historical pathological images and the historical contour images of the first area, construct a data set, and divide the data set into a training set and a validation set;

[0034] The constructed U 2 Net network model is trained to obtain the U 2 Net network model;

[0035] Using the verification set to verify the U 2 Net network model, according to the loss function to minimize the U to be verified 2 Net network model is optimized to obtain U 2 Net network model.

[0036] In one embodiment, fusing the mask of the first region, the mask of the second region, and the segmentation result of the first region to determine the target region includes:

[0037] fusing the mask of the first region and the segmentation result of the first region according to a preset fusion rule to obtain a total image of the first region;

[0038] Based on the overall image of the first area and the mask of the second area, a target area is determined.

[0039] In one embodiment, statistical analysis is performed on the pixel saturation of the target area to determine the pathological index, including:

[0040] Converting the pathological image to be processed into an HSV color gamut image;

[0041] Obtain the pixel value of the target area in the HSV color gamut image in the S channel;

[0042] The pixel saturation of the target area and the pixel value of the s channel are statistically analyzed to obtain a pathological index.

[0043] In a second aspect, an embodiment of the present application provides a pathological index determination device, the device comprising:

[0044] An acquisition module, used for acquiring a pathological image to be processed including a marked area;

[0045] A processing module, used for preprocessing the pathological image to be processed of the marked area to obtain a mask of the first area and a mask of the second area, and inputting the pathological image to be processed into a pre-trained deep learning segmentation model in parallel to obtain a segmentation result of the first area, wherein the deep learning segmentation model has image segmentation capability;

[0046] a fusion module, configured to fuse the mask of the first region, the mask of the second region, and the segmentation result of the first region to determine a target region;

[0047] The analysis module is used to perform statistical analysis on the pixel saturation of the target area to determine the pathological index.

[0048] In a third aspect, an embodiment of the present application provides a device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for determining pathological indicators as described in the first aspect above is implemented.

[0049] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is used to implement the pathological indicator determination method of the first aspect above.

[0050] The pathological index determination method, device, equipment and storage medium provided in the embodiment of the present application obtains the pathological image to be processed, then pre-processes the pathological image to be processed, obtains the mask of the first area and the mask of the second area, and inputs the pathological image to be processed into a pre-trained deep learning segmentation model in parallel to obtain the segmentation result of the first area, and fuses the mask of the first area, the mask of the second area and the segmentation result of the first area to determine the target area, and then statistically analyzes the pixel saturation of the target area to determine the pathological index. This technical solution can not only extract the layout details of the pathological image by pre-processing the pathological image to be processed, and perform image segmentation processing through the deep learning segmentation model in parallel, but also globally fuse and compare the image information obtained by the two, so as to accurately and quickly obtain the target area, and then accurately perform statistical analysis on the pixel saturation of the target area to determine the pathological index, provide accurate quantitative numerical results for doctors, reduce the occurrence of unstable pathological indicators due to artificial subjective factors, assist doctors in making efficient and accurate judgments, and improve the efficiency of pathological diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0052] Figure 1 A schematic diagram of the structure of the implementation environment of the pathological index determination method provided in the embodiment of the present application;

[0053] Figure 2 A schematic diagram of a process for determining pathological indicators provided in an embodiment of the present application;

[0054] Figure 3 A schematic diagram of the structure of a pathological indicator determination framework provided in an embodiment of the present application;

[0055] Figure 4 A schematic flow chart of a method for preprocessing a pathological image to be processed in a marked area provided in an embodiment of the present application;

[0056] Figure 5 A schematic diagram of a flow chart of a method for determining a segmentation result of a first region provided in an embodiment of the present application;

[0057] Figure 6 A schematic diagram of a structure for prediction by a model provided in an embodiment of the present application;

[0058] Figure 7 A schematic diagram of the structure of the meta tag learning structure provided in the embodiment of the present application;

[0059] Figure 8A mask image of a blood vessel contour area provided in an embodiment of the present application;

[0060] Fig. 9 A mask image of the blood mist contour area provided in an embodiment of the present application;

[0061] Fig.10 A schematic diagram of the structure of a pathological index determination device provided in another embodiment of the present application;

[0062] Fig.11 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the invention are shown in the accompanying drawings.

[0064] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0065] It is understandable that with the rapid development of artificial intelligence technology, medical technology is gradually turning to intelligence. In clinical practice, pathological images can be analyzed to diagnose and treat diseases intuitively and accurately. In order to improve the accuracy and work efficiency of pathological analysis, how to analyze pathological images is particularly important.

[0066] At present, one method in the related technology is to observe the pathological images of the eyeball and the like with the naked eye, check the distribution or morphology of the blood vessels, and the congestion condition, etc., to determine the pathological indicators. This method relies solely on manual experience, which is not only susceptible to the limitation of the subjective experience of doctors, but also with the increase in the number of clinical samples, conventional manual film viewing can no longer meet the clinical needs of the hospital pathology department. The computer-aided diagnosis algorithm based on machine learning has become one of the very popular frontier fields at home and abroad. Another method is to use traditional deep learning basic network structures such as VGGNet, ResNet, Inception and DenseNet. After a series of downsampling, the global semantic information of the original image is extracted. However, this method does not pay enough attention to local details and global contrast information, and the use of the above-mentioned basic network structures requires the use of pre-trained models on ImageNet. If the distribution of the target data to be predicted is significantly different from the distribution of the ImageNet data set, the prediction accuracy of the network structure is poor, thereby reducing the efficiency of pathological diagnosis.

[0067] Based on the above-mentioned defects, the present application provides a method for determining pathological indicators. Compared with the prior art, this technical solution can not only extract the layout details of the pathological image, but also globally compare the image information obtained by the two by preprocessing the pathological image to be processed and performing image segmentation processing through a deep learning segmentation model in parallel, so as to accurately and quickly acquire the target area, and then accurately perform statistical analysis on the pixel saturation of the target area to determine the pathological indicators, providing doctors with accurate quantitative numerical results, reducing the occurrence of unstable pathological indicators due to human subjective factors, assisting doctors in making efficient and accurate judgments, and improving the efficiency of pathological diagnosis.

[0068] Figure 1 : is a diagram of the implementation environment of a method for determining pathological indicators provided in an embodiment of the present application. Figure 1 As shown, the implementation environment architecture includes: a terminal 100 and a server 200.

[0069] The terminal 100 may be a terminal device in various AI application scenarios. For example, the terminal 100 may be a smart home device such as a smart TV or a smart TV set-top box, or the terminal 100 may be a mobile portable terminal such as a smart phone, a tablet computer, and an e-book reader, or the terminal 100 may be a smart wearable device such as smart glasses or a smart watch, which is not specifically limited in this embodiment.

[0070] Among them, the terminal 100 may be installed with an AI application based on deep learning.

[0071] The server 200 may be a single server, or a server cluster consisting of a plurality of servers, or the server 200 may include one or more virtualization platforms, or the server 200 may be a cloud computing service center.

[0072] The server 200 may be a server device that provides background services for the AI ​​application installed in the terminal 100 .

[0073] The terminal 100 and the server 200 establish a communication connection via a wired or wireless network. Optionally, the wireless network or wired network uses standard communication technology and / or protocols. The network is usually the Internet, but can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or any combination of a virtual private network.

[0074] In the process of providing AI application services, the above-mentioned AI application system based on deep learning can perform feature extraction and image segmentation on the pathological image to be processed through the deep learning segmentation model to obtain the target area, and determine the pathological index according to the target area to provide AI application services. Among them, the above-mentioned deep learning segmentation model can be set in the server 200, trained and applied by the server; or, the above-mentioned deep learning segmentation model can also be set in the terminal 100, and trained and updated by the server 200.

[0075] For ease of understanding and explanation, the following Figures 2 to 11 The pathological index determination method, device, equipment and storage medium provided in the embodiments of the present application are described in detail.

[0076] Figure 2 FIG. 2 is a flow chart of a method for determining pathological indicators according to an embodiment of the present application, which method can be executed by a device for determining pathological indicators. Figure 2 As shown, the method includes:

[0077] S101, obtaining a pathological image to be processed including a marked area.

[0078] Specifically, the annotated area may be a specific area manually annotated on the pathological image to be processed, and the pathological image to be processed including the annotated area may be a pathological image for which pathological indicators need to be determined, and statistical analysis of pathological indicators is performed in the annotated area. The pathological image to be processed may be an animal eyeball image or a heart image, etc. The number of the annotated areas may be one or more.

[0079] Optionally, the pathological image to be processed including the marked area can be imported and obtained through an external device, or can be obtained from a preset database.

[0080] In this step, after obtaining the original image data set, the coordinate value of the annotated area is obtained using the color threshold, and then the annotated area of ​​a preset size is intercepted in the original image data set, so that the pathological image to be processed including the annotated area is obtained through preprocessing operations. The preset size can be a size that is customized according to actual needs.

[0081] S102. Preprocess the pathological image to be processed in the marked area to obtain a mask of the first area and a mask of the second area, and input the pathological image to be processed into a pre-trained deep learning segmentation model in parallel to obtain a segmentation result of the first area. The deep learning segmentation model has feature extraction and image segmentation capabilities.

[0082] In the embodiments of this application, please refer to Figure 3As shown, after obtaining the pathological image to be processed including the marked area, the pathological image to be processed in the marked area is filtered and binarized to obtain a mask of the first area and a mask of the second area. For example, if the pathological image to be processed including the marked area is an animal eyeball image including a specific area, the mask of the first area obtained may be a blood vessel mask, and the mask of the second area may be a blood mist mask. Then, the pathological image to be processed is input into a pre-trained deep learning segmentation model in parallel for model prediction to obtain a segmentation result of the first area, and the mask of the first area, the mask of the second area and the segmentation result of the first area are fused to determine the target area, and then the pixel saturation of the target area is statistically analyzed to determine the pathological index.

[0083] Optionally, as a possible implementation method of this application, see Figure 4 As shown, Figure 4 A schematic flow chart of a method for preprocessing a pathological image to be processed in a marked area provided in an embodiment of the present application, the method comprising:

[0084] S201, using a morphological filter to filter the pathological image to be processed to obtain a filtered pathological image.

[0085] Specifically, the above-mentioned morphological filter may include a Sato filter, a Gaussian filter and a median filter. The pathological image to be processed may be processed by a Sato filter to filter out image interference information to obtain a first processed image, and then the first processed image may be linearly smoothed by a Gaussian filter to obtain a second processed image, and the second processed image may be subjected to noise information elimination processing by the median filter to obtain a filtered pathological image.

[0086] It should be noted that the above-mentioned Gaussian filter is a linear smoothing filter algorithm, which is suitable for eliminating Gaussian noise and is widely used in the noise reduction process of image processing. Gaussian filtering is the process of weighted averaging the first processed image. The value of each pixel is obtained by weighted averaging the value of itself and other pixels in the neighborhood. The specific operation can be: use a template (or convolution, mask) to scan each pixel in the first processed image, and replace the value of the central pixel of the template with the weighted average gray value of the pixels in the neighborhood determined by the template. The degree to which the image is smoothed after Gaussian filtering depends on the standard deviation. Its output is the weighted average of the pixels in the field, and the closer the pixel is to the center, the higher the weight. Therefore, compared with the mean filter, its smoothing effect is softer and the edge is better preserved.

[0087] The median filter uses the median filtering method, which is a nonlinear smoothing technology that sets the grayscale value of each pixel to the median of the grayscale values ​​of all pixels in a neighborhood window of the point. Median filtering is a nonlinear signal processing technology based on sorting statistics theory that can effectively suppress noise. The basic principle of median filtering is to replace the value of a point in a digital image or digital sequence with the median of the values ​​of each point in a neighborhood of the point, so that the surrounding pixel values ​​are close to the true value, thereby eliminating isolated noise points. The method is to use a two-dimensional sliding template with a certain structure to sort the pixels in the board according to the size of the pixel value, and generate a two-dimensional data sequence that is monotonically rising or falling.

[0088] For example, taking the pathological image to be processed as an eye image as an example, passing the eye image through a Sato filter can effectively filter out redundant interference information, thereby obtaining a preliminary image of the vascular area. The initial image of the vascular area is then post-processed through a Gaussian filter and a median filter in turn to obtain the vascular area.

[0089] S202 , binarizing the filtered pathological image by using a local adaptive threshold method to determine a binary image of the first region.

[0090] It should be noted that a binary image means that each pixel on the image has only two possible values ​​or grayscale states. In other words, the grayscale value of any pixel in the image is 0 or 255, representing black and white respectively.

[0091] In this step, after obtaining the filtered pathological image, the pixel values ​​on the image can be obtained, and the pixel values ​​are compared with the preset threshold value, and divided into pixel groups greater than the preset threshold value and pixel groups not greater than the preset threshold value, and the pixel groups greater than the preset threshold value are set to 255, and the pixel groups not greater than the preset threshold value are set to 0, so that the filtered pathological image is divided into regions with pixel values ​​of 0 and 255, and then a binary image of the first region is obtained. The binary image of the first region may be a binary image of a blood vessel region.

[0092] S203 , performing contour screening processing based on the binary image of the first region to determine a mask of the first region.

[0093] Specifically, after obtaining the binary image of the first region, the contour can be found on the binary image, and the target can be screened, and the preset algorithm in opencv can be called for processing to obtain the mask of the first region. The mask of the first region can be a blood vessel mask.

[0094] S204, converting the pathological image to be processed into a Lab color gamut image through the RGB channel.

[0095] It should be noted that the above Lab is an uncommon color space, which is a color system based on physiological characteristics. The L component in the Lab color space is used to represent the brightness of the pixel, and its value range is [0,100], which means from pure black to pure white; a represents the range from red to green, and its value range is [127,-128]; b represents the range from yellow to blue, and its value range is [127,-128]. The above RGB channels are composed of the red channel (R), the green channel (G), and the blue channel (B).

[0096] Optionally, in the process of converting the pathological image to be processed into a Lab color gamut image through the RGB channel, the color space of the RGB channel can be converted into the XYZ color space with the help of the XYZ color space, and then the XYZ color space is converted into the Lab color gamut image.

[0097] S205 , performing threshold processing on the a channel in the Lab color gamut image to obtain a binary image of the second region.

[0098] Specifically, the pixel value of the pathological image to be processed in the Lab color gamut image in the a channel can be obtained, and the pixel value of the a channel is compared with the preset pixel threshold, and then the area corresponding to the pixel value of the a channel less than the preset pixel threshold is binarized to generate a binary image of the second area. The preset pixel threshold can be pre-defined according to actual needs, for example, it can be 160. The binary image of the second area can be a binary image of the blood mist area.

[0099] S206 , performing contour screening processing based on the binary image of the second region to determine a mask of the second region.

[0100] In this step, after obtaining the binary image of the second area, the contour can be found on the binary image, and the target can be screened, and the preset algorithm in opencv can be called for processing to obtain the mask of the second area. The mask of the second area can be a blood mist mask.

[0101] In this embodiment, by pre-processing the pathological image to be processed including the marked area by filtering and binarization, the mask of the first area and the mask of the second area can be accurately determined, and thus the pathological index can be accurately determined.

[0102] Optionally, as a possible implementation method of this application, see Figure 5 As shown, Figure 5 A schematic flow chart of a method for determining a segmentation result of a first region provided in an embodiment of the present application, the method comprising:

[0103] S301, the pathological image to be processed is transmitted through U 2Net network model to obtain a significant probability map.

[0104] S302, through U 2 Net network model, upsamples the salient probability map to the same size as the pathological image to be processed, and determines the upsampling result.

[0105] S303 , fusing the up-sampling results through a cascade operation to obtain a fusion result.

[0106] S304: Process the fusion result through a convolution operation and an activation function to obtain a segmentation result of the first region.

[0107] In the embodiment of the present application, the above-mentioned deep learning segmentation model includes U 2 Net network model. 2 The UNet network model is constructed by the following steps: firstly, the historical pathological images are obtained and preprocessed to obtain the historical contour images of the first region; then, a data set is constructed based on the historical pathological images and the historical contour images of the first region, and the data set is divided into a training set and a validation set, and the training set is used to construct the UNet network model. 2 Net network model is trained to obtain the U 2 Net network model, and use the U 2 Net network model, according to the loss function minimization to treat the verification U 2 Net network model is optimized to obtain U 2 Net network model.

[0108] Specifically, when preprocessing the historical pathological image, a historical contour image of the first region can be obtained by morphological filtering and binarization, and the result is used as a label for supervised learning. The historical pathological image and the historical contour image of the first region are used to construct a data set, and the data set is divided into a training set and a validation set according to a preset ratio, and the preset ratio can be 85% for the training set and 15% for the validation set. Then, the training set is used to construct the U 2 Net network model is trained to obtain the U 2 Net network model, and use the validation set and evaluation indicators to search for the model with the best parameters. The U 2 Net network model is optimized to obtain U 2 Net network model. The evaluation indicator used by the model can be the Dice coefficient, the optimizer selects RMSporp, and the learning rate of the model is set to a dynamic learning rate.

[0109] Among them, the above U2 Net network model contains encoder, decoder, skip connection, residual structure and feature fusion block. The encoder and decoder are composed of encoder-decoder substructures, and each substructure is connected using a residual structure. The encoder and the decoder at the corresponding position use skip connections.

[0110] See also Figure 6 As shown, Figure 6 For U 2 Net network model network structure diagram, the U 2 The Net network model consists of three parts, including a six-level encoder, a five-level decoder, and a saliency map fusion model connected to the decoder and the last level encoder. The six-level encoder includes En_1, En_2, En_3, En_4, En_5, and En_6, and the five-level decoder includes De_1, De_2, De_3, De_4, and De_5.

[0111] It should be noted that the RSU structures of RSU-7, RSU-6, RSU-5, and RSU-4 are used in the above encoder En_1, En_2, En_3, and En_4 stages, respectively. The preceding numbers such as 7, 6, 5, and 4 are the heights L of the RSU. L is usually configured according to the spatial resolution of the input feature map. In En_5 and En_6, the resolution of the feature maps is relatively low, and further downsampling of these feature maps will result in the loss of useful context. Therefore, in the RSU-5 and RSU-6 stages, RSU-4F is used, where F indicates that RSU is an extended version, in which we use extended convolutions instead of merging and upsampling operations. This means that all intermediate feature maps of RSU-4F have the same resolution as their input feature maps. Among them, the new residual U-block (ReSidual U-block, RSU) can fuse features of receptive fields of different sizes to capture more contextual information of different scales.

[0112] The decoding stage has a similar structure to the symmetric encoding stage in En_6. In De_5, the RSU-4F of the expansion board is also used, which is similar to that used in the encoding stage En_5, En_6. Each decoder stage takes as input the concatenation of the upsampled feature map from the previous stage and the feature map from its symmetric encoder stage.

[0113] The last part is the saliency map fusion module. The U2-Net network first generates 6 output saliency probability maps from En_6, De_5, De_4, De_3, De_2 and De_1 through 3x3 convolution and Sigmoid function. Then, the logic map of the output saliency probability map (convolution output, before the Sigmoid function) is up-sampled to the same size as the input image to obtain a sampling result, and the sampling result is fused through a cascade operation to obtain a fusion result, and the fusion result is processed through a convolution operation and an activation function, and a 1x1 convolution layer and a Sigmoid function can be passed to generate the final saliency probability map Sfuse, thereby obtaining the segmentation result of the first region. The segmentation result of the first region can be the predicted probability of the label type of the first region, or the predicted category of the first region.

[0114] In this embodiment, the processed pathological image is input into U 2 Net network model, the encoder can be used to extract high-dimensional features from the pathological image to be processed, and the decoder can be used to improve the spatial resolution of the features. It can not only effectively extract local features of different sizes, but also obtain high-resolution feature maps, so that the segmentation result of the first area determined is more accurate, providing doctors with accurate quantitative numerical results, and assisting doctors in making efficient and accurate judgments.

[0115] S103: Fusing the mask of the first region, the mask of the second region, and the segmentation result of the first region to determine the target region.

[0116] Specifically, after obtaining the mask of the first region and the segmentation result of the first region, a fusion process is performed according to a preset fusion rule to obtain a total image of the first region, and then the target region is determined based on the total image of the first region and the mask of the second region.

[0117] The mask of the first region and the segmentation result of the first region may be subjected to a union operation, that is, when the mask of the first region or the segmentation result of the first region is positive, the contour image is determined to be the total image of the first region, and then the total image of the first region and the contour image of the second image are determined as the target region. Taking the pathological image to be processed as an eyeball image as an example, the target region may include a mask of a blood vessel region and a mask of a blood mist region.

[0118] S104: Perform statistical analysis on the pixel saturation of the target area to determine the pathological index.

[0119] In this step, after determining the target area, the pathological image to be processed is converted into an HSV color gamut image, and then based on the pixel saturation of the target area, the pixel value of the target area in the S channel in the HSV color gamut image is obtained, and the pixel saturation of the target area and the pixel value of the S channel are statistically analyzed to obtain the pathological index.

[0120] It should be noted that the above-mentioned HSV (Hue, Saturation, Value) color space is a color space created based on the intuitive characteristics of color, also known as the hexagonal cone model. The color parameters in this model are: hue (H), saturation (S), and lightness (V). Among them, saturation S indicates the degree to which the color is close to the spectral color. A color can be regarded as the result of a mixture of a certain spectral color and white. Among them, the greater the proportion of spectral color, the higher the degree to which the color is close to the spectral color, and the higher the saturation of the color. The higher the saturation, the deeper and brighter the color. The white light component of the spectral color is 0, and the saturation reaches the highest. The value range is usually 0% to 100%. The larger the value, the more saturated the color.

[0121] In the process of converting the pathological image to be processed into the HSV color gamut image, the conversion function in OpenCV can be called to convert the pathological image to be processed into the HSV color gamut image through the RGB channel.

[0122] Furthermore, the pixel values ​​of the s channel can be statistically analyzed to determine the pathological index corresponding to the saturation value, which can include: maximum value, minimum value, mean, median, mode, and standard deviation. And according to the target area, the blood vessel area, blood mist area, and target area area can be determined.

[0123] Exemplarily, taking the image of a rabbit eyeball as an example of a pathological image to be processed including an annotated area, the eyeball image can be preprocessed, and the redundant interference information can be effectively filtered out by passing the eyeball image through a Sato filter, thereby obtaining a preliminary image of the vascular area, and then the initial image of the vascular area is post-processed by a Gaussian filter and a median filter in turn, thereby obtaining an image of the vascular area, and then processed by a local adaptive threshold method to obtain a binary image of the vascular area, and the contour of the vascular image is found by calling opencv, and the contour is screened to obtain the vascular contour. The eyeball image is converted into a Lab color domain image through the RGB channel, and the a channel in the Lab color domain image is thresholded to obtain a binary image of the blood mist area, and then opencv is called to find the contour of the blood mist image, and the contour is screened to obtain the blood mist contour.

[0124] See also Figure 6As shown, in the process of inputting the eyeball image into the pre-trained deep learning segmentation model in parallel to obtain the segmentation result of the vascular area, two stages can be included: the initialization stage and the inference stage. First, enter the initialization stage, load the pre-trained deep learning segmentation model by starting the model loader in the disk, configure the relevant parameters for the model after loading and initialize the GPU; then enter the inference stage, call the data preprocessing program to obtain the eyeball image and crop it into the input size of the model to obtain the cropped eyeball image Patches, and send the cropped eyeball image to the model on the GPU for prediction to obtain rough vascular contour information, and smooth the vascular contour information by calling the post-processing program. After smoothing, the vascular mask of the eyeball image is obtained, and then the vascular mask and blood mist mask obtained by image processing are taken and the vascular mask obtained by model prediction are taken and the target area is obtained, and the pixel saturation of the target area is statistically analyzed to determine the pathological indicators. Among them, the mask map of the vascular area obtained can be referred to. Figure 8 As shown, the mask map of the blood mist area can be found in Fig. 9 shown.

[0125] The pathological index determination method provided in the embodiment of the present application obtains the pathological image to be processed, and then pre-processes the pathological image to be processed to obtain the mask of the first area and the mask of the second area, and inputs the pathological image to be processed into a pre-trained deep learning segmentation model in parallel to obtain the segmentation result of the first area, and fuses the mask of the first area, the mask of the second area and the segmentation result of the first area to determine the target area, and then statistically analyzes the pixel saturation of the target area to determine the pathological index. This technical solution can not only extract the layout details of the pathological image by pre-processing the pathological image to be processed, and perform image segmentation processing through the deep learning segmentation model in parallel, but also globally compare the image information obtained by the two, so as to accurately and quickly obtain the target area, and then accurately perform statistical analysis on the pixel saturation of the target area to determine the pathological index, provide accurate quantitative numerical results for doctors, reduce the occurrence of unstable pathological indicators due to artificial subjective factors, assist doctors in making efficient and accurate judgments, and improve the efficiency of pathological diagnosis.

[0126] It should be noted that although the operations of the method of the present invention are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all the operations shown must be performed to achieve the desired results. On the contrary, the steps depicted in the flow chart can be performed in a different order. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.

[0127] on the other hand, Fig.10 A schematic diagram of a pathological index determination device provided in an embodiment of the present application. The device may be a device in a terminal device, such as Fig.10 As shown, the device 400 includes:

[0128] An acquisition module 410 is used to acquire a pathological image to be processed including a marked area;

[0129] A processing module 420 is used to pre-process the pathological image to be processed in the marked area to obtain a mask of the first area and a mask of the second area, and to input the pathological image to be processed into a pre-trained deep learning segmentation model in parallel to obtain a segmentation result of the first area, wherein the deep learning segmentation model has image segmentation capability;

[0130] A fusion module 430 is used to fuse the mask of the first region, the mask of the second region and the segmentation result of the first region to determine the target region;

[0131] The analysis module 440 is used to perform statistical analysis on the pixel saturation of the target area to determine the pathological index.

[0132] Optionally, the processing module 420 is specifically used to:

[0133] The pathological image to be processed is filtered using a morphological filter to obtain a filtered pathological image;

[0134] Binarizing the filtered pathological image by a local adaptive threshold method to determine a binary image of the first region;

[0135] Performing contour screening processing based on the binary image of the first region to determine a mask of the first region;

[0136] Convert the pathological image to be processed into a Lab color gamut image through the RGB channel;

[0137] Thresholding the a channel in the Lab color gamut image to obtain a binary image of the second region;

[0138] A contour screening process is performed based on the binary image of the second region to determine a mask of the second region.

[0139] Optionally, the processing module 420 is further configured to:

[0140] Processing the pathological image to be processed through a Sato filter to filter out image interference information to obtain a first processed image;

[0141] Performing linear smoothing filtering on the first processed image through a Gaussian filter to obtain a second processed image;

[0142] The second processed image is subjected to a noise information elimination process through a median filter to obtain a filtered pathological image.

[0143] Optionally, the processing module 420 is further configured to:

[0144] Obtain the pixel value of the pathological image to be processed in the Lab color domain image in channel a;

[0145] Compare the pixel value of channel a with the preset pixel threshold;

[0146] The area corresponding to the pixel value of the a channel less than the preset pixel threshold is binarized to generate a binary image of the second area.

[0147] Optionally, the processing module 420 includes:

[0148] The first processing unit 421 is used to process the pathological image to be processed through U 2 The encoder and decoder in the Net network model are processed to obtain a significant probability map;

[0149] The up-sampling unit 422 is used to 2 The saliency map fusion model in the Net network model upsamples the saliency probability map to the same size as the pathological image to be processed, and determines the upsampling result;

[0150] A first fusion unit 423 is used to perform fusion processing on the up-sampling results through a cascade operation to obtain a fusion result;

[0151] The second processing unit 424 is used to process the fusion result through a convolution operation and an activation function to obtain a segmentation result of the first region.

[0152] Optional, U 2 Net network model is constructed through the following steps:

[0153] Access to historical pathology images;

[0154] Preprocessing the historical pathological image to obtain a historical contour image of the first area;

[0155] Based on the historical pathological images and the historical contour images of the first area, a data set is constructed, and the data set is divided into a training set and a validation set;

[0156] Use the training set to construct U 2 Net network model is trained to obtain the U 2 Net network model;

[0157] Using the U to be verified in the verification set 2Net network model, according to the loss function minimization to treat the verification U 2 Net network model is optimized to obtain U 2 Net network model.

[0158] Optionally, the fusion module 430 includes:

[0159] The second fusion unit 431 is used to fuse the mask of the first region and the segmentation result of the first region according to a preset fusion rule to obtain a total image of the first region;

[0160] The determination unit 432 is configured to determine a target region based on the total image of the first region and the mask of the second region.

[0161] Optionally, the analysis module 440 includes:

[0162] A conversion unit 441 is used to convert the pathological image to be processed into an HSV color gamut image;

[0163] The acquisition unit 442 is used to acquire the pixel value of the target area in the HSV color gamut image in the S channel;

[0164] The analysis unit 443 is used to perform statistical analysis on the pixel saturation of the target area and the pixel value of the s channel to obtain a pathological index.

[0165] The pathological index determination device provided in the embodiment of the present application acquires the pathological image to be processed through the acquisition module, and then pre-processes the pathological image to be processed through the processing module to obtain the mask of the first area and the mask of the second area, and inputs the pathological image to be processed into the pre-trained deep learning segmentation model in parallel to obtain the segmentation result of the first area, and fuses the mask of the first area, the mask of the second area and the segmentation result of the first area through the fusion module to determine the target area, and then statistically analyzes the pixel saturation of the target area through the analysis module to determine the pathological index. This technical solution can not only extract the layout details of the pathological image by pre-processing the pathological image to be processed and performing image segmentation processing through the deep learning segmentation model in parallel, but also globally compare the image information obtained by the two, so as to accurately and quickly acquire the target area, and then accurately determine the pathological index, provide accurate quantitative numerical results for doctors, reduce the occurrence of unstable pathological indicators due to artificial subjective factors, assist doctors in making efficient and accurate judgments, and improve the efficiency of pathological diagnosis.

[0166] On the other hand, the terminal device provided in an embodiment of the present application includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the program, the pathological indicator determination method as described above is implemented.

[0167] Reference below Fig.11 , Fig.11 A schematic diagram of the structure of a computer system of a terminal device according to an embodiment of the present application.

[0168] like Fig.11 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 603 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0169] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0170] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a machine-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 603, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present application are executed.

[0171] It should be noted that the computer-readable medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0172] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of the code, and the aforementioned module, program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0173] The units or modules involved in the embodiments described in the present application may be implemented by software or by hardware. The units or modules described may also be arranged in a processor, for example, they may be described as: a processor comprising: an acquisition module, a processing module, a fusion module and an analysis module. The names of these units or modules do not, in certain cases, constitute limitations on the units or modules themselves, for example, the acquisition module may also be described as "used to acquire a pathological image to be processed including an annotated area".

[0174] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or may exist independently and not be assembled into the electronic device. The above computer-readable storage medium stores one or more programs, and when the above programs are used by one or more processors to execute the pathological index determination method described in the present application:

[0175] Acquire a pathological image to be processed including the marked area;

[0176] Preprocessing the pathological image to be processed of the marked area to obtain a mask of the first area and a mask of the second area, and inputting the pathological image to be processed into a pre-trained deep learning segmentation model in parallel to obtain a segmentation result of the first area, wherein the deep learning segmentation model has feature extraction and image segmentation capabilities;

[0177] Fusing the mask of the first area, the mask of the second area, and the segmentation result of the first area to determine a target area;

[0178] The pixel saturation of the target area is statistically analyzed to determine the pathological index.

[0179] In summary, the pathological index determination method, device, equipment and storage medium provided in the embodiments of the present application obtain the pathological image to be processed, then pre-process the pathological image to be processed to obtain the mask of the first area and the mask of the second area, and input the pathological image to be processed into a pre-trained deep learning segmentation model in parallel to obtain the segmentation result of the first area, and fuse the mask of the first area, the mask of the second area and the segmentation result of the first area to determine the target area, and then perform statistical analysis on the pixel saturation of the target area to determine the pathological index. This technical solution can not only extract the layout details of the pathological image by pre-processing the pathological image to be processed and performing image segmentation processing through a deep learning segmentation model in parallel, but also globally compare the image information obtained by the two, so as to accurately and quickly obtain the target area, and then accurately determine the pathological index, provide doctors with accurate quantitative numerical results, reduce the occurrence of unstable pathological indicators due to artificial subjective factors, assist doctors in making efficient and accurate judgments, and improve the efficiency of pathological diagnosis.

[0180] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the present application.

Claims

1. A method for determining pathological indicators, characterized in that: The method includes: Acquire a pathological image to be processed including the marked area; Preprocessing the pathological image to be processed of the marked area to obtain a mask of the first area and a mask of the second area, and inputting the pathological image to be processed into a pre-trained deep learning segmentation model in parallel to obtain a segmentation result of the first area, wherein the deep learning segmentation model has feature extraction and image segmentation capabilities; Fusing the mask of the first area, the mask of the second area, and the segmentation result of the first area to determine a target area; Performing statistical analysis on the pixel saturation of the target area to determine pathological indicators; Preprocessing the pathological image to be processed in the marked area to obtain a mask of the first area and a mask of the second area includes: Using a morphological filter to filter the pathological image to be processed to obtain a filtered pathological image; Binarizing the filtered pathological image by a local adaptive threshold method to determine a binary image of the first region; Performing contour screening processing based on the binary image of the first area to determine a mask of the first area; Converting the pathological image to be processed into a Lab color gamut image through the RGB channel; Thresholding the a channel in the Lab color gamut image to obtain a binary image of the second region; Performing contour screening processing based on the binary image of the second area to determine a mask of the second area; The mask of the first area is a blood vessel mask, and the mask of the second area is a blood mist mask.

2. The method according to claim 1, characterized in that The morphological filter includes a Sato filter, a Gaussian filter and a median filter. The pathological image to be processed is filtered using the morphological filter to obtain a filtered pathological image, including: Processing the pathological image to be processed through the Sato filter to filter out image interference information to obtain a first processed image; Performing linear smoothing filtering on the first processed image through the Gaussian filter to obtain a second processed image; The second processed image is subjected to noise information elimination processing by the median filter to obtain a filtered pathological image.

3. The method according to claim 1, characterized in that Thresholding the a channel in the Lab color gamut image to obtain a binary image of the second region includes: Obtaining the pixel value of the pathological image to be processed in the Lab color gamut image in channel a; Compare the pixel value of the a channel with a preset pixel threshold; Binarization is performed on the region corresponding to the pixel value of the a channel being less than the preset pixel threshold to generate a binary image of the second region.

4. The method according to claim 1, characterized in that: The deep learning segmentation model includes U 2 Net network model, input the pathological image to be processed into the pre-trained U 2 Net network model, the segmentation results of the first region are obtained, including: The pathological image to be processed is passed through the U 2 The encoder and decoder in the Net network model are processed to obtain a significant probability map; Through the U 2 Net network model, upsampling the salient probability map to the same size as the pathological image to be processed, and determining the upsampling result; The up-sampling results are fused by cascading operations to obtain fused results; The fusion result is processed through a convolution operation and an activation function to obtain a segmentation result of the first region.

5. The method according to claim 4, characterized in that The U 2 Net network model is constructed through the following steps: Access to historical pathology images; Preprocessing the historical pathological image to obtain a historical contour image of the first area; Based on the historical pathological images and the historical contour images of the first area, construct a data set, and divide the data set into a training set and a validation set; Use the training set to construct U 2 Net network model is trained to obtain the U 2 Net network model; Using the verification set to verify the U 2 Net network model, according to the loss function to minimize the U to be verified 2 Net network model is optimized to obtain U 2 Net network model.

6. The method according to claim 1, characterized in that The step of fusing the mask of the first region, the mask of the second region, and the segmentation result of the first region to determine a target region includes: fusing the mask of the first region and the segmentation result of the first region according to a preset fusion rule to obtain a total image of the first region; Based on the overall image of the first area and the mask of the second area, a target area is determined.

7. The method according to claim 6, characterized in that Performing statistical analysis on the pixel saturation of the target area to determine pathological indicators includes: Converting the pathological image to be processed into an HSV color gamut image; Obtain the pixel value of the target area in the HSV color gamut image in the S channel; The pixel saturation of the target area and the pixel value of the s channel are statistically analyzed to obtain a pathological index.

8. A pathological index determination device, characterized in that: The device comprises: An acquisition module, used for acquiring the pathological image to be processed; A processing module, used for preprocessing the pathological image to be processed to obtain a mask of the first region and a mask of the second region, and inputting the pathological image to be processed into a pre-trained deep learning segmentation model in parallel to obtain a segmentation result of the first region, wherein the deep learning segmentation model has image segmentation capability; a fusion module, configured to fuse the mask of the first region, the mask of the second region, and the segmentation result of the first region to determine a target region; An analysis module, used for performing statistical analysis on the pixel saturation of the target area to determine pathological indicators; The processing module is specifically used for: The pathological image to be processed is filtered using a morphological filter to obtain a filtered pathological image; Binarizing the filtered pathological image by a local adaptive threshold method to determine a binary image of the first region; Performing contour screening processing based on the binary image of the first region to determine a mask of the first region; Convert the pathological image to be processed into a Lab color gamut image through the RGB channel; Thresholding the a channel in the Lab color gamut image to obtain a binary image of the second region; Performing contour screening processing based on the binary image of the second area to determine a mask of the second area; The mask of the first area is a blood vessel mask, and the mask of the second area is a blood mist mask.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the pathological indicator determination method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, wherein the computer program is used to implement the pathological index determination method according to any one of claims 1 to 7.

Images