Immunotherapy sensitivity judgment method and device, electronic equipment and medium

Abstract

The invention discloses a method and a device for judging immunotherapy sensitivity, electronic equipment and a medium. The method comprises the following steps: acquiring a digital slice image of a target slice from scanning equipment; performing image preprocessing on the digital slice image to obtain a target image; inputting the target image into a pre-trained feature extraction model to perform feature extraction of the nebula features, and obtaining segmented images of the nebula features; wherein the feature extraction model is obtained through slice sample image training based on a convolutional neural network; based on the segmented image and the digital slice image, obtaining an area proportion of nebula features; and according to the area ratio, combining with a preset threshold range to obtain the sensitivity level of the target object to immunotherapy. The technical scheme of the invention shows remarkable beneficial effects in the aspects of improving pathological analysis precision, accelerating evaluation speed, optimizing medical resource utilization, promoting medical research and the like, has important significance in improving the treatment effect of a patient and realizing precise medical treatment, and can be widely applied to the technical field of image processing.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device, electronic equipment and medium for judging sensitivity of immunotherapy. Background Art

[0002] Breast cancer is one of the most common malignant tumors in women worldwide. Immunotherapy, as an important treatment method in modern medicine, can significantly improve the survival rate and quality of life of breast cancer patients. However, the effect of immunotherapy varies from person to person, and there are significant differences in the sensitivity of different patients to immunotherapy. Therefore, how to accurately judge the sensitivity of patients to immunotherapy before treatment and then formulate an individualized treatment plan is a hot spot and difficulty in current clinical research.

[0003] Currently, the methods for determining the sensitivity of breast cancer patients to immunotherapy mainly include genetic testing, protein analysis in tumor tissue or blood, etc. Although these methods can predict the patient's response to immunotherapy to a certain extent, they still have the following disadvantages:

[0004] 1. High cost: Genetic testing and protein expression analysis require expensive reagents and equipment, which increases the financial burden on patients.

[0005] 2. Long time: Genetic testing and protein expression analysis usually take a long time and may delay treatment.

[0006] 3. Limited accuracy: The accuracy of genetic testing is limited by the quality of sample storage and the doctor's experience, and the accuracy of protein expression testing is limited by the quality of antibodies and samples. Summary of the invention

[0007] The main purpose of the embodiments of the present invention is to provide a method, device, electronic device and medium for determining sensitivity to immunotherapy, in order to solve at least one problem of the prior art. The present invention can efficiently and accurately determine the sensitivity to immunotherapy.

[0008] To achieve the above object, one aspect of an embodiment of the present invention provides a method for determining sensitivity to immunotherapy, the method comprising:

[0009] Acquire a digital slice image of a target slice from a scanning device; wherein the target slice is obtained based on a pathological HE-stained slice of the target object;

[0010] Perform image preprocessing on the digital slice image to obtain the target image;

[0011] Input the target image into a pre-trained feature extraction model to extract the features of the nebula sign, and obtain a segmented image of the nebula sign; wherein the feature extraction model is obtained by training a slice sample image based on a convolutional neural network, and the slice sample image is annotated with a nebula sign label;

[0012] Based on the segmented images and digital slice images, the area proportion of the nebula sign was obtained;

[0013] Based on the area ratio and the preset threshold range, the sensitivity level of the target object to immunotherapy is obtained.

[0014] In some embodiments, the method further comprises the following steps:

[0015] Setting a pathology slice scanner, and adjusting scanning parameters of the pathology slice scanner in response to a parameter setting instruction of the operation object;

[0016] Among them, the scanning parameters include scanning resolution and objective lens specifications;

[0017] The pathology slide scanner after adjusting the scanning parameters is set as the scanning device.

[0018] In some embodiments, acquiring a digital slice image of a target slice from a scanning device comprises the following steps:

[0019] Scanning all slice areas of the target slice by a scanning device to obtain a scanning result;

[0020] The scanning results of all slice areas are spliced ​​to obtain a digital slice image.

[0021] In some embodiments, performing image preprocessing on the digital slide image comprises at least one of the following steps:

[0022] Perform denoising operations on digital slice images;

[0023] Perform enhancement operations on digital slice images.

[0024] In some embodiments, the method further comprises the following steps:

[0025] Configure the convolutional neural network;

[0026] Input the slice sample image into the convolutional neural network for image segmentation to obtain the segmentation result;

[0027] According to the segmentation result and the label of the nebula sign in the slice sample image, the loss value is obtained using the loss function;

[0028] The parameters of the convolutional neural network are adjusted based on the loss value, and the feature extraction model is trained.

[0029] In some embodiments, obtaining the area ratio of the nebula sign based on the segmented image and the digital slice image includes the following steps:

[0030] Perform pixel statistics on the segmented image and the digital slice image in sequence to obtain a first pixel value and a second pixel value correspondingly;

[0031] The area ratio of the nebula sign in the digital slice image is obtained according to the ratio of the first pixel value to the second pixel value.

[0032] In some embodiments, the method further comprises the following steps:

[0033] Acquiring clinical data of multiple immunotherapy subjects; wherein the clinical data includes the area ratio of the nebula sign in the pathological HE staining section of each immunotherapy subject and the sensitivity result of immunotherapy;

[0034] A statistical analysis was performed on the correspondence between the area percentage values ​​of all immunotherapy subjects and the sensitivity results. Based on the results of the statistical analysis and the preset grade standards, a mapping relationship table of each sensitivity grade and the threshold range was compiled.

[0035] To achieve the above object, another aspect of an embodiment of the present invention provides a device for determining sensitivity to immunotherapy, the device comprising:

[0036] The first module is used to obtain a digital slice image of a target slice from a scanning device; wherein the target slice is obtained based on a pathological HE-stained slice of the target object;

[0037] The second module is used to perform image preprocessing on the digital slice image to obtain a target image;

[0038] The third module is used to input the target image into a pre-trained feature extraction model to extract the features of the nebula sign, and obtain a segmented image of the nebula sign; wherein the feature extraction model is obtained by training the slice sample image based on a convolutional neural network, and the slice sample image is annotated with a nebula sign label;

[0039] The fourth module is used to obtain the area ratio of the nebula sign based on the segmented image and the digital slice image;

[0040] The fifth module is used to obtain the sensitivity level of the target object to immunotherapy based on the area ratio and the preset threshold range.

[0041] In some embodiments, the apparatus further comprises:

[0042] The sixth module is used to set the pathology slice scanner, and adjust the scanning parameters of the pathology slice scanner in response to the parameter setting instruction of the operation object;

[0043] Among them, the scanning parameters include scanning resolution and objective lens specifications;

[0044] The seventh module is used to set the pathology slice scanner after completing the adjustment of the scanning parameters as a scanning device.

[0045] In some embodiments, the apparatus further comprises:

[0046] The eighth module is used to obtain clinical data of multiple immunotherapy subjects; wherein the clinical data includes the area ratio of the nebula sign in the pathological HE staining section of each immunotherapy subject and the sensitivity result of immunotherapy;

[0047] The ninth module is used to perform statistical analysis on the correspondence between the area percentage values ​​of all immunotherapy subjects and the sensitivity results. According to the results of the statistical analysis and the preset grade standards, a mapping relationship table of each sensitivity level and the threshold range is obtained.

[0048] To achieve the above objective, another aspect of an embodiment of the present invention provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above method when executing the computer program.

[0049] To achieve the above objective, another aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above method is implemented.

[0050] The present invention provides a method, device, electronic device and medium for judging the sensitivity of immunotherapy. The scheme obtains a digital slice image of a target slice from a scanning device; wherein the target slice is obtained based on a pathological HE-stained slice of a target object; image preprocessing is performed on the digital slice image to obtain a target image; the target image is input into a pre-trained feature extraction model to extract the features of a nebula sign, and a segmented image of the nebula sign is obtained; wherein the feature extraction model is obtained by training a slice sample image based on a convolutional neural network, and the slice sample image is annotated with a nebula sign label; based on the segmented image and the digital slice image, the area ratio of the nebula sign is obtained; according to the area ratio, the sensitivity level of the target object to immunotherapy is obtained in combination with a preset threshold range. The embodiments of the present invention include at least the following beneficial effects:

[0051] Improve assessment accuracy and efficiency: By using scanning equipment to obtain digital slice images, the conversion from traditional physical slices to digital information is realized, providing a high-quality image foundation for subsequent analysis. The image preprocessing step can effectively remove noise and enhance image features, laying a solid foundation for subsequent feature extraction, thereby improving the accuracy and efficiency of the entire assessment process.

[0052] Accurate feature extraction: Using a pre-trained feature extraction model based on a convolutional neural network and trained with a large number of slice sample images labeled with the nebula sign, the model can accurately identify and extract the nebula sign, a key pathological feature. The application of this technology greatly improves the accuracy and robustness of feature recognition and avoids the subjectivity and errors of manual interpretation.

[0053] Quantitative analysis improves decision support: By comparing the segmented image with the original digital slice image, the area ratio of the nebula sign can be accurately calculated. This quantitative indicator provides doctors with an objective and specific basis for evaluation. Combined with the preset threshold range, it can scientifically and accurately divide the sensitivity level of the target object to immunotherapy, providing strong data support for the formulation of personalized treatment plans.

[0054] Promote the optimal allocation of medical resources: This technical solution realizes the automation and intelligence of pathological analysis, reduces the high dependence on professional pathologists, and helps to alleviate the problem of tight medical resources. At the same time, fast and accurate evaluation results can accelerate the patient's treatment process and improve the overall efficiency of medical services.

[0055] Promoting progress in medical research: By analyzing and learning a large number of slice images, not only the feature extraction model is optimized, but also a rich data resource is provided for pathological research, which helps to deeply explore the association mechanism between the nebula sign and immunotherapy sensitivity, and further promote research progress in related fields.

[0056] In summary, the technical solution of the present invention has shown significant beneficial effects in improving the accuracy of pathological analysis, accelerating the evaluation speed, optimizing the utilization of medical resources and promoting medical research, and is of great significance for improving patient treatment effects and realizing precision medicine. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is a flow chart of a method for determining sensitivity to immunotherapy provided by an embodiment of the present invention;

[0058] Figure 2 is a schematic diagram of an example of the structural pathological characteristics of the nebula sign provided by an embodiment of the present invention;

[0059] Figure 3 is a schematic diagram of the architecture of a device for determining immunotherapy sensitivity provided by an embodiment of the present invention;

[0060] Figure 4 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the attached claims.

[0062] It is understood that the terms "first", "second", etc. used in the present invention can be used to describe various concepts in the present invention, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determination".

[0063] The terms "at least one", "multiple", "each", "any", etc. used in the present invention, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0064] Unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the present invention are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0065] The method for judging the sensitivity of immunotherapy provided by the embodiment of the present invention relates to the field of image processing technology. The method for judging the sensitivity of immunotherapy provided by the embodiment of the present invention can be applied to a terminal, can be applied to a server, or can be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The cloud server, the server can also be a node server in a blockchain network; the software can be an application for realizing the method for judging the sensitivity of immunotherapy, etc., but is not limited to the above forms.

[0066] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0067] Figure 1 is an optional flow chart of the method for determining the sensitivity of immunotherapy provided in an embodiment of the present invention. Figure 1 The method may include but is not limited to steps S100 to S500.

[0068] S100, acquiring a digital slice image of a target slice from a scanning device;

[0069] Wherein, the target slice is obtained based on the pathological HE-stained slice of the target object;

[0070] Optionally, in some embodiments, the method may further include a step of obtaining a pathological HE-stained section based on tumor tissue processing of the target object.

[0071] For example, in some specific implementations, obtaining pathological HE-stained sections can be achieved through the following process (specific parameters in the processing process can be adaptively adjusted according to actual application requirements):

[0072] 1) Specimen collection and fixation:

[0073] a) Collect an appropriate amount of sample (about 0.5-1 cm3) from the patient's tumor tissue;

[0074] b) Immediately place the sample in 10% neutral buffered formalin solution for fixation, the fixation time is generally 6-24 hours;

[0075] 2) Dehydration and clearing:

[0076] a) Dehydrate the fixed samples by immersing them in 70%, 80%, 95% and 100% alcohol solutions in sequence, with each immersion time being 1-2 hours;

[0077] b) then placing the sample in a xylene or dimethylbenzene solution for a transparent treatment for about 1-2 hours;

[0078] 3) Embedding:

[0079] a) Transfer the cleared sample into the melted paraffin solution, ensuring that it is completely immersed;

[0080] b) Soak at 60-65°C for 2-4 hours to allow the sample to completely penetrate the paraffin;

[0081] c) Take out the sample, quickly put it into a paraffin mold made of metal or plastic, and cool it to solidify;

[0082] 4) Slicing:

[0083] a) Use a rotary microtome to cut the embedded paraffin blocks at a thickness of 3-5 μm;

[0084] b) Be careful when slicing to avoid tissue breaks or wrinkles;

[0085] c) Carefully transfer the cut sections to a water bath to flatten the tissue;

[0086] 5) Dewaxing and staining:

[0087] a) Dewaxing the sections by immersing them in xylene or dimethylbenzene solution for about 5-10 minutes;

[0088] b) Then rehydrate with gradient alcohol solutions (100%, 95%, 80%, 70%), each for 2-3 minutes;

[0089] c) Finally, the sections were stained in a hematoxylin solution for 10-20 minutes, and then stained with eosin for 1-3 minutes;

[0090] 6) Sealing:

[0091] a) Wash the stained sections with water and dehydrate them with gradient alcohol;

[0092] b) Add neutral resin to seal the slide and cover with a cover glass;

[0093] c) Wait until the resin solidifies before observing.

[0094] Optionally, in some embodiments, the method may further include the following steps: setting a pathology slice scanner, adjusting scanning parameters of the pathology slice scanner in response to a parameter setting instruction of the operation object; wherein the scanning parameters include scanning resolution and objective lens specifications; and setting the pathology slice scanner that has completed the adjustment of the scanning parameters as a scanning device.

[0095] For example, in some specific implementations, the scanner settings may be implemented as follows:

[0096] a) Use a high-resolution fully automatic pathology slide scanner;

[0097] b) Set the scanning resolution to 0.25-0.5 μm / pixel and shoot with a 40X objective lens (the specific scanning parameters can be adaptively adjusted according to actual application requirements).

[0098] It should be noted that, in some embodiments, obtaining a digital slice image of a target slice from a scanning device may include the following steps: scanning all slice areas of the target slice by the scanning device to obtain a scanning result; and splicing the scanning results of all slice areas to obtain a digital slice image.

[0099] For example, in some specific implementations, the target slice is first placed in the scanning device, i.e., the slice is loaded, which can be realized by carefully loading the prepared slice onto the slide tray of the scanner, ensuring that the slice is flat and wrinkle-free and fits the slide perfectly. Then, automatic scanning is realized by the scanning device: the scanning device is started, and the scanner is automatically scanned at high resolution over the entire slice area. During the scanning process, the device will automatically focus, stitch, and generate a complete digital slice image.

[0100] S200, performing image preprocessing on the digital slice image to obtain a target image;

[0101] It should be noted that, in some embodiments, performing image preprocessing on the digital slice image may include at least one of the following steps: performing a denoising operation on the digital slice image; and performing an enhancement operation on the digital slice image.

[0102] For example, in some specific implementations, image preprocessing may be implemented as: performing preprocessing operations such as denoising and enhancement on the pathological HE-stained slice image to improve image quality.

[0103] S300, inputting the target image into a pre-trained feature extraction model to extract features of the nebula sign, and obtaining a segmented image of the nebula sign;

[0104] Among them, the feature extraction model is obtained by training the slice sample images based on the convolutional neural network, and the slice sample images are annotated with the nebula sign label;

[0105] It should be noted that, in some embodiments, the method may further include the following steps: configuring a convolutional neural network; inputting the slice sample image into the convolutional neural network for image segmentation to obtain a segmentation result; obtaining a loss value using a loss function based on the segmentation result and the label of the nebula sign in the slice sample image; adjusting the parameters of the convolutional neural network based on the loss value, and training to obtain a feature extraction model.

[0106] For example, in some specific embodiments, the slice sample image can come from the clinical data of the immunotherapy subject, and the tumor tissue of the immunotherapy subject is specifically sliced ​​and then scanned (the principle logic is the same as the principle logic of step S100 and its specific implementation method, which will not be repeated here). After the scanning is completed, a high-resolution digital slice image file will be obtained. Professional pathology image analysis software or experts in the corresponding field can be used to manually process the image to view, annotate, analyze, and obtain relevant information of the label, thereby assisting the training of the feature extraction model.

[0107] Furthermore, in some embodiments, in order to more specifically explain how to configure a convolutional neural network (CNN) and train a feature extraction model, the following is an example supplementary explanation based on the above content:

[0108] 1. Configure the convolutional neural network:

[0109] First, we need to build a convolutional neural network suitable for image segmentation tasks. This network can contain the following key layers:

[0110] Input layer: accepts a fixed-size slice sample image, such as a 256x256 pixel RGB image.

[0111] Convolutional layer: Use multiple convolution kernels to extract features from the image. These convolution kernels can have different sizes and strides to capture features at different scales.

[0112] Pooling layer: After the convolutional layer, a pooling operation (such as max pooling) is used to reduce the size of the feature map while retaining important features.

[0113] Fully connected layer: At the end of the network, a fully connected layer is used to map the convolutional and pooled features to the output space for image segmentation.

[0114] Output layer: Output a segmentation map of the same size as the input image, where each pixel represents the probability that the position belongs to the nebula sign.

[0115] In addition, some advanced modules such as residual connections (skip connections in ResNet), batch normalization, and Dropout can be added to the network to improve the performance and generalization ability of the network.

[0116] 2. Image segmentation and loss function:

[0117] Image segmentation: Input the slice sample image into the configured convolutional neural network, and the network will output a segmentation result, that is, the probability map of each pixel belonging to the nebula sign.

[0118] Loss function: In order to evaluate the difference between the segmentation result and the true label, a suitable loss function is used. For image segmentation tasks, commonly used loss functions include Cross-Entropy Loss and Dice Loss. These loss functions can measure the pixel-level difference between the predicted segmentation map and the true label.

[0119] 3. Parameter adjustment and model training:

[0120] Calculate the loss value: Based on the segmentation results and the true label of the nebula sign in the slice sample, the loss value is calculated using the loss function. This loss value reflects the degree of fit of the model to the training data under the current network parameters.

[0121] Parameter adjustment: Based on the calculated loss value, the gradient is calculated using the backpropagation algorithm, and the network parameters are updated through the optimizer (such as Adam, SGD, etc.). This process will be iterated many times until the loss value converges to a smaller value.

[0122] Model training: During the training process, it is necessary to continuously input slice sample images into the network, calculate the loss value, and adjust the parameters. At the same time, in order to avoid overfitting, data augmentation and regularization techniques can also be used. After enough training iterations, a trained feature extraction model will be obtained, which can accurately segment the nebula sign in the slice image.

[0123] Through the above steps, a convolutional neural network can be successfully configured and trained using slice sample images. During the training process, by continuously adjusting the network parameters to minimize the loss value, a feature extraction model that can accurately extract the nebula sign features is finally obtained. This model can be used for subsequent image processing and analysis tasks, such as evaluating the sensitivity level of the target object to immunotherapy.

[0124] In some specific application scenarios, manual recognition and machine learning algorithms can be used to analyze the pathological HE-stained sections of breast cancer patients and identify the structural pathological features of the "nebula sign" in the sections, such as Figure 2 The following is an example of the structural pathological features of the nebula sign. The specific steps are as follows:

[0125] a) Image preprocessing: Perform denoising, enhancement and other preprocessing operations on the pathological HE-stained slice images to improve image quality.

[0126] b) Feature extraction: The convolutional neural network (CNN) deep learning algorithm is used to extract the structural pathological features of the "nebula sign" from the image.

[0127] c) Feature recognition: The extracted features are classified and identified through the trained model to determine whether the “nebula sign” structural pathological features exist.

[0128] S400, based on the segmented image and digital slice image, obtain the area ratio of the nebula sign;

[0129] It should be noted that, in some embodiments, obtaining the area ratio of the nebula sign based on the segmented image and the digital slice image may include the following steps: performing pixel statistics on the segmented image and the digital slice image in turn to obtain a first pixel value and a second pixel value correspondingly; and obtaining the area ratio of the nebula sign in the digital slice image based on the ratio of the first pixel value to the second pixel value.

[0130] For example, in some specific implementations, the area calculation can be implemented as follows: pixel statistics are performed on the segmented area to calculate its area ratio in the entire slice image. The entire cancer tissue area can also be used as a baseline correction.

[0131] S500: Obtain the sensitivity level of the target object to immunotherapy based on the area ratio and a preset threshold range.

[0132] It should be noted that, in some embodiments, the method may further include the following steps: obtaining clinical data of multiple immunotherapy subjects; wherein the clinical data include the area ratio of the nebula sign in the pathological HE-stained sections of each immunotherapy subject and the sensitivity results of the immunotherapy; performing statistical analysis on the correspondence between the area ratio values ​​and the sensitivity results of all immunotherapy subjects, and obtaining a mapping relationship table between each sensitivity level and the threshold range based on the results of the statistical analysis and the preset level standards.

[0133] For example, in some specific embodiments, the statistical results can be input into a pre-established analysis model to determine the sensitivity of the patient to immunotherapy. In some specific application scenarios, the threshold can be analyzed for the optimal threshold based on the clinical data of the patient's immunotherapy effect, and the threshold can be screened using X-tile software. A threshold that can predict the sensitivity of the patient to immunotherapy is developed.

[0134] In order to explain the principle of the technical solution of the present invention in detail, the overall process of the present invention is described below in combination with some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and cannot be regarded as a limitation of the present invention.

[0135] First of all, it should be noted that the commonly used breast cancer immunotherapy sensitivity prediction schemes include:

[0136] 1.1.1 Genetic testing:

[0137] Genetic testing predicts the effect of immunotherapy by analyzing the mutation and expression of key genes in the patient's tumor cells and their microenvironment. Common genetic testing methods include: PD-L1 expression: PD-L1 (programmed death ligand-1) is the target of immune checkpoint inhibitors, and the expression level of PD-L1 in tumor cells and tumor microenvironment is related to the effect of immunotherapy. Tumor mutation burden (TMB): TMB refers to the number of mutations present in the tumor, and high TMB is usually associated with a better response to immunotherapy. Microsatellite instability (MSI): Tumors with high MSI are more likely to respond to immunotherapy.

[0138] 1.1.2 Protein expression analysis:

[0139] The expression of key proteins in the patient's tumor tissue is detected by immunohistochemistry (IHC) and other methods to predict the effect of immunotherapy. Commonly used protein markers include: PD-L1: Directly detect the expression level of PD-L1 protein. T cell infiltration: Analyze the number and activity of CD8+T cells infiltrating in tumor tissue.

[0140] 1.1.3 Blood biomarkers:

[0141] The effect of immunotherapy can be predicted by analyzing biomarkers in the patient's blood. Commonly used blood markers include: Circulating tumor DNA (ctDNA): Detects mutations in tumor DNA in the blood. Exosomes: The protein and nucleic acid components in exosomes can reflect the immune characteristics of the tumor.

[0142] However, although these methods can predict the patient's response to immunotherapy to a certain extent, they still have disadvantages such as high cost, long time and limited accuracy. In view of this, the present invention detects the most common pathological HE sections of the tumor, analyzes the unique and easily identifiable pathological structure therein, and judges whether the patient is effective for immunotherapy by the proportion of the structure on the section. The technical solution of the present invention can be specifically implemented as follows:

[0143] 1) Specimen collection and fixation:

[0144] a) Collect an appropriate amount of sample (about 0.5-1 cm3) from the patient's tumor tissue;

[0145] b) Immediately place the sample in 10% neutral buffered formalin solution for fixation, the fixation time is generally 6-24 hours;

[0146] 2) Dehydration and clearing:

[0147] a) Dehydrate the fixed samples by immersing them in 70%, 80%, 95% and 100% alcohol solutions in sequence, with each immersion time being 1-2 hours;

[0148] b) then placing the sample in a xylene or dimethylbenzene solution for a transparent treatment for about 1-2 hours;

[0149] 3) Embedding:

[0150] a) Transfer the cleared sample into the melted paraffin solution, ensuring that it is completely immersed;

[0151] b) Soak at 60-65°C for 2-4 hours to allow the sample to completely penetrate the paraffin;

[0152] c) Take out the sample, quickly put it into a paraffin mold made of metal or plastic, and cool it to solidify;

[0153] 4) Slicing:

[0154] a) Use a rotary microtome to cut the embedded paraffin blocks at a thickness of 3-5 μm;

[0155] b) Be careful when slicing to avoid tissue breaks or wrinkles;

[0156] c) Carefully transfer the cut sections to a water bath to flatten the tissue;

[0157] 5) Dewaxing and staining:

[0158] a) Dewaxing the sections by immersing them in xylene or dimethylbenzene solution for about 5-10 minutes;

[0159] b) Then rehydrate with gradient alcohol solutions (100%, 95%, 80%, 70%), each for 2-3 minutes;

[0160] c) Finally, the sections were stained in a hematoxylin solution for 10-20 minutes, and then stained with eosin for 1-3 minutes;

[0161] 6) Sealing:

[0162] a) Wash the stained sections with water and dehydrate them with gradient alcohol;

[0163] b) Add neutral resin to seal the slide and cover with a cover glass;

[0164] c) Observe after the resin solidifies;

[0165] 7) Scanner settings:

[0166] a) Use a high-resolution fully automatic pathology slide scanner;

[0167] b) Set the scanning resolution to 0.25-0.5 μm / pixel and shoot with a 40X objective lens (the specific scanning parameters can be adaptively adjusted according to actual application requirements).

[0168] 8) Slicing:

[0169] a) Carefully load the prepared slides onto the slide tray of the scanner, making sure the slides are flat and wrinkle-free and fit perfectly onto the slide.

[0170] 9) Automatic scanning:

[0171] a) Start the scanning device and let the scanner automatically scan the entire slice area with high resolution. During the scanning process, the device will automatically focus, stitch and generate a complete digital slice image.

[0172] 10) Image processing:

[0173] a) After the scan is completed, a high-resolution digital slice image file will be obtained, and professional pathology image analysis software can be used to view, annotate, and analyze the image.

[0174] 11) The present invention uses artificial recognition and establishes a machine learning algorithm to analyze the pathological HE-stained sections of breast cancer patients and identify the structural pathological characteristics of the "nebula sign" in the sections. The specific steps are as follows:

[0175] a) Image preprocessing: Perform denoising, enhancement and other preprocessing operations on the pathological HE-stained slice images to improve image quality.

[0176] b) Feature extraction: The convolutional neural network (CNN) deep learning algorithm is used to extract the structural pathological features of the "nebula sign" from the image.

[0177] c) Feature recognition: The extracted features are classified and identified through the trained model to determine whether the “nebula sign” structural pathological features exist.

[0178] 12) After identifying the structural pathological features of the "nebula sign", the present invention further performs statistics on the areas of these features. The specific method is as follows:

[0179] a) Regional segmentation: Use image segmentation algorithm to segment the area where the structural pathological features of the "nebula sign" are located from the image.

[0180] b) Area calculation: Perform pixel statistics on the segmented area and calculate its area ratio in the entire slice image.

[0181] c) Data analysis: Input the statistical results into a pre-established analysis model to determine the patient's sensitivity to immunotherapy.

[0182] In summary, the present invention provides a fast, low-cost, and highly accurate method for judging the sensitivity of patients to immunotherapy by identifying and counting the structural pathological features of the "nebula sign" in HE-stained sections of breast cancer pathology, which has significant clinical application value. Compared with the prior art, the present invention has at least the following beneficial effects:

[0183] 1. Low cost: The present invention uses pathological HE-stained slice images as analysis objects, does not require expensive gene detection and protein expression analysis equipment, and reduces costs.

[0184] 2. Fast and efficient: Through automated image processing and machine learning algorithms, the present invention can complete the identification and area statistics of the structural pathological features of the "nebula sign" in a relatively short time, thus improving efficiency.

[0185] 3. High accuracy: The present invention uses a deep learning algorithm to accurately identify tiny lesions in pathological sections, thereby improving the accuracy of judgment.

[0186] 4. High repeatability: The image processing and analysis method of the present invention has high repeatability and can stably identify the structural pathological characteristics of the "nebula sign" in different pathological sections.

[0187] like Figure 3 As shown, the embodiment of the present invention further provides a device 900 for determining sensitivity of immunotherapy, which may include:

[0188] The first module 901 is used to obtain a digital slice image of a target slice from a scanning device; wherein the target slice is obtained based on a pathological HE-stained slice of the target object;

[0189] The second module 902 is used to perform image preprocessing on the digital slice image to obtain a target image;

[0190] The third module 903 is used to input the target image into a pre-trained feature extraction model to extract the features of the nebula sign, and obtain a segmented image of the nebula sign; wherein the feature extraction model is obtained by training the slice sample image based on a convolutional neural network, and the slice sample image is annotated with a nebula sign label;

[0191] The fourth module 904 is used to obtain the area ratio of the nebula sign based on the segmented image and the digital slice image;

[0192] The fifth module 905 is used to obtain the sensitivity level of the target object to immunotherapy based on the area ratio and a preset threshold range.

[0193] In some embodiments, the apparatus may further include:

[0194] The sixth module is used to set the pathology slice scanner, and adjust the scanning parameters of the pathology slice scanner in response to the parameter setting instruction of the operation object;

[0195] Among them, the scanning parameters include scanning resolution and objective lens specifications;

[0196] The seventh module is used to set the pathology slice scanner after completing the adjustment of the scanning parameters as a scanning device.

[0197] In some embodiments, the apparatus may further include:

[0198] The eighth module is used to obtain clinical data of multiple immunotherapy subjects; wherein the clinical data includes the area ratio of the nebula sign in the pathological HE staining section of each immunotherapy subject and the sensitivity result of immunotherapy;

[0199] The ninth module is used to perform statistical analysis on the correspondence between the area percentage values ​​of all immunotherapy subjects and the sensitivity results. According to the results of the statistical analysis and the preset grade standards, a mapping relationship table of each sensitivity level and the threshold range is obtained.

[0200] The contents of the method embodiments of the present invention are all applicable to the device embodiments. The functions specifically implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0201] The embodiment of the present invention further provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method for determining immunotherapy sensitivity when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0202] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0203] See also Figure 4 , Figure 4 The hardware structure of an electronic device 1000 of another embodiment is illustrated. The electronic device 1000 includes:

[0204] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0205] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the method for determining the sensitivity of immunotherapy in the embodiments of the present invention;

[0206] Input / output interface 1003, used to implement information input and output;

[0207] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0208] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0209] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0210] An embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method for determining the sensitivity of immunotherapy is implemented.

[0211] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0212] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0213] The method for judging the sensitivity of immunotherapy, the device for judging the sensitivity of immunotherapy, the electronic device and the medium provided by the embodiment of the present invention obtain a digital slice image of the target slice from a scanning device; wherein the target slice is obtained based on the pathological HE-stained slice of the target object; the digital slice image is subjected to image preprocessing to obtain the target image; the target image is input into a pre-trained feature extraction model to extract the features of the nebula sign, and a segmented image of the nebula sign is obtained; wherein the feature extraction model is obtained by training the slice sample image based on a convolutional neural network, and the slice sample image is annotated with a label of the nebula sign; based on the segmented image and the digital slice image, the area ratio of the nebula sign is obtained; according to the area ratio, the sensitivity level of the target object to immunotherapy is obtained in combination with a preset threshold range. The technical solution of the present invention has shown significant beneficial effects in improving the accuracy of pathological analysis, accelerating the evaluation speed, optimizing the utilization of medical resources and promoting medical research, and is of great significance for improving the treatment effect of patients and realizing precision medicine.

[0214] The embodiments described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art can appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0215] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present invention and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0216] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0217] Those skilled in the art will appreciate that all or some of the steps, devices, and functional modules / units in the above disclosed methods may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0218] The terms "first", "second", "third", "fourth", etc. (if any) in the description of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0219] It should be understood that in the present invention, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0220] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0221] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0222] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0223] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store programs.

[0224] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the embodiments of the present invention is not limited thereby. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present invention shall be within the scope of the rights of the embodiments of the present invention.

Claims

1. A method for determining sensitivity to immunotherapy, characterized in that: The method comprises the following steps: Acquire a digital slice image of a target slice from a scanning device; wherein the target slice is obtained based on a pathological HE-stained slice of the target object; Performing image preprocessing on the digital slice image to obtain a target image; Inputting the target image into a pre-trained feature extraction model to extract features of the nebula sign, and obtaining a segmented image of the nebula sign; wherein the feature extraction model is obtained by training a slice sample image based on a convolutional neural network, and the slice sample image is annotated with a label of the nebula sign; Based on the segmented image and the digital slice image, obtaining the area ratio of the nebula sign; According to the area ratio and in combination with a preset threshold range, the sensitivity level of the target object to immunotherapy is obtained.

2. The method for determining immunotherapy sensitivity according to claim 1, characterized in that: The method further comprises the following steps: Setting a pathology slice scanner, and adjusting scanning parameters of the pathology slice scanner in response to a parameter setting instruction of an operation object; Wherein, the scanning parameters include scanning resolution and objective lens specifications; The pathological slice scanner that has completed the adjustment of the scanning parameters is set as the scanning device.

3. The method for determining immunotherapy sensitivity according to claim 1, characterized in that: The method of acquiring a digital slice image of a target slice from a scanning device comprises the following steps: Scanning all slice areas of the target slice by the scanning device to obtain a scanning result; The scanning results of all the slice areas are spliced ​​to obtain the digital slice image.

4. The method for determining immunotherapy sensitivity according to claim 1, characterized in that: The image preprocessing of the digital slice image comprises at least one of the following steps: Performing a denoising operation on the digital slice image; An enhancement operation is performed on the digital slice image.

5. The method for determining immunotherapy sensitivity according to claim 1, characterized in that: The method further comprises the following steps: configuring the convolutional neural network; Inputting the slice sample image into the convolutional neural network for image segmentation to obtain a segmentation result; Obtaining a loss value using a loss function according to the segmentation result and the label of the nebula sign in the slice sample image; The parameters of the convolutional neural network are adjusted based on the loss value, and the feature extraction model is obtained by training.

6. The method for determining immunotherapy sensitivity according to claim 1, characterized in that: The obtaining of the area ratio of the nebula sign based on the segmented image and the digital slice image comprises the following steps: Performing pixel statistics on the segmented image and the digital slice image in sequence to obtain a first pixel value and a second pixel value correspondingly; The area proportion of the nebula sign in the digital slice image is obtained according to the ratio of the first pixel value to the second pixel value.

7. The method for determining immunotherapy sensitivity according to claim 1, characterized in that: The method further comprises the following steps: Acquiring clinical data of multiple immunotherapy subjects; wherein the clinical data includes the area ratio of the nebula sign in the pathological HE-stained sections of each immunotherapy subject and the sensitivity result of immunotherapy; A statistical analysis is performed on the correspondence between the area percentage values ​​and the sensitivity results of all the immunotherapy subjects, and a mapping relationship table between each sensitivity level and the threshold range is obtained based on the results of the statistical analysis and a preset level standard.

8. A device for determining sensitivity to immunotherapy, characterized in that: The device comprises: The first module is used to obtain a digital slice image of a target slice from a scanning device; wherein the target slice is obtained based on a pathological HE-stained slice of the target object; The second module is used to perform image preprocessing on the digital slice image to obtain a target image; The third module is used to input the target image into a pre-trained feature extraction model to extract the features of the nebula sign, and obtain a segmented image of the nebula sign; wherein the feature extraction model is obtained by training a slice sample image based on a convolutional neural network, and the slice sample image is annotated with a label of the nebula sign; A fourth module is used to obtain the area ratio of the nebula sign based on the segmented image and the digital slice image; The fifth module is used to obtain the sensitivity level of the target object to immunotherapy based on the area ratio and a preset threshold range.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images