Postoperative recurrence prediction method and device, equipment and storage medium

Abstract

The invention discloses a postoperative recurrence prediction method and device, equipment and a storage medium. The method aims at obtaining a first image pyramid of a dyed slide image of a target object, obtaining a plurality of first image blocks through the first image pyramid, and extracting feature vectors of the first image blocks; obtaining a second image pyramid of the immunohistochemical slide image of the target object, obtaining a plurality of second image blocks through the second image pyramid, and extracting feature vectors of the second image blocks; determining a postoperative prediction result and a result confidence coefficient of the first image block based on the feature vector of the first image block, and determining a postoperative prediction result and a result confidence coefficient of the second image block based on the feature vector of the second image block; determining a target feature vector in the feature vectors of the plurality of first image blocks and the feature vectors of the plurality of second image blocks according to the result confidence of the first image blocks and the result confidence of the second image blocks; and determining a postoperative prediction result of the target object according to the feature vector and the target feature vector.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular, to a method, device, equipment, and storage medium for predicting postoperative recurrence. Background Art

[0002] Bladder cancer (BCa) is the ninth most common malignant tumor globally, and its incidence and mortality rate rank among the top in urinary system tumors. According to the depth of tumor invasion, bladder cancer can be divided into non-muscle-invasive bladder cancer (NMIBC) and muscle-invasive bladder cancer (MIBC). NMIBC accounts for approximately 75% of newly diagnosed bladder cancer cases, and its characteristic is that the tumor is limited to the bladder mucosa layer or submucosa layer without invading the bladder muscle layer. Although the malignancy of NMIBC is relatively low, it has a high recurrence rate and progression risk, bringing a heavy disease burden to patients. According to statistics, the recurrence rate of NMIBC patients within 5 years after the initial diagnosis is as high as 50%-70%, and about 10%-20% of these patients will progress to MIBC. This high recurrence rate and progression risk not only increase the psychological pressure of patients but also significantly increase the medical cost and social and economic burden. For patients with high-risk early recurrence of NMIBC, more aggressive treatment strategies should be adopted.

[0003] The standard treatment method for NMIBC is transurethral resection of bladder tumor (TURBT), which removes visible tumor tissue through surgery. However, even after TURBT treatment, NMIBC patients still face a high recurrence risk. It is reported that approximately 40% of patients will experience tumor recurrence within 1 year after surgery, and more than 20% of the recurrent patients will further progress to MIBC. This progression not only increases the complexity of treatment but also significantly reduces the survival rate of patients.

[0004] To reduce the recurrence and progression risk of NMIBC, a series of adjuvant treatments and follow-up programs are carried out on patients, including postoperative intravesical instillation of chemotherapy drugs or immunotherapy drugs, as well as regular cystoscopy and imaging follow-up. However, there are still about 30% of high-risk NMIBC patients who experience tumor recurrence within 2 years after starting maintenance BCG treatment. For patients who are non-responsive to BCG treatment, further intravesical BCG treatment often has limited effects. In addition, these invasive treatment and follow-up methods often bring great pain and economic burden to patients, further reducing the quality of life of bladder cancer patients.

[0005] Studies have shown that patients with early recurrence (usually within 1 year after surgery) have a higher recurrence frequency, worse bladder-preserving survival and overall survival than those with late recurrence (more than 1 year after surgery). That is, NMIBC patients at high risk of early recurrence require more aggressive treatment strategies to reduce the recurrence risk and improve the prognosis. Therefore, it is crucial to accurately stratify NMIBC patients based on the recurrence risk and treatment response to develop individualized treatment and follow-up strategies.

[0006] Currently, traditional postoperative recurrence prediction models mainly use scoring models, which perform risk stratification based on clinical and pathological characteristics. Although these models are widely used in clinical practice, their predictive ability is limited, especially in predicting early recurrence of high-grade NMIBC. Therefore, there is an urgent need for a more effective model to predict the recurrence of NMIBC patients after TURBT. Summary of the Invention

[0007] This application provides a postoperative recurrence prediction method, device, equipment and storage medium, which combines the global features and relatively highly correlated local features of multi-size images of stained glass slides and immunohistochemical glass slides to more comprehensively capture information related to postoperative recurrence, so as to accurately predict the postoperative recurrence situation of the target object in the early postoperative period, solve the problem that the scoring model in the prior art cannot accurately predict the recurrence situation in the early postoperative period, and improve the accuracy of postoperative recurrence prediction.

[0008] In the first aspect, this application provides a postoperative recurrence prediction method, including: Obtain a first image pyramid of the stained glass slide image of the target object, obtain a plurality of first image patches through the first image pyramid, and extract the feature vectors of each first image patch; Obtain a second image pyramid of the immunohistochemical glass slide image of the target object, obtain a plurality of second image patches through the second image pyramid, and extract the feature vectors of each second image patch; Determine the postoperative prediction result and result confidence of the corresponding first image patch based on the feature vector of each first image patch, and determine the postoperative prediction result and result confidence of the corresponding second image patch based on the feature vector of each second image patch; Determine the target feature vectors among the feature vectors of the plurality of first image patches and the feature vectors of the plurality of second image patches according to the result confidence of the plurality of first image patches and the result confidence of the plurality of second image patches; Determine the postoperative prediction result of the target object according to the feature vectors and the target feature vectors, and the postoperative prediction result is postoperative recurrence or no postoperative recurrence.

[0009] Second aspect, the present application provides a postoperative recurrence prediction device, including: A first image processing module, configured to obtain a first image pyramid of a stained glass slide image of a target object, obtain a plurality of first image patches through the first image pyramid, and extract feature vectors of each of the first image patches; A second image processing module, configured to obtain a second image pyramid of an immunohistochemical glass slide image of the target object, obtain a plurality of second image patches through the second image pyramid, and extract feature vectors of each of the second image patches; A first prediction module, configured to determine a postoperative prediction result and a result confidence level of a corresponding first image patch based on the feature vectors of each of the first image patches, and determine a postoperative prediction result and a result confidence level of a corresponding second image patch based on the feature vectors of each of the second image patches; A vector acquisition module, configured to determine target feature vectors from the feature vectors of the plurality of first image patches and the feature vectors of the plurality of second image patches according to the result confidence levels of the plurality of first image patches and the result confidence levels of the plurality of second image patches; A second prediction module, configured to determine a postoperative prediction result of the target object according to the feature vectors and the target feature vectors, where the postoperative prediction result is postoperative recurrence or non - recurrence after surgery.

[0010] Third aspect, the present application provides a postoperative recurrence prediction device, including: One or more processors; a storage device storing one or more programs, which when executed by the one or more processors, cause the one or more processors to implement the postoperative recurrence prediction method as described in the first aspect.

[0011] Fourth aspect, the present application provides a storage medium containing computer - executable instructions, where the computer - executable instructions are used to execute the postoperative recurrence prediction method as described in the first aspect when executed by a computer processor.

[0012] In this application, by obtaining the first image pyramid of the stained glass slide image of the target object, obtaining multiple first image patches through the first image pyramid, and extracting the feature vectors of each first image patch; obtaining the second image pyramid of the immunohistochemical glass slide image of the target object, obtaining multiple second image patches through the second image pyramid, and extracting the feature vectors of each second image patch; determining the postoperative prediction result and result confidence of the corresponding first image patch based on the feature vector of each first image patch, and determining the postoperative prediction result and result confidence of the corresponding second image patch based on the feature vector of each second image patch; determining the target feature vector among the feature vectors of multiple first image patches and the feature vectors of multiple second image patches according to the result confidence of multiple first image patches and the result confidence of multiple second image patches; determining that the postoperative prediction result of the target object is postoperative recurrence or no postoperative recurrence according to the feature vector and the target feature vector. Through the above technical means, the first image pyramid and the second image pyramid can be used to obtain the first image patches and the second image patches with different sizes and different positions, comprehensively capture the cell morphology and overall structure of the tissue in the stained glass slide image and the macroscopic distribution and molecular typing of antigens in the immunohistochemical glass slide image and other detailed information, enrich the features for predicting postoperative recurrence, help to more accurately discover the key information related to postoperative recurrence, and improve the prediction accuracy of the postoperative recurrence result. The higher the result confidence of the first image patch and the second image patch indicates the higher the correlation with the postoperative recurrence result. Obtaining the features of the image patches with higher correlation as the target feature vector according to the result confidence of the first image patch and the second image patch can accurately obtain the key information related to postoperative recurrence. Combining the key information and overall information in the stained glass slide image and the immunohistochemical glass slide image to predict the postoperative recurrence result, fusing the cell morphology and overall structure of the key tissue and the overall tissue in the stained glass slide image and the molecular typing of the key antigen and the overall antigen in the immunohistochemical glass slide image, and mining more comprehensive and in-depth complementary information, so as to more comprehensively evaluate the biological characteristics of the tumor in the patient's body and achieve accurate prediction of early postoperative recurrence. Brief Description of the Drawings

[0013] Figure 1 is a flowchart of a postoperative recurrence prediction method provided by an embodiment of the present application; Figure 2 is a schematic diagram of the first image pyramid and the sliding window provided by an embodiment of the present application; Figure 3 is a flowchart of obtaining the first image patch provided by an embodiment of the present application; Figure 4 is a flowchart of obtaining the second image patch provided by an embodiment of the present application; Figure 5 is a flowchart of determining the postoperative prediction result of the target object provided by an embodiment of the present application; Figure 6 It is a schematic structural diagram of an end-to-end network model provided by an embodiment of the present application; Figure 7 It is a flowchart for training an end-to-end network model provided by an embodiment of the present application; Figure 8 It is a schematic diagram of a backpropagation path provided by an embodiment of the present application; Figure 9 It is a schematic structural diagram of a postoperative recurrence prediction device provided by an embodiment of the present application; Figure 10 It is a schematic structural diagram of a postoperative recurrence prediction device provided by an embodiment of the present application. Specific embodiments

[0014] In order to make the objectives, technical solutions, and advantages of the present application clearer, the following further describes the specific embodiments of the present application in detail with reference to the accompanying drawings. It can be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Additionally, it should be noted that for the sake of description, only parts related to the present application are shown in the accompanying drawings rather than all the content. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations (or steps) as sequential processes, many of the operations can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the operations can be rearranged. When the operations are completed, the process can be terminated, but there may also be additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0015] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects and are not used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type and do not limit the number of objects. For example, the first object can be one or multiple. In addition, "and / or" in the specification and claims means at least one of the connected objects, and the character " / " generally indicates an "or" relationship between the associated objects before and after.

[0016] In a relatively relevant implementation, traditional postoperative recurrence prediction models mainly use scoring models, and the scoring models perform risk stratification based on clinical and pathological features. Although these models are widely used in clinical practice, their prediction ability is limited, especially in predicting the early recurrence of high-grade NMIBC.

[0017] With the rapid development of artificial intelligence technology, artificial intelligence technology has shown extensive applications in the fields of tumor diagnosis and postoperative prediction. It can extract valuable information from a large amount of clinical and pathological data and make accurate predictions through complex algorithm models. In the research of NMIBC, artificial intelligence technology has been widely applied to multiple aspects such as tumor staging, grading, recurrence prediction, and treatment response assessment.

[0018] Currently, there are multiple machine learning-based models for predicting the recurrence of NMIBC. These models usually conduct comprehensive analysis by combining the clinical characteristics of patients (such as age, gender, smoking history), pathological characteristics (such as tumor size, quantity, grade), and molecular markers (such as P53, Ki67, etc.). However, due to the heterogeneity of different research cohorts and the differences in data quality, the prediction performance of these models fluctuates greatly. In contrast, as a more advanced artificial intelligence technology, Deep Learning has gradually become a research hotspot in the tumor field because it can model non-linear parameters and show strong robustness on large-scale datasets.

[0019] In recent years, some studies have attempted to apply deep learning to the early recurrence prediction of NMIBC. For example, analyze H&E (hematoxylin and eosin) stained sections through a deep learning model to predict the recurrence-free survival (RFS) of NMIBC patients. Although the overall prediction performance of this model has been improved, it still performs mediocrely in predicting early recurrence, which indicates that relying solely on the pathological characteristics of H&E stained sections cannot fully capture the heterogeneity of tumors and the risk of early recurrence, and the prediction accuracy for the early recurrence results after surgery is relatively low.

[0020] To solve the above problems, this embodiment provides a postoperative recurrence prediction method to jointly capture the global features and relatively highly correlated local features of multi-size images of stained glass slides and immunohistochemical glass slides, more comprehensively capture the information related to postoperative recurrence, and accurately predict the postoperative recurrence situation of the target object in the early postoperative period, thereby improving the accuracy of postoperative recurrence prediction.

[0021] The postoperative recurrence prediction method provided in this embodiment can be executed by a postoperative recurrence prediction device. The postoperative recurrence prediction device can be implemented in a software and / or hardware manner. The postoperative recurrence prediction device can be composed of two or more physical entities or a single physical entity. For example, the postoperative recurrence prediction device can be an intelligent terminal with relatively strong processing capabilities such as a computer and a server. Among them, the server can be implemented by an independent server or a server cluster composed of multiple servers.

[0022] The postoperative recurrence prediction device is installed with at least one type of operating system. Based on the operating system, the postoperative recurrence prediction device can install at least one application program, which can be an application program built into the operating system or an application program downloaded from a third-party device or server. In this embodiment, the postoperative recurrence prediction device is at least installed with an application program that can execute the postoperative recurrence prediction method.

[0023] For ease of understanding, this embodiment takes a computer as the main body for executing the postoperative recurrence prediction method as an example for description.

[0024] Figure 1 The flowchart of a postoperative recurrence prediction method provided by an embodiment of the present application is given. Refer to Figure 1 , the postoperative recurrence prediction method specifically includes: S110. Obtain the first image pyramid of the stained glass slide image of the target object, obtain multiple first image patches through the first image pyramid, and extract the feature vectors of each first image patch.

[0025] Among them, the target object is a patient who has undergone transurethral resection of bladder tumor in the past year, that is, the target object is currently in the early stage of transurethral resection of bladder tumor. This embodiment aims to obtain the stained glass slide image and immunohistochemical glass slide image of the target object in the early postoperative period to predict the early postoperative recurrence result of the target object.

[0026] The stained glass slide image is a whole-slide digital pathology image (WSI, Whole-Slide-Image) of a tissue glass slide based on hematoxylin-eosin staining. Usually, devices such as digital pathology scanners are used to scan the stained tissue glass slide to obtain a high-resolution WSI image. The first image pyramid is a pyramid structure formed by multi-level downsampling of the originally scanned stained glass slide image. Among them, the originally scanned stained glass slide image is the bottom layer image of the first image pyramid and also the image with the largest resolution, and the resolution can generally reach more than 100000*100000. The remaining layer images in the first image pyramid except the bottom layer are obtained by downsampling, that is, shrinking, the lower layer image, so the resolution of the first image pyramid shrinks layer by layer, and the magnification ratio also shrinks layer by layer.

[0027] Optionally, after obtaining the original stained glass slide image by scanning, the target magnification factor of each level in the first image pyramid can be determined according to actual requirements and application scenarios, and the images of each level can be determined based on the target magnification factor of each level. For example, the bottom layer image, that is, the original stained glass slide image, can be set to a magnification factor of 40X, the second layer image to a magnification factor of 20X, and the top layer image to a magnification factor of 5X. Then, based on the magnification factors of each layer and the original stained glass slide image, a three-layer first image pyramid can be constructed. The second layer image is obtained by reducing the bottom layer image by 1 / 4, and the top layer image is obtained by reducing the second layer image by 1 / 16. Among them, an average filter downsampling or Gaussian filter downsampling algorithm can be used to downsample the bottom layer image to obtain the second layer image, and to downsample the second layer image to obtain the top layer image.

[0028] After that, use a sliding window with a preset fixed size to slide and extract images from each level image of the first image pyramid, and use the image intercepted by the sliding window as the first image block. After each first image block is obtained by the sliding window, it moves a preset step length to the next position to continue extracting images until the entire level image is traversed.

[0029] It should be noted that since the size of the sliding window is fixed and the resolutions of different level images are different, different numbers of first image blocks can be obtained for different level images, and the higher the resolution of the level image, the more first image blocks can be obtained. For example, Figure 2 is a schematic diagram of the first image pyramid and the sliding window provided by an embodiment of the present application. As Figure 2 shown, the first image pyramid 10 includes three level images. The three level images are, from left to right, the bottom layer image 11, the second layer image 12, and the top layer image 13. The resolutions of the bottom layer image 11, the second layer image 12, and the top layer image 13 decrease in sequence. When using the sliding window 14 to slide and extract images from the bottom layer image 11, the second layer image 12, and the top layer image 13, due to the different area ratios of the sliding window 14 to the bottom layer image 11, the second layer image 12, and the top layer image 13, in the case of the same step length, the bottom layer image 11 can be divided into the most first image blocks, the second layer image 12 is the second, and the top layer image 13 is the least. The more first image blocks are divided from the level image, the smaller the area corresponding to the first image block in the level image, and the microscopic single cell information can be extracted from the first image block. The fewer first image blocks are divided from the level image, the larger the area corresponding to the first image block in the level image, and the macroscopic overall structure information can be extracted from the first image block. In this embodiment, by obtaining the first image blocks of different level images, the first image blocks containing single cells can be obtained, and the first image blocks containing the overall structure can also be obtained, enriching the subsequent feature information for predicting postoperative recurrence.

[0030] Optionally, two hierarchical images can be obtained in the first image pyramid. The microscopic features of single cells can be extracted from the sliding window images corresponding to one hierarchical image, and the macroscopic features of the overall structure can be extracted from the sliding window images corresponding to the other hierarchical image. Based on these two hierarchical images, sliding window image extraction is performed to obtain the first image patches. For example, when the magnification of the stained glass slide image is 40 times, the magnifications of the two hierarchical images are 5 times and 20 times respectively. Among them, the microscopic features of single cells can be extracted from the sliding window images of the hierarchical image with a magnification of 5 times, and the macroscopic features of the overall structure can be extracted from the sliding window images of the hierarchical image with a magnification of 20 times. The purpose of this embodiment is to perform sliding window image extraction on the two hierarchical images to simplify the operation steps of sliding image extraction and improve the acquisition efficiency of the first image patches. Moreover, the sliding window images corresponding to these two hierarchical images can provide microscopic cell information and macroscopic structure information, providing sufficient image information for the subsequent prediction of postoperative recurrence results.

[0031] Since there are invalid regions and valid regions in the stained glass slide image, and the invalid regions do not contain tissues, if the first image patches are intercepted in the invalid regions, the number of the first image patches will increase and the accuracy of the first image patches will decrease, resulting in irrelevant information being introduced in the subsequent prediction and affecting the prediction accuracy. To this end, in order to improve the acquisition accuracy of the first image patches, the valid regions can be obtained in the hierarchical images of the first image pyramid, so as to obtain the first image patches in the valid regions. When determining the valid regions, the region mask of the corresponding valid region can be determined in any hierarchical image, and the region mask of the valid region is mapped to other hierarchical images according to the magnification between the hierarchical images, so as to obtain the valid regions of the other hierarchical images. Sliding window image extraction is performed in the valid regions of each hierarchical image to obtain the first image patches.

[0032] Furthermore, in order to improve the acquisition accuracy and acquisition efficiency of the first image patches, sliding window image extraction can be performed in the valid regions of two specific hierarchical images to obtain the first image patches. Optionally, Figure 3 is the flowchart of obtaining the first image patches provided by the embodiment of the present application. As Figure 3 shown, the steps of obtaining the first image patches specifically include S1101 - S1103: S1101. Obtain the first hierarchical image and the second hierarchical image in the first image pyramid. The magnification of the stained glass slide image is twice the magnification of the first hierarchical image, and the magnification of the first hierarchical image is four times the magnification of the second hierarchical image.

[0033] Exemplarily, the stained glass slide image serves as the bottom image of the first image pyramid, with the highest resolution and magnification. The hierarchical image with half the magnification of the stained glass slide image is taken as the first hierarchical image, which also has a sufficiently large magnification to extract the overall structural information from the sliding window images of the first hierarchical image. The hierarchical image with a quarter of the magnification of the stained glass slide image is taken as the second hierarchical image, which has a smaller magnification and can extract the cell monomer information from the sliding window images of the second hierarchical image. For example, when the magnification of the stained glass slide image is 40 times, the hierarchical image with a magnification of 20 times in the first image pyramid is taken as the first hierarchical image, and the hierarchical image with a magnification of 5 times in the first image pyramid is taken as the second hierarchical image.

[0034] S1102. Determine the effective region based on the third hierarchical image in the first image pyramid with a resolution in the thousands range, and map the effective region to the first hierarchical image and the second hierarchical image according to the scaling ratios between the third hierarchical image and the first hierarchical image and the second hierarchical image.

[0035] Exemplarily, select the third hierarchical image in the first image pyramid with a resolution in the thousands range. For example, take the third hierarchical image of 2000*2000 to determine the region mask of the effective region. The reason is that when the resolution of the hierarchical image is large, the positioning efficiency of the effective region is low, and when the resolution is small, the positioning accuracy of the effective region is low. Selecting the third hierarchical image in the thousands range can identify the region mask of the effective region with relatively high efficiency and accuracy.

[0036] When determining the effective region of the third hierarchical image, the pixel values of each pixel point in the third hierarchical image can be compared with a preset pixel threshold, and the effective region is formed based on the pixel points greater than or equal to the preset pixel threshold. Since the cell nuclei are stained under the stained glass slide, selecting an appropriate color threshold can distinguish the cell region from the background, thereby obtaining the effective region containing cells. In addition, morphological operations can be used to change the shape and structure of the objects in the image, so as to remove noise, fill holes, and connect broken regions, making the effective region clearer and more complete.

[0037] After that, according to the scaling ratio between the third hierarchical image and the first hierarchical image, map the region mask of the effective region of the third hierarchical image to the first hierarchical image to obtain the effective region in the first hierarchical image. According to the scaling ratio between the third hierarchical image and the second hierarchical image, map the region mask of the effective region of the third hierarchical image to the second hierarchical image to obtain the effective region in the second hierarchical image.

[0038] S1103. Perform sliding window cropping on the first-level image and the second-level image based on a preset window to obtain multiple image patches. When the image patch of the first-level image is in the corresponding valid area or the image patch of the second-level image is in the corresponding valid area, determine the corresponding image patch as the first image patch.

[0039] Exemplarily, the preset window is a sliding window of a preset size. Perform sliding cropping on the first-level image and the second-level image according to a preset step size through the sliding window to obtain multiple image patches. When the image patch of the first-level image intersects with the valid area of the first-level image, determine this image patch as the first image patch; when the image patch of the second-level image intersects with the valid area of the second-level image, determine this image patch as the first image patch.

[0040] In this embodiment, quickly locate the high-precision valid area by using the third-level image with a resolution in the thousands range, and map the valid area to the first-level image and the second-level image according to the scaling ratios between the third-level image, the first-level image, and the second-level image, so as to quickly locate the valid areas of the first-level image and the second-level image. Use the valid areas of the first-level image and the second-level image to perform sliding window image extraction to simultaneously obtain the first image patches containing microscopic cell information and macroscopic structure information, and improve the acquisition accuracy and efficiency of the first image patches.

[0041] After obtaining multiple first image patches, extract the feature vectors of each first image patch through a feature extraction network. Among them, the feature extraction network can be a deep neural network, such as ResNet or transformer, etc. It should be noted that since each first image patch comes from level images with different magnifications, if the same feature extraction network is directly used to extract features from all first image patches, it will lead to feature confusion. In this regard, different feature extraction networks can be used for different magnifications according to the magnification of the level image to which each first image patch belongs. For example, assume that first image patches are extracted from the first-level image and the second-level image. Then, use feature extraction network A to extract the feature vectors of the first image patches corresponding to the first-level image, and use feature extraction network B to extract the feature vectors of the first image patches corresponding to the second-level image.

[0042] S120. Obtain the second image pyramid of the immunohistochemical slide image of the target object, obtain multiple second image patches through the second image pyramid, and extract the feature vectors of each second image patch.

[0043] Exemplarily, the immunohistochemistry slide image is a whole digital pathological section image of a tissue slide based on immunohistochemistry (IHC). The core principle of immunohistochemistry IHC is based on the antigen-antibody reaction. Antigens (usually proteins) in tissue slides bind to specific antibodies, and through means such as enzymatic reactions or fluorescent labeling, detectable signals are generated at the binding sites. IHC can detect the expression levels of specific proteins in tumor tissues, providing important information for the molecular typing and prognosis evaluation of tumors. Markers (such as P53, CK20, Ki67) detected by IHC in tumor cells are closely related to the recurrence and progression of bladder cancer. In this regard, this embodiment uses immunohistochemistry slide images to detect the expression of specific proteins in tumor cells, and combines the microscopic cell information and macroscopic structure information detected by stained slide images to comprehensively evaluate the biological characteristics of tumor cells, so as to accurately predict the recurrence situation in the early postoperative period of patients.

[0044] Similarly, devices such as digital pathological scanners can be used to scan tissue slides of immunohistochemistry to obtain high-resolution WSI images. The second image pyramid is a pyramid structure formed by multi-level downsampling of the immunohistochemistry slide image obtained by the original scan. Among them, the immunohistochemistry slide image obtained by the original scan is the bottom layer image of the second image pyramid and also the image with the largest resolution, and the resolution can generally reach more than 100000*100000. The remaining layer images in the second image pyramid except the bottom layer are obtained by downsampling, that is, shrinking the lower layer image, so the resolution of the second image pyramid shrinks layer by layer, and the magnification ratio also shrinks layer by layer.

[0045] The generation process of the second image pyramid is similar to the generation process of the first image pyramid. For details, reference can be made to the generation process of the first image pyramid in step S110.

[0046] After that, a sliding window with a preset fixed size is used to slide and extract images from each level image of the second image pyramid, and the image intercepted by the sliding window is used as the second image block. After each second image block is obtained by the sliding window, it moves a preset step length to the next position to continue extracting images until the entire level image is traversed. Among them, the first image pyramid and the second image pyramid use sliding windows of the same size, and the sliding step length is also the same.

[0047] Optionally, since immunohistochemical slide images are intended to detect the expression of specific proteins in tumor cells therein, more attention is paid to the individual cells in immunohistochemical slide images. Therefore, only one level of images may be obtained for the second image pyramid. The micro features of individual cells can be extracted from the sliding window images corresponding to the intercepted images at this level. Second image patches are obtained by performing sliding window image extraction based on this level of images. For example, when the magnification of the immunohistochemical slide image is 40 times, second image patches are obtained by performing sliding window cropping on the level of images with a magnification of 20 times in the second image pyramid. This embodiment aims to perform sliding window image extraction on one level of images to simplify the operation steps of sliding image extraction and improve the acquisition efficiency of the second image patches. Moreover, the sliding window images corresponding to this level of images can provide microscopic cell information, providing sufficient image information for predicting the postoperative recurrence results.

[0048] Similarly, there are invalid areas and valid areas in immunohistochemical slide images. To improve the acquisition accuracy of the first image patches, the valid areas can be obtained from each level of images in the second image pyramid, and the second image patches can be obtained from the valid areas.

[0049] Furthermore, to improve the acquisition accuracy and acquisition efficiency of the second image patches, second image patches can be obtained by performing sliding window image extraction in the valid areas of specific level of images. Optionally, Figure 4 is the flowchart for obtaining the second image patches provided by the embodiments of the present application. As Figure 4 shown, the steps for obtaining the second image patches specifically include S1201 - S1203: S1201. Obtain the fourth level of images in the second image pyramid, where the magnification of the immunohistochemical slide image is twice the magnification of the fourth level of images.

[0050] Exemplarily, the immunohistochemical slide image is used as the bottom level image of the second image pyramid, which has the largest resolution and magnification. The level of images with half the magnification of the immunohistochemical slide image is taken as the fourth level of images. The second level of images also has a sufficiently large magnification, and the overall structural information can be extracted from the sliding window images of the second level of images. For example, when the magnification of the immunohistochemical slide image is 40 times, the level of images with a magnification of 20 times in the second image pyramid is taken as the fourth level of images.

[0051] S1202. Determine the valid area according to the fifth level of images in the second image pyramid with a resolution in the thousands range, and map the valid area to the fourth level of images according to the scaling ratio between the fifth level of images and the fourth level of images.

[0052] Exemplarily, select the fifth-level image with a resolution in the thousands range in the second image pyramid. For example, take the fifth-level image of 2000*2000 to determine the region mask of the effective region. Determine the region mask of the effective region of the fifth-level image through a preset image threshold or morphological operation, and then map the region mask of the effective region of the fifth-level image to the fourth-level image according to the scaling ratio between the fifth-level image and the fourth-level image to obtain the effective region in the fourth-level image.

[0053] S1203. Perform sliding window cropping on the fourth-level image based on a preset window to obtain multiple image patches. When the image patch of the fourth-level image is in the corresponding effective region, determine the corresponding image patch as the second image patch.

[0054] Exemplarily, the preset window is a sliding window of a preset size. Perform sliding cropping on the fourth-level image according to a preset step size through the sliding window to obtain multiple image patches. When the image patch of the fourth-level image intersects with the effective region of the fourth-level image, determine the image patch as the second image patch.

[0055] In this embodiment, by using the fifth-level image with a resolution in the thousands range to quickly locate the high-precision effective region, and mapping the effective region to the fourth-level image according to the scaling ratio between the fifth-level image and the fourth-level image to quickly locate the effective region of the fourth-level image. Use the effective region of the fourth-level image for sliding window image extraction to obtain the second image patch containing microscopic cell information, improving the acquisition accuracy and efficiency of the second image patch.

[0056] After obtaining multiple second image patches, extract the feature vectors of each second image patch through a feature extraction network. The feature extraction network can be a deep neural network, such as ResNet or transformer, etc. Similarly, when the second image patches come from hierarchical images with different resolutions, different feature extraction networks can be used to extract the feature vectors of the second image patches belonging to different hierarchical images.

[0057] Optionally, if the magnification factors of the hierarchical images to which the first image patch and the second image patch belong are the same, then the same feature extraction network can be used to extract the feature vectors of the first image patch and the second image patch. For example, use feature extraction network A to extract the feature vector of the first image patch corresponding to the first-level image. The magnification factor of the first-level image is 20 times, and the magnification factor of the fourth-level image is also 20 times. Then the second image patch in the corresponding fourth-level image can also be extracted using feature extraction network A.

[0058] Specifically, input the first image patch corresponding to the second-level image into the feature extraction network B to obtain the feature vector output by the feature extraction network B. Input the first image patch corresponding to the first-level image and the second image patch corresponding to the fourth-level image into the feature extraction network A to obtain the feature vector output by the feature extraction network A.

[0059] S130. Determine the postoperative prediction result and result confidence of the corresponding first image patch based on the feature vector of each first image patch, and determine the postoperative prediction result and result confidence of the corresponding second image patch based on the feature vector of each second image patch.

[0060] Exemplarily, input the feature vector of the first image patch into the pre-trained first classification network. Through the first classification network, predict the postoperative prediction result of the corresponding first image patch based on the input feature vector and output the result confidence. Input the feature vector of the second image patch into the same first classification network. Through the first classification network, predict the postoperative prediction result of the corresponding second image patch based on the input feature vector and output the result confidence.

[0061] Optionally, the first classification network can be a fully connected layer, a fully convolutional layer, or a combined network of a convolutional layer and a fully connected layer. The unification of the number of features can be achieved by pooling. For example, when the first classification network is a fully connected layer, the process of determining the postoperative prediction result and result confidence is as follows: The fully connected layer includes an input layer, a hidden layer, and an output layer. The first classification network receives the feature vector of the image patch through the input layer, transmits the received feature vector to the hidden layer. The hidden layer performs a full connection operation on the feature vector through the weight matrix to obtain an output vector, and performs a non-linear mapping on the output vector through an activation function to obtain the input vector of the next hidden layer. After the feature vector is processed layer by layer by multiple hidden layers, the final output vector is obtained. The output vector is passed to the output layer. The output layer uses neurons to convert the output vector into an output value, and uses the sigmoid function to map the output value to the interval (0,1), representing the abnormal probability of the cells in the image patch. When the abnormal probability is greater than 0.5, it indicates that there are abnormalities in the cells in the image patch, so as to determine that the postoperative prediction result of the image patch is recurrence, and the confidence of the postoperative prediction result is equal to the abnormal probability. When the abnormal probability is less than 0.5, it indicates that there are no abnormalities in the cells in the image patch, so as to determine that the postoperative prediction result of the image patch is non-recurrence, and the confidence of the postoperative prediction result is equal to 1 minus the abnormal probability.

[0062] It should be noted that since the magnification ratios of image patches at different levels are different and the contained features are also different, if the same classification network is used to process image patches at different levels, the classification network cannot make different treatments for features of different sizes, resulting in feature confusion. It is difficult for the network to accurately distinguish and learn different types of features, which affects the accuracy of classification. Therefore, different classification networks are used to classify image patches with different magnification ratios. Specifically, the feature vectors of the first image patch and the second image patch with the same magnification ratio are sequentially input into the corresponding first classification network to obtain the postoperative prediction result and the result confidence level output by the first classification network. Here, the magnification ratios of the first image patch and the second image patch refer to the magnification ratios of the images to which they belong. For example, when the first image patch belongs to the first-level image and the second image patch belongs to the fourth-level image, the magnification ratios of the first image patch and the second image patch are the same, both being 20 times. Therefore, the first image patch corresponding to the first-level image and the second image patch corresponding to the fourth-level image can be input into the first classification network A for classification to obtain the postoperative prediction result and the result confidence level output by the first classification network A. The magnification ratio of the first image patch corresponding to the second-level image is 5 times, and the first image patch corresponding to the second-level image can be input into the first classification network B for classification to obtain the postoperative prediction result and the result confidence level output by the first classification network B.

[0063] In this embodiment, different first classification networks are used to classify the first image patches and the second image patches with different magnification ratios to ensure that the same first classification network can analyze features of the same size and avoid the influence of feature confusion on the accuracy of classification.

[0064] S140. Determine the target feature vector from the feature vectors of the multiple first image patches and the feature vectors of the multiple second image patches according to the result confidence levels of the multiple first image patches and the result confidence levels of the multiple second image patches.

[0065] It is understandable that the higher the result confidence of the first image patch and the second image patch, the higher the correlation with the postoperative recurrence result, and the more crucial the information corresponding to the feature vectors of the first image patch and the second image patch. In this regard, according to the result confidence of multiple first image patches and the result confidence of multiple second image patches, feature vectors with higher result confidence can be selected from the feature vectors of multiple first image patches and the feature vectors of multiple second image patches as target feature vectors. For example, by sorting the result confidence of each feature vector from large to small, a preset number of feature vectors with higher result confidence ranking can be selected as target feature vectors. Alternatively, feature vectors with result confidence greater than a preset confidence threshold can be selected as target feature vectors. Since the weight of the feature vector with higher correlation will increase and the weight of the feature vector with lower correlation will decrease after obtaining the feature vector with higher correlation, the negative impact of invalid features on the accuracy of the prediction result can be reduced from the side.

[0066] Since the magnification ratios of image patches at different levels are different and the contained features are different. To ensure the diversity of features for subsequent prediction of postoperative recurrence results, the result confidence of the first image patches and the second image patches at the same magnification ratio can be sorted from large to small, and the feature vectors of multiple first image patches and / or second image patches with higher ranking can be determined as the target feature vectors corresponding to the magnification ratio. For example, after the first classification network B classifies the first image patches at a magnification of 5 to obtain the corresponding postoperative prediction results and result confidence, the feature vectors of the first image patches at a magnification of 5 are sorted in descending order according to the result confidence, and TN1 feature vectors with higher ranking are selected as the target feature vectors at a magnification of 5. After the first classification network A classifies the first image patches and the second image patches at a magnification of 20 to obtain the corresponding postoperative prediction results and result confidence, the feature vectors of the first image patches and the second image patches at a magnification of 20 are sorted in descending order according to the result confidence, and TN2 feature vectors with higher ranking are selected as the target feature vectors at a magnification of 20.

[0067] In this embodiment, by obtaining feature vectors with higher result confidence at different magnification ratios as target feature vectors, the target feature vectors include both microscopic cell features and macroscopic structural features, enriching the features for subsequent prediction of postoperative recurrence results while accurately obtaining the key information related to postoperative recurrence.

[0068] S150. Determine the postoperative prediction result of the target object according to the feature vector and the target feature vector, and the postoperative prediction result is postoperative recurrence or no postoperative recurrence.

[0069] Exemplarily, the feature vectors of each first image block and each second image block, along with the target feature vector, are fused to generate a multimodal feature vector. The multimodal feature vector incorporates the cellular detail features and overall structural features of the stained slide image, the molecular typing features of the immunohistochemistry slide image, and key features from both slide images. This multimodal feature vector is then fed into a second classification network. Based on the information about tumor cells contained in the multimodal feature vector, the second classification network can comprehensively assess the biological characteristics of tumor cells and predict early postoperative recurrence outcomes in patients.

[0070] Optionally, a convolution block can be used to fuse feature vectors or target feature vectors at the same magnification, and then a convolution block can be used to fuse feature vectors at different magnifications to fuse the cell morphology and overall structure of key tissues and overall tissues in stained slide images and the molecular typing of key antigens and overall antigens in immunohistochemical slide images, thereby mining more comprehensive and in-depth complementary information. Specifically, Figure 5 This is a flow chart of determining the postoperative prediction results of the target object provided by the embodiment of the present application. Figure 5 As shown, the step of determining the postoperative prediction result of the target object specifically includes S1501-S1504: S1501: Concatenate the feature vector of the first image block and the feature vector of the second image block at the same magnification and input the concatenated features into the corresponding first convolution block to obtain a first feature vector output by the first convolution block.

[0071] Exemplarily, the feature vectors of the 5-fold first image block are concatenated and input into the first convolution block B to obtain the first feature vector B output by the first convolution block B. The feature vectors of the 20-fold first image block and the second image block are concatenated and input into the first convolution block A to obtain the first feature vector I output by the first convolution block A.

[0072] S1502: splice the target feature vectors with the same magnification and input them into the corresponding second convolution block to obtain a second feature vector output by the second convolution block.

[0073] Exemplarily, the 5-fold target feature vectors are concatenated and input into the second convolution block B to obtain the second feature vector B output by the second convolution block B. The 20-fold target feature vectors are concatenated and input into the second convolution block A to obtain the second feature vector A output by the second convolution block A.

[0074] S1503: Concatenate the first eigenvector and the second eigenvector of each magnification and input the concatenated eigenvector into a third convolution block to obtain a multimodal eigenvector output by the third convolution block.

[0075] Exemplarily, the first feature vector A, the first feature vector B, the second feature vector A, and the second feature vector B are concatenated and then input into the third convolutional block to obtain the multi-modal feature vector output by the third convolutional block.

[0076] S1504. Input the multi-modal feature vector into the second classification network, and determine the postoperative prediction result of the target object through the second classification network.

[0077] Exemplarily, input the multi-modal feature vector into the second classification network, and the second classification network analyzes the multi-modal feature vector to predict whether the patient will relapse in the early postoperative period.

[0078] Similarly, the second classification network can be a fully connected layer, a fully convolutional layer, or a combined network of a convolutional layer and a fully connected layer. When the second classification network is a fully connected layer, the multi-modal feature vector is received by the input layer of the fully connected layer and then passed to the hidden layer. After being processed layer by layer by multiple hidden layers, the output vector is output to the output layer. The output layer converts the output vector into an output value through neurons, and the sigmoid function maps the output value to the interval (0, 1), representing the abnormal probability of cells and tissues expressed by the multi-modal feature vector. When the abnormal probability is greater than 0.5, it indicates that there are abnormalities in the cells and tissues, and thus the postoperative prediction result of the target object is recurrence. When the abnormal probability is less than 0.5, it indicates that there are no abnormalities in the cells and tissues in the image patch, and thus the postoperative prediction result of the target object is non-recurrence.

[0079] The feature extraction network, the first classification network, the first convolutional block, the second convolutional block, the third convolutional block, and the second classification network involved in this embodiment can be regarded as an end-to-end network model. This embodiment uses Figure 6 the structural schematic diagram of the shown end-to-end network model to describe the overall process of predicting the early postoperative recurrence result of the patient based on the stained glass slide image and the immunohistochemical glass slide image. As Figure 6As shown in the figure, the first image patches with a magnification of 5 times and the first image patches with a magnification of 20 times are obtained from the stained glass slide images, and the second image patches with a magnification of 20 times are obtained from the immunohistochemical glass slide images. The first image patches with a magnification of 5 times are input into the feature extraction network B to obtain the feature vectors with a magnification of 5 times output by the feature extraction network B. The first image patches with a magnification of 20 times and the second image patches are input into the feature extraction network A to obtain the feature vectors with a magnification of 20 times output by the feature extraction network A. The feature vectors with a magnification of 5 times are input into the first classification network B to obtain the recurrence prediction result and the result confidence level output by the first classification network B, and the feature vectors with higher confidence levels are obtained from the feature vectors with a magnification of 5 times according to the result confidence level as the target feature vectors with a magnification of 5 times. The feature vectors with a magnification of 20 times are input into the first classification network A to obtain the recurrence prediction result and the result confidence level output by the first classification network A, and the feature vectors with higher confidence levels are obtained from the feature vectors with a magnification of 20 times according to the result confidence level as the target feature vectors with a magnification of 20 times. The feature vectors with a magnification of 5 times are concatenated and then input into the first convolutional block B, the feature vectors with a magnification of 20 times are concatenated and then input into the first convolutional block A, the target feature vectors with a magnification of 5 times are concatenated and then input into the second convolutional block B, the target feature vectors with a magnification of 20 times are concatenated and then input into the second convolutional block A, the features output by the first convolutional block A, the first convolutional block B, the second convolutional block A, and the second convolutional block B are concatenated and then input into the third convolutional block, and the multi-modal feature vectors output by the third convolutional block are input into the second classification network to obtain the recurrence prediction result output by the second classification network.

[0080] Optionally, the result confidence level of the postoperative prediction result output by the second classification network can be adjusted based on the result confidence levels output by the first classification network A and the first classification network B to improve the reliability of the final prediction result.

[0081] Based on the above embodiments, Figure 7 is a flowchart of training an end-to-end network model provided by an embodiment of the present application. As Figure 7 shown, the steps of training the end-to-end network model include S210-S230: S210. Obtain the image pyramid of the sample image group, and obtain the sample feature set based on the image pyramid; wherein, the sample image group includes stained glass slide sample images and immunohistochemical glass slide sample images of the same object, and the label information of the sample feature set is the same as the label information of the sample image group.

[0082] Exemplarily, the stained glass slide sample image is upsampled based on a preset magnification factor to generate a corresponding first image pyramid, and the immunohistochemical glass slide sample image is upsampled based on the preset magnification factor to generate a corresponding image pyramid. Window sliding is performed on the hierarchical image with a 5-fold magnification factor in the first image pyramid to obtain multiple sample image patches with a 5-fold magnification factor, and the feature vectors of the sample image patches are extracted through the feature extraction network B to obtain the sample feature vectors with a 5-fold magnification factor. Window sliding is performed on the hierarchical images with a 20-fold magnification factor in the first image pyramid and the second image pyramid to obtain multiple sample image patches with a 20-fold magnification factor, and the feature vectors of the sample image patches are extracted through the feature extraction network A to obtain the sample feature vectors with a 20-fold magnification factor. The set of the sample feature vectors with a 20-fold magnification factor and the sample feature vectors with a 5-fold magnification factor is the sample feature set. The label information of the sample feature set is the label information on whether postoperative recurrence occurs in the object marked by the sample image group, and the sample image group is collected in the early postoperative period of the object.

[0083] It should be noted that the feature extraction network can be pre-trained or trained together with the first classification network during backpropagation optimization. In order to enable the feature extraction network to have better model performance, it can be pre-trained with more sample data to improve the generalization ability of the model. Correspondingly, during the subsequent backpropagation optimization of the first classification network, the network parameters of the feature extraction network are fixed and not updated.

[0084] S220. Input the sample feature set into the first classification network, and screen the target sample feature set from the sample feature set based on the postoperative prediction result and the result confidence level output by the first classification network.

[0085] Exemplarily, input the sample feature vectors with a 20-fold magnification factor into the first classification network A to obtain the postoperative prediction result and the result confidence level output by the first classification network A. Obtain multiple sample feature vectors with a higher ranking in descending order of the result confidence level and store them in the target sample feature set with a 20-fold magnification factor. Input the sample feature vectors with a 5-fold magnification factor into the first classification network B to obtain the postoperative prediction result and the result confidence level output by the first classification network B. Obtain multiple sample feature vectors with a higher ranking in descending order of the result confidence level and store them in the target sample feature set with a 5-fold magnification factor.

[0086] S230. Input the sample feature set into the first convolutional block, input the target sample feature set into the second convolutional block, splice the output features of the first convolutional block and the output features of the second convolutional block, and then input the spliced features into the third convolutional block to obtain the sample feature vectors output by the third convolutional block.

[0087] Exemplarily, the concatenated 20-fold sample feature vectors are input into the first convolutional block A, the concatenated 5-fold sample feature vectors are input into the first convolutional block B, the sample feature vectors in the 20-fold target sample feature set are input into the second convolutional block A, the sample feature vectors in the 5-fold target sample feature set are input into the second convolutional block B, and the features output by the first convolutional block A, the first convolutional block B, the second convolutional block A, and the second convolutional block B are concatenated and then input into the third convolutional block to obtain the sample feature vectors output by the third convolutional block.

[0088] S240. Input the sample feature vectors into the second classification network to obtain the postoperative prediction result and result confidence level output by the second classification network, and determine the total loss value according to the postoperative prediction results and result confidence levels output by the first classification network and the second classification network and the label information of the sample image group.

[0089] Exemplarily, the sample feature vectors output by the third convolutional block are input into the second classification network. The second classification network predicts the probability of the object having postoperative recurrence and the probability of not having postoperative recurrence based on the sample feature vectors, and determines the second loss value based on the cross-entropy loss function, the prediction probabilities of the second classification network, and the label information of the sample image group. According to the confidence level of the postoperative prediction result output by the first classification network, obtain multiple postoperative prediction results with higher confidence levels. Each postoperative prediction result includes the probability of the first classification network predicting the object having postoperative recurrence and the probability of not having postoperative recurrence. Based on the cross-entropy loss function, the prediction probabilities of the first classification network, and the label information of the sample image group, calculate the loss value of each postoperative prediction result, and use the average value of the loss values of multiple postoperative prediction results with higher confidence levels as the first loss value. Perform weighted summation on the first loss value and the second loss value to obtain the total loss value.

[0090] S250. Based on the total loss value, perform backpropagation starting from the second classification network to optimize the network parameters of the second classification network, the third convolutional block, the second convolutional block, and the first convolutional block. Based on the total loss value, perform backpropagation starting from the first classification network to optimize the network parameters of the first classification network.

[0091] For example, Figure 8 is a schematic diagram of the backpropagation path provided by an embodiment of the present application. As Figure 8As shown, when training the end-to-end model, it is divided into two backpropagation paths. If the feature extraction network is trained in advance, the backpropagation path A starts from the first classification network and ends at the first classification network. If the feature extraction network is not trained in advance, the backpropagation path A starts from the first classification network and ends at the feature extraction network). The backpropagation path B starts from the second classification network, passes through the third convolutional block, and finally ends at the second convolutional block and the first convolutional block. In this embodiment, the case where the feature extraction network is trained in advance is taken as an example for description. Based on the total loss value, backpropagation is performed along the backpropagation path A, and the network parameters of the first classification network are optimized during the backpropagation process to complete one training of the first classification network. Also, based on the total loss value, backpropagation is performed along the backpropagation path B, and the network parameters of the second classification network, the third convolutional block, the second convolutional block, and the first convolutional block are optimized during the backpropagation process to complete one training of the second classification network, the third convolutional block, the second convolutional block, and the first convolutional block.

[0092] When the number of training times reaches the upper limit, or when the network parameters of the first classification network, the second classification network, the third convolutional block, the second convolutional block, and the first convolutional block converge to meet the convergence condition, it is confirmed that the first classification network, the second classification network, the third convolutional block, the second convolutional block, and the first convolutional block are completed with training. The trained feature extraction network, the first classification network, the second classification network, the third convolutional block, the second convolutional block, and the first convolutional block can be deployed on a computer for real-time postoperative recurrence prediction.

[0093] It should be noted that in this embodiment, multiple sample image blocks are regarded as multiple instances, and backpropagation is performed according to the average loss value corresponding to the postoperative prediction results of multiple instances under the framework of multi-instance learning to optimize the first classification network and the feature extraction network. When training the second classification network, the backpropagation ends at the first convolutional block and the second convolutional block, and does not backpropagate to the feature extraction network and the first classification network involved in the multi-instance learning stage, ensuring the stability of the network part of feature extraction and classification processing determined in the multi-instance learning stage and avoiding its adverse effects in the subsequent processing process.

[0094] In summary, the postoperative recurrence prediction method provided in the embodiments of the present application obtains a first image pyramid of a stained slide image of a target object, obtains multiple first image blocks through the first image pyramid, and extracts a feature vector of each first image block; obtains a second image pyramid of an immunohistochemistry slide image of the target object, obtains multiple second image blocks through the second image pyramid, and extracts a feature vector of each second image block; determines a postoperative prediction result and a result confidence of the corresponding first image block based on the feature vector of each first image block, and determines a postoperative prediction result and a result confidence of the corresponding second image block based on the feature vector of each second image block; determines a target feature vector from the feature vectors of the multiple first image blocks and the feature vectors of the multiple second image blocks based on the result confidence of the multiple first image blocks and the result confidence of the multiple second image blocks; and determines, based on the feature vector and the target feature vector, whether the postoperative prediction result of the target object is postoperative recurrence or no recurrence. Through the above-described technical means, first and second image blocks of varying sizes and locations can be obtained using the first and second image pyramids. This comprehensively captures detailed information, such as the cellular morphology and overall structure of tissues in stained slide images, as well as the macroscopic distribution and molecular typing of antigens in immunohistochemical slide images. This enriches the features used to predict postoperative recurrence, helps more accurately identify key information related to postoperative recurrence, and improves the accuracy of postoperative recurrence prediction. A higher confidence level in the results of the first and second image blocks indicates a higher correlation with the postoperative recurrence outcome. Based on the confidence level of the results of the first and second image blocks, the features of the image blocks with high correlation are used as target feature vectors, accurately capturing key information related to postoperative recurrence. Predicting postoperative recurrence outcomes by combining key and overall information from stained slide images and immunohistochemical slide images. By integrating the cellular morphology and overall structure of key and overall tissues in stained slide images with the molecular typing of key and overall antigens in immunohistochemical slide images, more comprehensive and in-depth complementary information is extracted, enabling a more comprehensive assessment of the biological characteristics of the patient's tumor and achieving accurate prediction of early postoperative recurrence.

[0095] Based on the above embodiments, Figure 9 This is a schematic diagram of the structure of a postoperative recurrence prediction device provided in an embodiment of the present application. Figure 9 The postoperative recurrence prediction device provided in this embodiment specifically includes: a first image processing module 31 , a second image processing module 32 , a first prediction module 33 , a vector acquisition module 34 and a second prediction module 35 .

[0096] The first image processing module 31 is configured to obtain a first image pyramid of a stained glass slide image of a target object, obtain a plurality of first image blocks through the first image pyramid, and extract a feature vector of each first image block; The second image processing module 32 is configured to obtain a second image pyramid of the immunohistochemical slide image of the target object, obtain a plurality of second image patches through the second image pyramid, and extract the feature vectors of each of the second image patches; The first prediction module 33 is configured to determine the postoperative prediction result and result confidence of the corresponding first image patch based on the feature vector of each of the first image patches, and determine the postoperative prediction result and result confidence of the corresponding second image patch based on the feature vector of each of the second image patches; The vector acquisition module 34 is configured to determine target feature vectors from the feature vectors of the plurality of first image patches and the feature vectors of the plurality of second image patches according to the result confidence of the plurality of first image patches and the result confidence of the plurality of second image patches; The second prediction module 35 is configured to determine the postoperative prediction result of the target object according to the feature vector and the target feature vector, and the postoperative prediction result is postoperative recurrence or no postoperative recurrence.

[0097] Based on the above embodiments, the first image processing module 31 includes: a first-level image acquisition unit configured to obtain a first-level image and a second-level image in the first image pyramid, the magnification of the stained slide image being twice the magnification of the first-level image, and the magnification of the first-level image being four times the magnification of the second-level image; a first effective area determination unit configured to determine an effective area according to a third-level image in the first image pyramid with a resolution in the thousands range, and map the effective area to the first-level image and the second-level image according to the scaling ratio between the third-level image and the first-level image and the second-level image; a first image patch determination unit configured to perform sliding window cropping on the first-level image and the second-level image based on a preset window to obtain a plurality of image patches, and determine the corresponding image patch as a first image patch when the image patch of the first-level image is in the corresponding effective area or the image patch of the second-level image is in the corresponding effective area.

[0098] Based on the above embodiment, the second image processing module 32 includes: a second-level image acquisition unit, configured to acquire a fourth-level image in the second image pyramid, wherein the magnification of the immunohistochemistry slide image is twice that of the fourth-level image; a second effective area determination unit, configured to determine an effective area based on a fifth-level image having a resolution in the thousands in the second image pyramid, and map the effective area to the fourth-level image based on a scaling ratio between the fifth-level image and the fourth-level image; and a second image block determination unit, configured to perform sliding window cropping on the fourth-level image based on a preset window to obtain a plurality of image blocks, and determine the corresponding image block as the second image block when an image block of the fourth-level image is within the corresponding effective area.

[0099] Based on the above embodiment, the first prediction module 33 includes: a first prediction unit, which is configured to input the feature vector of the first image block and the feature vector of the second image block with the same magnification into the corresponding first classification network in sequence to obtain the postoperative prediction result and result confidence output by the first classification network.

[0100] Based on the above embodiment, the vector acquisition module 34 includes: a vector acquisition unit, which is configured to sort the result confidence of the first image block and the result confidence of the second image block at the same magnification from large to small, and determine the feature vectors of the multiple first image blocks and / or second image blocks with the highest sorting as the target feature vectors of the corresponding magnification.

[0101] Based on the above embodiment, the second prediction module 35 includes: a first convolution unit, configured to splice the feature vector of the first image block and the feature vector of the second image block of the same magnification and input them into the corresponding first convolution block to obtain the first feature vector output by the first convolution block; a second convolution unit, configured to splice the target feature vector of the same magnification and input them into the corresponding second convolution block to obtain the second feature vector output by the second convolution block; a third convolution unit, configured to splice the first feature vector and the second feature vector of each magnification and input them into the third convolution block to obtain the multimodal feature vector output by the third convolution block; a second prediction unit, configured to input the multimodal feature vector into a second classification network, and determine the postoperative prediction result of the target object through the second classification network.

[0102] Based on the above embodiments, the postoperative recurrence prediction device further includes a model training module, and the model training module includes: a first sample feature determination unit configured to obtain an image pyramid of a sample image group and obtain a sample feature set based on the image pyramid; wherein, the sample image group includes a stained glass slide sample image and an immunohistochemical glass slide sample image of the same object, and the label information of the sample feature set is the same as the label information of the sample image group; a second sample feature determination unit configured to input the sample feature set into a first classification network and screen a target sample feature set from the sample feature set based on the postoperative prediction result and result confidence output by the first classification network; a third sample feature determination unit configured to input the sample feature set into a first convolutional block, input the target sample feature set into a second convolutional block, splice the output features of the first convolutional block and the output features of the second convolutional block, and then input the spliced features into a third convolutional block to obtain a sample feature vector output by the third convolutional block; a loss value determination unit configured to input the sample feature vector into a second classification network, obtain the postoperative prediction result and result confidence output by the second classification network, and determine a total loss value according to the postoperative prediction results and result confidences output by the first classification network and the second classification network and the label information of the sample image group; a network parameter optimization unit configured to perform backpropagation starting from the second classification network based on the total loss value to optimize the network parameters of the second classification network, the third convolutional block, the second convolutional block, and the first convolutional block, and perform backpropagation starting from the first classification network based on the total loss value to optimize the network parameters of the first classification network.

[0103] As described above, the postoperative recurrence prediction device provided by the embodiments of the present application obtains the first image pyramid of the stained glass slide image of the target object, obtains multiple first image patches through the first image pyramid, and extracts the feature vectors of each first image patch; obtains the second image pyramid of the immunohistochemical glass slide image of the target object, obtains multiple second image patches through the second image pyramid, and extracts the feature vectors of each second image patch; determines the postoperative prediction result and result confidence of the corresponding first image patch based on the feature vector of each first image patch, and determines the postoperative prediction result and result confidence of the corresponding second image patch based on the feature vector of each second image patch; determines the target feature vector among the feature vectors of multiple first image patches and the feature vectors of multiple second image patches according to the result confidence of multiple first image patches and the result confidence of multiple second image patches; determines that the postoperative prediction result of the target object is postoperative recurrence or no postoperative recurrence according to the feature vector and the target feature vector. Through the above technical means, the first image pyramid and the second image pyramid can be used to obtain the first image patches and the second image patches with different sizes and different positions, comprehensively capture the cell morphology and overall structure of the tissue in the stained glass slide image and the details such as the macroscopic distribution and molecular typing of antigens in the immunohistochemical glass slide image, enrich the features for predicting postoperative recurrence, help to more accurately discover the key information related to postoperative recurrence, and improve the prediction accuracy of the postoperative recurrence result. The higher the result confidence of the first image patch and the second image patch indicates the higher the correlation with the postoperative recurrence result. The features of the image patches with higher correlation are obtained as the target feature vector according to the result confidence of the first image patch and the second image patch, and the key information related to postoperative recurrence can be accurately obtained. The key information and overall information in the stained glass slide image and the immunohistochemical glass slide image are combined to predict the postoperative recurrence result, so as to fuse the cell morphology and overall structure of the key tissue and the overall tissue in the stained glass slide image and the molecular typing of the key antigen and the overall antigen in the immunohistochemical glass slide image, and dig out more comprehensive and in-depth complementary information, so as to more comprehensively evaluate the biological characteristics of the tumor in the patient's body and achieve accurate prediction of early postoperative recurrence.

[0104] The postoperative recurrence prediction device provided by the embodiments of the present application can be used to execute the postoperative recurrence prediction method provided by the above embodiments, and has the corresponding functions and beneficial effects.

[0105] Figure 10 It is a schematic structural diagram of a postoperative recurrence prediction device provided by the embodiments of the present application. Refer to Figure 10, the postoperative recurrence prediction device includes: a processor 41, a memory 42, a communication device 43, an input device 44, and an output device 45. The number of processors 41 in the postoperative recurrence prediction device may be one or more, and the number of memories 42 in the postoperative recurrence prediction device may be one or more. The processor 41, memory 42, communication device 43, input device 44, and output device 45 of the postoperative recurrence prediction device may be connected via a bus or other means.

[0106] The memory 42, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the postoperative recurrence prediction method of any embodiment of the present application (for example, the first image processing module 31, the second image processing module 32, the first prediction module 33, the vector acquisition module 34, and the second prediction module 35). The memory 42 mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created according to the use of the device, etc. In addition, the memory 42 may include high-speed random access memory and may also include non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some instances, the memory may further include a memory remotely set relative to the processor, and these remote memories can be connected to the device via a network. Examples of the above network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and their combinations.

[0107] The communication device 43 is used for data transmission.

[0108] The processor 41 executes various functional applications and data processing of the device by running the software programs, instructions, and modules stored in the memory 42, that is, implements the above-mentioned postoperative recurrence prediction method.

[0109] The input device 44 can be used to receive input digital or character information, and generate key signal inputs related to the user settings and function controls of the device. The output device 45 may include display devices such as a display screen.

[0110] The above-provided postoperative recurrence prediction device can be used to execute the postoperative recurrence prediction method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0111] The embodiment of the present application further provides a storage medium containing computer-executable instructions. When the computer-executable instructions are executed by a computer processor, they are used to execute a postoperative recurrence prediction method. The postoperative recurrence prediction method includes: obtaining a first image pyramid of a stained glass slide image of a target object, obtaining a plurality of first image patches through the first image pyramid, and extracting the feature vectors of each first image patch; obtaining a second image pyramid of an immunohistochemical glass slide image of the target object, obtaining a plurality of second image patches through the second image pyramid, and extracting the feature vectors of each second image patch; determining the postoperative prediction result and result confidence of the corresponding first image patch based on the feature vector of each first image patch, and determining the postoperative prediction result and result confidence of the corresponding second image patch based on the feature vector of each second image patch; determining the target feature vector among the feature vectors of the plurality of first image patches and the feature vectors of the plurality of second image patches according to the result confidence of the plurality of first image patches and the result confidence of the plurality of second image patches; determining that the postoperative prediction result of the target object is postoperative recurrence or non-recurrence according to the feature vector and the target feature vector.

[0112] Storage medium - Any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media such as CD-ROMs, floppy disks or tape drives; computer system memory or random access memory such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory such as flash memory, magnetic media (such as hard disks or optical storage); registers or other similar types of memory elements, etc. The storage medium may also include other types of memory or combinations thereof. Additionally, the storage medium may be located in a first computer system in which the program is executed, or may be located in a different second computer system that is connected to the first computer system through a network (such as the Internet). The second computer system may provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media residing in different locations (such as in different computer systems connected through a network). The storage medium may store program instructions executable by one or more processors (such as specifically implemented as a computer program).

[0113] Of course, for a storage medium containing computer-executable instructions provided by the embodiment of the present application, its computer-executable instructions are not limited to the above postoperative recurrence prediction method, and can also execute the related operations in the postoperative recurrence prediction method provided by any embodiment of the present application.

[0114] The postoperative recurrence prediction device, postoperative recurrence prediction system, storage medium, and postoperative recurrence prediction equipment provided in the above embodiments can execute the postoperative recurrence prediction method provided in any embodiment of the present application. For technical details not described in detail in the above embodiments, reference may be made to the postoperative recurrence prediction method provided in any embodiment of the present application.

[0115] The above is only the preferred embodiment of the present application and the technical principles applied. The present application is not limited to the specific embodiments here. Various obvious changes, re-adjustments, and substitutions that can be made by those skilled in the art will not depart from the protection scope of the present application. Therefore, although the present application has been described in more detail through the above embodiments, the present application is not limited to the above embodiments. Without departing from the concept of the present application, more other equivalent embodiments can be included, and the scope of the present application is determined by the scope of the claims.

Claims

1. A method for predicting postoperative recurrence, characterized in that: include: obtaining a first image pyramid of a stained slide image of a target object, obtaining a plurality of first image blocks through the first image pyramid, and extracting a feature vector of each first image block; obtaining a second image pyramid of the immunohistochemistry slide image of the target object, obtaining a plurality of second image blocks through the second image pyramid, and extracting a feature vector of each second image block; Determining a postoperative prediction result and a result confidence level corresponding to the first image block based on the feature vector of each first image block, and determining a postoperative prediction result and a result confidence level corresponding to the second image block based on the feature vector of each second image block; determining a target feature vector from the feature vectors of the plurality of first image blocks and the feature vectors of the plurality of second image blocks according to the result confidences of the plurality of first image blocks and the result confidences of the plurality of second image blocks; A postoperative prediction result of the target object is determined according to the feature vector and the target feature vector, where the postoperative prediction result is postoperative recurrence or no postoperative recurrence.

2. The method for predicting postoperative recurrence according to claim 1, wherein: The acquiring a plurality of first image blocks through the first image pyramid includes: Acquire a first-level image and a second-level image in the first image pyramid, wherein the magnification of the stained slide image is twice that of the first-level image, and the magnification of the first-level image is four times that of the second-level image; determining a valid area based on a third-level image having a resolution in the thousands of levels in the first image pyramid, and mapping the valid area to the first-level image and the second-level image based on a scaling ratio between the third-level image and the first-level image and the second-level image; Based on a preset window, the first-level image and the second-level image are subjected to sliding window cropping to obtain multiple image blocks. When the image block of the first-level image is in a corresponding valid area or the image block of the second-level image is in a corresponding valid area, the corresponding image block is determined as the first image block.

3. The method for predicting postoperative recurrence according to claim 1, wherein: The acquiring a plurality of second image blocks through the second image pyramid includes: Acquire a fourth-level image in the second image pyramid, wherein the magnification of the immunohistochemistry slide image is twice the magnification of the fourth-level image; determining a valid area according to a fifth-level image having a resolution in the thousands in the second image pyramid, and mapping the valid area to the fourth-level image according to a scaling ratio between the fifth-level image and the fourth-level image; The fourth-level image is cropped by sliding a window based on a preset window to obtain a plurality of image blocks. When an image block of the fourth-level image is in a corresponding valid area, the corresponding image block is determined as a second image block.

4. The method for predicting postoperative recurrence according to claim 1, wherein: The determining of a postoperative prediction result and a result confidence level corresponding to the first image block based on the feature vector of each first image block, and determining a postoperative prediction result and a result confidence level corresponding to the second image block based on the feature vector of each second image block, includes: The feature vector of the first image block and the feature vector of the second image block of the same magnification are sequentially input into the corresponding first classification network to obtain the postoperative prediction result and result confidence output by the first classification network.

5. The method for predicting postoperative recurrence according to claim 1, wherein: The determining, according to the result confidences of the plurality of first image blocks and the result confidences of the plurality of second image blocks, a target feature vector from the feature vectors of the plurality of first image blocks and the feature vectors of the plurality of second image blocks, comprises: The result confidences of the first image blocks and the result confidences of the second image blocks at the same magnification are sorted from large to small, and the feature vectors of the first image blocks and / or the second image blocks with the highest sorting are determined as the target feature vectors of the corresponding magnification.

6. The method for predicting postoperative recurrence according to claim 4, wherein: Determining a postoperative prediction result of the target object based on the feature vector and the target feature vector includes: splicing the feature vector of the first image block and the feature vector of the second image block at the same magnification and inputting the concatenated features into the corresponding first convolution block to obtain a first feature vector output by the first convolution block; The target feature vectors with the same magnification are concatenated and input into the corresponding second convolution block to obtain a second feature vector output by the second convolution block; splicing the first eigenvector and the second eigenvector of each magnification and inputting the concatenated eigenvector into a third convolution block to obtain a multimodal eigenvector output by the third convolution block; The multimodal feature vector is input into a second classification network, and a postoperative prediction result of the target object is determined by the second classification network.

7. The method for predicting postoperative recurrence according to claim 6, wherein: The training steps of the first classification network, the first convolution block, the second convolution block, the third convolution block and the second classification network include: Obtaining an image pyramid of a sample image group, and obtaining a sample feature set based on the image pyramid; wherein the sample image group includes a stained glass slide sample image and an immunohistochemistry slide sample image of the same object, and label information of the sample feature set is the same as label information of the sample image group; Inputting the sample feature set into the first classification network, and screening a target sample feature set from the sample feature set based on the postoperative prediction result and result confidence output by the first classification network; Inputting the sample feature set into the first convolution block, inputting the target sample feature set into the second convolution block, concatenating the output features of the first convolution block and the output features of the second convolution block, and inputting the concatenated features into the third convolution block to obtain a sample feature vector output by the third convolution block; Inputting the sample feature vector into a second classification network to obtain a postoperative prediction result and a result confidence level output by the second classification network, and determining a total loss value based on the postoperative prediction results and result confidence levels output by the first and second classification networks and the label information of the sample image group; Based on the total loss value, backpropagation is started from the second classification network to optimize the network parameters of the second classification network, the third convolution block, the second convolution block and the first convolution block. Based on the total loss value, backpropagation is started from the first classification network to optimize the network parameters of the first classification network.

8. A postoperative recurrence prediction device, characterized in that: include: a first image processing module configured to obtain a first image pyramid of the stained slide image of the target object, obtain a plurality of first image blocks through the first image pyramid, and extract a feature vector of each of the first image blocks; a second image processing module configured to obtain a second image pyramid of the immunohistochemistry slide image of the target object, obtain a plurality of second image blocks through the second image pyramid, and extract a feature vector of each second image block; a first prediction module configured to determine a postoperative prediction result and a result confidence level corresponding to the first image block based on the feature vector of each first image block, and to determine a postoperative prediction result and a result confidence level corresponding to the second image block based on the feature vector of each second image block; a vector acquisition module configured to determine a target feature vector from the feature vectors of the plurality of first image blocks and the feature vectors of the plurality of second image blocks according to the result confidences of the plurality of first image blocks and the result confidences of the plurality of second image blocks; The second prediction module is configured to determine a postoperative prediction result of the target object according to the feature vector and the target feature vector, wherein the postoperative prediction result is postoperative recurrence or no recurrence.

9. A postoperative recurrence prediction device, characterized in that: include: one or more processors; A storage device stores one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the postoperative recurrence prediction method according to any one of claims 1 to 7.

10. A storage medium containing computer-executable instructions, characterized in that: When executed by a computer processor, the computer executable instructions are used to perform the postoperative recurrence prediction method according to any one of claims 1 to 7.

Images