Image processing method, computer-assisted cancer prognosis prediction method and system
By blocking and clustering images and combining image processing models, the accuracy of cancer prognosis prediction is improved, the problem of low image processing accuracy in the prior art is solved, and more accurate prediction results are achieved.
Patent Information
- Application Number
- CN202411237722.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-04
- Publication Date
- 2024-12-06
AI Technical Summary
In the prior art, image-based cancer prognosis prediction analysis technology has the problem of low image processing accuracy, especially in multiple stained images, which makes it difficult to effectively mine data information, resulting in inaccurate prediction results.
By obtaining the image set of the target object, dividing it into multiple block images, clustering it to determine the clustering area containing the target elements, and inputting it into the image processing model for fusion, predicting the target indicators within the preset time period, and improving Image processing accuracy.
Through clustering and fusion technology, the accuracy of image processing is significantly improved, and the survival rate and disease progress of cancer can be more accurately predicted, solving the problem of low image processing accuracy in the prior art.
Smart Images

Figure CN119092060A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to large model technology and image processing fields, and specifically, to an image processing method, a computer-assisted cancer prognosis prediction method and a system. Background Art
[0002] At present, image-based prognosis prediction and analysis technology mainly uses traditional statistical models to predict results through manual feature extraction and survival analysis statistical models. The advantage of this method is strong interpretability, but the limitation is that the manual features are limited and cannot capture complex morphological features, and cannot model complex feature relationships. In addition, the prognosis prediction and analysis technology of pathology images based on deep learning uses a deep learning framework to directly process raw image data. However, when faced with multiple stained images, existing algorithms are difficult to effectively mine information in the data, resulting in low accuracy in image processing.
[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention
[0004] The embodiments of the present application provide an image processing method, a computer-assisted cancer prognosis prediction method and a system to at least solve the technical problem of low accuracy of image processing in related technologies.
[0005] According to one aspect of an embodiment of the present application, there is provided an image processing method, comprising: acquiring an image set of a target object, wherein different images to be processed in the image set are used to represent images obtained by scanning different slices of the target object, and there are target elements in the target object that affect the target indicators corresponding to the target object; dividing the images to be processed in the image set to obtain a plurality of block images of the images to be processed; clustering the plurality of block images to obtain a target clustering region containing the target elements; inputting the target clustering region into an image processing model, and using the image processing model to fuse the image set to obtain a fusion result, wherein the fusion result is used to predict the target indicators of the target object in a preset future time period.
[0006] According to another aspect of an embodiment of the present application, an image processing method is provided, comprising: responding to an input instruction acting on an operation interface, displaying an image set of a target object on the operation interface, wherein different images to be processed in the image set are used to represent images obtained by scanning different slices of the target object, and there are target elements in the target object that affect the target indicators corresponding to the target object; responding to an image processing instruction acting on the operation interface, displaying a fusion result of the image set on the operation interface, wherein the fusion result is used to predict the target indicators of the target object within a preset future time period, and the fusion result is obtained by inputting a target clustering area into an image processing model and fusing the image set using the image processing model, and the target clustering area is an area containing the target element obtained by clustering multiple block images.
[0007] According to another aspect of an embodiment of the present application, an image processing method is provided, comprising: obtaining an image set of a target object by calling a first interface, wherein the first interface includes a first parameter, a parameter value of the first parameter is an image set, different images to be processed in the image set are used to characterize images obtained by scanning different slices of the target object, and there are target elements in the target object that affect the target index corresponding to the target object; dividing the images to be processed in the image set to obtain a plurality of block images of the images to be processed; clustering the plurality of block images to obtain a target clustering region containing the target element, wherein the target element is an element that affects the preset index corresponding to the target object; inputting the target clustering region into an image processing model, and using the image processing model to fuse the image set to obtain a fusion result, wherein the fusion result is used to predict the target index of the target object in a preset future time period; outputting the fusion result by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter is the fusion result.
[0008] According to another aspect of an embodiment of the present application, a computer-assisted cancer prognosis prediction method is provided, comprising: obtaining a set of digital pathology panoramic slice images of a biological object, wherein different digital pathology panoramic slice images in the set of digital pathology panoramic slice images are used to characterize images obtained by scanning different slices of the biological object, and there are cancer elements in the biological object that affect the corresponding survival rate of the biological object; dividing the digital pathology panoramic slice images in the set of digital pathology panoramic slice images to obtain a plurality of block images of the digital pathology panoramic slice images; clustering the plurality of block images to obtain a target clustering region containing the cancer elements; inputting the target clustering region into an image processing model, and using the image processing model to fuse the set of digital pathology panoramic slice images to obtain a fusion result; and predicting the survival rate of the biological object in a preset future time period based on the fusion result.
[0009] According to another aspect of an embodiment of the present application, a computer-assisted cancer prognosis prediction system is provided, comprising: an imaging device, used to collect a set of digital pathology panoramic slice images of a biological object, wherein different digital pathology panoramic slice images in the set of digital pathology panoramic slice images are used to characterize images obtained by scanning different slices of the biological object, and there are cancer elements in the biological object that affect the corresponding survival rate of the biological object; a processing device, connected to the imaging device, used to divide the digital pathology panoramic slice images in the set of digital pathology panoramic slice images to obtain a plurality of block images of the digital pathology panoramic slice images; clustering the plurality of block images to obtain a target clustering region containing the cancer elements; inputting the target clustering region into an image processing model, and using the image processing model to fuse the set of digital pathology panoramic slice images to obtain a fusion result; and predicting the survival rate of the biological object in a preset future time period based on the fusion result.
[0010] According to another aspect of the embodiments of the present application, a computer terminal is further provided, including: a memory storing an executable program; and a processor for running the program, wherein the method in each embodiment of the present application is executed when the program is running.
[0011] According to another aspect of an embodiment of the present application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the methods in each embodiment of the present application.
[0012] According to another aspect of the embodiments of the present application, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present application is implemented.
[0013] According to another aspect of an embodiment of the present application, a computer program product is also provided, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present application is implemented.
[0014] According to another aspect of the embodiments of the present application, a computer program is also provided, and when the computer program is executed by a processor, the methods in the various embodiments of the present application are implemented.
[0015] In an embodiment of the present application, an image set of a target object is obtained, wherein different images to be processed in the image set are used to represent images obtained by scanning different slices of the target object, and there is a target element in the target object that affects the target index corresponding to the target object; the images to be processed in the image set are divided to obtain multiple block images of the images to be processed; the multiple block images are clustered to obtain a target clustering area containing the target element; the target clustering area is input into an image processing model, and the image processing model is used to fuse the image set to obtain a fusion result, wherein the fusion result is used to predict the target index of the target object in a preset future time period, thereby achieving the purpose of improving the accuracy of image processing. It is easy to notice that the images to be processed in the image set can be divided to obtain multiple block images of the images to be processed, and by clustering the multiple block images, the target clustering area containing the target element can be quickly determined. Since the target element affects the target index corresponding to the target object, the image processing model can perform effective feature fusion on the image set according to the target clustering area, so that the obtained fusion result can more accurately predict the target index of the target object in the preset future time period, thereby improving the accuracy of image processing, thereby solving the technical problem of low accuracy of image processing in the related art.
[0016] It is easy to notice that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present application, and do not constitute a limitation of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0018] Figure 1 is a schematic diagram of an application scenario of an image processing method according to an embodiment of the present application;
[0019] Figure 2 is a flowchart of an image processing method according to an embodiment of the present application;
[0020] Figure 3 is a schematic diagram of a preprocessing structure of a pathological image according to an embodiment of the present application;
[0021] Figure 4 is a schematic diagram of a cancer prognosis prediction structure according to an embodiment of the present application;
[0022] Figure 5 It is a data comparison diagram of the present application and the current solution provided according to the embodiment of the present application;
[0023] Figure 6is a schematic diagram of a survival prediction analysis according to an embodiment of the present application;
[0024] Figure 7 is a flowchart of an image processing method according to an embodiment of the present application;
[0025] Figure 8 is a flowchart of an image processing method according to an embodiment of the present application;
[0026] Fig. 9 is a flowchart of a computer-assisted cancer prognosis prediction method according to an embodiment of the present application;
[0027] Fig.10 is a schematic diagram of an image processing device according to an embodiment of the present application;
[0028] Fig.11 is a schematic diagram of an image processing device according to an embodiment of the present application;
[0029] Fig.12 is a schematic diagram of an image processing device according to an embodiment of the present application;
[0030] Fig.13 is a schematic diagram of a computer-assisted cancer prognosis prediction device according to an embodiment of the present application;
[0031] Fig.14 is a structural diagram of a computer-assisted cancer prognosis prediction system according to an embodiment of the present application;
[0032] Fig.15 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0033] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0035] The technical solution provided in this application is mainly implemented by large-scale model technology. The large model here refers to a deep learning model with large-scale model parameters, which can usually contain hundreds of millions, tens of billions, hundreds of billions, trillions or even more than 10 trillion model parameters. The large model can also be called a cornerstone model / foundation model (Foundation Model). The large model is pre-trained through large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as large-scale language model (Large Language Model, LLM), multi-modal pre-training model (multi-modal pre-training model), etc.
[0036] It should be noted that when the large model is actually applied, the pre-trained model can be fine-tuned through a small number of samples, so that the large model can be applied to different tasks. For example, the large model can be widely used in natural language processing (NLP), computer vision, speech processing and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image caption (IC), image generation, etc. It can also be widely used in natural language processing tasks such as text-based sentiment classification, text summary generation, machine translation, etc. Therefore, the main application scenarios of the large model include but are not limited to digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc. In the embodiment of the present application, data processing through an image processing model in a medical scenario is taken as an example for explanation,
[0037] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:
[0038] Immunohistochemistry (IHC) is an immunological principle that uses modified antibodies to specifically bind to antigens in biological tissues to selectively identify antigens in tissue sections or smear cells and detect whether there is a "target antigen" in cells or tissues. This method can not only measure the expression of antigens, but also observe the location of antigens and improve the sensitivity and specificity of pathological diagnosis.
[0039] Hematoxylin-eosin staining (H&E) is a commonly used staining method in pathological histology. Hematoxylin staining mainly makes the cell nucleus appear blue, while eosin staining makes the cytoplasm appear red. This staining method helps to observe cell morphology, tissue structure and intercellular relationships.
[0040] Whole Slide Image (WSI) refers to the digital scanning of the entire tissue slice or smear to generate a high-resolution image file. This image can be enlarged, reduced, rotated, etc., which is convenient for observation and analysis of tissue slices.
[0041] The above-mentioned IHC and H&E are two commonly used staining methods in pathology. IHC is mainly used to detect the expression of specific proteins or antigens, while H&E is mainly used to observe cell morphology and tissue structure. WSI is a digital tissue section image that can improve the efficiency and accuracy of pathological diagnosis.
[0042] At present, cancer prognosis prediction research is crucial for judging disease progression, predicting treatment effects and patient survival, and is helpful for achieving personalized treatment. As the gold standard for most tumor diagnosis, pathological images provide rich information for distinguishing tumor types, evaluating the extent and boundaries of lesions, and determining molecular subtypes with their ultra-high cell-level resolution, which is a crucial modality in personalized treatment. In recent years, many exploratory studies have used deep learning to analyze H&E stained pathology images for prognosis prediction. However, clinical pathology is increasingly aware of the advantages of multi-IHC staining images. Compared with H&E, IHC staining not only uses hematoxylin staining to depict cells, tissues and tumor structures, but also further reveals specific biomarkers through specific staining, thereby achieving the identification of cells and cancer subtypes. For example, IHC can reveal the infiltration rate of cytotoxic immune cells (CD8-positive T cells), an indicator that has been shown to be associated with good prognosis prediction for a variety of solid tumors. At present, there are few studies on deep learning models for survival prediction based on multi-IHC staining. This application aims to fill this gap and proposes the first cancer survival analysis method based on multi-staining immunohistochemistry.
[0043] The existing computational pathology models / products mainly support the analysis of a single WSI, and it is difficult to effectively mine the relationship between multiple differently stained sections of the same wax block and perform survival analysis. In addition, most of the current models / products mainly support the analysis of H&E images. Due to differences in biological meanings, these models are less effective when used directly on multiple IHC stained images.
[0044] At present, pathological analysis plays a key guiding role in the clinical diagnosis and treatment of cancer. By using artificial intelligence modeling to analyze a variety of immunohistochemistry (IHC) pathological images, it can support doctors to deeply understand the patient's condition, optimize the treatment effect and prolong the patient's survival. In recent years, computational pathology has made some progress in using H&E staining pathology images for survival prediction. However, there are still few analysis methods for multiple IHC modalities. IHC can more clearly reveal the tumor microenvironment by staining various specific proteins, helping doctors to understand the patient's condition more accurately.
[0045] Prognostic prediction analysis technology based on IHC images. Currently, the prognostic prediction analysis based on IHC images in clinical research mostly adopts more traditional statistical models. Its paradigm can be summarized as first performing manual feature extraction, then establishing a survival analysis statistical model based on the features, and finally using a significance test tool to check the significance of each feature and the model typing performance. The manual features extracted are generally the total number of stained target cells, the number per unit area, the ratio of the number of target cells to the number of cancer cells, etc. The survival analysis statistical models include proportional hazard models (Cox regression models), etc. The advantage of this type of analysis technology is that it is highly interpretable, the manual features have clear meanings, and the proportion of the features in the statistical model can also be intuitively obtained through the size of the parameters after training. And its disadvantages are also obvious, a. Manual features are very limited and can only depict some simple features. The complex morphology and distribution characteristics of cells are difficult to design manual feature calculations. b. Statistical models can only model linear or certain characteristic nonlinear relationships between features, and cannot model more complex relationships between features. Therefore, this type of method has limited performance, is difficult to fit large-scale data, and has poor reproducibility on external validation sets.
[0046] In order to solve the above problems, the prognosis prediction and analysis technology of pathology images based on deep learning has flourished in recent years. The deep learning framework directly takes the original pathology images as input and tries to use data-driven fitting to predict the patient's prognosis. Due to the ultra-high resolution of pathology images, a picture usually has hundreds of millions to billions of pixels. Current hardware devices do not support deep learning models to learn directly on such a large scale of data.
[0047] This application innovatively and effectively integrates information from different IHC modality images and achieves advanced performance in prognosis prediction. This application provides a general multi-immunohistochemistry analysis framework that uses a multi-layer cross-regional attention mechanism to effectively extract prognosis prediction features within and between regions. This application makes full use of the biological significance of IHC staining to guide the network to automatically focus on important areas, that is, the target clustering areas containing target elements mentioned in this application, which significantly reduces the amount of model calculation without the use of additional annotations. This application also provides a specially improved spectral clustering algorithm that enables the model to better understand spatial partitioning information. This application can verify the performance superiority of this method in multimodal IHC analysis through sufficient experiments.
[0048] According to an embodiment of the present application, an image processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0049] Considering the huge amount of model parameters of the large model and the limited computing resources of the mobile terminal, the above-mentioned image processing method provided in the embodiment of the present application can be applied to the following Figure 1 The application scenarios shown are not limited to these. Figure 1 is a schematic diagram of an application scenario of an image processing method according to an embodiment of the present application. Figure 1 In the application scenario shown, the large model is deployed in the server 10, and the server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client device 20 here may include but is not limited to: a smart phone, a tablet computer, a laptop computer, a PDA, a personal computer, a smart home device, a vehicle-mounted device, etc. The client device 20 can interact with the user through a graphical user interface to implement the call of the large model, thereby implementing the method provided in the embodiment of the present application.
[0050] In the embodiment of the present application, the system composed of the client device and the server can perform the following steps: the client device executes uploading the image set of the target object, and the server executes obtaining the image set of the target object; the image to be processed in the image set is divided to obtain multiple block images of the image to be processed; the multiple block images are clustered to obtain a target clustering area containing the target element; the target clustering area is input into the image processing model, and the image set is fused using the image processing model to obtain a fusion result. It should be noted that the embodiment of the present application can be performed in the client device when the operating resources of the client device can meet the deployment and operation conditions of the large model.
[0051] Under the above operating environment, this application provides Figure 2 The image processing method shown. Figure 2 FIG. 1 is a flow chart of an image processing method according to an embodiment of the present application. Figure 2 As shown, the method may include the following steps:
[0052] Step S202: Acquire a set of images of the target object.
[0053] The different images to be processed in the image set are used to represent images obtained by scanning different slices of the target object, and the target object contains target elements that affect the target indicators corresponding to the target object.
[0054] The above-mentioned target objects include but are not limited to biological objects, such as human bodies, animal bodies, etc. The above-mentioned target objects can be living objects or non-living objects. The state of the specific target objects can be determined according to the actual scenario.
[0055] The target object set can be obtained by obtaining tissue on the target object. The size of the tissue can be 2CM x 2CM x 0.5CM, but the actual size can be determined according to the size that can be actually obtained and the required size. The size of the tissue is not specifically limited here.
[0056] For example, the tissue may be a portion of the stomach or pancreas removed by surgery, and then the location of interest is sampled (a piece of the size similar to the above is cut off). After the tissue is obtained, the tissue may be dehydrated and wax-wrapped, and finally the wrapped tissue may be sliced and scanned to obtain the above image set.
[0057] The above-mentioned image set may be images of different slices obtained by slicing and scanning the corresponding tissue of the target object. The different images to be processed contained in the image set may be different pathological slice images, for example, images of cytotoxic immune cells (CD8 positive T cells) and images of auxiliary immune cells (CD4). The above-mentioned image set may be obtained by scanning different slices of the target object using a scanning device. The above-mentioned scanning device may include but is not limited to digital devices such as digital pathology scanners and microscopes, and the shooting device is not specifically limited here.
[0058] In the scenario of cancer prognosis prediction, IHC can reveal the infiltration rate of CD8-positive T cells, which has been truly correlated with good prognosis prediction for a variety of solid tumors. Therefore, image processing can be performed by obtaining the above-mentioned CD8 so that the final result can be applied to the cancer prognosis prediction scenario.
[0059] The different slices of the target object mentioned above may refer to slice images of the target object containing the object to be studied. For example, if the object to be studied is cancer, the different slices mentioned above may be areas containing cancer cells, such as the stomach, liver, pancreas, etc.; if the object to be studied is a tumor, the different slices mentioned above may be areas containing solid tumors, such as the lungs, liver, kidneys, etc.
[0060] It should be noted that in the medical field, tissue on the target object can be obtained. The tissue itself is three-dimensional. The tissue can be encapsulated in wax and sliced to obtain different slices. The above-mentioned different slices can be obtained by processing with different reagents. Processing the slices with different reagents can color different cells, thereby facilitating the subsequent identification of the slice image.
[0061] The above-mentioned target indicators can be indicators of the target object that need to be predicted through the image set of the target object, wherein the target indicator can be the survival rate of the target object, the disease progression of the target object, the treatment response, the condition, etc. The target indicator can be determined here according to actual needs.
[0062] The target element mentioned above may be an element that affects the target index corresponding to the target object, wherein the target element may be a cancer element, a tumor element, a virus element, etc. The target element is not specifically limited here, and the target element mentioned above may be determined according to the target index to be predicted actually.
[0063] It should be noted that the above-mentioned cancer elements may be cancer cells, tumor elements may be tumor cells, and viral elements may be viral cells, but are not limited to this. Cancer elements may also be cells other than cancer cells that can affect the development of cancer, tumor elements may also be cells other than tumor cells that can affect the development of tumors, and viral elements may also be cells other than viral cells that can affect the development of viruses.
[0064] In an optional embodiment, when it is necessary to predict the target indicator of the target object, different slices related to the target indicator in the target image can be scanned to obtain an image set of the target image. Since the target element will affect the target indicator, the area where the target element is located in the image set of the target object can be analyzed so that the target indicator can be predicted based on the analysis results of the image set, thereby improving the accuracy of the prediction.
[0065] Step S204, dividing the image to be processed in the image set to obtain a plurality of block images of the image to be processed.
[0066] The multiple block images of the above-mentioned image to be processed can be obtained by dividing a single image to be processed. The sizes of the multiple block images can be determined according to the division method of the image to be processed.
[0067] In an optional embodiment, the above steps can be performed on multiple images to be processed in an image set, that is, for different images to be processed in the image set, different images to be processed can be divided to obtain multiple block images of different images to be processed. The image to be processed can be segmented according to a preset pixel scale to obtain a block image of a preset size. For example, the image to be processed can be segmented into 256x256 block images at a scale of 0.45μm / pixel. The image to be processed can also be segmented into block images of other sizes according to other pixel scales. This segmentation method can help decompose a larger image to be processed into multiple smaller block images to facilitate analysis of the image to be processed.
[0068] When processing large-size or high-resolution images, dividing the entire image into multiple small blocks can reduce computational complexity and memory consumption. By dividing the image into multiple small blocks, the processing algorithm can process each small block in parallel, thereby increasing the processing speed. The memory space required for each processing can be reduced. For some processing algorithms, dividing the image into multiple small blocks can improve the accuracy of the algorithm because the algorithm can process each small block more carefully. Block processing of the image to be processed helps improve processing efficiency, reduce memory consumption, improve algorithm accuracy, and facilitate the implementation of specific functions.
[0069] Step S206: clustering the multiple block images to obtain a target clustering region containing the target element.
[0070] After obtaining the plurality of block images in the above-mentioned image to be processed, the model can be used to extract features of the plurality of block images to obtain a plurality of block features corresponding to the plurality of block images, and the plurality of block features can be clustered to obtain a target clustering region containing the target element. For example, a deep network model (ResNet50) can be used to extract block features of the plurality of block images, and the deep network model can be a model trained using an image recognition dataset (ImageNet).
[0071] The target cluster region containing the target element may be a cluster region in which the target element accounts for a relatively large proportion among a plurality of cluster regions obtained by clustering a plurality of block images.
[0072] In an optional embodiment, block features corresponding to the plurality of block images may be clustered according to the similarity between the plurality of block images and the central pixel, and the number of block features in the clustering region may be controlled, thereby obtaining a target clustering region containing the target element. By determining the target clustering region containing the target element, it is convenient to quickly locate the region that needs to be analyzed emphatically when processing the to-be-processed images in the image set in the subsequent processing, thereby improving the efficiency and accuracy of image processing.
[0073] Step S208: input the target clustering region into the image processing model, and fuse the image set using the image processing model to obtain a fusion result.
[0074] The fusion result is used to predict the target index of the target object in a preset future time period.
[0075] The above-mentioned image processing model can be an image fusion model, through which different images to be processed in an image collection can be fused, thereby improving image quality, enhancing image details or obtaining more comprehensive information, so that the target indicators of the target object in a preset future time period can be more accurately predicted based on the fusion results.
[0076] The above-mentioned preset future time period can be a preset future time period of any length, and the preset future time period can be measured by time measurement standards such as minutes, hours, days, months, years, etc. The discrete data, change trend, etc. of the target indicator corresponding to the target object in the preset future time period can be predicted by the fusion result. There is no limitation on the output form of the predicted target indicator here.
[0077] In this process, the block features of the image to be processed in the image set are first assigned to different clustering regions through a clustering algorithm, that is, similar block features are grouped into one category. These clustering regions are then input into an image processing model, such as an image fusion model, and the model is used to process images in different clustering regions to finally obtain a fusion result.
[0078] In an optional embodiment, the target clustering region can be input into the image processing model so that when the image processing model fuses different images to be processed in the image set, it can combine the information of the target elements related to the predicted target indicators given by the target clustering region to perform targeted fusion, so that the obtained fusion result is a fusion result related to the target elements, thereby improving the accuracy and efficiency of the prediction when predicting the target indicators of the target object in a preset future time period based on the fusion result.
[0079] In the field of cancer prognosis prediction, the target element may be a cancer cell, the target object may be a biological object, and the different images to be processed in the image set may be CD8 images and CD4 images obtained by scanning different slices, and the images may be images that depict cell, tissue and tumor structure after hematoxylin staining. The CD4 image and CD8 image of the biological object may be obtained, and the CD4 image and the CD8 image may be divided respectively to obtain multiple block images of the CD4 image and multiple block images of the CD8 image, and feature extraction may be performed on the multiple block images of the CD4 image and the multiple block images of the CD8 image to obtain multiple block features of the CD4 image and multiple block features of the CD8 image, and the multiple block features of the CD4 image and the multiple block features of the CD8 image may be clustered respectively to obtain a target clustering region containing cancer cells, and the target clustering region may be input into an image processing model, and the CD4 image and the CD8 image may be fused using the image processing model to obtain a fusion result, and the fusion result may be represented by a fusion feature, and the survival state of the biological object in a preset future time period may be predicted by the fusion feature, and it should be noted that the survival state is associated with the above-mentioned cancer cells.
[0080] In the field of tumor prognosis prediction, the target element may be a tumor cell, the target object may be a biological object, and the different images to be processed in the image set may be CD8 images and CD4 images obtained by scanning different slices, and the images may be images that depict cells, tissues, and tumor structures after hematoxylin staining. The CD4 image and CD8 image of the biological object may be obtained, and the CD4 image and the CD8 image may be divided respectively to obtain multiple block images of the CD4 image and multiple block images of the CD8 image, and feature extraction may be performed on the multiple block images of the CD4 image and the multiple block images of the CD8 image to obtain multiple block features of the CD4 image and multiple block features of the CD8 image, and the multiple block features of the CD4 image and the multiple block features of the CD8 image may be clustered respectively to obtain a target clustering region containing tumor cells, and the target clustering region may be input into an image processing model, and the CD4 image and the CD8 image may be fused using the image processing model to obtain a fusion result, and the fusion result may be represented by a fusion feature, and the survival state of the biological object in a preset future time period may be predicted by the fusion feature, and it should be noted that the survival state is associated with the above-mentioned tumor cells.
[0081] Through the above steps, an image set of the target object is obtained, wherein different images to be processed in the image set are used to represent images obtained by scanning different slices of the target object, and there are target elements in the target object that affect the target index corresponding to the target object; the images to be processed in the image set are divided to obtain multiple block images of the images to be processed; the multiple block images are clustered to obtain a target clustering area containing the target element; the target clustering area is input into the image processing model, and the image set is fused using the image processing model to obtain a fusion result, wherein the fusion result is used to predict the target index of the target object in a preset future time period, thereby achieving the purpose of improving the accuracy of image processing. It is easy to notice that the images to be processed in the image set can be divided to obtain multiple block images of the images to be processed, and by clustering the multiple block images, the target clustering area containing the target element can be quickly determined. Since the target element affects the target index corresponding to the target object, the image processing model can perform effective feature fusion on the image set according to the target clustering area, so that the obtained fusion result can more accurately predict the target index of the target object in the preset future time period, thereby improving the accuracy of image processing, thereby solving the technical problem of low accuracy of image processing in the related technology.
[0082] In the above embodiment of the present application, the image processing model includes multiple fusion modules, which fuse the image set based on the target clustering area to obtain a fusion result, including: expanding the block features in the target clustering area based on preset features to obtain expanded features, wherein the preset features are features of the same type as the block features in the target clustering area; using multiple fusion modules to fuse the expanded features in different dimensions to obtain a fusion result.
[0083] The above-mentioned preset features may be features of the same type of block features in some of the above-mentioned target clustering regions made according to prior knowledge. In the field of cancer prognosis prediction or tumor prognosis prediction, the above-mentioned prior knowledge may be doctor prior knowledge.
[0084] The above-mentioned multiple fusion modules can fuse the extended features in different dimensions, so that the obtained fusion results can take into account various factors and achieve a more accurate fusion effect. The above-mentioned multiple fusion modules can be different types of fusion modules. The number of the above-mentioned multiple fusion modules is not limited. The number of fusion modules can be set according to the needs of fusion, and one or more different types of fusion modules can be set.
[0085] Optionally, different dimensions may refer to region-level fusion dimensions and object-level fusion dimensions. In the region-level fusion dimension, the extended features of different clustered regions in the same image to be processed may be fused to enhance the expression effect of the image to be processed. In the object-level fusion dimension, the extended features of target clustered regions in different images to be processed in a set of images corresponding to the target object may be fused to enhance the overall expression effect of the image set.
[0086] In the field of cancer prognosis prediction, the above-mentioned different dimensions can refer to the regional-level fusion dimension and the patient-level fusion dimension. In the regional-level fusion dimension, the expanded features of different clustered regions in the same image to be processed can be fused to enhance the expression effect of cells in the image to be processed. In the patient-level fusion dimension, the expanded features of the target clustered regions in different images to be processed in the image set corresponding to the biological object can be fused to enhance the overall expression effect of the image set in cancer elements.
[0087] In an optional embodiment, preset features containing prior knowledge can be used to expand the block features in the target cluster area to obtain expanded features, thereby enriching the feature representation in the target cluster area, so that the image processing model can better combine the features in the target cluster area to fuse the image set, obtain a more accurate fusion result, and thus achieve a more accurate prediction effect. By fusing the expanded features in different dimensions through multiple fusion modules, features related to the predicted target indicators can be captured from within the area and across image areas, thereby improving the accuracy of subsequent predictions of the target indicators.
[0088] In the above embodiment of the present application, the image processing model includes: a first fusion module and a second fusion module, and multiple fusion modules are used to fuse the extended features in different dimensions to obtain a fusion result, including: using the first fusion module to fuse the extended features to obtain the first fusion feature of the image set; using the second fusion module to fuse the first fusion feature of the image set to obtain a fusion result.
[0089] The first fusion module mentioned above may be a fusion module of regional features, which is mainly used to fuse the extended features of different cluster regions in the same image to be processed to obtain the first fusion feature. The first fusion module can achieve the effect of cross-regional feature fusion.
[0090] The second fusion module mentioned above may be a fusion module of patient-level features, which is mainly used to fuse the expanded features of clustered regions in different images to be processed to obtain a fusion result. The second fusion module may achieve the effect of cross-image feature fusion.
[0091] The above-mentioned feature fusion may include but is not limited to weighted averaging of features, feature concatenation, feature connection, etc., and the specific method of feature fusion is not limited here.
[0092] The advantage of the above feature fusion method is that it can combine feature information from different sources, thereby improving the representation ability and discrimination of image features. By fusing different features, we can make full use of their complementarity, making the fusion result more comprehensive and having higher representation ability. In addition, through the multi-level fusion process, the representation ability of features can be further improved and better adapted to the required task requirements.
[0093] Therefore, this feature fusion method has broad application prospects in the field of image processing and can improve the performance and accuracy of image recognition, classification, detection and other tasks. By rationally designing the fusion module and selecting the appropriate feature fusion strategy, the advantages of different features can be fully utilized to achieve better image analysis results.
[0094] In an optional embodiment, the first fusion module can be used to fuse the extended features of different target cluster regions in the same image to be processed to obtain the first fusion features of different images to be processed in the image set, which can achieve a cross-region fusion effect. The second fusion module can be used to fuse the extended features of target cluster regions of different images to be processed in the image set to obtain a fusion result, which can achieve a cross-image fusion effect.
[0095] In the above embodiment of the present application, the first fusion module includes multiple first sub-modules, the second fusion module includes multiple second sub-modules, and the multiple first sub-modules and the multiple second sub-modules are alternately connected; the first fusion module is used to fuse the extended features to obtain the first fusion features of the image set, including: using the first sub-module to fuse the first target features of the target clustering area in the image to be processed to obtain the first fusion features of the image to be processed, wherein the first target feature is the second fusion feature output by the previous second sub-module connected to the first sub-module, or the extended feature; using the second fusion module to fuse the first fusion features of the image set to obtain a fusion result, including: using the second sub-module to fuse the first fusion features of different images to be processed in the image set to obtain the second fusion features of the image set; when the second sub-module is the last sub-module, the second fusion feature is determined as the fusion result.
[0096] The first fusion module may include multiple first submodules, and the second fusion module may include multiple second submodules. By alternately connecting multiple first submodules and multiple submodules, the features in the image set can be more fully fused, thereby improving the fusion effect of the features. It should be noted that the number of the multiple first submodules and the multiple second submodules is not limited and can be adjusted according to actual needs.
[0097] In an optional embodiment, when the first submodule is the first module in the first fusion module, the first submodule can be used to fuse the expanded features of the target cluster area in the image to be processed to obtain the first fusion feature of the image to be processed, and the first fusion feature is input into the second submodule connected to the first submodule to obtain the second fusion feature of the image set, which is processed by the fusion features output by the remaining first submodule and the previous module of the remaining second submodule. When the second submodule is the last submodule among multiple second submodules, the second fusion feature is determined as the fusion result.
[0098] Exemplarily, the above-mentioned first fusion module may include two first sub-modules, and the above-mentioned second fusion module may include two second sub-modules. The first first sub-module is used to fuse the first target features of the target clustering area in the image to be processed, and the first fusion features of the image to be processed are output. The first second sub-module is used to fuse the first fusion features of multiple images to be processed in the image set, and the second fusion features of the image set are output. The second first sub-module is used to fuse the second fusion features, and the first fusion features of the image to be processed are output. The second second sub-module is used to fuse the first fusion features to obtain a fusion result.
[0099] In the above embodiment of the present application, the second submodule includes a self-attention layer and a feedforward layer, and the second submodule is used to fuse the first fusion features of different images to be processed in the image set to obtain the second fusion features of the image set, including: using the self-attention layer to perform feature recognition on the first fusion features of the image to be processed to obtain the target weight of the first fusion feature, wherein the target weight is used to represent the association information of the first fusion features in multiple images to be processed; using the feedforward layer to fuse the first fusion features of different images to be processed according to the target weight to obtain the second fusion feature.
[0100] The self-attention layer mentioned above is a self-attention mechanism that can automatically capture the dependencies between different positions in the input sequence. When processing images, the self-attention layer can help the model focus on the association information between different slices in the image, thereby better identifying features in the image.
[0101] The above-mentioned target weights can be a set of target weights obtained by processing the first fused features through the self-attention layer. These weights are used to represent the association information of the first fused features in multiple images to be processed. The size of the weight reflects the importance and degree of association between different features. Using the target weights to perform weighted summation of different features can effectively integrate features from different images or obtained through different processing methods, thereby improving the effect of feature fusion.
[0102] The above-mentioned feedforward layer is a neural network structure for performing linear or nonlinear transformation on input data. In the present application, the feedforward layer fuses the first fused features of different images to be processed according to the target weight, that is, different features are weighted summed according to their relevance and importance to obtain the second fused feature. The second fused feature obtained after the fusion of the feedforward layer has better feature expression ability and information integration ability. By arranging the fusion of the self-attention layer and the feedforward layer in the second submodule, the relevance between different slices in the image can be better captured, and the expression ability of the feature can be improved.
[0103] It should be noted that the parameters of the self-attention layer and the feedforward layer can be adjusted according to actual needs to adapt to different image processing scenarios and task requirements.
[0104] In an optional embodiment, the first fused features of the image to be processed are firstly identified, and the target weights are obtained using the self-attention layer. These target weights can indicate the degree of association between the first fused features in different images to be processed, that is, which features are more important for the model's judgment. Then, the first fused features of different images to be processed are fused according to the target weights using the feedforward layer to obtain the second fused features. In this way, the model can comprehensively consider the features of different images, and adjust the contributions of different features according to the target weights, further improving the representation ability of the features and the performance of the model.
[0105] By using the second submodule to fuse the first fusion features of different images to be processed in the image set, the model can better utilize the correlation information between different images and improve the representation ability of features, thereby achieving better results in recognition and classification tasks. For example, in image classification tasks, by fusing the features of different images, the model can more accurately distinguish images of different categories and improve the accuracy and stability of classification. Therefore, using the self-attention layer and the feedforward layer to fuse image features can effectively improve the performance of the model in image processing tasks.
[0106] In the above embodiment of the present application, the block features in the target clustering area are expanded based on the preset features to obtain the expanded features, including: expanding the block features in the target clustering area based on the preset features to obtain the number of expanded features and the expanded feature positions, wherein the number of expanded features is used to represent the number of preset features expanded in the target clustering area, and the expanded feature position is used to represent the position of the preset features expanded in the target clustering area; the block features, the number of expanded features and the expanded feature positions in the target clustering area are spliced to obtain the expanded features.
[0107] Expanding the block features in the target clustering region by using the above-mentioned preset features can be used to enrich or expand the block features in the target clustering region, thereby improving the quality or accuracy of clustering.
[0108] After the block features are expanded, specific implementation details of the expanded features may be determined, that is, determining whether to add a plurality of expanded features and the specific positions of the expanded features in the target clustering area.
[0109] The aforementioned number of expanded features may refer to the number of newly added preset features, and the aforementioned positions of expanded features may refer to the distribution of the newly added preset features in the target clustering area.
[0110] It should be noted that the number of extended features can be used to determine the richness and possible accuracy. More extended features may mean higher clustering accuracy, but may also increase computational complexity. The extended feature position can refer to how the preset features are combined with the existing data blocks, thereby affecting the clustering results. Reasonable feature positions can make clustering more balanced and accurate.
[0111] In an optional embodiment, the block features, the number of expanded features, the location of expanded features, and the expanded preset features in the target clustering area can be spliced and combined to obtain a richer data feature set containing original features and new expanded features. By expanding the features, the inherent structure of the data can be better captured, thereby improving the accuracy of fusion. The expanded features may contain information that is lacking in the original data, which helps to represent the data more comprehensively. By increasing the number and diversity of features, subsequent fusion modules can better adapt to the complexity and heterogeneity of the data during processing. The expanded features can be adjusted according to different data distributions, making the fusion process more flexible and adaptable. The quality and accuracy of the fusion can be improved by expanding the block features in the target clustering area through preset features.
[0112] In another optional embodiment, the block features in the target cluster area can be expanded by the preset features to obtain the number of expanded features and the expanded feature positions. The number of expanded features indicates the number of preset features expanded in the target cluster area, and the expanded feature positions indicate the positions of these expanded features in the target cluster area. Then, the block features, the number of expanded features and the expanded feature positions are spliced to obtain the final expanded features.
[0113] By splicing the block features, the number of expanded features, and the location of expanded features in the target clustering area, richer feature information can be extracted, making the features of the target clustering area more comprehensive and detailed. By expanding the preset features, the understanding and analysis capabilities of the target clustering area can be increased, which helps to improve the recognition accuracy and performance of the target. At the same time, by splicing block features, expanding the number of features, and expanding the location of features, feature information at different levels can be effectively integrated, and the ability to express features can be improved, thereby better supporting subsequent tasks and applications. It can also effectively enrich the feature information in the target clustering area, improve the ability to express features, and thus enhance the recognition and analysis capabilities of the target.
[0114] In the above embodiment of the present application, clustering multiple block images to obtain a target clustering area containing a target element includes: extracting features from the multiple block images to obtain multiple block features; clustering the multiple block features to obtain multiple clustering areas, wherein the block features in different clustering areas belong to different categories; and determining the target clustering area containing the target element among the multiple clustering areas.
[0115] The above feature extraction of multiple block images can convert useful information (such as texture, color, shape, etc.) in the block images into numerical feature vectors, which is convenient for subsequent data processing and analysis. Through the segmentation and feature extraction of block images, different slices in the image can be analyzed more carefully, thereby improving the accuracy of target detection.
[0116] The above clustering can cluster multiple block features, with the goal of grouping block images with similar features into one category. There are many clustering methods, such as K-means, density-based clustering spatial application algorithm (DBSCAN), etc. The purpose of clustering is to divide different slices in the image into different clustering areas.
[0117] The above-mentioned different clustering regions can be obtained by clustering based on the similarity and position of block features. The similarity of block features in the same clustering region is relatively large, and the positions of block features in the same clustering region are close to each other. In the same range, the categories to which block features in different clustering regions belong may be different. In different ranges, since the positions are close to each other, there may be clustering regions belonging to the same category.
[0118] In an optional embodiment, feature extraction can be performed on multiple block images to abstract and refine the information of the image, thereby better describing the content of the image. Clustering these features can classify similar features into one category, thereby finding similar areas in the image. By processing the image in blocks and performing feature extraction, the processing time for the entire image can be reduced and the processing efficiency can be improved. Similar features can be classified into one category through cluster analysis, thereby more accurately locating the position of the target element. Block processing and cluster analysis of images can reduce the difficulty of processing complex images and improve the accuracy and reliability of processing.
[0119] In the above embodiment of the present application, multiple block features are clustered to obtain multiple clustering regions, including: constructing a correlation matrix of the multiple block features; determining the Laplace matrix of the multiple block features based on the correlation matrix; and performing feature decomposition on the multiple block features based on the Laplace matrix to obtain multiple clustering regions.
[0120] The above correlation matrix can quantify the relationship between multiple block features. The correlation coefficient (such as Pearson correlation coefficient) can be calculated for multiple block features to form a symmetric matrix, in which different elements of the matrix represent the correlation strength of the corresponding block feature pairs. Through the correlation matrix, highly correlated features can be identified, which helps to reduce the dimensionality of the data, avoid overfitting, and improve the generalization ability of the model.
[0121] The above Laplace matrix is used in graph theory to describe the properties of a graph. Applying it to the feature space here can help us understand the structure of the feature space. The Laplace matrix can help identify the clustering structure in the feature space and provide a basis for subsequent feature decomposition and clustering.
[0122] The above-mentioned feature decomposition can refer to a dimensionality reduction and feature extraction method for multiple block features. By decomposing the feature matrix, the hidden structure and pattern in the multiple block features can be obtained, so as to better understand the data and perform tasks such as classification and clustering.
[0123] In the above process, by constructing the correlation matrix and Laplace matrix of the block features, the relationship and similarity between different block features can be effectively captured. In this way, by performing eigendecomposition on the Laplace matrix, multiple clustering regions can be obtained, that is, the data can be divided into different categories or clusters. This method can better cluster the data and improve the accuracy and effect of clustering. The above-mentioned eigendecomposition can reduce the dimension of the data, extract the most important feature information, make clustering more accurate and effective, discover the hidden structures and patterns in multiple block features, help understand the internal relationships and laws of multiple block features, and help visualize and explain multiple block features, and more intuitively display the distribution and clustering results of multiple block features. In clustering tasks, eigendecomposition can improve the stability and robustness of clustering, thereby obtaining more reasonable and reliable clustering results.
[0124] In summary, eigendecomposition plays an important role in data clustering tasks, which can help improve the accuracy and effect of clustering and better understand and utilize data. Therefore, combining eigendecomposition with data clustering is an effective method to obtain more accurate and meaningful clustering results.
[0125] In the process of eigendecomposing multiple block features based on the Laplace matrix, the principal components in the feature space can be found through eigendecomposition. These principal components can represent the main change direction of the features, and the Laplace matrix is eigenvalue decomposed to obtain a set of eigenvectors and corresponding eigenvalues. These eigenvectors can guide the understanding of the structure of the feature space. Eigendecomposition can help identify the intrinsic structure of the data, remove redundant information, and may reveal hidden patterns in the data.
[0126] The purpose of cluster analysis is to group multiple block features so that the block features in the same group are more similar than those in other groups. Using the feature vectors obtained by feature decomposition, clustering algorithms can be applied to determine the clustering structure of the data. Clustering in the above way can understand the intrinsic grouping of multiple block features, which helps to improve the interpretability and predictive ability of the model, while reducing the feature dimension and improving computational efficiency.
[0127] The Laplacian matrix of multiple block features is determined based on the correlation matrix to transform the relationship between different block features into a graph structure, which is used to represent the connection strength between different block features. By solving the eigenvectors and eigenvalues of the Laplacian matrix, multiple clustering regions can be obtained, that is, block features with similar characteristics will be divided into the same clustering region.
[0128] In an optional embodiment, the mutual relationship between different block features can be obtained by constructing a correlation matrix between multiple block features. This correlation matrix can represent the degree of connection between different block features, thereby providing important information for subsequent clustering. Then the Laplace matrix of the block features is determined by the correlation matrix. The Laplace matrix is a commonly used graph theory tool for describing structural information in data. Here, the Laplace matrix can help capture the similarities and differences between multiple block features, so as to better perform clustering. Finally, multiple clustering regions can be obtained by performing eigendecomposition on the Laplace matrix. Eigendecomposition is a commonly used linear algebra method that can decompose a matrix into eigenvectors and eigenvalues. Here, eigendecomposition can help decompose data into different clustering regions, thereby achieving effective clustering of data. This process combines methods such as correlation matrix, Laplace matrix and eigendecomposition, and can effectively implement cluster analysis of block features, thereby obtaining the above-mentioned multiple clustering regions.
[0129] In the above embodiment of the present application, determining a target clustering region containing a target element among multiple clustering regions includes: determining at least one initial clustering region containing a target element among the multiple clustering regions; screening at least one initial clustering region based on the number of block features in the at least one initial clustering region to obtain a target clustering region, wherein the number of block features in the target clustering region is greater than the number of block features in other clustering regions, and the other clustering regions are other clustering regions in the at least one initial clustering region except the target clustering region.
[0130] In an optional embodiment, after obtaining multiple clustering regions, the multiple clustering regions can be preliminarily screened to obtain at least one initial clustering region containing a target element among the multiple clustering regions, and the target element is associated with the demand for subsequent prediction target indicators. The at least one initial clustering region obtained by screening can better meet the prediction requirements of subsequent target indicators. In the case where there are multiple initial clustering regions, the initial clustering regions need to be further screened to retain the target clustering regions containing a large number of target elements. The multiple initial clustering regions can be sorted according to the number of block features in the initial clustering regions, and the multiple target clustering regions ranked first can be retained.
[0131] The first preset number of the target clustering areas retained above can be determined according to actual conditions. The first preset number can be 20, 21, 24, etc. This is only an example, and the specific number can be determined according to needs.
[0132] In the process of clustering multiple block features, the number of block features in the clustering area can be controlled, and the number of block features contained in different clustering areas can be controlled to be less than a second preset number. The second preset number can be 400, 500, etc. This is only an example, and the specific number can be determined according to needs.
[0133] In the field of cancer analysis, biological objects may include CDM4 pathology images and CDM8 pathology images. Figure 3 is a schematic diagram of a preprocessing structure of a pathological image according to an embodiment of the present application, such as Figure 3 As shown, the CDM4 pathology image and the CDM8 pathology image can be segmented into small blocks of 256x256 at a scale of 0.45 μm / pixel to obtain multiple block images of the CDM4 pathology image and multiple block images of the CDM8 pathology image. The feature extraction module is used to perform feature extraction (Patch Embeddings) on the multiple block images to obtain multiple block features. The multiple block features can be clustered to obtain multiple clustering regions.
[0134] In the process of clustering multiple block features, the correlation matrix between multiple block features is first established. The correlation between multiple block features is calculated by the similarity of their features and the distance between the central pixels. Then, the Laplacian matrix of the correlation matrix is calculated and k clusters are obtained by eigendecomposition. In the clustering process, the number of block features contained in the cluster category can be controlled to be equal to 400. Through the block feature clustering algorithm, the block features will be clustered into multiple regions according to similarity and position. According to clinical prior knowledge, the area where cancer cells gather in the tissue has a higher value in determining the prognosis of the patient. IHC staining markers can be used to segment and classify cells, determine the location and category of each cell, and then count the number of cancer cells in different clustering areas, and finally include the first 24 areas containing more cancer cells, that is, the target clustering area.
[0135] Figure 4 is a schematic diagram of a cancer prognosis prediction structure according to an embodiment of the present application, such as Figure 4 As shown in the figure, in order to better predict the prognosis, some manual features containing the doctor's prior knowledge can be used to expand the block features. The expanded information can include the number of cancer cells and immune cells in the block feature (Cell Count Embeddings) and the relative position of the block feature in the clustering area. The expanded features obtained after expansion are spliced by the original block features (Region Embeddings), the encoded cell number (Patch Position Embeddings) and the encoded relative position (Patch Position Embeddings).
[0136] Figure 4 The first fusion module includes regional-level fusion, and the first fusion module may include two first submodules, and the first submodule may be a deep model (Transformer) structure. In the first fusion module, a multi-layer deep model structure may be used to fuse information. Different slices may share a Transformer network. Different slices are equipped with an independent tag token (CLSToken) for carrying the fused information. In this example, the final implementation uses two Transformer layers.
[0137] Figure 4 It also includes a second fusion module for patient-level fusion. The second fusion module includes two second submodules. A second submodule for patient-level feature fusion is added after the first submodule to fuse cross-region and cross-IHC staining information. The two first submodules and the two second submodules are alternately connected, and the last second submodule outputs the fusion result. This fusion module is implemented by using a self-attention mechanism for CLSTokens of different slices in different images of a patient. This allows information-related areas to interact and fuse information organically in this module. The patient-level feature fusion module uses an additional CLSToken to carry the fused patient-level information. Finally, this patient-level CLSToken is used for prognosis calculation.
[0138] When processing cells in pathological images, in order to retrieve slice-by-slice cell counts, that is, to determine the number of cells, the watershed algorithm of the pathological image analysis software (QuPath) was first used to detect and segment cells using the calculated optical density of the hematoxylin channel from the pathological image. Next, statistics of different color channels were extracted from the cell nucleus pixels and classified by thresholding their average color values. These categories include cancer cells (stained red), immune cells (stained brown), uncertain cells (unclear cancer cells or immune cells), and other cells (stromal cells and other cells without immunohistochemical staining).
[0139] Figure 5 It is a data comparison diagram of the present application and the current solution provided according to the embodiment of the present application, such as Figure 5As shown, in multiple staining (CD4 and CD8), the current method includes a deep learning model, a patch-based graph convolutional network (Patch GCN), a correlated multi-instance learning (Trans MIL), and a hierarchical visual transformer (HVTSurv), and its parameters (M) are 0.07, 1.19, 2.67, and 3.25, and the number of floating-point operations (G) are 1.1, 18.3, 53.2, and 65.5, respectively. The performance is 0.5310±0.0054, 0.5461±0.0150, 0.5300±0.0247, and 0.5645±0.0082, respectively. Among them, the parameter of this application is 0.68, the number of floating-point operations of this application is 23.3, and the performance of this application is 0.6373±0.0142. In single stain (CD8), the current method includes deep learning model, patch-based graph convolutional network, correlated multi-instance learning, hierarchical visual converter, and its parameters (M) are 0.07, 1.19, 2.67, 3.25, the number of floating point operations (G) are 0.6, 9.4, 48.4, 33.9, and the performance is 0.5310±0.0054, 0.5461±0.0150, 0.5300±0.0247, 0.5645±0.0082, respectively. Among them, the parameter of this application is 0.68, the number of floating point operations of this application is 23.3, and the performance of this application is 0.6005±0.0117.
[0140] Figure 5 The main performance comparisons of this application are summarized, including the results of single staining (CD8) and multiple staining (CD4 and CD8) settings. There are two reasons why CD8 was chosen for single staining experiments during the experiment: the first reason is that CD8 is usually a stronger baseline compared to CD4, and the second reason is that CD8 has more published evidence to be an effective prognostic predictor, while the prognostic value of CD4 is weak and usually comes from its interaction with other immune-targeted staining. In the single staining scenario, the method of this application has more advantages, Figure 5 We demonstrate the effectiveness of our clustering and patch enrichment strategy, even without exploiting cross-staining context. Our model outperforms by nearly 0.08 CI, highlighting the critical role of incorporating IHC-based priors. Compared to current approaches, our scheme has the additional advantage of modeling global relationships through spatial awareness. Furthermore, our approach is notable for its large data requirements compared to other Transformer-based approaches, highlighting the value of incorporating meaningful prior knowledge to improve data efficiency.
[0141] Figure 6 is a schematic diagram of a survival prediction analysis according to an embodiment of the present application, such as Figure 6As shown, (a) represents the prediction effect of the patch-based graph convolutional network (Patch GCN), (b) represents the prediction effect of the correlated multi-instance learning (Trans MIL), (c) represents the prediction effect of the hierarchical visual transformer (HVT Surv), and (d) represents the prediction effect of the method of the present application. After stratifying the patients into low risk and high risk according to their median predicted risk scores, the non-parametric statistical method curve (kaplan-meier) of survival analysis is visualized. The horizontal axis in the figure can be expressed in months, including 0 months, 20 months, 40 months, 60 months, and 80 months. The vertical axis can represent the value of the risk, including 0, 0.2, 0.4, 0.6, 0.8, and 1.0. Kaplan-Meier is a non-parametric statistical method for survival analysis, which is used to estimate the probability of reaching an event (such as death or disease recurrence) in time. This method can be used to analyze survival time data, draw survival curves, and compare the differences in survival time between different groups. The method of the present application can better distinguish risk groups, which is a statistically significant prediction compared to the current method.
[0142] There are still many challenges in the deep learning pathology image prognosis prediction technology. In particular, when faced with a variety of IHC staining images, existing algorithms cannot effectively mine the information in the data. There are three main problems: a. Since the number of small blocks is usually thousands to tens of thousands, the current information fusion algorithm cannot well consider the relationship between the position and content of the small blocks, resulting in limited performance. b. Due to various factors including data collection difficulties, existing algorithms mostly only consider the analysis of a single H&E staining image. When faced with a variety of IHC staining images, existing algorithms cannot effectively mine the complex connections between multiple types of cells marked by different stains.
[0143] In this application, in order to analyze multi-stained IHC images, the existing algorithm for analyzing a single H&E image can be first trained on the IHC dataset, and then some ensemble learning strategies, such as prediction averaging or voting, can be used to simply fuse the prediction results of different IHC stainings.
[0144] This application can introduce doctor priors and IHC image staining labeling information, use a specially designed feature- and position-based spectral clustering algorithm, and intersperse regional and patient-level information fusion modules based on self-attention mechanisms to effectively extract and fuse information within the IHC image and across multiple IHC staining images, ultimately obtaining better survival analysis performance. This application is a general survival analysis framework for multi-stained IHC analysis. The carefully designed regional and patient-level feature fusion modules can effectively capture prognostic features from within and across image regions. This application effectively utilizes IHC staining labeling information for cell classification, and uses doctor prior knowledge to perform regional preliminary screening of oversized pathological images. While effectively reducing the computational burden, no additional labeling work is required. This application introduces a specially designed spectral clustering algorithm that simultaneously considers small-block image features and spatial location information to obtain more meaningful clustering results.
[0145] According to an embodiment of the present application, an image processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is used in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0146] Figure 7 is a flowchart of an image processing method according to an embodiment of the present application, such as Figure 7 As shown, the method includes:
[0147] Step S702: In response to an input instruction on the operation interface, a set of images of the target object is displayed on the operation interface.
[0148] The different images to be processed in the image set are used to represent images obtained by scanning different slices of the target object, and the target object contains target elements that affect the target indicators corresponding to the target object.
[0149] The above-mentioned operation interface can be used to display the image set of the target object, wherein the operation interface can include a variety of different controls for the user to operate, so as to display the target image set of the target object on the operation interface.
[0150] Step S704, in response to the image processing instruction acting on the operation interface, the fusion result of the image set is displayed on the operation interface.
[0151] Among them, the fusion result is used to predict the target index of the target object in a preset future time period. The fusion result is obtained by inputting the target clustering area into the image processing model and fusing the image set using the image processing model. The target clustering area is the area containing the target elements obtained by clustering multiple block images.
[0152] The above-mentioned image processing instructions may be instructions generated for fusing a set of images. Multiple block images may be clustered according to the image processing instruction lines to obtain a target clustering region containing target elements. The target clustering region may be input into an image processing model, and the image set may be fused through the image processing model to obtain a fusion result.
[0153] Through the above steps, in response to the input instruction on the operation interface, an image set of the target object is displayed on the operation interface, wherein different images to be processed in the image set are used to represent images obtained by scanning different slices of the target object, and there are target elements in the target object that affect the target indicators corresponding to the target object; in response to the image processing instruction on the operation interface, a fusion result of the image set is displayed on the operation interface, wherein the fusion result is used to predict the target indicators of the target object in a preset future time period, and the fusion result is obtained by inputting the target clustering area into the image processing model and fusing the image set using the image processing model, and the target clustering area is an area containing the target elements obtained by clustering multiple block images, thereby achieving the purpose of improving the accuracy of image processing. It is easy to notice that the image to be processed in the image set can be divided to obtain multiple block images of the image to be processed. By clustering the multiple block images, the target clustering area containing the target element can be quickly determined. Since the target element affects the target indicator corresponding to the target object, the image processing model can effectively fuse the features of the image set according to the target clustering area, so that the obtained fusion result can more accurately predict the target indicator of the target object in a preset future time period, thereby improving the accuracy of image processing and solving the technical problem of low accuracy of image processing in related technologies.
[0154] According to an embodiment of the present application, an image processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is used in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0155] Figure 8 is a flowchart of an image processing method according to an embodiment of the present application, such as Figure 8 As shown, the method includes:
[0156] Step S802, acquiring an image set of the target object by calling the first interface;
[0157] Among them, the first interface includes a first parameter, the parameter value of the first parameter is an image set, different images to be processed in the image set are used to represent images obtained by scanning different slices of the target object, and there are target elements in the target object that affect the corresponding target indicators of the target object.
[0158] The above-mentioned first interface can be an interface for data interaction between the cloud server and the client. The client can pass the image set of the target object into the interface function as the first parameter of the interface function to achieve the purpose of uploading the image set of the target object to the cloud server.
[0159] Step S804, dividing the image to be processed in the image set to obtain a plurality of block images of the image to be processed;
[0160] Step S806, clustering the multiple block images to obtain a target clustering region containing the target element;
[0161] The target element is an element that affects the preset indicator corresponding to the target object.
[0162] Step S808, inputting the target clustering region into the image processing model, and using the image processing model to fuse the image set to obtain a fusion result;
[0163] The fusion result is used to predict the target index of the target object in a preset future time period.
[0164] Step S810: output the fusion result by calling the second interface.
[0165] The second interface includes a second parameter, and a parameter value of the second parameter is a fusion result.
[0166] The above-mentioned second interface can be an interface for data interaction between the cloud server and the client. The cloud server can pass the fusion result to the interface function as the second parameter of the interface function to achieve the purpose of sending the fusion result to the client.
[0167] Through the above steps, the image set of the target object is obtained by calling the first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter is an image set, different images to be processed in the image set are used to characterize images obtained by scanning different slices of the target object, and there are target elements in the target object that affect the target index corresponding to the target object; the images to be processed in the image set are divided to obtain multiple block images of the images to be processed; the multiple block images are clustered to obtain a target clustering area containing the target element, wherein the target element is an element that affects the preset index corresponding to the target object; the target clustering area is input into the image processing model, and the image processing model is used to fuse the image set to obtain a fusion result, wherein the fusion result is used to predict the target index of the target object in a preset future time period; the fusion result is output by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the fusion result, thereby achieving the purpose of improving the accuracy of image processing. It is easy to notice that the image to be processed in the image set can be divided to obtain multiple block images of the image to be processed. By clustering the multiple block images, the target clustering area containing the target element can be quickly determined. Since the target element affects the target indicator corresponding to the target object, the image processing model can effectively fuse the features of the image set according to the target clustering area, so that the obtained fusion result can more accurately predict the target indicator of the target object in a preset future time period, thereby improving the accuracy of image processing and solving the technical problem of low accuracy of image processing in related technologies.
[0168] According to an embodiment of the present application, a computer-assisted cancer prognosis prediction method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is used in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0169] Fig. 9 is a flowchart of a computer-assisted cancer prognosis prediction method according to an embodiment of the present application, such as Fig. 9 As shown, the method includes:
[0170] Step S902, obtaining a set of digital pathological panoramic slice images of a biological object;
[0171] Among them, different digital pathology panoramic slice images in the digital pathology panoramic slice image set are used to represent images obtained by scanning different slices of a biological object, and there are cancer elements in the biological object that affect the corresponding survival rate of the biological object.
[0172] The above-mentioned different digital pathology panoramic slice images may include but are not limited to auxiliary immune cell digital pathology panoramic slice images and cytotoxic immune cell digital pathology panoramic slice images. The digital pathology panoramic slice images may also include T cell surface antigen (CD3) digital pathology panoramic slice images, monocyte and macrophage antigen (CD68) digital case panoramic slice images, B cell surface antigen (CD20) digital case panoramic slice images, programmed death protein 1 (PD1) digital case panoramic slice images, programmed death ligand 1 (PDL1) digital case panoramic slice images, human epidermal growth factor receptor 2 (HER2) digital case panoramic slice images, etc. The type of digital pathology panoramic slice images is not limited here. The type of digital case panoramic slice image can be determined according to the type of cancer prognosis.
[0173] The biological object mentioned above may be a patient. The digital pathology panoramic slice image of auxiliary immune cells may be a CD4 pathology image, and the digital pathology panoramic slice image of cytotoxic immune cells may be a CD8 pathology image.
[0174] Step S904, dividing the digital pathology panoramic slice image in the digital pathology panoramic slice image set to obtain a plurality of block images of the digital pathology panoramic slice image;
[0175] Step S906, clustering the multiple block images to obtain a target clustering region containing cancer elements;
[0176] Step S908, inputting the target clustering region into the image processing model, and using the image processing model to fuse the digital pathology panoramic slice image set to obtain a fusion result;
[0177] Step S910: predicting the survival rate of biological objects in a preset future time period based on the fusion result.
[0178] Cancer prognosis prediction refers to the possibility of a patient's disease progression and survival predicted by a doctor based on the patient's condition, pathological characteristics, treatment response and other factors after the patient has cancer. Cancer prognosis prediction is usually divided into three types: good prognosis prediction, moderate prognosis prediction and poor prognosis prediction, which helps doctors formulate reasonable treatment plans and patients and their families make corresponding decisions. Cancer prognosis prediction depends not only on the characteristics of the disease itself, but also on the patient's age, gender, physical condition, treatment method and other factors.
[0179] Through the above steps, a set of digital pathology panoramic slice images of a biological object is obtained, wherein different digital pathology panoramic slice images in the set of digital pathology panoramic slice images are used to represent images obtained by scanning different slices of the biological object, and there are cancer elements in the biological object that affect the corresponding survival rate of the biological object; the digital pathology panoramic slice images in the set of digital pathology panoramic slice images are divided to obtain a plurality of block images of the digital pathology panoramic slice images; the plurality of block images are clustered to obtain a target clustering area containing cancer elements; the target clustering area is input into an image processing model, and the set of digital pathology panoramic slice images is fused using the image processing model to obtain a fusion result; the survival rate of the biological object in a preset future time period is predicted based on the fusion result, thereby achieving the purpose of improving the accuracy of image processing. It is easy to notice that the image to be processed in the image set can be divided to obtain multiple block images of the image to be processed. By clustering the multiple block images, the target clustering area containing the target element can be quickly determined. Since the target element affects the target indicator corresponding to the target object, the image processing model can effectively fuse the features of the image set according to the target clustering area, so that the obtained fusion result can more accurately predict the target indicator of the target object in a preset future time period, thereby improving the accuracy of image processing and solving the technical problem of low accuracy of image processing in related technologies.
[0180] In the above embodiment of the present application, the digital pathology panoramic slice image set includes at least one of the following: auxiliary immune cell digital pathology panoramic slice images, and cytotoxic immune cell digital pathology panoramic slice images.
[0181] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0182] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.
[0183] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.
[0184] According to the above embodiment of the present application, an image processing device for implementing the above image processing method is also provided. Fig.10 is a schematic diagram of an image processing device according to an embodiment of the present application, such as Fig.10 As shown, the device 1000 includes: an acquisition module 1002 , a division module 1004 , a clustering module 1006 , and a fusion module 1008 .
[0185] Among them, the acquisition module is used to acquire an image set of the target object, wherein different images to be processed in the image set are used to represent images obtained by scanning different slices of the target object, and there are target elements in the target object that affect the target indicators corresponding to the target object; the division module is used to divide the images to be processed in the image set to obtain multiple block images of the images to be processed; the clustering module is used to cluster the multiple block images to obtain a target clustering area containing the target elements; the fusion module is used to input the target clustering area into the image processing model, and use the image processing model to fuse the image set to obtain a fusion result, wherein the fusion result is used to predict the target indicator of the target object in a preset future time period.
[0186] It should be noted that the acquisition module 1002, the division module 1004, the clustering module 1006, and the fusion module 1008 correspond to steps S202 to S208 in the above embodiment, and the four modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be run in the server 10 provided in the above embodiment as part of the device.
[0187] In the above embodiment of the present application, the image processing model includes multiple fusion modules, and the fusion module is also used to expand the block features in the target clustering area based on preset features to obtain expanded features, wherein the preset features are features of the same type as the block features in the target clustering area; and multiple fusion modules are used to fuse the expanded features in different dimensions to obtain a fusion result.
[0188] In the above embodiment of the present application, the fusion module is also used to use the first fusion module to fuse the expanded features to obtain the first fusion features of the image set; and use the second fusion module to fuse the first fusion features of the image set to obtain the fusion result.
[0189] In the above embodiment of the present application, the fusion module is also used to use the first fusion module to fuse the extended features to obtain the first fused features of the image set, including: using the first submodule to fuse the first target features of the target cluster area in the image to be processed to obtain the first fused features of the image to be processed, wherein the first target feature is the second fused feature output by the previous second submodule connected to the first submodule, or the extended feature; using the second fusion module to fuse the first fused features of the image set to obtain a fusion result, including: using the second submodule to fuse the first fused features of different images to be processed in the image set to obtain the second fused features of the image set; when the second submodule is the last submodule, determining the second fused feature as the fusion result.
[0190] In the above embodiment of the present application, the fusion module is also used to use the self-attention layer to perform feature recognition on the first fusion feature of the image to be processed, and obtain the target weight of the first fusion feature, wherein the target weight is used to represent the association information of the first fusion feature in multiple images to be processed; and use the feedforward layer to fuse the first fusion features of different images to be processed according to the target weight to obtain the second fusion feature.
[0191] In the above embodiment of the present application, the fusion module is also used to expand the block features in the target clustering area based on preset features to obtain the number of expanded features and the expanded feature positions, wherein the number of expanded features is used to indicate the number of preset features expanded in the target clustering area, and the expanded feature positions are used to indicate the positions of the preset features expanded in the target clustering area; the block features, the number of expanded features and the expanded feature positions in the target clustering area are spliced to obtain the expanded features.
[0192] In the above embodiments of the present application, the clustering module is also used to extract features from multiple block images to obtain multiple block features; cluster the multiple block features to obtain multiple clustering regions, wherein the block features in different clustering regions belong to different categories; and determine a target clustering region containing a target element among the multiple clustering regions.
[0193] In the above embodiment of the present application, the clustering module is also used to construct a correlation matrix of multiple block features; determine the Laplace matrix of multiple block features based on the correlation matrix; and perform feature decomposition on the multiple block features based on the Laplace matrix to obtain multiple clustering regions.
[0194] In the above embodiment of the present application, the clustering module is also used to determine at least one initial clustering region among multiple clustering regions that contains the target element; based on the number of block features in at least one initial clustering region, at least one initial clustering region is screened to obtain a target clustering region, wherein the number of block features in the target clustering region is greater than the number of block features in other clustering regions, and the other clustering regions are other clustering regions in at least one initial clustering region except the target clustering region.
[0195] According to an embodiment of the present application, an image processing device for implementing the above-mentioned image processing method is also provided. Fig.11 is a schematic diagram of an image processing device according to an embodiment of the present application, such as Fig.11 As shown, the device 1100 includes: a first display module 1102 and a second display module 1104 .
[0196] Among them, the first display module is used to respond to the input instructions acting on the operation interface, and display the image set of the target object on the operation interface, wherein the different images to be processed in the image set are used to represent the images obtained by scanning different slices of the target object, and there are target elements in the target object that affect the corresponding target indicators of the target object; the second display module is used to respond to the image processing instructions acting on the operation interface, and display the fusion result of the image set on the operation interface, wherein the fusion result is used to predict the target indicator of the target object in a preset future time period, and the fusion result is obtained by inputting the target clustering area into the image processing model and fusing the image set using the image processing model, and the target clustering area is an area containing the target element obtained by clustering multiple block images.
[0197] It should be noted that the first display module 1102 and the second display module 1104 of the acquisition module correspond to steps S702 to S704 in the above embodiment, and the examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be run in the server 10 provided in the above embodiment as part of the device.
[0198] According to an embodiment of the present application, an image processing device for implementing the above-mentioned image processing method is also provided. Fig.12 is a schematic diagram of an image processing device according to an embodiment of the present application, such as Fig.12 As shown, the device 1200 includes: an acquisition module 1202 , a division module 1204 , a clustering module 1206 , a fusion module 1208 , and an output module 1210 .
[0199] Among them, the acquisition module is used to obtain an image set of the target object by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter is an image set, different images to be processed in the image set are used to characterize images obtained by scanning different slices of the target object, and there are target elements in the target object that affect the target index corresponding to the target object; the division module is used to divide the images to be processed in the image set to obtain multiple block images of the images to be processed; the clustering module is used to cluster the multiple block images to obtain a target clustering area containing the target element, wherein the target element is an element that affects the preset index corresponding to the target object; the fusion module is used to input the target clustering area into the image processing model, and use the image processing model to fuse the image set to obtain a fusion result, wherein the fusion result is used to predict the target index of the target object in a preset future time period; the output module is used to output the fusion result by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the fusion result.
[0200] It should be noted that the acquisition module 1202, the division module 1204, the clustering module 1206, the fusion module 1208, and the output module 1210 correspond to steps S802 to S810 in the above embodiment, and the five modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be run in the server 10 provided in the above embodiment as part of the device.
[0201] According to an embodiment of the present application, a computer-assisted cancer prognosis prediction device for implementing the above-mentioned computer-assisted cancer prognosis prediction method is also provided. Fig.13 is a schematic diagram of a computer-assisted cancer prognosis prediction device according to an embodiment of the present application, such as Fig.13 As shown, the device 1300 includes: an acquisition module 1302 , a division module 1304 , a clustering module 1306 , a fusion module 1308 , and a prediction module 1310 .
[0202] Among them, the acquisition module is used to acquire a set of digital pathology panoramic slice images of biological objects, wherein different digital pathology panoramic slice images in the digital pathology panoramic slice image set are used to represent images obtained by scanning different slices of the biological object, and there are cancer elements in the biological object that affect the corresponding survival rate of the biological object; the division module is used to divide the digital pathology panoramic slice images in the digital pathology panoramic slice image set to obtain multiple block images of the digital pathology panoramic slice images; the clustering module is used to cluster the multiple block images to obtain a target clustering area containing cancer elements; the fusion module is used to input the target clustering area into the image processing model, and use the image processing model to fuse the digital pathology panoramic slice image set to obtain a fusion result; the prediction module is used to predict the survival rate of the biological object in a preset future time period based on the fusion result.
[0203] It should be noted that the acquisition module 1302, the division module 1304, the clustering module 1306, the fusion module 1308, and the prediction module 1310 correspond to steps S902 to S910 in the above embodiment, and the five modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be run in the server 10 provided in the above embodiment as part of the device.
[0204] According to an embodiment of the present application, a computer-assisted cancer prognosis prediction system is also provided. Fig.14 is a structural diagram of a computer-assisted cancer prognosis prediction system according to an embodiment of the present application, such as Fig.14 As shown, the system 1400 includes: an imaging device 1402 and a processing device 1404 .
[0205] Among them, the imaging device 1402 is used to collect a set of digital pathology panoramic slice images of a biological object, wherein different digital pathology panoramic slice images in the digital pathology panoramic slice image set are used to represent images obtained by scanning different slices of the biological object, and there are cancer elements in the biological object that affect the corresponding survival rate of the biological object; the processing device 1404 is connected to the imaging device, and is used to divide the digital pathology panoramic slice images in the digital pathology panoramic slice image set to obtain multiple block images of the digital pathology panoramic slice images; cluster the multiple block images to obtain a target clustering area containing cancer elements; input the target clustering area into the image processing model, and use the image processing model to fuse the digital pathology panoramic slice image set to obtain a fusion result; and predict the survival rate of the biological object in a preset future time period based on the fusion result.
[0206] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in the above embodiments, as well as the application scenario and implementation process, but is not limited to the scheme provided in the above embodiments.
[0207] The embodiment of the present application may provide an electronic device, which may be any electronic device in a group of electronic devices. Optionally, in this embodiment, the electronic device may also be replaced by a terminal device such as a mobile terminal.
[0208] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.
[0209] In this embodiment, the computer terminal can execute the program code in the method.
[0210] Optionally, Fig.15 is a structural block diagram of an electronic device according to an embodiment of the present application. Fig.15 As shown, the electronic device A may include: one or more ( Fig.15 (only one is shown) processor 102, memory 104, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.
[0211] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implementing the methods in the above embodiments. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0212] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain an image set of the target object, wherein different images to be processed in the image set are used to represent images obtained by scanning different slices of the target object, and there are target elements in the target object that affect the target indicators corresponding to the target object; divide the images to be processed in the image set to obtain multiple block images of the images to be processed; cluster the multiple block images to obtain a target clustering area containing the target elements; input the target clustering area into the image processing model, and use the image processing model to fuse the image set to obtain a fusion result, wherein the fusion result is used to predict the target indicator of the target object in a preset future time period.
[0213] In an embodiment of the present application, an image set of a target object is obtained, wherein different images to be processed in the image set are used to represent images obtained by scanning different slices of the target object, and there is a target element in the target object that affects the target index corresponding to the target object; the images to be processed in the image set are divided to obtain multiple block images of the images to be processed; the multiple block images are clustered to obtain a target clustering area containing the target element; the target clustering area is input into an image processing model, and the image processing model is used to fuse the image set to obtain a fusion result, wherein the fusion result is used to predict the target index of the target object in a preset future time period, thereby achieving the purpose of improving the accuracy of image processing. It is easy to notice that the images to be processed in the image set can be divided to obtain multiple block images of the images to be processed, and by clustering the multiple block images, the target clustering area containing the target element can be quickly determined. Since the target element affects the target index corresponding to the target object, the image processing model can perform effective feature fusion on the image set according to the target clustering area, so that the obtained fusion result can more accurately predict the target index of the target object in the preset future time period, thereby improving the accuracy of image processing, thereby solving the technical problem of low accuracy of image processing in the related art.
[0214] Those skilled in the art will understand that Fig.15 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, and other terminal devices. Fig.15 It does not limit the structure of the above electronic device. For example, the electronic device A may also include Fig.15 More or fewer components (such as network interfaces, display devices, etc.) shown in the figure, or having Fig.15 Different configurations are shown.
[0215] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0216] The embodiment of the present application further provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the method provided in the above embodiment.
[0217] Optionally, in this embodiment, the above storage medium may be located in any electronic device in a group of electronic devices in a computer network, or in any mobile terminal in a group of mobile terminals.
[0218] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain an image set of the target object, wherein different images to be processed in the image set are used to represent images obtained by scanning different slices of the target object, and there are target elements in the target object that affect the target indicators corresponding to the target object; divide the images to be processed in the image set to obtain multiple block images of the images to be processed; cluster the multiple block images to obtain a target clustering area containing the target elements; input the target clustering area into the image processing model, and use the image processing model to fuse the image set to obtain a fusion result, wherein the fusion result is used to predict the target indicator of the target object in a preset future time period.
[0219] In an embodiment of the present application, an image set of a target object is obtained, wherein different images to be processed in the image set are used to represent images obtained by scanning different slices of the target object, and there is a target element in the target object that affects the target index corresponding to the target object; the images to be processed in the image set are divided to obtain multiple block images of the images to be processed; the multiple block images are clustered to obtain a target clustering area containing the target element; the target clustering area is input into an image processing model, and the image processing model is used to fuse the image set to obtain a fusion result, wherein the fusion result is used to predict the target index of the target object in a preset future time period, thereby achieving the purpose of improving the accuracy of image processing. It is easy to notice that the images to be processed in the image set can be divided to obtain multiple block images of the images to be processed, and by clustering the multiple block images, the target clustering area containing the target element can be quickly determined. Since the target element affects the target index corresponding to the target object, the image processing model can perform effective feature fusion on the image set according to the target clustering area, so that the obtained fusion result can more accurately predict the target index of the target object in the preset future time period, thereby improving the accuracy of image processing, thereby solving the technical problem of low accuracy of image processing in the related art.
[0220] The embodiment of the present application further provides a computer program product. Optionally, in this embodiment, the computer program product may include a computer program, and the computer program implements the method provided in the embodiment when executed by a processor.
[0221] The embodiments of the present application also provide a computer program product. Optionally, the computer program product may include a non-volatile computer-readable storage medium, which may be used to store a computer program, and when the computer program is executed by a processor, the method provided in the embodiments is implemented.
[0222] The embodiment of the present application further provides a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, the method provided in the above embodiment is implemented.
[0223] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.
[0224] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0225] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0226] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0227] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.
[0228] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. An image processing method, characterized in that: include: Acquire an image set of a target object, wherein different images to be processed in the image set are used to represent images obtained by scanning different slices of the target object, and the target object contains a target element that affects a target indicator corresponding to the target object; Dividing the image to be processed in the image set to obtain a plurality of block images of the image to be processed; Clustering the multiple block images to obtain a target clustering region containing the target element; The target clustering region is input into an image processing model, and the image set is fused using the image processing model to obtain a fusion result, wherein the fusion result is used to predict the target indicator of the target object in a preset future time period.
2. The image processing method according to claim 1, characterized in that: The image processing model includes a plurality of fusion modules, which fuse the image set based on the target cluster area to obtain a fusion result, including: Expanding the block features in the target clustering region based on preset features to obtain expanded features, wherein the preset features are features of the same type as the block features in the target clustering region; The expanded features are fused in different dimensions using the multiple fusion modules to obtain the fusion result.
3. The image processing method according to claim 2, characterized in that: The image processing model includes: a first fusion module and a second fusion module, and the plurality of fusion modules are used to fuse the extended features in different dimensions to obtain the fusion result, including: Using the first fusion module to fuse the extended features to obtain a first fusion feature of the image set; The second fusion module is used to fuse the first fusion features of the image set to obtain the fusion result.
4. The image processing method according to claim 3, characterized in that: The first fusion module includes a plurality of first submodules, the second fusion module includes a plurality of second submodules, and the plurality of first submodules and the plurality of second submodules are alternately connected; Using the first fusion module to fuse the extended features to obtain a first fused feature of the image set, including: using the first submodule to fuse the first target feature of the target cluster region in the image to be processed to obtain the first fused feature of the image to be processed, wherein the first target feature is a second fused feature output by a previous second submodule connected to the first submodule, or the extended feature; Using the second fusion module to fuse the first fusion features of the image set to obtain the fusion result, including: using the second submodule to fuse the first fusion features of different to-be-processed images in the image set to obtain the second fusion features of the image set; In the case where the second submodule is the last submodule, the second fusion feature is determined as the fusion result.
5. The image processing method according to claim 4, characterized in that: The second submodule includes a self-attention layer and a feedforward layer, and the second submodule is used to fuse the first fused features of different to-be-processed images in the image set to obtain the second fused features of the image set, including: Using the self-attention layer to perform feature recognition on the first fused feature of the image to be processed, and obtain a target weight of the first fused feature, wherein the target weight is used to represent association information of the first fused feature in a plurality of the images to be processed; The feed-forward layer is used to fuse the first fusion features of different images to be processed according to the target weight to obtain the second fusion feature.
6. The image processing method according to claim 2, characterized in that: Expanding the block features in the target clustering area based on the preset features to obtain expanded features includes: Expanding the block features in the target clustering area based on the preset features to obtain the number of expanded features and the expanded feature positions, wherein the number of expanded features is used to indicate the number of the preset features expanded in the target clustering area, and the expanded feature positions are used to indicate the positions of the preset features expanded in the target clustering area; The block features in the target cluster area, the number of the extended features and the positions of the extended features are concatenated to obtain the extended features.
7. The image processing method according to any one of claims 1 to 6, characterized in that: Clustering the multiple block images to obtain a target clustering region containing the target element includes: Extracting features from the plurality of block images to obtain a plurality of block features; Clustering the multiple block features to obtain multiple cluster regions, wherein the block features in different cluster regions belong to different categories; The target cluster region including the target element is determined among the plurality of cluster regions.
8. The image processing method according to claim 7, characterized in that: Clustering the multiple block features to obtain multiple clustering regions includes: constructing a correlation matrix of the plurality of block features; determining a Laplacian matrix of the plurality of block features based on the correlation matrix; The plurality of block features are subjected to feature decomposition based on the Laplacian matrix to obtain the plurality of cluster regions.
9. The image processing method according to claim 8, characterized in that: Determining the target cluster region including the target element among the multiple cluster regions includes: Determine at least one initial clustering region including the target element among the multiple clustering regions; The at least one initial clustering region is screened based on the number of block features in the at least one initial clustering region to obtain a target clustering region, wherein the number of block features in the target clustering region is greater than the number of block features in other clustering regions, and the other clustering regions are other clustering regions in the at least one initial clustering region except the target clustering region.
10. An image processing method, characterized in that: include: In response to an input instruction acting on an operation interface, an image set of a target object is displayed on the operation interface, wherein different images to be processed in the image set are used to represent images obtained by scanning different slices of the target object, and the target object contains a target element that affects a target indicator corresponding to the target object; In response to an image processing instruction acting on the operation interface, a fusion result of the image set is displayed on the operation interface, wherein the fusion result is used to predict the target indicator of the target object within a preset future time period, and the fusion result is obtained by inputting a target clustering area into an image processing model and fusing the image set using the image processing model, and the target clustering area is an area containing the target element obtained by clustering multiple block images.
11. An image processing method, characterized in that: include: Acquire an image set of the target object by calling a first interface, wherein the first interface includes a first parameter, a parameter value of the first parameter is the image set, different images to be processed in the image set are used to represent images obtained by scanning different slices of the target object, and the target object has a target element that affects a target indicator corresponding to the target object; Dividing the image to be processed in the image set to obtain a plurality of block images of the image to be processed; Clustering the multiple block images to obtain a target clustering region containing the target element, wherein the target element is an element that affects a preset indicator corresponding to the target object; Inputting the target clustering region into an image processing model, and using the image processing model to fuse the image set to obtain a fusion result, wherein the fusion result is used to predict the target index of the target object in a preset future time period; The fusion result is output by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter is the fusion result.
12. A computer-assisted cancer prognosis prediction method, characterized in that: include: Acquiring a set of digital pathology panoramic slice images of a biological object, wherein different digital pathology panoramic slice images in the set of digital pathology panoramic slice images are used to represent images obtained by scanning different slices of the biological object, and cancer elements that affect the corresponding survival rate of the biological object exist in the biological object; Dividing the digital pathology panoramic slice image in the digital pathology panoramic slice image set to obtain a plurality of block images of the digital pathology panoramic slice image; Clustering the multiple block images to obtain a target clustering region containing the cancer element; Inputting the target clustering region into an image processing model, and using the image processing model to fuse the digital pathology panoramic slice image set to obtain a fusion result; The survival rate of the biological object in a preset future time period is predicted based on the fusion result.
13. The computer-assisted cancer prognosis prediction method according to claim 12, characterized in that: The digital pathology panoramic slice image set includes at least one of the following: auxiliary immune cell digital pathology panoramic slice images, and cytotoxic immune cell digital pathology panoramic slice images.
14. A computer-assisted cancer prognosis prediction system, characterized in that: include: An imaging device for collecting a set of digital pathology panoramic slice images of a biological object, wherein different digital pathology panoramic slice images in the set of digital pathology panoramic slice images are used to represent images obtained by scanning different slices of the biological object, and cancer elements that affect the corresponding survival rate of the biological object exist in the biological object; A processing device is connected to the imaging device and is used to divide the digital pathology panoramic slice image in the digital pathology panoramic slice image set to obtain a plurality of block images of the digital pathology panoramic slice image; cluster the plurality of block images to obtain a target clustering area containing the cancer element; input the target clustering area into an image processing model, and fuse the digital pathology panoramic slice image set using the image processing model to obtain a fusion result; and predict the survival rate of the biological object in a preset future time period based on the fusion result.
15. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 12.
16. A computer terminal, characterized in that: include: processor; A memory, connected to the processor, for providing the processor with instructions for executing the method according to any one of claims 1 to 12.
17. A computer program product, characterized in that It comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 12.
