Method, device and storage medium for predicting the degree of abnormal cell metastasis

Abstract

A method for predicting the degree of abnormal cell metastasis based on medical data using artificial intelligence is provided. The method includes: obtaining medical data of multiple modalities related to abnormal cells; extracting the single modality features of each modality of medical data from the multiple modalities of medical data; fusing the extracted single modality features to obtain cross-modality fusion features of multiple modalities of medical data; adjusting each single modality feature using the cross-modality fusion feature as a global feature; obtaining multi-modality features of multiple modalities of medical data based on the adjusted single modality features; and predicting the degree of abnormal cell metastasis based on the multi-modality features. Through the method and device for predicting the degree of abnormal cell metastasis based on medical data, the fusion of the patient's pathological information of different modalities is realized, the patient's multi-modality information resources are fully utilized, and the prediction accuracy of abnormal cell lymphatic metastasis is improved.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing, and in particular to a method, an apparatus, a computing device and a computer-readable storage medium for predicting the degree of abnormal cell metastasis. Background Art

[0002] At present, artificial intelligence technology has been widely used in various medical engineering. Artificial intelligence networks have been used to assist doctors in making decisions based on the characteristics of different diseases.

[0003] In clinical diagnosis, doctors usually make decisions based on the pathological images of the patient's lesion tissue and the patient's clinical diagnosis information. In some related technologies, neural network models (such as fully connected neural networks) are used to predict the state of the patient's lesion tissue based on the patient's clinical information. However, since the content of the clinical information is a limited description of the patient's condition, it cannot fully reflect the microenvironment of the patient's lesion tissue, which limits the performance of the neural network model. In other related technologies, pathological images are used to predict the state of the patient's lesion tissue. However, since the pathological image only reflects the pathological environment of the lesion tissue site, the patient's physiological state is not included in the decision-making process. For some schemes that consider both the patient's clinical information and pathological images, the pathological images involved (especially the pathological feature part) usually require manual interpretation. Since such interpretation has high requirements for pathologists and takes a long time, and there are differences between different laboratories, it will cause the interpretation indicators to fluctuate. This may lead to wrong decisions. Summary of the invention

[0004] When classifying abnormal cells, artificial intelligence in the prior art mostly only classifies based on one pathological information, without combining other pathological information (e.g., the patient's clinical information) for classification, resulting in low classification accuracy. In the prior art artificial intelligence network, the classification logic and network are too simple, resulting in low classification accuracy.

[0005] In view of this, the present invention provides a method, apparatus, computing device and computer-readable storage medium for predicting the degree of abnormal cell metastasis, which is expected to alleviate or overcome some or all of the above-mentioned deficiencies and other possible deficiencies.

[0006] According to the first aspect of the present invention, a method for predicting the degree of abnormal cell metastasis based on medical data is provided. The method comprises: obtaining medical data of multiple modalities related to abnormal cells; extracting the single modality features of each modality of medical data from the multiple modalities of medical data; fusing the extracted single modality features to obtain cross-modality fusion features of the medical data of multiple modalities; adjusting each single modality feature using the cross-modality fusion feature as a global feature; obtaining multi-modality features of the medical data of multiple modalities based on the adjusted single modality features; and predicting the degree of abnormal cell metastasis based on the multi-modality features.

[0007] In some embodiments, the multiple modalities include at least a first modality and a second modality, the medical data of the first modality corresponds to image data of abnormal cells, and the medical data of the second modality corresponds to clinical information data of patients to whom the abnormal cells belong, wherein the medical data of the first modality corresponds to image data of abnormal cells at multiple zoom scales, and the abnormal cell images at each of the multiple zoom scales include multiple images.

[0008] In some embodiments, extracting single-modal features of medical data of each modality from medical data of multiple modalities includes: extracting features of the first modality using a first feature extraction network for the first modality; and extracting features of the second modality using a second feature extraction network for the second modality.

[0009] In some embodiments, the abnormal cell images at each zoom scale in the multiple zoom scales include multiple images with different numbers.

[0010] In some embodiments, the first feature extraction network includes an image feature extractor pre-trained based on a first data set, and the second feature extraction network includes a table feature extractor pre-trained based on a second data set.

[0011] In some embodiments, cross-modal fusion features of medical data of multiple modalities are obtained by fusing the extracted single-modal features, including: weighted fusion of features of multiple images of abnormal cells at each scaling scale under the first modality to obtain weighted fusion features of abnormal cells at each scale; and splicing the weighted fusion features of each scale under the first modality with the features of the second modality to obtain multi-modal global features.

[0012] In some embodiments, weighted fusion is performed on features of multiple images of abnormal cells at each zoom scale under the first modality to obtain weighted fusion features of abnormal cells at each scale, including: average pooling is performed on features of multiple images of abnormal cells at each zoom scale under the first modality to obtain weighted fusion features of abnormal cells at each scale.

[0013] In some embodiments, using cross-modal fusion features as global features to adjust each single modal feature includes: performing local-global information fusion based on the cross-modal fusion features as global features and features of each image at each scale under the first modality to obtain local-global information fusion features of each image at each scale under the first modality; performing weighted averaging on the local-global information fusion features of each image at each scale under the first modality to obtain adjusted features of each scale under the first modality; and adjusting the features of the second modality based on the cross-modal fusion features as global features to obtain adjusted features under the second modality.

[0014] In some embodiments, adjusting the features of the second modality based on the cross-modal fusion features as global features to obtain adjusted features of the second modality includes: calculating the multimodal global features through a fully connected network, and adjusting the features of the second modality using the scaling coefficients obtained by the Sigmoid activation function to obtain adjusted features of the second modality.

[0015] In some embodiments, obtaining multimodal features of medical data of multiple modalities based on adjusted single-modal features includes: splicing adjusted features of each scale under the first modality and features of the second modality to obtain multimodal features of medical data of multiple modalities.

[0016] In some embodiments, classifying the degree of abnormal cell metastasis based on multimodal fusion features includes: inputting the multimodal fusion features into a classifier to predict the degree of abnormal cell metastasis; wherein the classifier is pre-trained based on the features of samples in a training set as input and the labels of the samples as supervisory signals.

[0017] In some embodiments, the classifier includes at least a primary classifier and a secondary classifier.

[0018] In some embodiments, acquiring medical data of multiple modalities related to abnormal cells further includes: determining a region of interest in a tissue slice; and acquiring images of the region of interest at different zoom scales through a sliding window based on the region of interest in the tissue slice.

[0019] According to the second aspect of the present invention, a device for predicting the degree of abnormal cell metastasis based on medical data is provided. The device includes: an acquisition module, a single-modal feature extraction module, a cross-modal fusion module, an adjustment module, a multi-modal feature module and a prediction module. The acquisition module is configured to acquire medical data of multiple modalities related to abnormal cells; the single-modal feature extraction module is configured to extract the single-modal features of each modality of medical data from the multiple modalities of medical data; the cross-modal fusion module is configured to obtain the cross-modal fusion features of the medical data of multiple modalities by fusing the extracted single-modal features; the adjustment module is configured to adjust each single-modal feature using the cross-modal fusion feature as a global feature; the multi-modal feature module is configured to obtain the multi-modal features of the medical data of multiple modalities based on the adjusted single-modal features; the prediction module is configured to predict the degree of abnormal cell metastasis based on the multi-modal features.

[0020] According to a third aspect of the present invention, a computer device is provided, characterized in that it comprises a memory and a processor, wherein a computer program is stored in the memory. When the computer program is executed by the processor, the processor executes the steps of the method described in the above aspects.

[0021] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the processor executes the steps of the method described in the above aspects.

[0022] Through the method and device for predicting the degree of abnormal cell metastasis based on medical data implemented by artificial intelligence, the fusion of different modalities of pathological information of patients is realized, the multimodal information resources of patients are fully utilized, and the prediction accuracy of abnormal cell metastasis is improved. Specifically, the technical solution of the present invention adopts an end-to-end learning method, and the clinical information used is easy to obtain and does not require additional collection. The present invention combines the characteristics of multimodal data analysis and combines it into the framework of multi-instance learning analysis, using the patient's clinical information and corresponding pathological images to predict lymph node metastasis in the early stage, assisting doctors in preoperative decision-making, thereby providing more accurate treatment solutions and improving the quality of life of patients. The multimodal multi-instance method uses the gold standard for clinical cancer diagnosis (biopsy pathological sections) and the patient's clinical diagnosis information, and applies deep learning to combine the information of the two to predict whether lymph node metastasis will occur before surgery, effectively assisting preoperative decision-making.

[0023] These and other advantages of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Embodiments of the present application will now be described in more detail and with reference to the accompanying drawings, in which:

[0025] Figure 1 An exemplary application scenario according to an embodiment of the present invention is shown;

[0026] Figure 2 A flow chart showing a method for predicting the extent of abnormal cell metastasis based on medical data;

[0027] Figure 3 A schematic block diagram of a method according to an embodiment of the present invention is shown;

[0028] Figure 4 Shows Figure 3 A schematic diagram of the specific structure of the multi-modal multi-instance module in FIG.

[0029] Figure 5 The image heat maps obtained by processing according to the method of the present invention and the image single-modality method are shown;

[0030] Figure 6 A schematic diagram showing a device for predicting the extent of abnormal cell metastasis; and

[0031] Figure 7 A schematic block diagram of a computing system capable of implementing a method for predicting the extent of abnormal cell metastasis according to some embodiments of the present invention is shown. DETAILED DESCRIPTION

[0032] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this application will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0033] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0034] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0035] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0036] It should be understood that although the terms first, second, third, etc. may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another component. Therefore, the first component discussed below can be referred to as the second component without departing from the teachings of the concepts of the present application. As used herein, the term "and / or" and similar terms include all combinations of any, multiple, and all of the associated listed items.

[0037] Those skilled in the art will appreciate that the drawings are merely schematic diagrams of example embodiments, and the modules or processes in the drawings are not necessarily necessary for implementing the present application, and therefore cannot be used to limit the scope of protection of the present application.

[0038] Before describing the embodiments of the present invention in detail, some related concepts are first explained:

[0039] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0040] Clinical Information (CI) refers to relevant information that characterizes the patient's physiological indicators, including but not limited to height, weight, blood pressure, medical history, genomic information, transcriptomics, metabolomics, proteomics, and information obtained from intelligent clinical tests.

[0041] Whole slide image (WSI) is the mode closest to direct microscopic diagnosis and is the mainstream technology of current digital pathology. Pathological analysis based on WSI is an important means of diagnosing diseases, especially malignant tumors. WSI meets the requirements of high-definition, high-speed, and high-throughput adjustment, making it possible to digitize traditional pathological sections and laying a solid foundation for the development and application of digital pathology. In digital pathology, a common digitization strategy is to scan pathological sections 200 times or 400 times, and the data generated by the scan is called WSI. The rapid development of digital pathology based on whole slide digital images (WSI) has enabled artificial intelligence (AI) to make many advances in computer-aided diagnosis.

[0042] The region of interest (ROI) is an area to be processed that is delineated from the processed image in the form of a box, circle, ellipse, irregular polygon, etc. in image processing using machine vision. In the present invention, it specifically refers to the area of ​​the pathological (such as cancer) tissue in the WSI image.

[0043] TabNet feature extraction network is a high-performance and interpretable deep tabular data learning neural network proposed by Google. TabNet uses sequential attention to select the function to be inferred from each feature, making it easier to use the model's learning ability for the most significant features, which can achieve interpretability and more effective learning.

[0044] In the medical field (taking cancer as an example), doctors often need to obtain pathological images of the patient's cancerous tissue in order to formulate the next treatment plan. For example, highly suspected lesion tissue can be directly obtained by puncture. In the pathology department, the obtained lesion tissue is embedded in paraffin and stained with hematoxylin and eosin to make pathological biopsy slides, and then biopsy pathological images are generated. On the other hand, when formulating a treatment plan, the doctor will also comprehensively examine the patient's overall health level and analyze the patient's clinical information to determine the patient's condition.

[0045] Breast cancer is the most common cancer in women. Most breast cancers are already in the invasive stage when they are discovered, which means that the abnormal cells have spread beyond the breast cancer or lobules to the surrounding tissues. During the treatment of breast cancer, some patients' conditions involve axillary lymph node dissection. Axillary lymph node dissection can cause many short-term and long-term complications. Therefore, before the dissection, the metastatic status of the lymph nodes needs to be accurately classified to avoid unnecessary surgery. Currently, for the task of classifying the degree of lymph node metastasis in breast cancer, routine clinical testing either uses only pathological images or only clinical information, and therefore cannot achieve high accuracy. For solutions that take into account both the patient's clinical information and pathological images, the pathological images still need to be manually interpreted, so an end-to-end automated decision process cannot be achieved.

[0046] According to the present invention, by utilizing a multimodal model to simulate the doctor's interpretation method and applying deep learning in combination with multimodal data, multimodal data (such as multi-scale pathological image information and clinical information) can be simultaneously incorporated into the decision-making process, thereby enabling an end-to-end prediction of the state of lesion tissue (such as breast cancer lymph node metastasis) without the need for pathologists to interpret the condition.

[0047] Figure 1An exemplary application scenario 100 according to an embodiment of the present invention is shown. The application scenario 100 includes data 101, a server 102, a network 103 and a display 104. Data 101 refers to data input to the server 102 for training or testing the abnormal cell metastasis degree prediction model. For example, data 101 may include a training set, a validation set and a test set. The data type may include, but is not limited to, image information, table information and sound information. In the application scenario of the present invention, data 101 such as a puncture image of a patient's cancer tissue, an echo of a patient's relevant part and a patient's clinical information may be sent to the server 102 through the network 103, or directly input to the server 102 through an input device of the server 102. The network 103 may be, for example, a wide area network (WAN), a local area network (LAN), a wireless network, a public telephone network, an intranet and any other type of network known to those skilled in the art. It is understood that in other application scenarios, the data 101 may be directly input to the server 201 without passing through the network 103. After receiving the data, the server 102 can classify it according to the neural network in one embodiment of the present invention, and visually output the intermediate calculation content to the display 104 for display, to assist the doctor in classifying the data. The server 102 is a hardware device that can apply the technical solution according to one embodiment of the present invention, and at least a pre-trained neural network according to one embodiment of the present invention is stored thereon. As understood by those of ordinary skill in the art, the server 102 is not limited to a single server device, but can also be a server cluster or a distributed device. The display 104 can receive display data from the server 102, and display the analysis results of the degree of abnormal cell metastasis according to the display data.

[0048] Figure 2 A flowchart of a method 200 for predicting the degree of abnormal cell metastasis based on medical data is shown. The method 200 can use artificial intelligence to predict the degree of abnormal cell metastasis based on the following steps. Figure 1 The system is implemented on the server 102 shown.

[0049] In step 201, the server 102 obtains medical data of multiple modalities related to abnormal cells from tissue sections. In one embodiment, the multiple modalities include at least a first modality and a second modality. The medical data of the first modality corresponds to image data related to abnormal cells. The medical data of the second modality corresponds to clinical information data of the patient to whom the abnormal cells belong. Here, the operator (e.g., a pathologist) first collects biopsy pathology images of the patient. The operator outlines the cancer area as the ROI and obtains the clinical information corresponding to the ROI at the same time. In one embodiment, the medical data of the first modality corresponds to image data of abnormal cells at multiple zoom scales. The operator zooms the WSI to 20 times, 10 times, and 5 times, respectively. Here, the pathological image data can use other numbers of magnifications (e.g., 5 zoom scales) or other magnification values. For different zoom scales, a sliding window of 512×512 size is used, and a step size of 512 is slid in the effective ROI area to obtain small pictures (Patch) at different magnifications, which are saved to disk for subsequent processing. Other sizes can be used for the patch of pathological image acquisition here. In one embodiment, the abnormal cell image at each zoom scale in multiple zoom scales includes multiple images. In addition, the table of clinical information is cleaned and digitized by category for the subsequent numerical calculation part. In the context of this article, modality refers to a form of pathological information of a patient. As understood by those skilled in the art, medical data of multiple modalities are not limited to medical data of the first modality and medical data of the second modality. For example, data of other modalities such as medical data of gene sequencing modality and medical data of proteohistochemistry modality may also be included. The ROI size at different magnifications is not necessarily the same, and the ROI size for different patients is also different. In one embodiment, the multiple images included in the abnormal cell image at each zoom scale in multiple zoom scales may be multiple images of different numbers.

[0050] In step 202, the unimodal features of the medical data of each modality are extracted from the medical data of multiple modalities. In one embodiment, the first feature extraction network is used to extract the features of the first modality for the first modality. In one embodiment, the first feature extraction network includes an image feature extractor pre-trained based on the first data set. For the pathological image part, EfficientNet-B0 pre-trained on the ImageNet data set is used as an image feature extractor to extract features for all patches. All features acquired by each WSI at different magnifications will be packaged together and saved to disk for subsequent training or prediction. The image feature extractor in this scheme can use other convolutional neural networks (such as ResNet, DenseNet, EfficientNet, InceptionNet, VGG), in which the network initialization method can be based on ImageNet data set pre-training, random initialization and pre-training based on other data sets. The subsequent algorithm will set up a separate network branch for each magnification for processing. In one embodiment, a second feature extraction network is used to extract the features of the second modality for the second modality. TabNet, a tabular feature extractor, is used to extract features from the patient's clinical information to generate an embedded feature representation of the clinical information. For feature extraction of clinical information, other table models or fully connected neural networks can be used.

[0051] In the pathological image part, image features are extracted from images at multiple magnifications. Since the ROI sizes at different magnifications are inconsistent and the ROI sizes of different patients are also different, the number of features extracted in this step will vary greatly. An effective feature selection mechanism is needed here to retain effective sample features. At the same time, this method uses data from multiple modalities, and it is also necessary to effectively integrate information from different modalities to complete the prediction together. The multimodal multi-example module will solve the aforementioned feature selection and modality information integration problems. Therefore, in step 203, the cross-modal fusion features of medical data of multiple modalities are obtained by fusing the extracted single-modal features. There will be a set of image features of varying numbers at each magnification. Here, multiple features at each magnification are fused into a single feature by global average pooling. Then, the table features and the features of images of different magnifications are connected, and the features are fused using a fully connected network layer to obtain cross-modal fusion features. The features of multiple images of abnormal cells at each zoom scale under the first modality are weighted fused to obtain weighted fusion features of abnormal cells at each scale; the weighted fusion features of each scale under the first modality are spliced ​​with the features of the second modality to obtain cross-modal fusion features. Here, weighted fusion can be performed by global average pooling of the features of multiple images of abnormal cells at each zoom scale under the first modality.

[0052] In step 204, each single modality feature is adjusted using the cross-modality fusion feature as a global feature. In one embodiment, the adjustment process includes: based on the cross-modality fusion feature as a global feature, local-global information fusion is performed with the features of each picture of each scale under the first modality to obtain the local-global information fusion features of each picture of each scale under the first modality; the local-global information fusion features of each picture of each scale under the first modality are weighted averaged to obtain the adjusted features of each scale under the first modality; and based on the cross-modality fusion feature as a global feature, the features of the second modality are adjusted to obtain the adjusted features under the second modality. In one embodiment, the features of the second modality are adjusted based on the cross-modality fusion feature as a global feature to obtain the adjusted features under the second modality, including: calculating the multimodal global features through a fully connected network, and adjusting the features of the second modality using the scaling coefficient obtained by the Sigmoid activation function to obtain the adjusted features under the second modality. In the network branch of the table feature, after the cross-modal fusion feature passes through the fully connected network layer, the Sigmoid activation function is used to obtain the scaling factor of the branch feature and scale the table feature. This scaling is reflected in the guidance of the cross-modal data to the learning process of the single-modal branch. In the network branch of the image feature, it is necessary to process the features at each magnification separately and adjust the effective features. The cross-modal fusion features are spliced ​​with the patch features at each magnification, and the fully connected network layer is used for local and global feature fusion. The fused features are then processed using a multi-layer fully connected network followed by a Softmax activation function to obtain the weight representing the relative importance of each patch. For the features of each magnification, the weight is multiplied back to the features of the patch, and then all the patch features are summed to obtain the embedded feature representation of the pathological image at that magnification.

[0053] In step 205, multimodal features of medical data of multiple modalities are obtained based on the adjusted single-modal features. In one embodiment, obtaining multimodal features of medical data of multiple modalities based on the adjusted single-modal features includes: splicing the adjusted features of each scale under the first modality and the features of the second modality to obtain multimodal features of medical data of multiple modalities.

[0054] In step 206, the degree of abnormal cell metastasis is predicted based on the multimodal features. After the previous processing, the fused table embedding features and the embedded features of the image at each magnification have been obtained. In the classification output part, the table features and image features are spliced, and a multi-layer fully connected network is used as the classifier Gc (also called the main classifier) ​​of the entire model to predict breast cancer lymph node metastasis and obtain the probability of metastasis. In one embodiment, classifying the degree of abnormal cell metastasis based on multimodal fusion features includes: inputting the multimodal fusion features into the classifier to predict the degree of abnormal cell metastasis; wherein the classifier is pre-trained based on the features of the samples in the training set as input, and the labels of the samples are used as supervisory signals. In another embodiment, the classifier includes at least a main classifier and an auxiliary classifier. The reason for using the auxiliary classifier here is that the network training process may fluctuate in the multi-example fusion part. In order to stabilize this process, based on the idea of ​​deep supervision, another classifier Gt is used during training to add additional supervisory information to the branch of the table features.

[0055] This method realizes the fusion of the patient's pathological information of different modalities, makes full use of the patient's multimodal information resources, and improves the prediction accuracy of abnormal cell metastasis. Specifically, the technical solution of the present invention adopts an end-to-end learning method, and the clinical information used is easy to obtain and does not require additional collection. The present invention combines the characteristics of multimodal data analysis and combines it into the framework of multi-instance learning analysis. It uses the patient's clinical information and corresponding pathological images to predict lymph node metastasis in the early stage, assisting doctors in making preoperative decisions, thereby providing more accurate treatment solutions and improving the patient's quality of life. This multimodal multi-instance method uses the gold standard for clinical cancer diagnosis (biopsy pathological sections) and the patient's clinical diagnosis information, and applies deep learning to combine the information of the two to predict whether lymph node metastasis will occur before surgery, effectively assisting preoperative decision-making.

[0056] Figure 3 A schematic block diagram of a method according to an embodiment of the present invention is shown. In the method, a biopsy pathology image of a patient is first collected, and an operator (eg, a pathologist) delineates a cancer region as a ROI. Other information is obtained at the same time.

[0057] Then, the acquired multimodal information is preprocessed. The operator scales the WSI of the ROI to different scaling sizes. Here, the scaling sizes are 20 times, 10 times and 5 times as examples. For the images 301, 302 and 303 with 20 times, 10 times and 5 times, a sliding window is used to slide in the effective ROI area with a certain step size to obtain small images 304, 305 and 306 at different magnifications. For different magnifications, the size of the sliding window can be 512 (pixels) × 512 (pixels), and the step size can be 512. For different magnifications, the sliding window and step size can also be set to other values. The small images at different magnifications are saved to the disk for subsequent processing. The table 310 of clinical information is cleaned and digitized according to categories for the subsequent numerical calculation part.

[0058] Next, feature extraction is performed on medical data of different modalities, namely pathological images and clinical information. For the first modality of medical data (i.e. pathological image data), we use Figure 3 The feature extraction network shown in performs feature extraction on all small images to obtain embedded features 307, 308 and 309 of the extracted images at different magnifications. The feature extraction network here can use EfficientNet-B0 pre-trained with the first data set (e.g., ImageNet data set). All features acquired by each WSI at different magnifications are packaged together and saved to disk for subsequent training or prediction. For the medical data of the second modality (i.e., clinical information data 310), a tabular feature extraction network (e.g., TabNet network) is used to extract features from the patient's clinical information to generate an embedded feature representation 311 of the clinical information.

[0059] Finally, the features of each modality are fused through the multimodal multi-instance module 312. The multimodal multi-instance module 312 fuses the image embedding features 307, 308 and 309 and the embedded feature representation 311 of the clinical information to obtain the fused table embedding features 316 and the fused embedded features 313, 314 and 315 of the image at each magnification. The specific structure of the multimodal multi-instance module 312 will be described in Figure 4A more detailed description is given in . At 317, the embedded features 313, 314 and 315 of the fused images at each magnification and the fused table embedded features 316 are spliced. The spliced ​​features are classified using classifier Gc 318 to predict the lymph node metastasis of abnormal cells and obtain the probability of metastasis. Here, the classifier 318 can adopt a multi-layer fully connected network. In the part of multi-example fusion, the training process of the network may fluctuate. In order to stabilize this process, based on the idea of ​​deep supervision, an auxiliary classifier 319 is used during training. The auxiliary classifier 319 classifies the fused table embedded features 316 to add additional supervision information.

[0060] Figure 4 Shows Figure 3 Schematic diagram of the specific structure of the multimodal multi-instance module 312 in FIG. Here, taking an image at three zoom scales as an example, the multimodal multi-instance module 312 first aggregates the embedded features 307, 308, and 309 of the image respectively. Since there may be different numbers of image features at each zoom scale. Figure 4 The left side of shows the process of feature aggregation. Here, the global average pooling method is used to fuse multiple features 307, 308, 309 at each magnification into a single feature u1, u2, u3. Then, the table feature ht is connected with the fused features u1, u2, u3 of the images of different magnifications. At 401, the connected features are fused using a fully connected network to obtain a cross-modal fused feature Fglobal.

[0061] Next, the cross-modal fusion features are used as global features to adjust each single-modal feature. In one embodiment, using the cross-modal fusion features as global features to adjust each single-modal feature includes: based on the cross-modal fusion features as global features, local-global information fusion is performed with the features of each picture at each scale under the first modality to obtain the local-global information fusion features of each picture at each scale under the first modality; the local-global information fusion features of each picture at each scale under the first modality are weighted averaged to obtain the adjusted features of each scale under the first modality; and the features of the second modality are adjusted based on the cross-modal fusion features as global features to obtain the adjusted features under the second modality.

[0062] In another embodiment, the features of the second modality are adjusted based on the cross-modal fusion features as global features to obtain the adjusted features of the second modality, including: calculating the multimodal global features through a fully connected network, and adjusting the features of the second modality using the scaling coefficients obtained by the Sigmoid activation function to obtain the adjusted features of the second modality.

[0063] In the network branch of the table feature, the cross-modal fusion feature Fglobal passes through the fully connected network layer at 403, and then uses the Sigmoid activation function to obtain the scaling factor of the branch and scale the table feature based on the scaling factor. The scaling sparsity is reflected in the guidance of the cross-modal data on the learning process of the single-modal branch.

[0064] In the image feature network branch, the features at each magnification need to be processed separately to screen out effective features. Here, the obtained cross-modal fusion features are divided into sub-cross-modal fusion features o1, o2 and o3 with the same number of zoom scales. O1, o2 and o3 are spliced ​​with the patch features at each magnification respectively. Next, a fully connected network is used to fuse local and global features, and the fused features are processed using a multi-layer fully connected network followed by a Softmax activation function to obtain weights representing the relative importance of each patch. For the features of each zoom scale, the weight is multiplied back to the features of the patch, and then all the patch features are summed to obtain the embedded feature representation of the pathological image at that magnification.

[0065] This method obtains the weights that represent the relative importance of patches at each magnification. Compared with previous single-modality image algorithms, this method can better judge the importance of patches. Based on this, this method can associate the explanations of corresponding positions and visualize the influence of patches at different positions on the results, which can further assist doctors in making preoperative decisions.

[0066] After completing the model training, the trained model weights are loaded in the testing phase and the above process is repeated to predict the possibility of lymph node metastasis.

[0067] It is worth noting that the multimodal fusion method in this scheme can be replaced by a fully connected neural network fusion to obtain a multimodal representation, or a weighted summation of single-modal prediction probabilities. The task of this scheme is to predict breast cancer lymph node metastasis, which can be replaced by lymph node metastasis prediction tasks or classification tasks for other cancer types.

[0068] The method of the present invention and other methods are compared and evaluated below.

[0069] In this comparative evaluation, the method of the present invention is compared with six other methods, which are:

[0070] (1) Single-modality method based on tabular modality. In this method, only the tabular modality representing the patient's clinical information is used as input for classification. Its network mainly uses the TabNet feature extraction network to extract the modal features of the tabular modality, and directly performs classification operations based on the modal features to obtain classification results.

[0071] (2) Single-modality method based on image modality. In this method, only the 5x image modality of the WSI image representing the abnormal cells of the patient is used as input for classification. Its network mainly uses the EfficientNet-B0 feature extraction network to extract the modality features of the 5x image modality, and directly performs classification operations based on the modality features to obtain classification results.

[0072] (3) Single-modality method based on image modality. In this method, only the 10x image modality of the WSI image representing the patient's abnormal cells is used as input for classification. Its network mainly uses the EfficientNet-B0 feature extraction network to extract the modality features of the 10x image modality, and directly performs classification operations based on the modality features to obtain classification results.

[0073] (4) Single-modality method based on image modality. In this method, only the 20x image modality of the WSI image representing the abnormal cells of the patient is used as input for classification. Its network mainly uses the EfficientNet-B0 feature extraction network to extract the modality features of the 20x image modality, and directly performs classification operations based on the modality features to obtain classification results.

[0074] (5) Multimodal method based on image modality. In this method, 5, 10 and 20 magnification image modalities of WSI images representing abnormal cells of patients are used as input for classification. Its network mainly uses the EfficientNet-B0 feature extraction network to extract the modal features of these three image modalities, and then fuses the features of these three modalities into a feature representation through a neural network, and directly performs classification operations based on the one feature representation to obtain classification results. Similar to the above method, the evaluation effect of this method is an embodiment of the above-mentioned single-modal simple network method. This method classifies according to one pathological information, and does not classify in combination with the overall situation of various pathological information, and the classification resources are not fully utilized.

[0075] (6) Simple multimodal method. In this method, 5, 10 and 20 magnification image modalities of WSI images representing abnormal cells of the patient and a tabular modality representing the clinical information of the patient are used as input for classification. Its network mainly includes using the EfficientNet-B0 feature extraction network to extract the modal features of these three image modalities, and using the TabNet feature extraction network to extract the features of the tabular modality. Based on the features of these four modalities, the features of these four modalities are fused into a feature representation through a fully connected network, and the classification operation is directly performed based on the one feature representation to obtain the classification result. The evaluation effect of this method is an embodiment of the above-mentioned multimodal simple network including method. The classification logic and network of this method are too rough and simple, and cannot fully adapt to the complexity of multi-category data.

[0076] The comparative evaluation results are shown in Table 1, where AUC, F1-score, precision, and recall are indicators commonly used in the prior art to evaluate classification performance. Among them, precision refers to how many of all predicted positive examples are correctly predicted positive examples. The precision can be denoted as Pr. For example, there are 10 positive examples and 5 negative examples. Among them, 8 of the 10 positive examples are correctly predicted positive examples, and 2 of the 5 negative examples are predicted to be positive, then Pr is {8 / (8+2)=0.8}; Recall refers to the proportion of correctly predicted positive examples among all positive examples, denoted as R. The comprehensive value is the comprehensive value obtained based on the precision Pr and the recall R, which can be denoted as F1. For example, the comprehensive value can be the harmonic mean of the precision Pr and the recall R. As shown in the following formula:

[0077] F1 = precision * recall * 2 / (precision + recall) (1)

[0078] Among them, the larger the precision, recall or F1 value is, the better the effect of the corresponding model is. Generally speaking, the F1 value is a better representation of the classification effect because it combines the other two evaluation indicators.

[0079] AUC (Area Under the receiver operating characteristic Curve) is also used as an evaluation indicator to measure the effect of each solution. AUC measures the probability that the classifier correctly judges the value of the positive sample to be approximately the value of the negative sample when a positive sample (y = 1) and a negative sample (y = 0) are randomly selected. Therefore, the larger the AUC value of the classifier, the higher the accuracy.

[0080]

[0081] Table 1 Comparative evaluation table.

[0082] By comparison, it can be found that the AUC, F1-score and precision indexes of the present invention are obviously higher than those of any other methods, and the recall rate also performs well. The method of the present invention improves the classification accuracy.

[0083] Figure 5 The image thermal maps processed by the method of the present invention and the image single-modality method are shown. Figure 5 Block 501 is a heat map after the image at 5 times is processed by the single-modal method based on the A-MIL model; block 502 is a heat map after the image at 10 times is processed by the single-modal method based on the A-MIL model; block 503 is a heat map after the image at 20 times is processed by the single-modal method based on the A-MIL model; block 504 is a heat map after the image at 5 times is processed by the method of the present invention; block 505 is a heat map after the image at 10 times is processed by the method of the present invention; block 506 is a heat map after the image at 20 times is processed by the method of the present invention. Figure 5 Different grayscales correspond to different weight coefficients. The colors in blocks 501 to 503 are relatively close, and different image blocks cannot be distinguished well. Blocks 504 to 506 can distinguish image blocks well, and useless image blocks can be selected. It can be seen that the method according to the present invention can integrate the information of each modality to more accurately distinguish the degree of metastasis of abnormal cells compared with the single-modality method with the same magnification.

[0084] Figure 6 A device 600 for predicting the degree of abnormal cell metastasis is shown. The device 600 includes: an acquisition module 601, a unimodal feature extraction module 602, a cross-modal fusion module 603, an adjustment module 604, a multimodal feature module 605 and a prediction module 606. The acquisition module 601 is configured to acquire medical data of multiple modalities related to abnormal cells. The unimodal feature extraction module 602 is configured to extract unimodal features of the medical data of each modality from the medical data of the multiple modalities. The cross-modal fusion module 603 is configured to obtain cross-modal fusion features of the medical data of the multiple modalities by fusing the extracted unimodal features. The adjustment module 604 is configured to adjust each unimodal feature using the cross-modal fusion feature as a global feature. The multimodal feature module 605 is configured to obtain multimodal features of the medical data of the multiple modalities based on the adjusted unimodal features. The prediction module 606 is configured to predict the degree of abnormal cell metastasis according to the multimodal features.

[0085] The device realizes the fusion of the patient's pathological information of different modalities, fully utilizes the patient's multimodal information resources, and improves the classification accuracy. Specifically, an end-to-end learning method is adopted, and the clinical information used is easy to obtain and does not require additional collection. The device combines the characteristics of multimodal data analysis and combines it into the framework of multi-instance learning analysis. It uses the patient's clinical information and corresponding pathological images to predict lymph node metastasis in the early stage, assisting doctors in preoperative decision-making, thereby providing more accurate treatment solutions and improving the patient's quality of life. This multimodal multi-instance method uses the gold standard for clinical cancer diagnosis (biopsy pathological sections), combined with the patient's clinical diagnosis information, and applies deep learning to combine the information of the two to predict whether lymph node metastasis will occur before surgery, effectively assisting preoperative decision-making.

[0086] Figure 7 An example system 700 is illustrated, which includes an example computing device 710 that represents one or more systems and / or devices that can implement the various techniques described herein. Computing device 710 can be, for example, a server of a service provider, a device associated with a server, a system on a chip, and / or any other suitable computing device or computing system. Figure 6 The described apparatus 600 for predicting the degree of abnormal cell metastasis may take the form of a computing device 710. Alternatively, the apparatus 600 for predicting the degree of abnormal cell metastasis may be implemented as a computer program in the form of an application 716.

[0087] The example computing device 710 as shown includes a processing system 711, one or more computer-readable media 712, and one or more I / O interfaces 713 that are communicatively coupled to each other. Although not shown, the computing device 710 may also include a system bus or other data and command transmission system that couples various components to each other. The system bus may include any one or combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and / or a processor or local bus utilizing any one of a variety of bus architectures. Various other examples are also contemplated, such as control and data lines.

[0088] Processing system 711 represents the functionality of performing one or more operations using hardware. Thus, processing system 711 is illustrated as including hardware elements 714 that may be configured as processors, functional blocks, and the like. This may include other logic devices implemented in hardware as application specific integrated circuits or formed using one or more semiconductors. Hardware elements 714 are not limited by the materials from which they are formed or the processing mechanisms employed therein. For example, a processor may be composed of (multiple) semiconductors and / or transistors (e.g., electronic integrated circuits (ICs)). In such a context, processor executable instructions may be electronic executable instructions.

[0089] Computer readable media 712 is illustrated as including memory / storage 715. Memory / storage 715 represents memory / storage capacity associated with one or more computer readable media. Memory / storage 715 may include volatile media (such as random access memory (RAM)) and / or non-volatile media (such as read-only memory (ROM), flash memory, optical disks, magnetic disks, etc.). Memory / storage 715 may include fixed media (e.g., RAM, ROM, fixed hard drive, etc.) and removable media (e.g., flash memory, removable hard drive, optical disk, etc.). Computer readable media 712 may be configured in various other ways as further described below.

[0090] One or more I / O interfaces 713 represent functionality that allows a user to input commands and information to the computing device 710 using various input devices and optionally also allows information to be presented to the user and / or other components or devices using various output devices. Examples of input devices include a keyboard, a cursor control device (e.g., a mouse), a microphone (e.g., for voice input), a scanner, touch functionality (e.g., a capacitive or other sensor configured to detect physical touch), a camera (e.g., that can detect motion that does not involve touch as gestures using visible or invisible wavelengths (such as infrared frequencies), etc.). Examples of output devices include a display device (e.g., a projector), a speaker, a printer, a network card, a tactile response device, etc. Thus, the computing device 710 can be configured in various ways as further described below to support user interaction.

[0091] The computing device 710 also includes an application 716. The application 716 may be, for example, a software instance of the apparatus 600 for predicting the extent of abnormal cell metastasis, and in combination with other elements in the computing device 710 implements the technology described herein.

[0092] Various techniques may be described herein in the general context of software hardware elements or program modules. Generally, these modules include routines, programs, objects, elements, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The terms "module", "function" and "component" as used herein generally represent software, firmware, hardware or a combination thereof. The features of the techniques described herein are platform-independent, meaning that these techniques can be implemented on a variety of computing platforms with a variety of processors.

[0093] An implementation of the described modules and techniques may be stored on or transmitted across some form of computer-readable media. Computer-readable media may include various media accessible by computing device 710. By way of example and not limitation, computer-readable media may include “computer-readable storage media” and “computer-readable signal media.”

[0094] In contrast to a simple signal transmission, carrier wave, or signal itself, "computer-readable storage medium" refers to a medium and / or device capable of persistently storing information, and / or a tangible storage device. Therefore, a computer-readable storage medium refers to a non-signal-bearing medium. Computer-readable storage media include hardware such as volatile and non-volatile, removable and non-removable media and / or storage devices implemented in a method or technology suitable for storing information (such as computer-readable instructions, data structures, program modules, logic elements / circuits, or other data). Examples of computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage device, hard disk, cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, or other storage device, tangible medium, or article of manufacture suitable for storing desired information and accessible by a computer.

[0095] "Computer-readable signal media" refers to signal-bearing media that is configured to send instructions to the hardware of the computing device 710, such as via a network. Signal media typically can embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, data signal, or other transport mechanism. Signal media also include any information delivery media. The term "modulated data signal" refers to a signal that has one or more of its characteristics set or changed so as to encode information into the signal. By way of example and not limitation, communication media include wired media such as a wired network or direct connection and wireless media such as acoustic, RF, infrared, and other wireless media.

[0096] As before, hardware elements 714 and computer-readable media 712 represent instructions, modules, programmable device logic and / or fixed device logic implemented in hardware form, which in some embodiments can be used to implement at least some aspects of the technology described herein. Hardware elements can include integrated circuits or systems on a chip, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), and other implementations in silicon or components of other hardware devices. In this context, hardware elements can be used as processing devices for executing program tasks defined by instructions, modules and / or logic embodied by hardware elements, as well as hardware devices for storing instructions for execution, such as the computer-readable storage media described previously.

[0097] The aforementioned combination may also be used to implement various techniques and modules herein. Therefore, software, hardware or program modules and other program modules may be implemented as one or more instructions and / or logic embodied on some form of computer-readable storage medium and / or by one or more hardware elements 714. Computing device 710 may be configured to implement specific instructions and / or functions corresponding to software and / or hardware modules. Therefore, for example, by using a computer-readable storage medium and / or hardware element 714 of a processing system, a module may be implemented as a module executable by computing device 710 as software, at least in part, in hardware. Instructions and / or functions may be executable / operable by one or more articles (e.g., one or more computing devices 710 and / or processing systems 711) to implement the techniques, modules and examples described herein.

[0098] In various embodiments, computing device 710 can be implemented in various configurations. For example, computing device 710 can be implemented as a computer-type device including a personal computer, a desktop computer, a multi-screen computer, a laptop computer, a netbook, etc. Computing device 710 can also be implemented as a mobile device-type device including mobile devices such as mobile phones, portable music players, portable game devices, tablet computers, multi-screen computers, etc. Computing device 710 can also be implemented as a television-type device, which includes a device with or connected to a generally larger screen in a casual viewing environment. These devices include televisions, set-top boxes, game consoles, etc.

[0099] The techniques described herein may be supported by these various configurations of computing device 710 and are not limited to the specific examples of the techniques described herein. Functionality may also be implemented in whole or in part on the "cloud" 720 by using a distributed system, such as through platform 722 as described below.

[0100] Cloud 720 includes and / or represents a platform 722 for resources 724. Platform 722 abstracts the underlying functionality of the hardware (e.g., servers) and software resources of cloud 720. Resources 724 may include applications and / or data that may be used when executing computer processing on a server remote from computing device 710. Resources 724 may also include services provided over the Internet and / or over a subscriber network such as a cellular or Wi-Fi network.

[0101] The platform 722 may abstract resources and functionality to connect the computing device 710 with other computing devices. The platform 722 may also be used to abstract hierarchies of resources to provide hierarchies of corresponding levels of demand encountered for resources 724 implemented via the platform 722. Thus, in an interconnected device embodiment, implementation of the functionality described herein may be distributed throughout the system 700. For example, functionality may be implemented in part on the computing device 710 and through the platform 722 that abstracts functionality of the cloud 720.

[0102] It should be understood that, for the sake of clarity, embodiments of the present application are described with reference to different functional units. However, it will be apparent that, without departing from the present application, the functionality of each functional unit can be implemented in a single unit, in multiple units or as a part of other functional units. For example, the functionality that is described as being performed by a single unit can be performed by multiple different units. Therefore, reference to a specific functional unit is only considered as a reference to the appropriate unit for providing the described functionality, rather than indicating a strict logical or physical structure or organization. Therefore, the present application can be implemented in a single unit, or can be physically and functionally distributed between different units and circuits.

[0103] Although the present application has been described in conjunction with some embodiments, it is not intended to be limited to the specific form set forth herein. On the contrary, the scope of the present application is limited only by the appended claims. Additionally, although individual features may be included in different claims, these may possibly be advantageously combined, and the inclusion in different claims does not imply that the combination of features is not feasible and / or advantageous. The order of features in the claims does not imply any specific order in which the features must work. In addition, in the claims, the word "comprising" does not exclude other elements, and the term "one" or "an" does not exclude a plurality. The figure marks in the claims are provided only as clear examples and should not be interpreted as limiting the scope of the claims in any way.

Claims

1. A method for predicting the degree of abnormal cell metastasis based on medical data, comprising: Acquiring medical data of multiple modalities related to abnormal cells, the multiple modalities including at least a first modality and a second modality; Extracting unimodal features of medical data of each modality from the medical data of the multiple modalities respectively; Obtaining cross-modal fusion features of the medical data of the multiple modalities by fusing the extracted single-modal features; The cross-modal fusion feature is used as a global feature to adjust each single-modal feature, which includes: Performing local-global information fusion based on the cross-modal fusion feature as a global feature and the features of each picture at each scale under the first modality, to obtain the local-global information fusion feature of each picture at each scale under the first modality; performing weighted averaging on the local-global information fusion features of each picture at each scale under the first modality to obtain adjusted features at each scale under the first modality; and Adjusting the features of the second modality based on the cross-modal fusion features as global features to obtain adjusted features of the second modality; Obtaining multimodal features of the medical data of the multiple modalities based on the adjusted single-modal features; The degree of abnormal cell metastasis is predicted based on the multimodal features.

2. The method according to claim 1, wherein the medical data of the first modality corresponds to image data of abnormal cells, and the medical data of the second modality corresponds to clinical information data of a patient to which the abnormal cells belong, The medical data of the first modality corresponds to image data of abnormal cells at multiple zoom scales, and the abnormal cell images at each zoom scale of the multiple zoom scales include multiple images.

3. The method according to claim 2, wherein extracting the single modality features of each modality of medical data from the medical data of the multiple modalities comprises: Extracting features of the first modality using a first feature extraction network for the first modality; and A second feature extraction network is used for the second modality to extract features of the second modality. 4 . The method according to claim 2 , wherein the abnormal cell images at each of the plurality of zoom scales include a plurality of images having different numbers.

5. The method according to claim 3, wherein the first feature extraction network comprises an image feature extractor pre-trained based on a first data set, and the second feature extraction network comprises a table feature extractor pre-trained based on a second data set.

6. The method according to claim 3, wherein the cross-modal fusion features of the medical data of multiple modalities obtained by fusing the extracted single-modal features include: Performing weighted fusion on features of multiple images of the abnormal cells at each scaling scale under the first modality to obtain weighted fusion features of the abnormal cells at each scaling scale; The weighted fusion features of each scale under the first modality and the features of the second modality are concatenated to obtain cross-modal fusion features.

7. The method according to claim 6, wherein the weighted fusion of features of multiple images of the abnormal cells at each scaling scale under the first modality to obtain weighted fusion features of the abnormal cells at each scaling scale comprises: Average pooling is performed on features of multiple images of the abnormal cells at each scaling scale under the first modality to obtain weighted fusion features of the abnormal cells at each scale.

8. The method according to claim 1, wherein the step of adjusting the feature of the second modality based on the cross-modal fusion feature as a global feature to obtain the adjusted feature under the second modality comprises: The multimodal global features are calculated through a fully connected network, and the features of the second modality are adjusted using a scaling coefficient obtained by a Sigmoid activation function to obtain adjusted features under the second modality.

9. The method according to claim 1, wherein the step of obtaining the multimodal features of the medical data of the multiple modalities based on the adjusted single-modal features comprises: The adjusted features of each scale under the first modality and the features of the second modality are spliced ​​to obtain multimodal features of the medical data of the multiple modalities.

10. The method according to claim 1, wherein the predicting the degree of metastasis of the abnormal cells according to the multimodal features comprises: Inputting the multimodal features into a classifier to predict the degree of metastasis of the abnormal cells; The classifier is pre-trained based on the features of samples in the training set as input, and the labels of the samples are used as supervisory signals.

11. The method according to any one of claims 1 to 10, wherein acquiring medical data of multiple modalities related to abnormal cells further comprises: Identify regions of interest in tissue sections; Based on the region of interest in the tissue slice, images of the region of interest at different zoom scales are acquired through a sliding window.

12. A device for predicting the degree of abnormal cell metastasis based on medical data, comprising: An acquisition module configured to acquire medical data of multiple modalities related to abnormal cells, wherein the multiple modalities include at least a first modality and a second modality; A single modality feature extraction module is configured to extract single modality features of medical data of each modality from the medical data of the multiple modalities; A cross-modal fusion module, configured to obtain cross-modal fusion features of the medical data of the multiple modalities by fusing the extracted single-modal features; An adjustment module is configured to use the cross-modal fusion feature as a global feature to adjust each single-modal feature, and includes: Performing local-global information fusion based on the cross-modal fusion feature as a global feature and the features of each picture at each scale under the first modality, to obtain the local-global information fusion feature of each picture at each scale under the first modality; performing weighted averaging on the local-global information fusion features of each picture at each scale under the first modality to obtain adjusted features at each scale under the first modality; and Adjusting the features of the second modality based on the cross-modal fusion features as global features to obtain adjusted features of the second modality; A multimodal feature module, configured to obtain multimodal features of the medical data of the multiple modalities based on the adjusted single-modal features; The prediction module is configured to predict the degree of abnormal cell metastasis based on the multimodal features.

13. A computer device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the steps of the method according to any one of claims 1 to 11.

Images