Method, device, equipment and medium for predicting cancer metastasis of target tissue

Abstract

The application discloses a method and device for predicting cancer metastasis of target tissue, equipment and medium, and relates to the field of machine learning. The method comprises the following steps: obtaining an electronic medical image and an electronic clinical index of a user account; determining an image feature of the user account according to the electronic medical image of the user account; determining a clinical feature of the user account according to the electronic clinical index of the user account; fusing the image feature and the clinical feature to generate an overall feature of the user account; classifying the overall feature to generate a classification score of the user account; and determining that the target tissue of the user account has cancer metastasis in response to the classification score being greater than a classification threshold. The application determines whether the target tissue has cancer metastasis according to the overall feature, and the determination result is more accurate.

Description

Technical Field

[0001] This application relates to the field of machine learning, and in particular to a method, apparatus, device, and medium for predicting cancer metastasis in a target tissue. Background Technology

[0002] Cancer cells are highly invasive and can metastasize to distant sites via lymphatic pathways. Clinically, it is necessary to predict the risk of lymph node metastasis in order to determine whether to perform lymph node dissection and other appropriate treatments. Therefore, preoperative diagnosis of lymph node metastasis is crucial for developing treatment plans for cancer patients.

[0003] The relevant technology uses AI (Artificial Intelligence) to extract image features from three images of cancer patients: computed tomography (CT), magnetic resonance imaging (MRI), and ultrasound (US). The image features of the three images are then fused to obtain the overall features. Based on the overall features, it is then determined whether the target tissue of the cancer patient will metastasize to lymph nodes.

[0004] However, the relevant technology only uses single-modal data related to images from cancer patients, resulting in low accuracy of the judgment results. Summary of the Invention

[0005] This application provides a method, apparatus, device, and medium for predicting cancer metastasis in a target tissue. The method can fuse image features and clinical features of a user account to generate overall features, and then determine the cancer metastasis status of the target tissue based on these overall features, resulting in a more accurate assessment. The technical solution is as follows:

[0006] According to one aspect of this application, a method for predicting cancer metastasis in a target tissue is provided, the method comprising:

[0007] Obtain electronic medical images and electronic clinical indicators from user accounts;

[0008] Based on the electronic medical images of the user account, determine the image features of the user account; based on the electronic clinical indicators of the user account, determine the clinical features of the user account.

[0009] By fusing the image features and the clinical features, the overall features of the user account are generated;

[0010] The overall features are classified to generate a classification score for the user account;

[0011] In response to the classification score being greater than the classification threshold, it is determined that the target tissue of the user account has undergone cancer metastasis.

[0012] The image features are used to represent the features of the electronic medical image related to the cancer metastasis status of the target tissue, and the clinical features are used to represent the features of the electronic clinical indicators related to the cancer metastasis status of the target tissue.

[0013] According to one aspect of this application, a device for predicting cancer metastasis in a target tissue is provided, the device comprising:

[0014] The acquisition module is used to acquire electronic medical images and electronic clinical indicators of the user account;

[0015] The feature extraction module is used to determine the image features of the user account based on the electronic medical image of the user account; and to determine the clinical features of the user account based on the electronic clinical indicators of the user account.

[0016] The fusion module is used to fuse the image features and the clinical features to generate the overall features of the user account;

[0017] The fusion module is also used to classify the overall features and generate a classification score for the user account;

[0018] The fusion module is further configured to determine that the target tissue of the user account has undergone cancer metastasis in response to the classification score being greater than the classification threshold; wherein the image features are used to represent features related to the cancer metastasis status of the electronic medical image and the target tissue, and the clinical features are used to represent features related to the cancer metastasis status of the electronic clinical indicators and the target tissue.

[0019] According to another aspect of this application, a training method for a model predicting cancer metastasis in a target tissue is provided, the model comprising an image feature extraction module, a clinical feature extraction module, and a feature fusion module, the method comprising:

[0020] Obtain a sample training set, which includes real annotations of sample user accounts, sample electronic medical images of sample user accounts, and sample electronic clinical indicators.

[0021] The image feature extraction module is invoked to determine the sample image features based on the sample medical image; the clinical feature extraction module is invoked to determine the sample clinical features based on the sample clinical indicators.

[0022] The feature fusion module is invoked to fuse the sample image features and the sample clinical features to generate the overall sample features of the sample user account;

[0023] The feature fusion module is invoked to classify the overall features of the sample and generate a sample classification score for the user account of the sample.

[0024] The model for predicting cancer metastasis in the target tissue is trained based on the difference between the sample classification score and the true label.

[0025] The sample image features are used to represent the features related to the cancer metastasis status of the sample electronic medical image and the target tissue, and the sample clinical features are used to represent the features related to the cancer metastasis status of the sample electronic clinical indicators and the target tissue.

[0026] According to another aspect of this application, a training apparatus for a model predicting cancer metastasis in a target tissue is provided, the model including an image feature extraction module, a clinical feature extraction module, and a feature fusion module, the method comprising:

[0027] The sample acquisition module is used to acquire a sample training set, which includes real annotations of sample user accounts, sample electronic medical images of sample user accounts, and sample electronic clinical indicators.

[0028] The sample feature extraction module is used to call the image feature extraction module to determine the sample image features based on the sample medical image; and to call the clinical feature extraction module to determine the sample clinical features based on the sample clinical indicators.

[0029] The sample fusion module is used to call the feature fusion module to fuse the sample image features and the sample clinical features to generate the overall sample features of the sample user account;

[0030] The sample fusion module is also used to call the feature fusion module to classify the overall features of the sample and generate a sample classification score for the user account of the sample.

[0031] The training module is used to train the model for predicting cancer metastasis in the target tissue based on the difference between the sample classification score and the true label; wherein, the sample image features are used to represent the features related to the cancer metastasis status of the sample electronic medical image and the target tissue, and the sample clinical features are used to represent the features related to the cancer metastasis status of the sample electronic clinical indicators and the target tissue.

[0032] According to another aspect of this application, a computer device is provided, comprising: a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, wherein the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the method for predicting cancer metastasis of a target tissue as described above, or the method for training a model for predicting cancer metastasis of a target tissue.

[0033] According to another aspect of this application, a computer storage medium is provided, wherein at least one piece of program code is stored in the computer-readable storage medium, the program code being loaded and executed by a processor to implement the method for predicting cancer metastasis in a target tissue as described above, or the method for training a model for predicting cancer metastasis in a target tissue.

[0034] According to another aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method for predicting cancer metastasis in a target tissue as described above, or the method for training a model for predicting cancer metastasis in a target tissue.

[0035] The beneficial effects of the technical solutions provided in this application include at least the following:

[0036] This application embodiment can fuse image features and clinical features of a user account to generate an overall feature, and then determine the cancer metastasis status of the target tissue of the user account based on the overall feature. Since the overall feature of this technical solution integrates features from different modalities, it includes information from different modalities, thus enabling a more comprehensive assessment of the cancer metastasis status of the target tissue and more accurate prediction results. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a schematic diagram of the structure of a computer system provided in an exemplary embodiment of this application;

[0039] Figure 2 This is a schematic diagram of a model for predicting cancer metastasis in a target tissue provided in an exemplary embodiment of this application;

[0040] Figure 3 This is a flowchart illustrating a method for predicting cancer metastasis in a target tissue, provided in an exemplary embodiment of this application.

[0041] Figure 4 This is a flowchart illustrating a feature fusion method provided in an exemplary embodiment of this application;

[0042] Figure 5 This is a flowchart illustrating an exemplary embodiment of an image feature extraction method provided in this application;

[0043] Figure 6 This is a flowchart illustrating a method for predicting cancer metastasis in a target tissue, provided in an exemplary embodiment of this application.

[0044] Figure 7 This is a schematic diagram of a state heatmap provided in an exemplary embodiment of this application;

[0045] Figure 8 This is a flowchart illustrating a method for determining cancer metastasis in lymph node tissue provided in an exemplary embodiment of this application;

[0046] Figure 9 This is a flowchart illustrating a method for determining the tumor type of a target tissue according to an exemplary embodiment of this application;

[0047] Figure 10 This is a flowchart illustrating a training method for a model that predicts cancer metastasis in a target tissue, provided in an exemplary embodiment of this application.

[0048] Figure 11 This is a schematic diagram of the structure of a predictive device for cancer metastasis in a target tissue provided in an exemplary embodiment of this application;

[0049] Figure 12 This is a schematic diagram of the structure of a training device for a model that predicts cancer metastasis in a target tissue, provided in an exemplary embodiment of this application.

[0050] Figure 13 This is a schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0052] First, let's introduce the terms used in the embodiments of this application:

[0053] Cancer metastasis: In the medical field, cancer cells have the ability to metastasize within a living organism. Cancer cells or tumors can spread from the original lesion site to other parts of the body. Cancer metastasis refers to the spread of cancer cells or tumors from the target tissue to other tissues or organs within the body. For example, lymph node metastasis is a type of cancer metastasis; lymph node metastasis refers to the spread of carcinoma in situ to lymph nodes.

[0054] Cancer metastasis status: Used to assess whether cancer metastasis will occur in biological tissues. If the cancer metastasis status is "metastasis," it means that cancer will metastasize in the tissue; if the cancer metastasis status is "non-metastasis," it means that cancer will not metastasize in the tissue.

[0055] Attention mechanisms assign weights to different parts of the input to identify the parts of interest. For example, when applied to feature extraction models, the input includes both features of interest and features of little interest. The attention mechanism assigns greater weights to the features of interest and less weights to the features of little interest, thereby highlighting the features of interest.

[0056] A histogram, also known as a quality distribution chart, is a statistical reporting graph that uses a series of vertical bars or lines of varying heights to represent the distribution of data. Generally, the horizontal axis represents the data type, and the vertical axis represents the distribution.

[0057] The Picture Archiving and Communication System (PACS) is a product of the intersection of radiology, medical imaging, digital imaging technology, computer technology, and communication technology. PACS converts medical images and indicator data into digital form, and completes the functions of image information acquisition, storage, management, processing, and transmission, so that medical images and indicator data can be effectively managed and fully utilized.

[0058] Figure 1 A schematic diagram of the structure of a computer system provided in an exemplary embodiment of this application is shown. The computer system 100 includes a terminal 120 and a server 140.

[0059] Terminal 120 runs an application related to determining cancer metastasis in a target tissue. This application can be a small app within a web application, a dedicated application, or a web client. For example, a user performs data processing-related operations on terminal 120; for instance, after terminal 120 determines the electronic medical images and electronic clinical indicators of a user account, terminal 120 obtains information on whether cancer metastasis has occurred in the target tissue of that user account. Terminal 120 is at least one of a smartphone, tablet computer, e-book reader, MP3 player, MP4 player, laptop computer, and desktop computer.

[0060] Terminal 120 is connected to server 140 via a wireless network or a wired network.

[0061] Server 140 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Server 140 is used to provide background services and send the cancer metastasis status of the target organization to terminal 120. Optionally, server 140 undertakes the main computing work, and terminal 120 undertakes the secondary computing work; or, server 140 undertakes the secondary computing work, and terminal 120 undertakes the main computing work; or, server 140 and terminal 120 jointly perform computing using a distributed computing architecture.

[0062] Figure 2 The diagram shows a model for predicting cancer metastasis in a target tissue according to an embodiment of this application. The model includes a clinical feature extraction module 201, an image feature extraction module 202, and a feature fusion module 203.

[0063] The clinical feature extraction module 201 is used to extract clinical features from electronic clinical indicators. The input to the clinical feature extraction module 201 is electronic clinical indicator 204, and the output is clinical feature 205. For example... Figure 2As shown, the electronic clinical indicator 204 includes, but is not limited to, at least one of age, carcinoembryonic antigen (CEA), a specific marker (CA125, the name of a glycoprotein), cancer antigen (CA19-9, a tumor marker associated with cancer), and Cyan fluorescent protein (CFP). Clinical characteristics are used to represent features of the electronic clinical indicator that are associated with the cancer metastasis status of the target tissue. Here, the target tissue refers to tissue containing cancer cells or tumors, or tissue that may have metastasized.

[0064] The image feature extraction module 202 is used to extract image features from the electronic medical image. The input to the image feature extraction module 202 is the electronic medical image 206, and the output is the image features 207. For example... Figure 2 As shown, in the image feature extraction module 202, the region of interest (ROI) in the electronic medical image 206 is first determined, and the ROI in the electronic medical image 206 is divided into n medical image blocks 208. The overlap ratio between each of the n medical image blocks 208 and the ROI is greater than a preset ratio. Optionally, the ROI extraction network is invoked to determine the ROI in the electronic medical image. After obtaining the n medical image blocks 208, the Multiple Instance Learning (MIL) network 209 is invoked to process the n medical image blocks 208 to obtain image features 207. In an optional implementation, in Figure 2 In this process, an image feature extraction network is invoked to process n medical image patches, outputting n image patch features 210 corresponding to the n medical image patches. From the n image patch features 210, i image patch features 211 are determined. Based on an attention mechanism, the i image patch features are fused into image features 207. Optionally, a ResNet-18 network is invoked to process the n medical image patches, outputting n image patch features corresponding to the n medical image patches. i image patch features are determined from the n image patch features based on histograms or maximum mean difference. Based on an attention mechanism, the i image patch features are fused into image features.

[0065] The feature fusion module 203 is used to fuse clinical features and image features to obtain overall features, and to determine the cancer metastasis status of the target tissue based on these overall features. The inputs to the feature fusion module 203 are clinical features 205 and image features 207. For example... Figure 2As shown, based on a gating attention mechanism, image feature 207 and clinical feature 205 are attention-weighted to obtain weighted image feature 211 and weighted clinical feature 212. Based on a tensor fusion mechanism, the weighted image feature 212 and weighted clinical feature 213 are fused to obtain overall feature 213. Overall feature 213 is passed through at least two fully connected layers to obtain classification score 214. If the classification score 214 is greater than the classification threshold, it is determined that the target tissue of the user account has cancer metastasis; if the classification score 214 is less than the classification threshold, it is determined that the target tissue of the user account has not cancer metastasis.

[0066] Figure 3 This illustration shows a flowchart of a method for predicting cancer metastasis in a target tissue according to an embodiment of this application. The method can be... Figure 1 The method, executed by the terminal 120, server 140, or other computer device shown, includes the following steps:

[0067] Step 302: Obtain electronic medical images and electronic clinical indicators for the user account.

[0068] Electronic medical images refer to medical images stored in electronic data form. In one implementation, electronic medical images are medical images acquired from a PACS (Picture Archiving and Communication System). In another implementation, electronic medical images are provided by other devices; for example, device A acquires a medical image, uploads the medical image to a server in electronic data form so that other devices can use the medical image, and device B downloads the medical image from the server to obtain the electronic medical image.

[0069] Electronic clinical indicators (ECAs) refer to clinical indicators stored in electronic data form. In one implementation, ECAs are clinical indicators obtained from a PACS (Picture Archiving and Communication System). In another implementation, ECAs are provided by other devices; for example, device C collects clinical indicators, uploads these indicators to a server as electronic data so that other devices can use them, and device D downloads the indicators from the server to obtain the electronic ECAs.

[0070] The target tissue can be any tissue within a living organism. For example, a target tissue could be a lymph node containing cancerous tissue. A skilled technician may select the target tissue based on specific needs. In one particular implementation, the target tissue refers to lymph node tissue.

[0071] Optionally, the electronic medical image is an electronic biopsy pathology image or an electronic gross section pathology image.

[0072] In one alternative implementation, when the target tissue is a lymph node of cancerous tissue, the electronic clinical indicators include at least one of age, CEA, CA125, CA19-9, and CFP. In practice, technicians can adjust the electronic clinical indicators according to actual needs.

[0073] In one alternative implementation, electronic medical images and electronic clinical indicators of the user account are obtained from the medical system.

[0074] Step 304: Determine the image features of the user account based on the electronic medical image of the user account.

[0075] Image features are used to represent characteristics of electronic medical images related to the cancer metastasis status of target tissue. For example, in... Figure 2 In the diagram, image feature 207 is denoted as h. bio ∈R 4 .

[0076] Step 306: Determine the clinical characteristics of the user account based on the electronic clinical indicators of the user account.

[0077] Clinical features are used to represent characteristics associated with the cancer metastasis status of a target tissue in relation to electronic clinical indicators. For example, in... Figure 2 In the text, clinical feature 205 is denoted as h. img ∈R 64 In one alternative implementation, a fully connected feedforward network is invoked to process the electronic clinical indicators of the user account and output the clinical characteristics of the user account.

[0078] Optionally, when clinical indicators are missing, the corresponding clinical features are also missing. The missing clinical indicators are determined using a multiple imputation method based on chain equations. For example, a normal linear regression model is used to determine the imputation model corresponding to the missing clinical feature; the imputation model is then invoked to determine the missing clinical indicator. For example, the normal linear regression model determines that the missing clinical feature A is a count variable, therefore the imputation model is a Poisson regression imputation model. The normal linear regression model determines that the missing clinical feature B is a dichotomous variable, therefore the imputation model is a logistic regression imputation model.

[0079] Step 308: Fuse image features and clinical features to generate overall features of the user account.

[0080] Overall features are used to represent the features corresponding to the cancer metastasis status of the target tissue under multiple modalities. For example, in Figure 2 In the above, the overall feature 214 is denoted as h. fusion .

[0081] Optionally, methods for fusing image features and clinical features include at least one of the following:

[0082] 1. Fusion of image and clinical features based on gating attention mechanism and tensor fusion mechanism;

[0083] 2. Fusing image features and clinical features through feature addition;

[0084] 3. Fusing image features and clinical features by concatenating feature vectors;

[0085] 4. Utilize neural networks to fuse image features and clinical features.

[0086] Taking the fusion of image features and clinical features based on gating attention mechanism and tensor fusion mechanism as an example, the steps include: based on gating attention mechanism, performing attention weighting on image features and clinical features to obtain weighted image features and weighted clinical features; based on tensor fusion mechanism, fusing weighted image features and weighted clinical features to obtain overall features.

[0087] Step 310: Classify the overall features and generate classification scores for user accounts.

[0088] In one specific implementation, at least two fully connected layers are invoked to process the overall features and output the classification score of the user account.

[0089] The classification score is used to represent the probability that a user account's target tissue has metastasized to cancer. Optionally, the classification score can range from [0, 1].

[0090] Step 312: In response to a classification score greater than the classification threshold, determine that the target tissue of the user account has cancer metastasis.

[0091] The classification threshold can be set by technical personnel.

[0092] Optionally, in response to a classification score less than or equal to a classification threshold, it is determined that the target tissue of the user account has not undergone cancer metastasis.

[0093] In one specific implementation, a classifier is invoked to process the classification score and output a classification result. In response to the classification result being the first classification result, it is determined that the target tissue of the user account has undergone cancer metastasis. For example, the classification score is 0.7. Specifically, when the classification score is greater than 0.5, the classifier outputs 1; when the classification score is less than or equal to 0.5, the classifier outputs 0. Therefore, the classification result is 1, confirming that the target tissue has undergone cancer metastasis.

[0094] In summary, this embodiment can fuse image features and clinical features of a user account to generate overall features, and then determine the cancer metastasis status of the target tissue based on these overall features. Because this technical solution integrates features from different modalities, the overall features include information from different modalities, thus enabling a more comprehensive assessment of the cancer metastasis status of the target tissue and resulting in more accurate predictions.

[0095] In the following embodiments, a method is provided for fusing image features and clinical features to generate overall features of a user account. This method uses a gating attention mechanism and a tensor fusion mechanism to fuse image features and clinical features.

[0096] Figure 4 A flowchart illustrating a feature fusion method according to an embodiment of this application is shown. This method can be... Figure 1 The method, executed by the terminal 120, server 140, or other computer device shown, includes the following steps:

[0097] Step 401: Based on the gating attention mechanism, perform attention weighting on image features and clinical features to determine image weights and clinical weights.

[0098] In one specific implementation, a gating network is invoked to perform attention weighting on image features and clinical features, and output image weights and clinical weights.

[0099] like Figure 2 As shown, based on the gating attention mechanism, attention weights are applied to image features and clinical features to obtain image weights σ1 and clinical weights σ2.

[0100] Step 402: Calculate the product of the image weight and the image feature to obtain the weighted image feature.

[0101] For example, such as Figure 2 As shown, the product of image weight σ² and clinical feature 205 is calculated to obtain the weighted image feature h'. img ∈R 64 .

[0102] Step 403: Calculate the product of clinical weights and clinical features to obtain the weighted clinical features.

[0103] For example, such as Figure 2 As shown, the product of clinical weight σ1 and image feature 207 is calculated to obtain the weighted clinical feature h'. bio ∈R 4 .

[0104] Steps 402 and 403 can be executed in any order. You can execute step 402 first and then step 403, or you can execute step 403 first and then step 402, or you can execute steps 402 and 403 simultaneously.

[0105] Step 404: Calculate the Kronecker product of the weighted image features and the weighted clinical features to obtain the features after tensor fusion.

[0106] For example, such as Figure 2 As shown, the Kronecker product of the weighted image features and the weighted clinical features is calculated to obtain the tensor-fused features, where the tensor-fused features are denoted as h. fu ∈R 264 .

[0107] Step 405: Combine the weighted image features and weighted clinical features into the tensor fusion features to obtain the overall features.

[0108] like Figure 2 As shown, the weighted image features and weighted clinical features are merged into the tensor fusion features to obtain the overall features, which are represented as h. fusion =h' bio ×h' img (∈R 325 ).

[0109] It should be noted that the feature h after tensor fusion fu ∈R 264 Weighted clinical features h' bio ∈R 4 The weighted image features h' img ∈R 64 Therefore, after directly merging the weighted image features and weighted clinical features into the tensor fusion features, interpolation or zero-padding is still required to obtain the overall feature h. fusion ∈R 325 .

[0110] In another optional implementation, a 1 is added to both the weighted image features and the weighted clinical features. A Kronecker product is then performed on the weighted image features and the weighted clinical features after adding the 1 to obtain the overall features. For example, if the weighted image features are [A, B, C, D] and the weighted clinical features are [E, F, G, H], adding a 1 to both the weighted image features and the weighted clinical features yields [1, A, B, C, D] and [1, E, F, G, H]. When calculating the Kronecker product, a matrix-form overall feature can be obtained. Since a 1 is added to both the weighted image features and the weighted clinical features, this matrix-form overall feature retains the original image features and clinical features.

[0111] In summary, this embodiment provides a method for fusing image features and clinical features to obtain overall features. This method can combine information from image features and clinical features to obtain overall features that include multimodal information. Through these overall features, a more accurate prediction result of the cancer metastasis status of the target tissue can be obtained.

[0112] In the following embodiments, a method for extracting image features is provided, which is implemented using a ResNet-18 (Residual Network-18, where 18 represents the number of convolutional and fully connected layers in the residual network) network. This method can extract image features from electronic medical images with relatively high accuracy.

[0113] Figure 5 A schematic flowchart of an image feature extraction method according to an embodiment of this application is shown. This method can be... Figure 1 The method, executed by the terminal 120, server 140, or other computer device shown, includes the following steps:

[0114] Step 501: Divide the region of interest in the electronic medical image into n medical image blocks.

[0115] In this context, the overlap ratio between each of the n medical image blocks and the region of interest is greater than a preset ratio. The preset ratio can be determined by technicians based on actual needs. For example, a preset ratio of 20% is used.

[0116] Optionally, a region of interest extraction network is invoked to process the electronic medical image and determine the region of interest within it. An image segmentation network is then invoked to process the region of interest in the electronic medical image, resulting in n medical image blocks.

[0117] For example, the region of interest in an electronic medical image is divided into n 512×512 pixel medical image blocks. The size of the medical image blocks can be set by the technician.

[0118] Step 502: Call the image feature extraction network to process the n medical image blocks and output the n image block features corresponding to the n medical image blocks.

[0119] In this embodiment, as Figure 2 As shown, the ResNet-18 network is invoked to process n medical image blocks 208, and the output is n image block features 210 corresponding to the n medical image blocks 208.

[0120] Alternatively, at least one of SqueezeNet (Squeeze Network), ResNet (Residual Network), DenseNet (Dense Network), InceptionNet (Inception Network), and VGG (Visual Geometry Group network) can be invoked to process n medical image patches and output n image patch features corresponding to the n medical image patches.

[0121] Step 503: Filter the n image patch features to obtain the filtered n image patch features.

[0122] It should be noted that the features of the n image patches are filtered using histograms and the maximum mean difference.

[0123] Optionally, the i-th feature in the n image patch features is removed, where i is an integer greater than 0 and less than n+1. For example, the image patch features have a total of 512 bits, and the 3rd, 4th, and 5th features in the image patch features are removed.

[0124] Optionally, preset positional features are retained from the features of n image patches. These preset positional features are determined during the training of the model for predicting cancer metastasis in the target tissue.

[0125] Specifically, the sample training set includes real annotations of sample user accounts and sample electronic medical images of sample user accounts; n sample image block features of sample electronic medical images of each sample user account are determined; based on the i-th feature among the n sample image block features of each sample user account, the histogram corresponding to the i-th feature under each sample user account is obtained; based on the aforementioned histogram and real annotations, the maximum mean discrepancy (MMD) of the i-th feature is determined; in response to the maximum mean discrepancy of the i-th feature being greater than or equal to the maximum mean discrepancy threshold, the i-th feature is used as a preset feature.

[0126] For example, the sample training set includes sample user account group A and sample user account group B. Sample user account group A consists of samples with cancer metastasis, and sample user account group B consists of samples without cancer metastasis. Taking sample user account group A as an example, the electronic medical image A of the i-th user sample in group A... i There are 20 sample image patch feature vectors. From these 20 feature vectors, the first feature of each sample image patch is extracted, resulting in 20 features. A histogram F is then obtained based on these 20 image patch features. Ai The histograms F for the remaining sample user accounts were obtained using the same method. A2 …F AM F B1 …F BN Then, the maximum mean difference between the histograms of sample group A and sample group B is calculated. If the maximum mean difference is greater than or equal to the maximum mean difference threshold, it indicates that the first feature in the aforementioned sample image patch features is helpful in predicting cancer metastasis, and the first feature is used as the preset feature. If the maximum mean difference is less than the maximum mean difference threshold, it indicates that the first feature in the aforementioned sample image patch features is not helpful in predicting cancer metastasis. Since sample user account group A represents samples with cancer metastasis, and sample user account group B represents samples without cancer metastasis, the maximum mean difference between histogram A and histogram B in this example can represent the degree of influence of the first feature on predicting cancer metastasis.

[0127] Step 504: Based on the attention mechanism, the features of the selected n image patches are fused into image features.

[0128] Optionally, an attention-based feature fusion network can be invoked to fuse the features of the selected n image patches into image features.

[0129] In summary, this embodiment provides a method for extracting image features, which can extract relatively accurate image features from medical images. This allows the fused overall features to more comprehensively assess the cancer metastasis status of the target tissue, resulting in more accurate predictions.

[0130] In the following embodiments, the predicted occurrence of cancer metastasis for each image patch in a medical image can be determined. This facilitates technicians or users in accurately locating the site of potential cancer metastasis in the target tissue.

[0131] Figure 6 This illustration shows a flowchart of a method for predicting cancer metastasis in a target tissue according to an embodiment of this application. The method can be... Figure 1 The method, executed by the terminal 120, server 140, or other computer device shown, includes the following steps:

[0132] Step 601: Determine the clinical characteristics of the user account based on the electronic clinical indicators of the user account.

[0133] Clinical features are used to represent the characteristics of electronic clinical indicators that are related to the cancer metastasis status of the target tissue.

[0134] Step 602: Select the target medical image block from the n medical image blocks.

[0135] The target medical image block is any one of n medical image blocks, and the target medical image block can be selected by technicians or users.

[0136] Step 603: Call the image feature extraction network to process the target medical image block and output the target image block features corresponding to the target medical image block.

[0137] In this embodiment, the ResNet-18 network is invoked to process the target medical image patch and output the target image patch features corresponding to the target medical image patch.

[0138] Optionally, at least one of SqueezeNet (Squeeze Network), ResNet (Residual Network), DenseNet (Dense Network), InceptionNet (Inception Network), and VGG (Visual Geometry Group network) can be invoked to process the target medical image patch and output the target image patch features corresponding to the target medical image patch.

[0139] Step 604: Fuse the target image patch features and clinical features to generate the overall image patch features of the user account.

[0140] Optionally, based on a gating attention mechanism, attention weighting is applied to the target image patch features and clinical features to obtain weighted target image patch features and weighted clinical features; based on a tensor fusion mechanism, the weighted target image patch features and weighted clinical features are fused to obtain the overall features of the target image patch.

[0141] Optionally, methods for fusing image features and clinical features include at least one of the following:

[0142] 1. Fusing image features and clinical features through feature addition;

[0143] 2. Fusing image features and clinical features by concatenating feature vectors;

[0144] 3. Utilize neural networks to fuse image features and clinical features.

[0145] Step 605: Classify the overall features of the image patch and generate the image patch classification score for the user account.

[0146] In one specific implementation, at least two fully connected layers are invoked to process the overall features and output the image patch classification score of the user account.

[0147] The image patch classification score is used to represent the probability that the tissue corresponding to the target image patch has undergone cancer metastasis. Optionally, the value range of the image patch classification score is the interval [0, 1].

[0148] Step 606: In response to the image patch classification score being greater than the classification threshold, determine that the target tissue corresponding to the target medical image patch has undergone cancer metastasis.

[0149] Optionally, steps 601 to 605 are repeated to determine the cancer metastasis status corresponding to each medical image patch; a metastasis status heatmap is generated based on the cancer metastasis status corresponding to each medical image patch. For example, such as... Figure 7 As shown, medical image 701 corresponds to a non-metastatic state of the target tissue, and heatmap 702 is the heatmap corresponding to medical image 701. Medical image 703 corresponds to a target tissue where cancer has metastasized, and heatmap 704 is the heatmap corresponding to medical image 703.

[0150] In summary, this embodiment can provide the cancer metastasis status of the tissue corresponding to the target medical image block in the medical image. Since this technical solution integrates features from different modalities, the overall features include information from different modalities. Therefore, the overall features can more comprehensively assess the cancer metastasis status of the target tissue, and the prediction results are more accurate.

[0151] The embodiments of this application can also be used to determine the cancer metastasis status of lymph node tissue:

[0152] Figure 8 This illustration shows a flowchart of a method for determining cancer metastasis in lymph node tissue according to an embodiment of this application. The method can be... Figure 1 The method, executed by the terminal 120, server 140, or other computer device shown, includes the following steps:

[0153] Step 801: Obtain electronic lymph node images and electronic clinical indicators of the user account's lymph node tissue.

[0154] The target tissue can be any tissue within a living organism. For example, a target tissue could be a lymph node containing cancerous tissue. Technicians can select the target tissue based on their specific needs.

[0155] Optionally, the electronic lymph node image is an electronically processed pathological image or gross section pathological image used to test for lymph node metastasis. For example, to determine whether lymph node metastasis has occurred in colorectal cancer, a section is taken from the colorectal area to determine whether the cancerous growth in the colorectal area has metastasized to the lymph nodes.

[0156] In one optional implementation, the electronic clinical indicators include at least one of age, CEA, CA125, CA19-9, and CFP. In practice, technicians can adjust the electronic clinical indicators according to actual needs.

[0157] In one alternative implementation, electronic lymph node images and electronic clinical indicators of the target tissue of the user account are obtained from the medical system.

[0158] Step 802: Determine the lymph node image features of the user account based on the electronic lymph node image of the user account.

[0159] Lymph node image features are used to represent features of electronic lymph node images that are related to the cancer metastasis status of lymph nodes.

[0160] Step 803: Determine the clinical characteristics of the user account based on the electronic clinical indicators of the user account.

[0161] Clinical features are used to represent characteristics related to the cancer metastasis status of the target tissue in relation to electronic clinical indicators. In one optional implementation, a fully connected feedforward network is invoked to process the electronic clinical indicators of the user account and output the clinical features of the user account.

[0162] Step 804: Integrate lymph node image features and clinical features to generate overall features of the user account.

[0163] The overall features are used to represent the features corresponding to the cancer metastasis status of the target tissue under multiple modalities.

[0164] Optionally, methods for fusing image features and clinical features include at least one of the following:

[0165] 1. Fusion of image and clinical features based on gating attention mechanism and tensor fusion mechanism;

[0166] 2. Fusing image features and clinical features through feature addition;

[0167] 3. Fusing image features and clinical features by concatenating feature vectors;

[0168] 4. Utilize neural networks to fuse image features and clinical features.

[0169] Taking the fusion of image features and clinical features based on gating attention mechanism and tensor fusion mechanism as an example, the steps include: based on gating attention mechanism, performing attention weighting on image features and clinical features to obtain weighted image features and weighted clinical features; based on tensor fusion mechanism, fusing weighted image features and weighted clinical features to obtain overall features.

[0170] Step 805: Classify the overall features and generate the lymph node classification score for the user account.

[0171] In one specific implementation, at least two fully connected layers are invoked to process the overall features and output the lymph node classification score of the user account.

[0172] The lymph node classification score is used to represent the probability that a user's lymph node tissue is in a state of cancer metastasis. Optionally, the lymph node classification score ranges from [0, 1].

[0173] Step 806: In response to a lymph node classification score greater than the lymph node classification threshold, determine that the user account's lymph node tissue has metastasized to cancer.

[0174] The lymph node classification threshold can be set by technical personnel. Optionally, in response to a lymph node classification score being less than or equal to the lymph node classification threshold, it is determined that the user account's lymph node tissue has not metastasized.

[0175] In one specific implementation, a classifier is invoked to process the lymph node classification score and output the lymph node classification result. In response to the lymph node classification result being the first lymph node classification result, it is determined that the user account's lymph node tissue has undergone cancer metastasis. For example, the lymph node classification score is 0.7. Specifically, when the lymph node classification score is greater than 0.5, the classifier outputs 1; when the lymph node classification score is less than or equal to 0.5, the classifier outputs 0. Therefore, the obtained lymph node classification result is 1, which confirms that the lymph node tissue has undergone cancer metastasis.

[0176] In summary, this embodiment can integrate lymph node image features and clinical features of a user account to generate overall features, and then determine the cancer metastasis status of the user account's lymph node tissue based on these overall features. Because this technical solution integrates features from different modalities, the overall features include information from different modalities, thus enabling a more comprehensive assessment of the cancer metastasis status of lymph node tissue and resulting in more accurate predictions.

[0177] The embodiments of this application can also be used to determine the tumor type of a target tissue:

[0178] Figure 9 A flowchart illustrating a method for determining the tumor type of a target tissue according to an embodiment of this application is shown. This method can be... Figure 1 The method, executed by the terminal 120, server 140, or other computer device shown, includes the following steps:

[0179] Step 901: Obtain electronic medical images and electronic clinical indicators of tumor tissue from the user account.

[0180] The target tissue can be any tissue within a living organism. For example, a target tissue could be a lymph node containing cancerous tissue. Technicians can select the target tissue based on their specific needs.

[0181] Optionally, the electronic medical image is an electronic biopsy pathology image or an electronic gross section pathology image.

[0182] In one alternative implementation, when the target tissue is a lymph node of cancerous tissue, the electronic clinical indicators include at least one of age, CEA, CA125, CA19-9, and CFP. In practice, technicians can adjust the electronic clinical indicators according to actual needs.

[0183] In one alternative implementation, electronic medical images and electronic clinical indicators of the target organization are obtained from the medical system from the user account.

[0184] Step 902: Determine the image features of the user account based on the electronic medical image of the user account.

[0185] Image features are used to represent features associated with the cancer metastasis status of a target tissue in electronic medical images.

[0186] Step 903: Determine the clinical characteristics of the user account based on the electronic clinical indicators of the user account.

[0187] Clinical features are used to represent characteristics related to the cancer metastasis status of the target tissue in relation to electronic clinical indicators. In one optional implementation, a fully connected feedforward network is invoked to process the electronic clinical indicators of the user account and output the clinical features of the user account.

[0188] Step 904: Fuse image features and clinical features to generate overall features of the user account.

[0189] The overall features are used to represent the features corresponding to the cancer metastasis status of the target tissue under multiple modalities.

[0190] Optionally, methods for fusing image features and clinical features include at least one of the following:

[0191] 1. Fusion of image and clinical features based on gating attention mechanism and tensor fusion mechanism;

[0192] 2. Fusing image features and clinical features through feature addition;

[0193] 3. Fusing image features and clinical features by concatenating feature vectors;

[0194] 4. Utilize neural networks to fuse image features and clinical features.

[0195] Taking the fusion of image features and clinical features based on gating attention mechanism and tensor fusion mechanism as an example, the steps include: based on gating attention mechanism, performing attention weighting on image features and clinical features to obtain weighted image features and weighted clinical features; based on tensor fusion mechanism, fusing weighted image features and weighted clinical features to obtain overall features.

[0196] Step 905: Classify the overall features and generate a tumor classification score for the user account.

[0197] In one specific implementation, at least two fully connected layers are invoked to process the overall features and output the tumor classification score of the user account.

[0198] The tumor classification score is used to represent the probability that a user account's tumor tissue is of type I. Optionally, the cancer classification score can range from [0, 1].

[0199] Step 906: In response to a tumor classification score greater than the tumor classification threshold, determine that the user account's tumor tissue belongs to the first type.

[0200] The tumor classification threshold can be set by technical personnel. Optionally, in response to a tumor classification score less than or equal to the tumor classification threshold, it is determined that the user account's tumor tissue belongs to the second type. For example, the first type is malignant tumors, and the second type is benign tumors.

[0201] In one specific implementation, a classifier is invoked to process the tumor classification score and output a tumor classification result. In response to the tumor classification result being a first tumor classification result, it is determined that the user account's tumor tissue belongs to the first type. For example, the tumor classification score is 0.7. Specifically, when the tumor classification score is greater than 0.5, the classifier outputs 1; when the tumor classification score is less than or equal to 0.5, the classifier outputs 0. Therefore, the obtained tumor classification result is 1, confirming that the tumor tissue belongs to the first type.

[0202] In summary, this embodiment can fuse image features and clinical features of a user account to generate overall features, and then determine the type of tumor tissue of the user account based on the overall features. Because this technical solution integrates features from different modalities, the overall features include information from different modalities, thus enabling a more comprehensive classification of tumor tissue and resulting in more accurate classification results.

[0203] Figure 10 This illustration shows a flowchart of a training method for a model predicting cancer metastasis in a target tissue, according to an embodiment of this application. The method can be... Figure 1 The method, executed by the terminal 120, server 140, or other computer device shown, predicts cancer metastasis in target tissue using a model comprising an image feature extraction module, a clinical feature extraction module, and a feature fusion module. The method includes the following steps:

[0204] Step 1001: Obtain the sample training set, which includes the real annotations of sample user accounts, sample electronic medical images of sample user accounts, and sample electronic clinical indicators.

[0205] The target tissue can be any tissue within a living organism. For example, a target tissue could be a lymph node containing cancerous tissue. Technicians can select the target tissue based on their specific needs.

[0206] Optionally, the electronic medical image of the sample is an electronic biopsy pathology image or an electronic gross section pathology image.

[0207] In one alternative implementation, when the target tissue is a lymph node of cancerous tissue, the electronic clinical indicators include at least one of age, CEA, CA125, CA19-9, and CFP. In practice, technicians can adjust the electronic clinical indicators according to actual needs.

[0208] In one alternative implementation, sample electronic medical images and sample electronic clinical indicators of the target organization are obtained from the medical system from the sample user account.

[0209] Step 1002: Call the image feature extraction module to determine the features of the sample image based on the sample medical image.

[0210] Sample image features are used to represent features in electronic medical images of samples that are related to the cancer metastasis status of the target tissue.

[0211] Step 1003: Call the clinical feature extraction module to determine the clinical features of the sample based on the sample's clinical indicators.

[0212] Clinical features are used to represent the characteristics related to the cancer metastasis status of the target tissue in electronic clinical indicators. In one optional implementation, a fully connected feedforward network is invoked to process the sample electronic clinical indicators of the sample user account, outputting the sample clinical features of the sample user account. Optionally, when clinical indicators are missing, the corresponding clinical features are also missing. A chain equation-based multiple imputation method is used to imputate the missing clinical features.

[0213] Step 1004: Call the feature fusion module to fuse sample image features and sample clinical features to generate the overall sample features of the sample user account.

[0214] The overall features of the sample are used to represent the features corresponding to the cancer metastasis status of the target tissue under multiple modalities.

[0215] Optionally, the method for fusing sample image features and sample clinical features includes at least one of the following:

[0216] 1. Fusion of image and clinical features based on gating attention mechanism and tensor fusion mechanism;

[0217] 2. Fusing image features and clinical features through feature addition;

[0218] 3. Fusing image features and clinical features by concatenating feature vectors;

[0219] 4. Utilize neural networks to fuse image features and clinical features.

[0220] Step 1005: Call the feature fusion module to classify the overall features of the sample and generate the sample classification score for the sample user account.

[0221] In one specific implementation, at least two fully connected layers are invoked to process the overall features and output the classification score of the user account.

[0222] The classification score is used to represent the probability that a user account's target tissue has metastasized to cancer. Optionally, the classification score can range from [0, 1].

[0223] Step 1006: Train the model for predicting cancer metastasis in the target tissue based on the difference between the sample classification score and the true label.

[0224] Optionally, the model for predicting cancer metastasis in the target tissue can be trained using an error backpropagation algorithm based on the difference between the sample classification score and the true label.

[0225] In summary, this embodiment provides a method for training a model to predict cancer metastasis in a target tissue. The trained model can determine the cancer metastasis status of the target tissue based on overall characteristics, resulting in a more accurate prediction.

[0226] The following are device embodiments of this application. For details not described in detail in the device embodiments, please refer to the corresponding descriptions in the above method embodiments. They will not be repeated here.

[0227] Figure 11 A schematic diagram of a predictive device for cancer metastasis in a target tissue, provided in an exemplary embodiment of this application, is shown. This device can be implemented as all or part of a computer device through software, hardware, or a combination of both. The device 1100 includes:

[0228] The acquisition module 1101 is used to acquire electronic medical images and electronic clinical indicators of the user account;

[0229] The feature extraction module 1102 is used to determine the image features of the user account based on the electronic medical image of the user account; and to determine the clinical features of the user account based on the electronic clinical indicators of the user account.

[0230] The fusion module 1103 is used to fuse the image features and the clinical features to generate the overall features of the user account;

[0231] The fusion module 1103 is also used to classify the overall features and generate a classification score for the user account;

[0232] The fusion module 1103 is further configured to determine that the target tissue of the user account has undergone cancer metastasis in response to the classification score being greater than the classification threshold; wherein the image features are used to represent features related to the cancer metastasis status of the electronic medical image and the target tissue, and the clinical features are used to represent features related to the cancer metastasis status of the electronic clinical indicators and the target tissue.

[0233] In an optional implementation, the fusion module 1103 is further configured to perform attention weighting on the image features and the clinical features based on a gating attention mechanism to obtain weighted image features and weighted clinical features; and to fuse the weighted image features and the weighted clinical features based on a tensor fusion mechanism to obtain the overall features.

[0234] In an optional implementation, the fusion module 1103 is further configured to perform attention weighting on the image features and the clinical features based on the gating attention mechanism to determine the image weights and clinical weights; calculate the product of the image weights and the image features to obtain the weighted image features; and calculate the product of the clinical weights and the clinical features to obtain the weighted clinical features.

[0235] In an optional implementation, the fusion module 1103 is further configured to calculate the Kronecker product of the weighted image features and the weighted clinical features to obtain the tensor fused features; and to merge the weighted image features and the weighted clinical features into the tensor fused features to obtain the overall features.

[0236] In an optional implementation, the feature extraction module 1102 is further configured to divide the electronic medical image into n medical image blocks, where n is a positive integer; call an image feature extraction network to process the n medical image blocks and output n image block features corresponding to the n medical image blocks; filter the n image block features to obtain filtered n image block features; and fuse the filtered n image block features into the image features based on an attention mechanism.

[0237] In an optional implementation, the feature extraction module 1102 is further configured to filter the n image patch features based on histograms and maximum mean differences to obtain the filtered n image patch features.

[0238] In an optional embodiment, the feature extraction module 1102 is further configured to divide the region of interest in the electronic medical image into n medical image blocks, wherein the overlap ratio between each of the n medical image blocks and the region of interest is greater than a preset ratio.

[0239] In an optional implementation, the feature extraction module 1102 is further configured to: extract a target medical image block from the n medical image blocks; call the image feature extraction network to process the target medical image block and output the target image block features corresponding to the target medical image block; the fusion module 1103 is further configured to: fuse the target image block features and the clinical features to generate the overall image block features of the user account; classify the overall image block features to generate the image block classification score of the user account; and, in response to the image block classification score being greater than the classification threshold, determine that the target tissue corresponding to the target medical image block has undergone cancer metastasis.

[0240] In an optional implementation, the feature extraction module 1102 is further configured to invoke a fully connected feedforward network to process the electronic clinical indicators of the user account and output the clinical features of the user account.

[0241] In summary, this embodiment can fuse image features and clinical features of a user account to generate overall features, and then determine the cancer metastasis status of the target tissue based on these overall features. Because this technical solution integrates features from different modalities, the overall features include information from different modalities, thus enabling a more comprehensive assessment of the cancer metastasis status of the target tissue and resulting in more accurate predictions.

[0242] The following are device embodiments of this application. For details not described in detail in the device embodiments, please refer to the corresponding descriptions in the above method embodiments. They will not be repeated here.

[0243] Figure 12 This illustration shows a schematic diagram of a training apparatus for a model predicting cancer metastasis in a target tissue, provided in an exemplary embodiment of this application. The apparatus can be implemented as all or part of a computer device through software, hardware, or a combination of both. The apparatus 1200 includes:

[0244] The sample acquisition module 1201 is used to acquire a sample training set, which includes real annotations of sample user accounts, sample electronic medical images of sample user accounts, and sample electronic clinical indicators.

[0245] The sample feature extraction module 1202 is used to call the image feature extraction module to determine the sample image features based on the sample medical image; and to call the clinical feature extraction module to determine the sample clinical features based on the sample clinical indicators.

[0246] The sample fusion module 1203 is used to call the feature fusion module to fuse the sample image features and the sample clinical features to generate the overall sample features of the sample user account;

[0247] The sample fusion module 1203 is also used to call the feature fusion module to classify the overall features of the sample and generate a sample classification score for the sample user account.

[0248] The training module 1204 is used to train the model for predicting cancer metastasis in the target tissue based on the difference between the sample classification score and the true label; wherein, the sample image features are used to represent the features related to the cancer metastasis status of the sample electronic medical image and the target tissue, and the sample clinical features are used to represent the features related to the cancer metastasis status of the sample electronic clinical indicators and the target tissue.

[0249] In summary, this embodiment can fuse image features and clinical features of a user account to generate overall features, and then determine the cancer metastasis status of the target tissue based on these overall features. Because this technical solution integrates features from different modalities, the overall features include information from different modalities, thus enabling a more comprehensive assessment of the cancer metastasis status of the target tissue and resulting in more accurate predictions.

[0250] Figure 13 This is a schematic diagram illustrating the structure of a computer device according to an exemplary embodiment. The computer device 1300 includes a Central Processing Unit (CPU) 1301, a system memory 1304 including Random Access Memory (RAM) 1302 and Read-Only Memory (ROM) 1303, and a system bus 1305 connecting the system memory 1304 and the CPU 1301. The computer device 1300 also includes a basic input / output system (I / O system) 1306 to facilitate information transfer between various devices within the computer device, and a mass storage device 1307 for storing the operating system 1313, application programs 1314, and other program modules 1315.

[0251] The basic input / output system 1306 includes a display 1308 for displaying information and an input device 1309 for user input, such as a mouse or keyboard. Both the display 1308 and the input device 1309 are connected to the central processing unit 1301 via an input / output controller 1310 connected to the system bus 1305. The basic input / output system 1306 may also include the input / output controller 1310 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1310 also provides output to a display screen, printer, or other types of output devices.

[0252] The mass storage device 1307 is connected to the central processing unit 1301 via a mass storage controller (not shown) connected to the system bus 1305. The mass storage device 1307 and its associated computer device-readable media provide non-volatile storage for the computer device 1300. That is, the mass storage device 1307 may include computer device-readable media (not shown), such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0253] Without loss of generality, the computer device readable medium may include computer device storage media and communication media. Computer device storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer device readable instructions, data structures, program modules, or other data. Computer device storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM, digital video disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer device storage media are not limited to the above-mentioned types. The system memory 1304 and mass storage device 1307 described above can be collectively referred to as memory.

[0254] According to various embodiments of this disclosure, the computer device 1300 can also be connected to a remote computer device on a network, such as the Internet. That is, the computer device 1300 can be connected to the network 1311 via a network interface unit 1312 connected to the system bus 1305, or it can use the network interface unit 1312 to connect to other types of networks or remote computer device systems (not shown).

[0255] The memory also includes one or more programs stored in the memory. The central processing unit 1301 executes the one or more programs to implement all or part of the steps of the above-mentioned method for predicting cancer metastasis in the target tissue, or the training method for the model for predicting cancer metastasis in the target tissue.

[0256] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the prediction method for cancer metastasis of a target tissue or the training method for a model for predicting cancer metastasis of a target tissue provided in the above-described method embodiments.

[0257] This application also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method for predicting cancer metastasis in a target tissue provided in the above method embodiments, or the method for training a model for predicting cancer metastasis in a target tissue.

[0258] This application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method for predicting cancer metastasis in a target tissue, or the method for training a model for predicting cancer metastasis in a target tissue, as provided in the above embodiments.

[0259] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0260] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0261] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for predicting cancer metastasis in a target tissue, characterized in that, The method includes: Obtain the digitized biopsy pathology images and digitized clinical indicators of the user account; The digitized biopsy pathology images are divided into n medical image blocks, where n is a positive integer. An image feature extraction network is invoked to process the n medical image blocks, outputting n image block features corresponding to the n medical image blocks. A preset position feature is retained among the n image block features to obtain the filtered n image block features. Based on an attention mechanism, the filtered n image block features are fused into the image features. The preset position feature is determined through the following steps: determining the n sample image block features of the digitized biopsy pathology images of each sample user account; obtaining the histogram corresponding to the i-th feature under each sample user account based on the i-th feature among the n sample image block features of each sample user account; determining the maximum mean difference of the i-th feature between the sample user account group with cancer metastasis and the sample user account group without cancer metastasis based on the histogram and the ground truth annotation; and using the i-th feature as the preset position feature in response to the maximum mean difference of the i-th feature being greater than or equal to the maximum mean difference threshold. The clinical characteristics of the user account are determined based on the electronic clinical indicators of the user account. Based on a gated attention mechanism, the image features and clinical features are weighted by attention to determine image weights and clinical weights; the product of the image weights and the image features is calculated to obtain weighted image features; the product of the clinical weights and the clinical features is calculated to obtain weighted clinical features; the Kronecker product of the weighted image features and the weighted clinical features is calculated to obtain tensor-fused features; the weighted image features and the weighted clinical features are merged into the tensor-fused features to obtain overall features; the overall features are classified to generate a classification score for the user account. In response to the classification score being greater than the classification threshold, it is determined that the target tissue of the user account has undergone cancer metastasis. Take a target medical image block from the n medical image blocks; call the image feature extraction network to process the target medical image block and output the target image block features corresponding to the target medical image block; fuse the target image block features and the clinical features to generate the overall image block features of the user account; classify the overall image block features to generate the image block classification score of the user account; in response to the image block classification score being greater than the classification threshold, determine that the target tissue corresponding to the target medical image block has undergone cancer metastasis; determine the cancer metastasis status corresponding to each medical image block; generate a metastasis status heatmap based on the cancer metastasis status corresponding to each medical image block; The image features are used to represent the features related to the cancer metastasis status of the target tissue in the digitized biopsy pathology image, and the clinical features are used to represent the features related to the cancer metastasis status of the target tissue in the digitized clinical indicators.

2. The method according to claim 1, characterized in that, The process of dividing the digitized biopsy pathology image into n medical image blocks includes: The region of interest in the digitized biopsy pathology image is divided into n medical image blocks, and the overlap ratio between each of the n medical image blocks and the region of interest is greater than a preset ratio.

3. The method according to claim 1 or 2, characterized in that, The step of determining the clinical characteristics of a user account based on its electronic clinical indicators includes: A fully connected feedforward network is invoked to process the electronic clinical indicators of the user account and output the clinical characteristics of the user account.

4. A training method for a model predicting cancer metastasis in a target tissue, characterized in that, The model for predicting cancer metastasis in target tissues includes an image feature extraction module, a clinical feature extraction module, and a feature fusion module. The method includes: Obtain a sample training set, which includes the real annotation of the sample user account, the biopsy pathological image of the sample user account after digitization, and the sample electronic clinical indicators. The image feature extraction module is invoked to divide the digitized biopsy pathology image into n sample medical image blocks, where n is a positive integer. An image feature extraction network is invoked to process the n sample medical image blocks, outputting n sample image block features corresponding to the n sample medical image blocks. The n sample image block features of the digitized biopsy pathology image for each sample user account are determined. Based on the i-th feature among the n sample image block features for each sample user account, a histogram corresponding to the i-th feature under each sample user account is obtained. Based on the histogram and the ground truth annotation, the maximum mean difference of the i-th feature between the sample user account group with cancer metastasis and the sample user account group without cancer metastasis is determined. In response to the maximum mean difference of the i-th feature being greater than or equal to a maximum mean difference threshold, the i-th feature is used as a preset feature. The preset feature among the n sample image block features is retained to obtain the filtered n sample image block features. Based on an attention mechanism, the filtered n image block features are fused into sample image features. The clinical feature extraction module is invoked to determine the clinical features of the sample based on the electronic clinical indicators of the sample. The feature fusion module is invoked, and based on a gated attention mechanism, attention weights are applied to the sample image features and the sample clinical features to determine image weights and clinical weights. The product of the image weights and the sample image features is calculated to obtain weighted sample image features. The product of the clinical weights and the sample clinical features is calculated to obtain weighted sample clinical features. The Kronecker product of the weighted sample image features and the weighted sample clinical features is calculated to obtain tensor-fused features. The weighted sample image features and the weighted sample clinical features are then merged into the tensor-fused features to obtain the overall sample features. The feature fusion module is invoked to classify the overall features of the sample and generate a sample classification score for the user account of the sample. The model for predicting cancer metastasis in the target tissue is trained based on the difference between the sample classification score and the true label. Wherein, the sample image features are used to represent the features related to the cancer metastasis status of the target tissue in the digitized biopsy pathological image of the sample; and the sample clinical features are used to represent the features related to the cancer metastasis status of the target tissue in the digitized clinical indicators of the sample. The image feature extraction network and the feature fusion module are invoked in the following steps: a target medical image block is selected from n medical image blocks; the image feature extraction network is invoked to process the target medical image block, outputting target image block features corresponding to the target medical image block; the feature fusion module is invoked to fuse the target image block features and clinical features to generate overall image block features for the user account; the overall image block features are classified to generate an image block classification score for the user account; in response to the image block classification score being greater than a classification threshold, it is determined that the target tissue corresponding to the target medical image block has undergone cancer metastasis; the cancer metastasis status corresponding to each medical image block is determined; and a metastasis status heatmap is generated based on the cancer metastasis status corresponding to each medical image block.

5. A device for predicting cancer metastasis in a target tissue, characterized in that, The device includes: The acquisition module is used to acquire the digitized biopsy pathology images and digitized clinical indicators of the user account; The feature extraction module is used to divide the digitized biopsy pathology image into n medical image blocks, where n is a positive integer; call an image feature extraction network to process the n medical image blocks and output n image block features corresponding to the n medical image blocks; retain preset position features in the n image block features to obtain the filtered n image block features; and fuse the filtered n image block features into the image features based on an attention mechanism; the preset position features are determined through the following steps: determining n sample digitized biopsy pathology images of each sample user account. This image patch feature; based on the i-th feature among the n sample image patch features of each sample user account, obtain the histogram corresponding to the i-th feature under each sample user account; based on the histogram and the true annotation, determine the maximum mean difference of the i-th feature between the sample user account group with cancer metastasis and the sample user account group without cancer metastasis; in response to the maximum mean difference of the i-th feature being greater than or equal to the maximum mean difference threshold, use the i-th feature as a preset feature; determine the clinical characteristics of the user account based on the electronic clinical indicators of the user account; The fusion module is used to perform attention weighting on the image features and clinical features based on a gated attention mechanism to determine image weights and clinical weights; calculate the product of the image weights and the image features to obtain weighted image features; calculate the product of the clinical weights and the clinical features to obtain weighted clinical features; calculate the Kronecker product of the weighted image features and the weighted clinical features to obtain tensor-fused features; and merge the weighted image features and the weighted clinical features into the tensor-fused features to obtain the overall features. The fusion module is also used to classify the overall features and generate a classification score for the user account; The fusion module is further configured to determine that the target tissue of the user account has cancer metastasis in response to the classification score being greater than the classification threshold; wherein the image features are used to represent the features related to the cancer metastasis status of the target tissue in the digitized biopsy pathology image, and the clinical features are used to represent the features related to the cancer metastasis status of the target tissue in the digitized clinical indicators; The feature extraction module is further configured to: extract a target medical image block from the n medical image blocks; call the image feature extraction network to process the target medical image block and output the target image block features corresponding to the target medical image block; the fusion module is further configured to: fuse the target image block features and the clinical features to generate the overall image block features of the user account; classify the overall image block features to generate the image block classification score of the user account; in response to the image block classification score being greater than a classification threshold, determine that the target tissue corresponding to the target medical image block has undergone cancer metastasis; determine the cancer metastasis status corresponding to each medical image block; and generate a metastasis status heatmap based on the cancer metastasis status corresponding to each medical image block.

6. A training device for a model predicting cancer metastasis in a target tissue, characterized in that, The model for predicting cancer metastasis in target tissues includes an image feature extraction module, a clinical feature extraction module, and a feature fusion module. The device includes: The sample acquisition module is used to acquire a sample training set, which includes the real annotation of the sample user account, the biopsy pathological image of the sample user account after digitization, and the sample electronic clinical indicators. The sample feature extraction module is used to divide the digitized biopsy pathology images of the samples into n sample medical image blocks, where n is a positive integer; call the image feature extraction network to process the n sample medical image blocks and output n sample image block features corresponding to the n sample medical image blocks; determine the n sample image block features of the digitized biopsy pathology images of each sample user account; obtain the histogram corresponding to the i-th feature under each sample user account based on the i-th feature in the n sample image block features of each sample user account; and calculate the histogram based on the histogram and the true label. Note that the maximum mean difference of the i-th feature is determined between the sample user account group with cancer metastasis and the sample user account group without cancer metastasis; in response to the maximum mean difference of the i-th feature being greater than or equal to the maximum mean difference threshold, the i-th feature is used as a preset feature; the preset feature is retained among the n sample image patch features to obtain the filtered n sample image patch features; based on the attention mechanism, the filtered n image patch features are fused into sample image features; the clinical feature extraction module is called to determine the sample clinical features according to the sample electronic clinical indicators; The sample fusion module is used to call the feature fusion module, and based on a gated attention mechanism, to perform attention weighting on the sample image features and the sample clinical features to determine image weights and clinical weights; calculate the product of the image weights and the sample image features to obtain weighted sample image features; calculate the product of the clinical weights and the sample clinical features to obtain weighted sample clinical features; calculate the Kronecker product of the weighted sample image features and the weighted sample clinical features to obtain tensor-fused features; and merge the weighted sample image features and the weighted sample clinical features into the tensor-fused features to obtain the overall sample features. The sample fusion module is also used to call the feature fusion module to classify the overall features of the sample and generate a sample classification score for the sample user account. The training module is used to train the model for predicting cancer metastasis in the target tissue based on the difference between the sample classification score and the true annotation; wherein, the sample image features are used to represent the features related to the cancer metastasis status of the target tissue in the digitized biopsy pathology image of the sample, and the sample clinical features are used to represent the features related to the cancer metastasis status of the target tissue in the digitized clinical indicators of the sample. The image feature extraction network and the feature fusion module are invoked in the following steps: a target medical image block is selected from n medical image blocks; the image feature extraction network is invoked to process the target medical image block, outputting target image block features corresponding to the target medical image block; the feature fusion module is invoked to fuse the target image block features and clinical features to generate overall image block features for the user account; the overall image block features are classified to generate an image block classification score for the user account; in response to the image block classification score being greater than a classification threshold, it is determined that the target tissue corresponding to the target medical image block has undergone cancer metastasis; the cancer metastasis status corresponding to each medical image block is determined; and a metastasis status heatmap is generated based on the cancer metastasis status corresponding to each medical image block.

7. A computer device, characterized in that, The computer device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method for predicting cancer metastasis in a target tissue as described in any one of claims 1 to 3, or the method for training a model for predicting cancer metastasis in a target tissue as described in claim 4.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one piece of program code, which is loaded and executed by a processor to implement the method for predicting cancer metastasis in a target tissue as described in any one of claims 1 to 3, or the method for training a model for predicting cancer metastasis in a target tissue as described in claim 4.

9. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, they implement the method for predicting cancer metastasis in the target tissue as described in any one of claims 1 to 3, or the training method for the model for predicting cancer metastasis in the target tissue as described in claim 4.

Images