Gastrointestinal tract tumor image multi-attribute classification method and device based on multi-modal fusion and medium

Abstract

The invention discloses a gastrointestinal tumor image multi-attribute classification method and device based on multi-modal fusion, and a medium. The method comprises the following steps: constructing a multi-modal gastrointestinal tumor data set; performing hierarchical operation on the data set; constructing an image feature extraction network, and respectively extracting EUS image and WLE image features in the data set by using the image feature extraction network; constructing a multi-attribute classification network, performing multi-attribute classification training on the tumor attributes by using the image features, and after the training is completed, predicting the tumor image to be detected by using the network to obtain the tumor attributes after network prediction completion; and performing cross-modal attention fusion and tumor property classification under lesion attribute guidance, and finally outputting tumor property classification of the to-be-detected tumor image. According to the method, through cross-modal attention fusion operation, the model learns complementary contents in the EUS mode and the WLE mode, the attributes of tumor deletion are supplemented, and the properties of the tumor of the patient are classified by utilizing the supplemented attribute data, so that the method is used for assisting a diagnosis result.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal medical image attribute classification, and in particular to a multi-attribute classification method, device and medium for gastrointestinal tumor images based on multimodal fusion. Background Art

[0002] Gastrointestinal stromal tumor (GIST) is a common gastrointestinal and mesenteric tumor and is considered a potential malignant tumor. In endoscopic ultrasound (EUS) imaging, the imaging features of GIST and other tumors such as leiomyoma overlap, making it difficult for endoscopists to distinguish the pathological nature of the tumor. It is also very difficult for doctors to diagnose gastrointestinal submucosal tumors under white light endoscopy (WLE).

[0003] The existing single-modality EUS model does not meet clinical requirements for GIST recognition accuracy, while EUS combined with WLE imaging can provide powerful complementary diagnostic information, which will be more helpful for the diagnosis of GISTs. At the same time, in the process of diagnosing tumors, doctors will use EUS and WLE imaging to diagnose the nature of the tumor and generate a formatted tumor attribute diagnosis report. Therefore, the model's recognition of tumor attributes is helpful for the final tumor property identification. At the same time, the obtained tumor attributes can also be used to generate a formatted diagnosis report, improving the model's interpretability. Summary of the invention

[0004] In order to solve the problems existing in the prior art, the present invention provides a multi-attribute classification method, device and medium for gastrointestinal tumor images based on multimodal fusion, which can classify tumor attributes such as echo size, origin level, echo heterogeneity, boundary condition, tumor shape and protrusion direction of gastrointestinal tumors, and further classify the tumor properties based on these attributes, thereby solving the problems mentioned in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a multi-attribute classification method for gastrointestinal tumor images based on multimodal fusion, comprising the following steps:

[0006] S1. Construct a multimodal gastrointestinal tumor dataset.

[0007] S2, perform layered operations on the data set;

[0008] S3, construct an image feature extraction network, and use the image feature extraction network to extract the features of EUS images and WLE images in the dataset respectively;

[0009] S4, constructing a multi-attribute classification network, using image features to perform multi-attribute classification training on tumor attributes, after the training is completed, using the network to predict the tumor image to be tested, and obtaining the tumor attributes after network prediction completion;

[0010] S5. Cross-modal attention fusion and tumor property classification guided by lesion attributes: The network-predicted and completed tumor attributes are used to guide the attention of image features and perform cross-modal attention fusion on EUS and WLE modalities to obtain the vector after cross-modal attention fusion. The vector then passes through the three-layer fully connected layer of the multi-attribute classification network and finally outputs the tumor property classification of the tumor image to be tested for auxiliary diagnosis results.

[0011] Preferably, in step S1, a multimodal data set of a patient is collected, the data set including an EUS image, a WLE image and a text description of the two images of the patient, and all data are paired according to a combination of EUS image, EUS text, WLE image and WLE text;

[0012] Then, image and text preprocessing is performed: for images, resizing and standardization operations are performed to make them consistent with the network input; for text, typos are identified and text format is standardized, and the attribute feature labels of the tumor in the text description are extracted.

[0013] Preferably, in step S2, the data set is stratified using the number of extracted attribute feature labels, specifically including: when both the EUS image attribute label and the WLE image attribute label are not missing, the data is classified as zero-layer data; when the number of missing EUS image attribute labels and WLE image attribute labels is two, the data is classified as second-layer data.

[0014] Preferably, in step S3, the image feature extraction network includes an EUS image feature extraction network and a WLE image feature extraction network, both of which use ImageNet and the trained Resnet18 network as the basic feature extraction skeleton to obtain EUS image features and WLE image features:

[0015] M EUS =R EUS (I EUS )

[0016] M WLE =R WLE (I WLE )

[0017] F EUS =AvgPool(M EUS )

[0018] F WLE =AvgPool(M WLE )

[0019] in Respectively represent the EUS image feature vector and the WLE image feature vector, They represent the EUS image feature extraction network and the WLE image feature extraction network respectively. They represent the EUS image feature map and the WLE image feature map respectively, and AvgPool(·) is the average pooling operation.

[0020] Preferably, in step S4, the multi-attribute classification network includes EUS and WLE attribute classification networks, which are composed of three fully connected layers, wherein EUS has six attributes and WLE has three attributes. Therefore, a total of nine three-layer fully connected layers are used in the attribute classification layer for tumor multi-attribute classification; wherein the first layer is a 512*1024 fully connected layer, the second layer is a 1024*1024 fully connected layer, and the third layer is a 1024*Dim cls The fully connected layer, where Dim cls is the number of classifications for a single attribute of the tumor;

[0021] During training, Adam is used to update the multi-attribute classification network. If all tumor attribute labels can be extracted from the description text, all predicted results and true attribute labels are used to train the model. If the label is missing in the text, the corresponding weight is added during training. The specific loss function is as follows:

[0022]

[0023]

[0024] Among them, n and m represent the number of EUS attributes and the number of WLE attributes respectively, GT represents the true label, and y c Represents the true value of an attribute;

[0025] During training, the model is first trained using the data from the zeroth layer, and then the data from the first and second layers are added in turn to train the model until all layers of data are added for training. Except for the zeroth layer, when each layer of data is added, the model trained in the previous layer of data is first used to predict the missing attribute labels in the data of that layer, and the corresponding pseudo labels are generated at the missing locations. The specific process is as follows:

[0026]

[0027] in Represent the EUS and WLE attributes predicted by the i-th layer model respectively They represent the classification networks of the j-th attribute of EUS and WLE respectively;

[0028] Use the labels predicted by the model to replace the missing labels in this layer of data:

[0029]

[0030] in They represent the jth attribute of the updated EUS and WLE of the i-th layer respectively. The data with the missing parts filled with pseudo labels are added to the training set of the model, and subsequent model training is performed. After the training is completed, the network is used to predict the features of the tumor image to be tested, and the tumor attributes after the network prediction is completed are obtained.

[0031] Preferably, in step S5, the following is specifically included:

[0032] S51, extracting features of the tumor attributes after network prediction completion using the LSTM network to obtain a tumor attribute feature vector of the corresponding image. The specific process is as follows:

[0033] F EAttr =L EUS (Att EUS )

[0034] F WAttr =L WLE (Att WLE )

[0035] in, are EUS attribute feature vector and WLE attribute feature vector respectively, are the EUS attributes and WLE attributes after network prediction and completion, L EUS , L WLE They are EUS attribute feature extractor and WLE attribute feature extractor respectively;

[0036] S52, using the EUS and WLE attribute features to guide attention on the feature maps of the EUS and WLE images, the process is as follows:

[0037] Since the number of channels between the EUS feature map and the EUS report features obtained by feature extraction does not match, we first align the number of channels in the EUS feature map and the length of the report features:

[0038]

[0039] in They are all learnable parameters, aligning the attribute features of EUS and WLE with the image feature channels;

[0040] Then the aligned attribute features Each element in is broadcast to formed in the space Then, all the information is aggregated into the final attribute attention feature map using a 1×1 convolution kernel, as shown below:

[0041]

[0042] Bro(·) represents the broadcast operation, Conv(·) represents the aggregation of multiple channel information into one channel using a 1×1 convolution kernel. is the guided feature map after being guided by attribute features;

[0043] A EUS , A WLE Each element in and M EUS 、M WLE The elements of each channel in are weighted averaged to obtain the image features guided by attribute features, as follows:

[0044]

[0045] in is the final image feature after being guided by attribute features;

[0046] S53, cross-modal feature attention fusion, the specific process includes: first, the EUS features and WLE features guided by attribute features are projected into the bilinear feature space through two low-rank matrices, as shown below:

[0047]

[0048] in is the low-rank projection matrix, is the output of the bilinear layer, ° is the Hadmard product operation, is a vector of all 1s;

[0049] The o low-rank matrices are expanded by dimensionality reduction to obtain a two-dimensional projection matrix:

[0050]

[0051] By using the low-rank projection matrix after dimensionality reduction and combining it with the addition pooling operation, we can get the vector after cross-modal attention fusion:

[0052]

[0053] Where SumPooling(x,k) means using a one-dimensional non-overlapping window of size k to perform an additive pooling operation on x;

[0054] S54, after passing through the three-layer fully connected layer of the multi-attribute classification network, the tumor property classification of the tumor image to be tested is finally output, and the cross entropy loss function used is as follows:

[0055]

[0056] Where N represents the number of samples, y i represents the true label of the tumor property of the i-th sample, F i Represents the fusion feature of the i-th sample.

[0057] On the other hand, to achieve the above-mentioned purpose, the present invention also provides the following technical solution: an electronic device, the electronic device comprising: a processor; and a memory for storing one or more programs;

[0058] When the one or more programs are executed by the processor, the processor executes the multi-attribute classification method for gastrointestinal tumor images based on multi-modal fusion.

[0059] On the other hand, to achieve the above-mentioned purpose, the present invention also provides the following technical solution: a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-attribute classification method for gastrointestinal tumor images based on multimodal fusion.

[0060] The beneficial effects of the present invention are as follows: the present invention can enable the model to learn the complementary contents in the EUS modality and the WLE modality through the cross-modal attention fusion operation, supplement the missing attributes of the tumor through the given EUS images and WLE images, and use the supplemented attribute data to finally classify the nature of the patient's tumor, improve the accuracy of classification through the multi-modal advantages, and assist doctors in further making effective judgments on the patient's condition. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 Schematic diagram of the process steps of a multi-attribute classification method for gastrointestinal tumor images based on multi-modal fusion in an embodiment of the present invention;

[0062] Figure 2 This is a schematic diagram of the structure of an electronic device in an embodiment of the present invention;

[0063] In the figure, 210 is a processor; 220 is a storage. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] The present invention provides a technical solution: a multi-attribute classification method for gastrointestinal tumor images based on multimodal fusion, please refer to Figure 1 , including the following steps:

[0066] S1. Construction of a multimodal gastrointestinal tumor dataset

[0067] Collect multimodal datasets of patients from real clinical scenarios. The datasets include EUS images, WLE images, and text descriptions of the two images. All data are paired according to the combination of EUS images, EUS texts, WLE images, and WLE texts.

[0068] Then, image and text preprocessing is performed: for images, resizing and standardization operations are performed to make them consistent with the network input; for text, typos are identified and text format is standardized, and the attribute feature labels of the tumor in the text description are extracted.

[0069] S2. Perform layered operations on the data set

[0070] The data set is stratified using the number of extracted attribute feature labels, specifically: when both the EUS image attribute label and the WLE image attribute label are not missing, the data is classified as the zeroth layer data; when the number of missing EUS image attribute labels and WLE image attribute labels is two, the data is classified as the second layer data.

[0071] S3. Construct an image feature extraction network and use it to extract the features of EUS images and WLE images in the dataset.

[0072] The image feature extraction network includes an EUS image feature extraction network and a WLE image feature extraction network, both of which use the ImageNet and trained Resnet18 network as the basic feature extraction skeleton to obtain EUS image features and WLE image features:

[0073] M EUS =R EUS (I EUS )

[0074] M WLE =R WLE (I WLE )

[0075] F EUS =AvgPool(M EUS )

[0076] F WLE =AvgPool(M WLE )

[0077] in Respectively represent the EUS image feature vector and the WLE image feature vector, They represent the EUS image feature extraction network and the WLE image feature extraction network respectively. They represent the EUS image feature map and the WLE image feature map respectively, and AvgPool(·) is the average pooling operation.

[0078] S4. Construct a multi-attribute classification network and use image features to perform multi-attribute classification training on tumor attributes. After the training is completed, use the network to predict the tumor image to be tested and obtain the tumor attributes after network prediction completion.

[0079] The multi-attribute classification network includes EUS and WLE attribute classification networks, which are composed of three fully connected layers, where EUS has six attributes and WLE has three attributes. Therefore, a total of nine three-layer fully connected layers are used in the attribute classification layer for tumor multi-attribute classification; the first layer is a 512*1024 fully connected layer, the second layer is a 1024*1024 fully connected layer, and the third layer is a 1024*Dim cls The fully connected layer, where Dim cls is the number of classifications for a single attribute of the tumor;

[0080] During training, Adam is used to update the multi-attribute classification network. If all tumor attribute labels can be extracted from the description text, all predicted results and true attribute labels are used to train the model. If the label is missing in the text, the corresponding weight is added during training. The specific loss function is as follows:

[0081]

[0082]

[0083] Among them, n and m represent the number of EUS attributes and the number of WLE attributes respectively, GT represents the true label, and y c Represents the true value of an attribute;

[0084] During training, the model is first trained using the data from the zeroth layer, and then the data from the first and second layers are added in turn to train the model until all layers of data are added for training. Except for the zeroth layer, when each layer of data is added, the model trained in the previous layer of data is first used to predict the missing attribute labels in the data of that layer, and the corresponding pseudo labels are generated at the missing locations. The specific process is as follows:

[0085]

[0086] in Represent the EUS and WLE attributes predicted by the i-th layer model respectively They represent the classification networks of the j-th attribute of EUS and WLE respectively;

[0087] Use the labels predicted by the model to replace the missing labels in this layer of data:

[0088]

[0089] in They represent the jth attribute of the updated EUS and WLE of the i-th layer respectively. The data with the missing parts filled with pseudo labels are added to the training set of the model, and subsequent model training is performed. After the training is completed, the network is used to predict the features of the tumor image to be tested, and the tumor attributes after the network prediction is completed are obtained.

[0090] S5. Cross-modal attention fusion and tumor property classification guided by lesion attributes: The network-predicted and completed tumor attributes are used to guide the attention of image features and perform cross-modal attention fusion on EUS and WLE modalities to obtain the vector after cross-modal attention fusion. The vector then passes through the three-layer fully connected layer of the multi-attribute classification network and finally outputs the tumor property classification of the tumor image to be tested for auxiliary diagnosis results.

[0091] The details include:

[0092] S51, extracting features of the tumor attributes after network prediction completion using the LSTM network to obtain a tumor attribute feature vector of the corresponding image. The specific process is as follows:

[0093] F EAttr =L EUS (Att EUS )

[0094] F WAttr =L WLE (Att WLE )

[0095] in, are EUS attribute feature vector and WLE attribute feature vector respectively, are the EUS attributes and WLE attributes after network prediction and completion, L EUS , L WLE They are EUS attribute feature extractor and WLE attribute feature extractor respectively;

[0096] S52, using the EUS and WLE attribute features to guide attention on the feature maps of the EUS and WLE images, the process is as follows:

[0097] Since the number of channels between the EUS feature map and the EUS report features obtained by feature extraction does not match, we first align the number of channels in the EUS feature map and the length of the report features:

[0098]

[0099] in They are all learnable parameters, aligning the attribute features of EUS and WLE with the image feature channels;

[0100] Then the aligned attribute features Each element in is broadcast to formed in the space Then, all the information is aggregated into the final attribute attention feature map using a 1×1 convolution kernel, as shown below:

[0101]

[0102] Bro(·) represents the broadcast operation, Conv(·) represents the aggregation of multiple channel information into one channel using a 1×1 convolution kernel. is the guided feature map after being guided by attribute features;

[0103] A EUS , A WLE Each element in and M EUS 、M WLE The elements of each channel in are weighted averaged to obtain the image features guided by attribute features, as follows:

[0104]

[0105] in It is the final image feature guided by the attribute features. This process guides the attention of the feature map of the image through multimodal attribute features, thereby preventing the model from paying attention to other areas outside the lesion, which can effectively improve the diagnostic accuracy of the model.

[0106] S53, cross-modal feature attention fusion, the specific process includes: first, the EUS features and WLE features guided by attribute features are projected into the bilinear feature space through two low-rank matrices, as shown below:

[0107]

[0108] in is the low-rank projection matrix, is the output of the bilinear layer, ° is the Hadmard product operation, is a vector of all 1s;

[0109] The o low-rank matrices are expanded by dimensionality reduction to obtain a two-dimensional projection matrix:

[0110]

[0111]

[0112] By using the low-rank projection matrix after dimensionality reduction and combining it with the addition pooling operation, we can get the vector after cross-modal attention fusion:

[0113]

[0114] Where SumPooling(x,k) means using a one-dimensional non-overlapping window of size k to perform an additive pooling operation on x; through the cross-modal attention fusion operation, the model can learn the complementary content in the EUS modality and the WLE modality, thereby improving the accuracy of diagnosis through multi-modal advantages.

[0115] S54, after passing through the three-layer fully connected layer of the multi-attribute classification network, the tumor property classification of the tumor image to be tested is finally output, and the cross entropy loss function used is as follows:

[0116]

[0117] Where N represents the number of samples, y i represents the true label of the tumor property of the i-th sample, F i Represents the fusion feature of the i-th sample.

[0118] Application verification: The real data set collected from clinical practice was divided into training set and test set in a ratio of 7:3. During training, each layer in the first 5 layers was trained for 30 epochs, and the learning rate was 1×10 -4 , momentum is 0.9; the last layer training epoch is 40, and the learning rate is 1×10 -5 . The model optimization algorithm is SGD, and the final test result model prediction accuracy is 84.4%. When using the model, the collected multimodal images are first converted into 224*224 pixel images, and easy-to-observe tumor attribute features such as tumor color, surface mucosal condition, tumor echo size, origin level, etc. are given. These data are input into the model together. The model can supplement the missing attributes of the tumor through the given EUS images and WLE images. The attribute results can also assist doctors to further effectively judge the patient's condition, and use the supplemented attribute data to make a final diagnosis of the nature of the patient's tumor and give the model's diagnosis results.

[0119] Based on the same inventive concept as the above method embodiment, the present application embodiment also provides an electronic device, such as Figure 2 As shown, the device includes: a processor 210; and a memory 220, for storing one or more programs;

[0120] When the one or more programs are executed by the processor 210, the processor executes the multi-attribute classification method for gastrointestinal tumor images based on multi-modal fusion.

[0121] The multi-attribute classification method for gastrointestinal tumor images based on multimodal fusion specifically includes the following:

[0122] S1. Construct a multimodal gastrointestinal tumor dataset.

[0123] S2, perform layered operations on the data set;

[0124] S3, construct an image feature extraction network, and use the image feature extraction network to extract the features of EUS images and WLE images in the dataset respectively;

[0125] S4, constructing a multi-attribute classification network, using image features to perform multi-attribute classification training on tumor attributes, after the training is completed, using the network to predict the tumor image to be tested, and obtaining the tumor attributes after network prediction completion;

[0126] S5. Cross-modal attention fusion and tumor property classification guided by lesion attributes: The network-predicted and completed tumor attributes are used to guide the attention of image features and perform cross-modal attention fusion on EUS and WLE modalities to obtain the vector after cross-modal attention fusion. The vector then passes through the three-layer fully connected layer of the multi-attribute classification network and finally outputs the tumor property classification of the tumor image to be tested for auxiliary diagnosis results.

[0127] Based on the same inventive concept as the above-mentioned method embodiment, the embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by the processor 210, the multi-attribute classification method of gastrointestinal tumor images based on multimodal fusion is implemented.

[0128] The multi-attribute classification method for gastrointestinal tumor images based on multimodal fusion specifically includes the following:

[0129] S1. Construct a multimodal gastrointestinal tumor dataset.

[0130] S2, perform layered operations on the data set;

[0131] S3, construct an image feature extraction network, and use the image feature extraction network to extract the features of EUS images and WLE images in the dataset respectively;

[0132] S4, constructing a multi-attribute classification network, using image features to perform multi-attribute classification training on tumor attributes, after the training is completed, using the network to predict the tumor image to be tested, and obtaining the tumor attributes after network prediction completion;

[0133] S5. Cross-modal attention fusion and tumor property classification guided by lesion attributes: The network-predicted and completed tumor attributes are used to guide the attention of image features and perform cross-modal attention fusion on EUS and WLE modalities to obtain the vector after cross-modal attention fusion. The vector then passes through the three-layer fully connected layer of the multi-attribute classification network and finally outputs the tumor property classification of the tumor image to be tested for auxiliary diagnosis results.

[0134] In several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed apparatus and method can also be implemented in other ways. The apparatus and method embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the apparatus, method and computer program product according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0135] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0136] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, an electronic device, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, RandomAccess Memory), a disk or an optical disk. It should be noted that, in this article, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0137] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0138] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0139] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-attribute classification method for gastrointestinal tumor images based on multimodal fusion, characterized in that: The steps include: S1. Construct a multimodal gastrointestinal tumor dataset. S2, perform layered operations on the data set; S3, construct an image feature extraction network, and use the image feature extraction network to extract the features of EUS images and WLE images in the dataset respectively; S4, constructing a multi-attribute classification network, using image features to perform multi-attribute classification training on tumor attributes, after the training is completed, using the network to predict the tumor image to be tested, and obtaining the tumor attributes after network prediction completion; S5. Cross-modal attention fusion and tumor property classification guided by lesion attributes: The network-predicted and completed tumor attributes are used to guide the attention of image features and perform cross-modal attention fusion on EUS and WLE modalities to obtain the vector after cross-modal attention fusion. The vector then passes through the three-layer fully connected layer of the multi-attribute classification network and finally outputs the tumor property classification of the tumor image to be tested for auxiliary diagnosis results.

2. The multi-attribute classification method for gastrointestinal tumor images based on multimodal fusion according to claim 1, characterized in that: In step S1, a multimodal data set of a patient is collected, the data set includes an EUS image, a WLE image and a text description of the two images of the patient, and all data are paired according to the combination of EUS image, EUS text, WLE image and WLE text; Then, image and text preprocessing is performed: for images, resizing and standardization operations are performed to make them consistent with the network input; for text, typos are identified and text format is standardized, and the attribute feature labels of the tumor in the text description are extracted.

3. The multi-attribute classification method for gastrointestinal tumor images based on multimodal fusion according to claim 1, characterized in that: In step S2, the data set is stratified using the number of extracted attribute feature labels, specifically including: when both the EUS image attribute label and the WLE image attribute label are not missing, the data is classified as the zeroth layer data; when the number of missing EUS image attribute labels and WLE image attribute labels is two, the data is classified as the second layer data.

4. The multi-attribute classification method for gastrointestinal tumor images based on multimodal fusion according to claim 1, characterized in that: In step S3, the image feature extraction network includes an EUS image feature extraction network and a WLE image feature extraction network, both of which use the ImageNet and trained Resnet18 network as the basic feature extraction skeleton to obtain EUS image features and WLE image features: M EUS =R EUS (I EUS ) M WLE =R WLE (I WLE ) F EUS =AvgPool(M EUS ) F WLE =AvgPool(M WLE ) in Respectively represent the EUS image feature vector and the WLE image feature vector, They represent the EUS image feature extraction network and the WLE image feature extraction network respectively. They represent the EUS image feature map and the WLE image feature map respectively, and AvgPool(·) is the average pooling operation.

5. The multi-attribute classification method for gastrointestinal tumor images based on multimodal fusion according to claim 1, characterized in that: In step S4, the multi-attribute classification network includes EUS and WLE attribute classification networks, which are composed of three fully connected layers, where EUS has six attributes and WLE has three attributes. Therefore, a total of nine three-layer fully connected layers are used in the attribute classification layer for tumor multi-attribute classification; the first layer is a 512*1024 fully connected layer, the second layer is a 1024*1024 fully connected layer, and the third layer is a 1024*Dim cls The fully connected layer, where Dim cls is the number of classifications for a single attribute of the tumor; During training, Adam is used to update the multi-attribute classification network. If all tumor attribute labels can be extracted from the description text, all predicted results and true attribute labels are used to train the model. If the label is missing in the text, the corresponding weight is added during training. The specific loss function is as follows: Among them, n and m represent the number of EUS attributes and the number of WLE attributes respectively, GT represents the true label, and y c Represents the true value of the attribute; During training, the model is first trained using the data from the zeroth layer, and then the data from the first and second layers are added in turn to train the model until all layers of data are added for training. Except for the zeroth layer, when each layer of data is added, the model trained in the previous layer of data is first used to predict the missing attribute labels in the data of that layer, and the corresponding pseudo labels are generated at the missing locations. The specific process is as follows: in Represent the EUS and WLE attributes predicted by the i-th layer model respectively They represent the classification networks of the j-th attribute of EUS and WLE respectively; Use the labels predicted by the model to replace the missing labels in this layer of data: in They represent the jth attribute of the updated EUS and WLE of the i-th layer respectively. The data with the missing parts filled with pseudo labels are added to the training set of the model, and subsequent model training is performed. After the training is completed, the network is used to predict the features of the tumor image to be tested, and the tumor attributes after the network prediction is completed are obtained.

6. The multi-attribute classification method for gastrointestinal tumor images based on multimodal fusion according to claim 1, characterized in that: In step S5, the specific steps include: S51, extracting features of the tumor attributes after network prediction completion using the LSTM network to obtain a tumor attribute feature vector of the corresponding image. The specific process is as follows: F EAttr =L EUS (To EUS ) F WAttr =L WLE (To WLE ) in, are EUS attribute feature vector and WLE attribute feature vector respectively, are the EUS attributes and WLE attributes after network prediction and completion, L EUS , L WLE They are EUS attribute feature extractor and WLE attribute feature extractor respectively; S52, using the EUS and WLE attribute features to guide attention on the feature maps of the EUS and WLE images, the process is as follows: Since the number of channels between the EUS feature map and the EUS report features obtained by feature extraction does not match, we first align the number of channels in the EUS feature map and the length of the report features: in They are all learnable parameters, aligning the attribute features of EUS and WLE with the image feature channels; Then the aligned attribute features Each element in is broadcast to formed in the space Then, all the information is aggregated into the final attribute attention feature map using a 1×1 convolution kernel, as shown below: Bro(·) represents the broadcast operation, Conv(·) represents the aggregation of multiple channel information into one channel using a 1×1 convolution kernel. is the guided feature map after being guided by attribute features; A EUS , A WLE Each element in and M EUS 、M WLE The elements of each channel in are weighted averaged to obtain the image features guided by attribute features, as follows: in is the final image feature after being guided by attribute features; S53, cross-modal feature attention fusion, the specific process includes: first, the EUS features and WLE features guided by attribute features are projected into the bilinear feature space through two low-rank matrices, as shown below: in is the low-rank projection matrix, is the output of the bilinear layer, is the Hadmard product operation, is a vector of all 1s; The o low-rank matrices are expanded by dimensionality reduction to obtain a two-dimensional projection matrix: By using the low-rank projection matrix after dimensionality reduction and combining it with the addition pooling operation, we can get the vector after cross-modal attention fusion: Where SumPooling(x,k) means using a one-dimensional non-overlapping window of size k to perform an additive pooling operation on x; S54, after passing through the three-layer fully connected layer of the multi-attribute classification network, the tumor property classification of the tumor image to be tested is finally output, and the cross entropy loss function used is as follows: Where N represents the number of samples, y i represents the true label of the tumor property of the i-th sample, F i Represents the fusion feature of the i-th sample.

7. An electronic device, characterized in that: The electronic device comprises: a processor (210); and a memory (220) for storing one or more programs; When the one or more programs are executed by the processor (210), the processor executes the multi-attribute classification method for gastrointestinal tumor images based on multi-modal fusion as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by the processor (210), the method for multi-attribute classification of gastrointestinal tumor images based on multi-modal fusion as described in any one of claims 1 to 6 is implemented.

Images