User attribute multi-modal feature-based parotid benign and malignant tumor identification method
By employing a multimodal parotid gland tumor identification method based on user attributes, and utilizing CNN networks and user attribute information to fuse imaging and text features, this method addresses the issue of reliance on doctors' subjective judgment in parotid gland tumor diagnosis, achieving more efficient parotid gland tumor classification and treatment support.
Patent Information
- Application Number
- CN202410114088.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-27
- Publication Date
- 2025-12-30
AI Technical Summary
Current medical imaging examinations rely on doctors' subjective judgment in the diagnosis of parotid gland tumors, making it difficult to make a comprehensive preliminary diagnosis of malignant tumors. Furthermore, they do not make full use of patient user attribute information, resulting in low diagnostic efficiency and a high risk of misdiagnosis.
A method for identifying benign and malignant parotid gland tumors based on user attribute multimodal features is adopted. The method extracts radiological image features through CNN convolutional neural network, combines user text attribute information, models attention features and fuses multimodal features, optimizes weight parameters, and predicts the benign and malignant classification labels of parotid gland tumors.
It improves the accuracy and interpretability of parotid gland tumor identification, provides more precise diagnostic and treatment support, and reduces the risk of misdiagnosis.
Smart Images
Figure BDA0004684541440000031 
Figure BDA0004684541440000034 
Figure BDA0004684541440000036
Abstract
Description
Technical Field
[0001] This invention relates to the field of parotid gland benign and malignant tumor identification technology, specifically a method for identifying parotid gland benign and malignant tumors based on user attribute multimodal features. Background Technology
[0002] The parotid glands, being the largest pair of salivary glands in the human body, have a relatively high incidence of tumors, making them a common and frequently occurring disease in medicine. Benign tumors are more common. However, once a malignant tumor develops, it seriously threatens the patient's life and health. While imaging examinations are an important means of preoperative diagnosis of parotid gland tumors, the information obtained can sometimes be limited. Moreover, relying solely on imaging examinations depends on the doctor's subjective judgment and extensive clinical experience, making it difficult to comprehensively complete the initial diagnosis of malignant tumors. Furthermore, the diagnosis of parotid gland tumors requires doctors to observe and assess various patient indicators over a long period, reducing diagnostic efficiency, especially when imaging features are atypical, which can easily lead to misdiagnosis of parotid gland tumors.
[0003] In recent years, the development of artificial intelligence and deep learning technologies has made the intelligent extraction of features from medical imaging images more efficient and easier. Deep learning technology can extract digital features from images and has been widely applied in image classification and segmentation. Multimodal feature fusion technology can not only extract rich high-order semantic features from multiple source images, but also model potential features related to label categories (e.g., benign, malignant) based on patient user attributes (e.g., age, gender, smoking history), achieving the fusion of interactive features from different modalities. Therefore, learning multimodal features from medical images and user attributes is particularly important for improving the accuracy of tumor identification.
[0004] In fact, for the accurate diagnosis and treatment planning of parotid gland tumors, doctors often conduct a comprehensive analysis based on patient attribute information and medical image data to make an accurate judgment. Based on this approach, we designed a method for identifying benign and malignant parotid gland tumors based on user attribute multimodal features. By modeling the user's imaging features and the patient's attribute features, we establish the correlation between imaging features and attribute features, learn the user's parotid gland multimodal features, and then predict and assess the benign or malignant nature of the patient's parotid gland tumor. Summary of the Invention
[0005] To address the issue that existing medical image tumor classification technologies neglect the role of patient user attributes in improving tumor classification effectiveness, this invention provides a method for identifying benign and malignant parotid gland tumors based on multimodal features of user attributes.
[0006] This invention is achieved using the following technical solution:
[0007] A method for identifying benign and malignant parotid gland tumors based on user attribute multimodal features, the method comprising the following steps:
[0008] Step 1: Obtain the user's parotid gland image features by modeling based on the user's parotid gland imaging images;
[0009] Step 2: Model the user's text attribute information to obtain the user's text attribute features;
[0010] Step 3: Model the attention feature representation of parotid gland image features based on different textual attribute information of the user;
[0011] Step 4: Integrate user attribute attention features and parotid gland image features to model the user's multimodal parotid gland features;
[0012] Step 5: Calculate the classification label score of parotid gland tumors based on the user's parotid gland multimodal features, and optimize the weight parameters;
[0013] Step 6: Input the parotid gland image features and text attribute features of the new user, and calculate the parotid gland tumor category label based on the optimized weight parameters;
[0014] Step 7: Indicate whether the parotid gland tumor is malignant based on the category label.
[0015] Furthermore, the specific steps of step 1 are as follows:
[0016] For the user's parotid gland imaging images, convolution, pooling, and fully connected operations are performed using a CNN convolutional neural network to form a feature vector h. p =f(W c X+b), obtain the user's parotid gland image features h p Where f represents the activation function, W c Let X represent the weight parameters for learning, X represent the input parotid gland image features, and b represent the offset.
[0017] Furthermore, the specific steps of step 2 are as follows:
[0018] Step 2.1: Initialize the user's text attribute value IDs, construct a dictionary of attribute value IDs, and perform one-hot encoding as follows:
[0019]
[0020] In the formula: The one-hot encoding representation of the attribute;
[0021] Step 2.2: Use random initialization of the lookup table to obtain the attribute value feature vector matrix of the attribute object. This implements feature representation of the user's attribute object values, defined as:
[0022]
[0023] In the formula: Indicates attribute t a Vector representation of;
[0024] Step 2.3: Considering all the specific values of the user's attributes, initialize the user's attribute vector representation u. A Defined as:
[0025]
[0026] In the formula: The vector representation of attribute t1. Indicates attribute t n The vector representation of .
[0027] Furthermore, the specific steps of step 3 are as follows:
[0028] Step 3.1: To observe the correlation between different textual attributes of users and parotid gland image features, the correlation metric between user attributes and sentences is defined as:
[0029]
[0030] In the formula: v p W represents the weight vector learned. p W t Let b represent the weight matrix of the neural network. s Indicates the offset;
[0031] Step 3.2: Define the attention weights for the attributes as follows:
[0032]
[0033] In the formula: Indicates attribute t a In parotid gland image features h p Attention weights The attribute t representing user u a The vector representation of , The attribute t representing user u i Vector representation of;
[0034] Step 3.3: Considering all possible attribute values of the user, the attribute features related to the parotid gland image features are aggregated and represented as u. t Defined as:
[0035]
[0036] In the formula: Indicates attribute t i In parotid gland image features h p Attention weights Indicates attribute t i The vector representation of .
[0037] Furthermore, the specific steps of step 4 are as follows:
[0038] Considering the parotid gland image feature representation and the associated user text attribute feature representation, the multimodal feature representation can be formalized by concatenation:
[0039] s = h p ||u t ;
[0040] In the formula: s represents the user's parotid gland multimodal features.
[0041] Furthermore, the specific steps of step 5 are as follows:
[0042] Step 5.1: Feed the user's parotid gland multimodal features s into a multilayer perceptron, taking into account the bias terms b of different layers. k To learn the parotid gland benign / malignant rating based on multimodal features, the rating prediction function is defined as:
[0043]
[0044] In the formula: W represents the benign or malignant score of a user's parotid gland sample. 1 W k b represents the weight parameters of the first and kth layers of the neural network; 1 b k This represents the offset between the first and kth layers of the neural network;
[0045] Step 5.2: Determine the benign or malignant score of the user's parotid gland sample. Based on this, the objective loss function for classifying labels is calculated as follows: Optimize weight parameter W c W p W t W 1 W k .
[0046] Furthermore, the specific steps of step 6 are as follows:
[0047] For new users' parotid gland image features and text attribute features, based on the optimized weight parameter W c W p W t W 1 W kSteps 2 through 5 are used to calculate the benign or malignant score of the parotid gland sample from the new user.
[0048] Furthermore, the specific steps of step 7 are as follows:
[0049] Benign / malignant score of parotid gland samples from new users The marker is set; if the value is greater than 0.5, it is marked as benign; otherwise, it is marked as malignant.
[0050] Compared with existing medical image tumor classification technologies, this invention has the following advantages: First, the significant difference between the method provided in this invention and existing methods lies in the design of a method that integrates user attribute attention features and parotid gland image features to model the user's parotid gland multimodal features. It establishes a parotid gland tumor classification label scoring prediction model based on image and user attribute multimodal features, proposes a parotid gland tumor identification strategy based on image and user attribute multimodal features, and interprets the classification results based on imaging images and user attribute multimodal features. Second, this invention integrates user attribute attention features and parotid gland image features to learn the user's parotid gland multimodal features, and interprets the classification results based on imaging images and user attribute multimodal features, improving the interpretability of deep learning tumor identification and providing significant support for obtaining more accurate parotid gland tumor diagnosis and treatment results. Detailed Implementation
[0051] A method for identifying benign and malignant parotid gland tumors based on user attribute multimodal features, the method comprising the following steps:
[0052] Step 1: Obtain the user's parotid gland image features by modeling based on the user's parotid gland imaging images;
[0053] Step 2: Model the user's text attribute information to obtain the user's text attribute features;
[0054] Step 3: Model the attention feature representation of parotid gland image features based on different textual attribute information of the user;
[0055] Step 4: Integrate user attribute attention features and parotid gland image features to model the user's multimodal parotid gland features;
[0056] Step 5: Calculate the classification label score of parotid gland tumors based on the user's parotid gland multimodal features, and optimize the weight parameters;
[0057] Step 6: Input the parotid gland image features and text attribute features of the new user, and calculate the parotid gland tumor category label based on the optimized weight parameters;
[0058] Step 7: Indicate whether the parotid gland tumor is malignant based on the category label.
[0059] The specific steps of step 1 are as follows:
[0060] For the user's parotid gland imaging images, convolution, pooling, and fully connected operations are performed using a CNN convolutional neural network to form a feature vector h. p =f(W c X+b), obtain the user's parotid gland image features h p Where f represents the activation function, W c Let X represent the weight parameters for learning, X represent the input parotid gland image features, and b represent the offset.
[0061] The specific steps of step 2 are as follows:
[0062] Step 2.1: Initialize the user's text attribute value IDs, construct a dictionary of attribute value IDs, and perform one-hot encoding as follows:
[0063]
[0064] In the formula: The one-hot encoding representation of the attribute;
[0065] Step 2.2: Use random initialization of the lookup table to obtain the attribute value feature vector matrix of the attribute object. This implements feature representation of the user's attribute object values, defined as:
[0066]
[0067] In the formula: Indicates attribute t a Vector representation of;
[0068] Step 2.3: Considering all the specific values of the user's attributes, initialize the user's attribute vector representation u. A Defined as:
[0069]
[0070] In the formula: The vector representation of attribute t1. Indicates attribute t n The vector representation of .
[0071] The specific steps of step 3 are as follows:
[0072] Step 3.1: To observe the correlation between different textual attributes of users and parotid gland image features, the correlation metric between user attributes and sentences is defined as:
[0073]
[0074] In the formula: vp W represents the weight vector learned. p W t Let b represent the weight matrix of the neural network. s Indicates the offset;
[0075] Step 3.2: Define the attention weights for the attributes as follows:
[0076]
[0077] In the formula: Indicates attribute t a In parotid gland image features h p Attention weights The attribute t representing user u a The vector representation of u ti The attribute t representing user u i Vector representation of;
[0078] Step 3.3: Considering all possible attribute values of the user, the attribute features related to the parotid gland image features are aggregated and represented as u. t Defined as:
[0079]
[0080] In the formula: Indicates attribute t i In parotid gland image features h p Attention weights Indicates attribute t i The vector representation of .
[0081] The specific steps of step 4 are as follows:
[0082] Considering the parotid gland image feature representation and the associated user text attribute feature representation, the multimodal feature representation can be formalized by concatenation:
[0083] s = h p ||u t ;
[0084] In the formula: s represents the user's parotid gland multimodal features.
[0085] The specific steps of step 5 are as follows:
[0086] Step 5.1: Feed the user's parotid gland multimodal features s into a multilayer perceptron, taking into account the bias terms b of different layers. k To learn the parotid gland benign / malignant rating based on multimodal features, the rating prediction function is defined as:
[0087]
[0088] In the formula: W represents the benign or malignant score of a user's parotid gland sample. 1 W k b represents the weight parameters of the first and kth layers of the neural network; 1 b k This represents the offset between the first and kth layers of the neural network;
[0089] Step 5.2: Determine the benign or malignant score of the user's parotid gland sample. Based on this, the objective loss function for classifying labels is calculated as follows: Optimize weight parameter W c W p W t W 1 W k .
[0090] The specific steps of step 6 are as follows:
[0091] For new users' parotid gland image features and text attribute features, based on the optimized weight parameter W c W p W t W 1 W k Steps 2 through 5 are used to calculate the benign or malignant score of the parotid gland sample from the new user.
[0092] The specific steps of step 7 are as follows:
[0093] Benign / malignant score of parotid gland samples from new users The marker is set; if the value is greater than 0.5, it is marked as benign; otherwise, it is marked as malignant.
[0094] To verify the effectiveness of this invention, analysis was conducted based on imaging images and attribute data of a subset of users. Six user attributes were included: user ID, user age, user gender, user smoking history, surface morphology of the user's parotid gland tumor, and user facial nerve symptoms. Experiments were performed on the dataset with a 6:2:2 split between training, validation, and test sets. Evaluation metrics included Accuracy, Precision, Recall, and F1. To further verify the effectiveness and advancement of this invention, several existing rating and recommendation prediction models were compared: VisionTransformer, DenseNet, ResNet34, and EfficientNet. Experimental results are shown in Table 1.
[0095] Table 1 Experimental Results
[0096]
[0097] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of the present invention is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but all such changes and modifications fall within the scope of protection of the present invention.
Claims
1. A method for identifying parotid gland benign and malignant tumors based on user attribute multi-modal features, characterized in that: The method comprises the following steps: Step 1: obtaining the parotid image features of the user according to the modeling of the parotid image of the user; Step 2: obtaining the text attribute features of the user according to the modeling of the text attribute information of the user; Step 3: modeling the attention feature representation of different text attribute information of the user to the parotid image features; Step 4: fusing the attribute attention features and the parotid image features of the user to model the parotid multi-modal features of the user; Step 5: calculating the classification label score of the parotid tumor according to the parotid multi-modal features of the user, and optimizing the weight parameters; Step 6: inputting the parotid image features and text attribute features of the new user, and calculating the class label of the parotid tumor according to the optimized weight parameters; Step 7: according to the class label of the parotid tumor, whether it is a malignant tumor is marked.
2. The method of claim 1, wherein the method is based on user attribute multi-modal features of parotid gland benign and malignant tumor recognition. The specific steps of step 1 are: For the user's parotid imaging image, the CNN convolutional neural network is used for convolution, pooling, full connection operation to form a feature vector h p = f(W c X + b), the user's parotid image feature h p is obtained; wherein f represents an activation function, W c represents a learned weight parameter, X represents an input parotid image feature, and b represents an offset.
3. The method of claim 2, wherein the method is based on user attribute multi-modal features of parotid gland benign and malignant tumor recognition. The specific steps of step 2 are: Step 2.1: initialize the text attribute value ID of the user, construct the ID dictionary of the attribute value, and one-hot encode as: In the formulae: one-hot encoding representation of the attributes; Step 2.2: Get the attribute value feature vector matrix of the attribute object using random initialization lookup table The attribute object value of the user is characterized, defined as: In the formulae: represents the attribute t a a vector representation; Step 2.3: Initialize the user's attribute vector representation u considering all specific values of the user's attributes A is defined as: wherein: represents a vector representation of the attribute t1, represents a vector representation of the attribute t n .
4. The method of claim 3, wherein the method is characterized by: The specific steps of step 3 are: Step 3.1: in order to observe the correlation between different text attributes of the user and the parotid image features, the correlation between the user attributes and the sentence is defined as: where: v p represents a weight vector of the learning, W p , W t represents a weight matrix of the neural network, b s represents an offset Step 3.2: define the attention weight of the attribute as: wherein: represents the attribute t a the attention weight of the parotid gland image feature h p , represents the vector representation of the attribute t a of the user u represents the vector representation of the attribute t i of the user u Step 3.3: Considering all attribute values of the user, the attribute feature aggregation related to the parotid image feature is denoted as u t is defined as: where: represents the attribute t i In the parotid gland image feature h p the attention weight of, represents the attribute t i vector representation.
5. The method of claim 4, wherein the method is characterized by: The specific steps of step 4 are: Considering the parotid image feature representation and the related user text attribute feature representation, the multi-modal feature representation can be formalized by splicing: s = h p ‖u t ; In the formula: s represents the parotid multi-modal features of the user.
6. The method of claim 5, wherein the method is based on user attribute multi-modal features of parotid gland benign and malignant tumor recognition. The specific steps of step 5 are: Step 5.1: Send the user's parotid gland multi-modal features s into the multi-layer perceptron, considering the bias term b of different layers k , learn the parotid gland benign and malignant score based on multi-modal features, and the score prediction function is defined as: In the formula: W represents the benign and malignant score value of the user parotid sample 1 W k W represents the weight parameters of the first layer and the kth layer of the neural network; b 1 b k W represents the offset of the first layer and the kth layer of the neural network Step 5.2: On the basis of the benign and malignant score values of the user parotid gland sample, the target loss function of the classification label is calculated as Optimizing the weight parameters W c , W p , W t , W 1 , W k . 7. The method of claim 6, wherein the method is based on user attribute multimodal features of parotid gland benign and malignant tumor recognition. The specific steps of step 6 are: For the new user's parotid image features and text attribute features, according to the optimized weight parameters W c , W p , W t , W 1 , W k , through steps 2-5, the benign and malignant score values of the new user's parotid sample are calculated 8. The method of claim 7, wherein the method is based on user attribute multimodal features of parotid gland benign and malignant tumor recognition. The specific steps of step 7 are: Based on new user's parotid sample benign and malignant score value The label is benign if greater than 0.5, otherwise malignant.