A training method of a multi-modal eye orbit disease reasoning model and application thereof

Abstract

The technical scheme of the present application is to provide a training method of a multi-modal eye socket disease reasoning model, characterized in that the training method comprises the following steps: determining N two-dimensional features and M three-dimensional features for identifying eye socket diseases; eliminating redundant features in the N two-dimensional features and the M three-dimensional features; labeling two-dimensional image data sets and three-dimensional data sets; training two-dimensional and three-dimensional eye socket disease reasoning models, and then fusing them into a multi-modal eye socket disease reasoning model. Another technical scheme of the present application is to provide an application of the above-mentioned training method of the multi-modal eye socket disease reasoning model in an eye socket disease reasoning system. Compared with the conventional unified labeling method, the present application adopts a new strategy of labeling texture features for two-dimensional images and labeling structural features for three-dimensional models, which can reduce the noise introduced by forcibly processing different dimensional data, and has good accuracy and stability.

Description

Technical Field

[0001] This invention relates to a method for training a multimodal orbital disease inference model, and the application of the multimodal orbital disease inference model obtained based on the training method in an orbital disease inference system. Background Technology

[0002] Orbital disease is a complex and dangerous ophthalmic condition that can lead to blindness, disability, and death. Patients often present at an advanced stage with poor prognosis, making early identification crucial. The development of AI technology offers new possibilities for early identification of orbital diseases. However, the clinical signs of orbital diseases are complex and diverse. Early identification requires integrating three-dimensional spatial data, including eyeball protrusion and eyelid parameters, with two-dimensional texture data, including conjunctival color and pupil orientation. Multimodal fusion of these two dimensions is an effective method for addressing this challenge.

[0003] However, orbital diseases possess extremely rich information in both 2D images and 3D models, with basic building blocks such as pixels and point clouds often numbering in the tens of thousands. Orbital disease identification requires numerous spatial and textural features. Simultaneously collecting these features from both types of images for training would result in an excessively large number of features used for training, increasing computational complexity, inducing the "curse of dimensionality," and negatively impacting classifier performance. Furthermore, since 3D data consists of grids and lacks color and other textural information, 2D images cannot adequately represent 3D spatial information. Forcibly unifying features extracted from images of different dimensions would introduce significant noise, leading to poor model performance. Moreover, directly inputting 2D and 3D data for multimodal fusion results in poor model interpretability. Clinicians lack an understanding of the technological intersection, making it difficult to build trust in the model and severely hindering its application and widespread adoption. Summary of the Invention

[0004] The purpose of this invention is to provide an image data processing and model training method based on prior clinical knowledge to enhance the accuracy, stability, and interpretability of orbital disease identification.

[0005] To achieve the above objectives, one technical solution of the present invention provides a training method for a multimodal orbital disease inference model. This multimodal orbital disease inference model integrates an orbital disease inference model one based on two-dimensional orbital images and an orbital disease inference model two based on three-dimensional orbital images. The training method includes the following steps:

[0006] Step 1: Determine the N two-dimensional features and M three-dimensional features used to identify orbital diseases;

[0007] Step 2: Remove redundant two-dimensional features from the N two-dimensional features, including the following steps:

[0008] Step 201: Calculate the improved variance ratio between feature subsets belonging to N two-dimensional features in the two-dimensional image dataset. The improved variance ratio is the ratio of the inter-class variance to the intra-class variance of the feature subset. Select feature subsets based on the improved variance ratio: if the improved variance ratio between the feature subset belonging to the current two-dimensional feature and the feature subset belonging to another two-dimensional feature is not greater than a preset threshold, then the current two-dimensional feature and the other two-dimensional feature are deleted or removed.

[0009] Step 202: Using the AdaBoost algorithm to further select feature subsets based on step 201: A weak classifier is built for each two-dimensional feature selected in step 201. In each iteration, the weak classifier with the smallest weighted classification error is searched from all possible feature spaces. This weak classifier is then used to update the weights so that the weights of misclassified samples are increased. After multiple iterations, a weak classifier is formed. At the same time, N1 two-dimensional features from the remaining two-dimensional features after the processing in step 201 are selected.

[0010] Removing redundant 3D features from the M-item 3D features includes the following steps:

[0011] Using a feature selection algorithm optimized based on specific outlier samples, after removing some outlier 3D data samples from the 3D dataset, the mean class center of the remaining 3D data samples is calculated. Then, the variance of the class belonging to each 3D feature is calculated, and then a threshold is selected using the statistically significant normal distribution 3σ principle. 3D features outside the threshold are regarded as redundant features. After removing redundant features, M1 3D features out of M 3D features are selected.

[0012] Step 3: The two-dimensional image dataset and the three-dimensional dataset used for training the multimodal orbital disease inference model are processed as follows:

[0013] For two-dimensional image datasets:

[0014] To determine whether each two-dimensional image data sample in the two-dimensional image dataset has N1 two-dimensional features, the annotation results are used to determine whether the patient has orbital disease. Based on the determination results, a classification label is added to each two-dimensional image data sample, and the two-dimensional image dataset is divided into a two-dimensional training set and a two-dimensional test set.

[0015] For 3D datasets:

[0016] To determine whether each 3D data sample in the 3D image dataset has M1 3D features, the labeling results are used to determine whether the sample has orbital disease. Based on the determination results, a classification label is added to each 3D data sample, and the 3D image dataset is divided into a 3D training set and a 3D test set.

[0017] Step 4: Train orbital disease inference model 1 using a two-dimensional training set and a two-dimensional test set, and train orbital disease inference model 2 using a three-dimensional training set and a three-dimensional test set. Then, merge the trained orbital disease inference model 1 and orbital disease inference model 2 into a multimodal orbital disease inference model.

[0018] Preferably, in step 1, the N two-dimensional features include abnormal eye position, conjunctival hyperemia, orbital venous malformation, eyelid defects, entropion, ectropion, eyelid swelling, and eyelid masses.

[0019] Preferably, in step 1, the M-item three-dimensional features include protruding eyeballs, sunken eyeballs, abnormal position of the inner and outer canthi, eyeball displacement, orbitofacial asymmetry, increased interorbital distance, eyelid retraction, and ptosis.

[0020] Another technical solution of the present invention is to provide an application of the above-mentioned training method for the multimodal orbital disease inference model in an orbital disease inference system, characterized in that the orbital disease inference system includes:

[0021] A two-dimensional eye socket image extraction unit is used to extract two-dimensional images of the eye socket region of a human face;

[0022] A three-dimensional orbital reconstruction unit is used to extract three-dimensional orbital depth features;

[0023] The multimodal orbital disease inference model trained using the above training method is used to predict whether or not someone has orbital disease after inputting two-dimensional images and three-dimensional orbital depth features into the multimodal orbital disease inference model.

[0024] Compared with existing technical solutions, the present invention has the following advantages:

[0025] 1) Compared with conventional unified annotation methods, this invention innovatively adopts a new strategy of annotating texture features of two-dimensional images and structural features of three-dimensional models. This not only reduces the noise introduced by forcibly processing data of different dimensions and avoids the impact of noise on model training, but also takes advantage of the strengths of two-dimensional and three-dimensional data in different types of features. By utilizing the rich and obvious texture features of two-dimensional images and the fact that structural features of three-dimensional models can better reflect the spatial relationship of data, different processing methods and models are used to train data of different dimensions, making the training more reliable and having better accuracy and stability.

[0026] 2) This invention uses feature selection to reduce the dimensionality of sample features, eliminating redundant features and thus reducing the number of photo features processed each time, thereby reducing the data size. This helps to shorten modeling time and improve modeling efficiency, ensuring that the information obtained is concentrated on only some key features. This not only reduces the number of features used in training, effectively avoiding the "curse of dimensionality," but also reduces the difficulty of the learning task, making it easier to discover the patterns inherent in the data. In terms of subsequent processing, this invention uses the FSSO algorithm to remove outlier sample data before processing the remaining samples. Compared with conventional algorithms, this method can make the sample mean closer to the true patient data mean, less affected by some samples with large deviations, and greatly improve the performance of the model.

[0027] 3) Based on clinical expert consensus, the texture features that 2D images excel at and the structural features that 3D models are uniquely characterized are labeled separately, which improves the interpretability of the model, provides clinicians with a knowledge intersection to understand the working principle of the model, and is more in line with human understanding of image features of different dimensions. This allows us to modify the model at the semantic level, providing conditions for the promotion and practical application of the model. Attached Figure Description

[0028] Figure 1 This illustrates the selection of features for two-dimensional data;

[0029] Figure 2 This illustrates the selection of features for three-dimensional data. Detailed Implementation

[0030] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0031] In machine learning, for a given learning task, only some attributes in the attribute set may be effective, while others are useless redundancies. These attributes are called "features." Features that are useful for the learning task are called "relevant features," while useless features are called "irrelevant features." The process of extracting a subset of relevant features from a given attribute set is called "feature selection."

[0032] Based on whether they depend on the class labels of the samples, feature selection algorithms can be divided into supervised feature selection algorithms, unsupervised feature selection algorithms, and semi-supervised feature selection algorithms. Supervised feature selection algorithms can be further divided into filtered feature selection methods, wrapped feature selection methods, and embedded feature selection methods. Among them, filtered feature selection methods do not depend on the performance of any classifier, but score each feature by measuring the essential attributes of the training samples, so filtered feature selection methods are usually computationally efficient. Wrapped feature selection methods are based on random search sequences and need to select the subset of features with the highest scores based on the classification results, so this type of feature selection method has a large computational cost. Embedded feature selection methods combine the feature selection problem and the classification problem into a single objective programming problem. Although its computational cost is greatly reduced compared to wrapped feature selection methods, it is still not as good as the filtered model.

[0033] In the process of orbital disease identification, this invention first extracts features that contribute significantly to the identification of orbital diseases and then calibrates the collected dataset. Using the Delphi method to synthesize the opinions of multiple orbital disease experts, and focusing on the clinical manifestations, imaging data, and physical examination data of common clinical orbital disease cases, and referring to internationally accepted diagnoses of the diseases, combined with the identifiable physical signs of the diseases and the dominant information types upon which identification depends, a total of 16 features were determined. These include: 8 two-dimensional features, including abnormal eye position, conjunctival hyperemia, orbital venous malformation, eyelid defects, entropion, ectropion, eyelid swelling, and eyelid masses; and 8 three-dimensional features, including proptosis, enophthalmos, abnormal inner and outer canthal positions, eye displacement, orbitofacial asymmetry, increased interorbital distance, eyelid retraction, and ptosis.

[0034] Then, each of the 4000 facial images in the database was labeled. In subsequent processing, the data volume based on 16 features was large, and not all features were strongly correlated with the determination of whether someone had orbital disease. Therefore, feature selection was used to remove some redundant features from the 16 features to reduce computational load, simplify the model, reduce the risk of overfitting, and improve computational efficiency. To avoid the one-sidedness of a single feature extraction method and prevent the removal of key features, this invention uses a combination of methods to improve accuracy, mainly filtering and wrapping methods.

[0035] In this embodiment, the removal of redundant features from the eight two-dimensional features based on two-dimensional data features includes the following steps:

[0036] Texture is an important visual cue, a ubiquitous yet difficult-to-describe feature in images, representing phenomena such as uniformity, detail, and coarseness. Common texture features include gray-level co-occurrence matrix, autoregressive texture model, Tamura texture features, and wavelet transform. This invention leverages the obvious advantages of two-dimensional image texture features, using these features as two-dimensional data features to extract eight two-dimensional features.

[0037] Wrapping is a common feature extraction method that uses machine learning algorithms to evaluate the effectiveness of feature subsets and detect the interaction relationships between two or more features, thereby optimizing the selected feature subset for the model's performance. The feature subset search method used in this invention combines sequential backward search (starting with the entire set and subtracting one feature at a time) and forward search (starting with an empty set and adding one feature at a time) from heuristic search, gradually reducing the feature subset until the optimal feature subset is found. The time complexity of this operation is O(n^2). 2 ).

[0038] Combination Figure 1 This invention first uses the Augmented Variance Rate (AVR) method for feature subset selection. AVR is the ratio of between-class variance to within-class variance of a feature. Generally, the higher the AVR value of a feature, the greater its discriminative power. This invention calculates the average and variance of the grayscale and contrast of two-dimensional image data samples in a two-dimensional image dataset, and uses AVR to compare the AVR of images that conform to and do not conform to the current two-dimensional feature, checking if the value is large enough. If the AVR value is not large, it indicates that the discriminative power of this two-dimensional feature is not significant, and this two-dimensional feature can be removed.

[0039] The present invention then uses AdaBoost for further feature subset selection: a weak classifier is built for each two-dimensional feature. In each iteration, the weak classifier with the smallest weighted classification error is searched from all possible feature spaces. This weak classifier is then used to update the weights so that the weights of misclassified samples are increased. After multiple iterations, a weak classifier is formed, and six of the original eight two-dimensional features are selected.

[0040] Through the above process, the present invention eliminates two redundant two-dimensional features based on two-dimensional texture features from the eight two-dimensional features.

[0041] In this embodiment, the removal of redundant features from the eight three-dimensional features based on three-dimensional data features includes the following steps:

[0042] Three-dimensional data is typically composed of point clouds or meshes, which can reflect the spatial structural characteristics of the data. To ensure accuracy, the three-dimensional dataset of this invention uses a high density of basic building blocks such as point clouds, resulting in a large sample size. Furthermore, since three-dimensional point cloud data contains rich intrinsic attributes, this invention employs a feature selection algorithm based on a filtering model.

[0043] Combination Figure 2 The filtering method used in this invention is a univariate filtering method, which does not consider the interrelationships between features. It sorts features according to the correlation or mutual information between the feature variables and the target variable, filtering out the least relevant feature variables. This approach is computationally efficient and less prone to overfitting. This invention uses the most intuitive variance selection method, employing a feature selection based on specific outliers (FSSO) algorithm. After removing some outlier 3D data samples, the mean class center of the remaining 3D data samples is calculated, thus improving data reliability and result accuracy. Then, the variance of each 3D feature is calculated, and a threshold is selected using the statistically significant normal distribution 3σ principle. 3D features outside the threshold are considered redundant. In this embodiment, two 3D features out of eight that exceeded the set threshold were removed.

[0044] During training, this invention annotates the two-dimensional image dataset and the three-dimensional dataset according to different two-dimensional and three-dimensional features, thereby reducing noise and improving model accuracy.

[0045] For two-dimensional image datasets, this invention determines whether each two-dimensional image data sample has orbital disease based solely on the image's texture features and annotates it with both two-dimensional features and the presence of orbital disease. For three-dimensional datasets, however, this invention determines whether each three-dimensional data sample has orbital disease based on structural features and annotates it with both three-dimensional features and the presence of orbital disease. This avoids the drawbacks of two-dimensional texture features being severely affected by resolution and potentially by lighting and reflection conditions. Furthermore, the texture reflected in two-dimensional image data may not be the true texture of a three-dimensional object's surface, so texture features are not suitable for labeling three-dimensional data. Similarly, since two-dimensional image data lacks the unique structural features of three-dimensional objects, such as convexity / concave texture and reflectivity, structural features cannot be used as a criterion for labeling both two-dimensional and three-dimensional datasets simultaneously. This strategy of using different annotation methods for features of different dimensions avoids introducing noisy data, and because the selected two-dimensional and three-dimensional feature criteria are applicable to the corresponding machine learning networks, it greatly benefits the subsequent training processes for each dimension.

[0046] Because different dimensional datasets use different annotation methods, and the training process is data-driven, the subsequent machine learning training process will also differ.

[0047] For two-dimensional image data, this invention first uses MTCNN (Multi-Task Convolutional Neural Network) to perform face detection, alignment, and cropping on the two-dimensional image dataset. Then, it utilizes the HOG (Histogram of Oriented Gradients) active appearance model to locate key points at specific face angles, obtaining stable and reliable facial key points. Based on the key point information of the eye socket contour, the eye socket area is located and segmented. From a face photo, two sub-images are obtained: rectangular images containing the left and right eye socket areas respectively, thus completing the extraction of the eye socket area. Next, this invention labels whether the extracted eye socket images have two-dimensional features and judges whether the labeled results indicate the presence of eye socket disease. Then, the results are divided into training and test sets in an 8:2 ratio. A ResNet-34 network is trained to obtain an eye socket disease inference model for two-dimensional eye socket images, defined as model a in this embodiment. For three-dimensional data, this invention extracts three-dimensional (mesh) eye socket depth features based on the LSA-3DMM method. This invention cascades the LSA-Conv feature extraction network with a fully connected layer to obtain a classification network for processing three-dimensional mesh data. After reconstructing 4000 orbital mesh datasets in 3D, we labeled them to determine whether they possessed each 3D feature, and judged the negative and positive results of orbital diseases based on the labeling results. Then, the dataset was divided into training and test sets in an 8:2 ratio. The above classification network was trained to obtain a 3D orbital mesh orbital disease inference model, defined as model b in this embodiment.

[0048] After completing the above two-dimensional and three-dimensional data processing, multimodal fusion can be performed. The fully connected layers in both model a and model b are removed, and the feature vectors output from the GAP (Global Average Pooling) layers in both models are concatenated to form new feature vectors. A binary classification (having orbital disease and not having orbital disease) fully connected layer is added, and the concatenated feature vector is input into this fully connected layer for classification to determine whether the person's eye has orbital disease. Thus, model a and model b are combined into a multimodal orbital disease inference model. Finally, the fully connected layers of the multimodal orbital disease inference model are trained using the two-dimensional orbital image and the reconstructed three-dimensional orbital mesh as inputs to model a and model b, respectively. After 200 rounds of training, the trained multimodal orbital disease inference model is obtained.

Claims

1. A training method for a multimodal orbital disease inference model, wherein the multimodal orbital disease inference model integrates an orbital disease inference model one based on two-dimensional orbital images and an orbital disease inference model two based on three-dimensional orbital images, characterized in that, The training method includes the following steps: Step 1: Determine the N two-dimensional features and M three-dimensional features used to identify orbital diseases; Step 2: Remove redundant two-dimensional features from the N two-dimensional features, including the following steps: Step 201: Calculate the improved variance ratio between feature subsets belonging to N two-dimensional features in the two-dimensional image dataset. The improved variance ratio is the ratio of the inter-class variance to the intra-class variance of the feature subset. Select feature subsets based on the improved variance ratio: if the improved variance ratio between the feature subset belonging to the current two-dimensional feature and the feature subset belonging to another two-dimensional feature is not greater than a preset threshold, then the current two-dimensional feature and the other two-dimensional feature are deleted or removed. Step 202: Using the AdaBoost algorithm to further select feature subsets based on step 201: A weak classifier is built for each two-dimensional feature selected in step 201. In each iteration, the weak classifier with the smallest weighted classification error is searched from all possible feature spaces. This weak classifier is then used to update the weights so that the weights of misclassified samples are increased. After multiple iterations, a weak classifier is formed. At the same time, N1 two-dimensional features from the remaining two-dimensional features after the processing in step 201 are selected. Removing redundant 3D features from the M-item 3D features includes the following steps: Using a feature selection algorithm optimized based on specific outlier samples, after removing some outlier 3D data samples from the 3D dataset, the mean class center of the remaining 3D data samples is calculated. Then, the variance of the class belonging to each 3D feature is calculated, and then a threshold is selected using the statistically significant normal distribution 3σ principle. 3D features outside the threshold are regarded as redundant features. After removing redundant features, M1 3D features out of M 3D features are selected. Step 3: The two-dimensional image dataset and the three-dimensional dataset used for training the multimodal orbital disease inference model are processed as follows: For two-dimensional image datasets: To determine whether each two-dimensional image data sample in the two-dimensional image dataset has N1 two-dimensional features, the annotation results are used to determine whether the patient has orbital disease. Based on the determination results, a classification label is added to each two-dimensional image data sample, and the two-dimensional image dataset is divided into a two-dimensional training set and a two-dimensional test set. For 3D datasets: To determine whether each 3D data sample in the 3D image dataset has M1 3D features, the labeling results are used to determine whether the sample has orbital disease. Based on the determination results, a classification label is added to each 3D data sample, and the 3D image dataset is divided into a 3D training set and a 3D test set. Step 4: Train orbital disease inference model 1 using a two-dimensional training set and a two-dimensional test set, and train orbital disease inference model 2 using a three-dimensional training set and a three-dimensional test set. Then, merge the trained orbital disease inference model 1 and orbital disease inference model 2 into a multimodal orbital disease inference model.

2. The training method for a multimodal orbital disease inference model as described in claim 1, characterized in that, In step 1, the N two-dimensional features include abnormal eye position, conjunctival congestion, orbital venous malformation, eyelid defects, entropion, ectropion, eyelid swelling, and eyelid masses.

3. The training method for a multimodal orbital disease inference model as described in claim 1, characterized in that, In step 1, the M-item three-dimensional features include protruding eyeballs, enophthalmos, abnormal position of the inner and outer canthi, eyeball displacement, orbitofacial asymmetry, increased interorbital distance, eyelid retraction, and ptosis.

Images