Multi-modal eye socket classification and recognition method based on edge AI, terminal and medium

Abstract

This invention relates to a multimodal orbital classification and recognition method, terminal, and medium based on edge AI. The method, applied to a mobile terminal, includes the following steps: acquiring a two-dimensional color image of a face to be recognized; performing region segmentation on the two-dimensional color image of the face to obtain a color image of the orbital region; acquiring three-dimensional face data corresponding to the two-dimensional color image of the face; obtaining a face depth image based on the three-dimensional face data; performing region segmentation on the face depth image to obtain a depth image of the orbital region; and using the color image of the orbital region and the depth image of the orbital region as input to a lightweight multimodal recognition model to obtain the classification result. Compared with existing technologies, this invention has advantages such as high reliability and small scale.

Description

Technical Field

[0001] This invention relates to medical image processing methods, and more particularly to a multimodal orbital classification and recognition method, terminal, and medium based on edge AI. Background Technology

[0002] Orbital diseases are complex ophthalmic conditions that can lead to blindness, disability, and death. Their main clinical features include abnormal eye position, eyelid swelling, and protruding eyeballs. In severe cases, they can threaten visual function and appearance, placing a heavy burden on patients, their families, and society. Early screening can significantly reduce the difficulty of treatment, improve patient prognosis, and alleviate medical costs. However, effective screening methods are currently lacking internationally. While corneal diseases, fundus diseases, and cataracts can all have improved prognoses through effective early screening, the absence of orbital disease screening severely impacts the quality of life for patients with these conditions. Furthermore, the development of orbital disease diagnosis and treatment is uneven across different levels of medical institutions. Effective screening should ideally employ convenient, rapid, and easily implemented methods to truly promote early diagnosis and treatment of orbital diseases.

[0003] Changes in the midface shape are a key characteristic of orbital diseases, providing insights for effective early screening. Based on this, orbital features can be extracted after acquiring facial images, and an artificial intelligence neural network model can be trained to automate the identification of orbital diseases. Current models are primarily based on single two-dimensional or three-dimensional facial images. However, the facial features of patients with orbital diseases may exhibit both obvious three-dimensional spatial relationships, such as protruding and enophthalmos, and color and texture features more easily displayed in two-dimensional images, such as ptosis and conjunctival hyperemia. Different patients have different clinical characteristics; therefore, establishing a single-dimensional model is insufficient and may still result in significant detection errors. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a highly reliable and small-scale multimodal orbital classification and recognition method, terminal and medium based on edge AI.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] A multimodal orbital classification and recognition method based on edge AI, applied to mobile terminals, includes the following steps:

[0007] A two-dimensional color image of a face to be identified is obtained, and the two-dimensional color image of the face is segmented to obtain a color image of the eye socket region;

[0008] Obtain three-dimensional face data corresponding to the two-dimensional face color image, obtain a face depth image based on the three-dimensional face data, perform region segmentation on the face depth image, and obtain a depth image of the eye socket region;

[0009] The color image and depth image of the orbital region are used as input to a lightweight multimodal recognition model to obtain the classification result.

[0010] The lightweight multimodal recognition model extracts features from both the color image and the depth image of the orbital region, respectively, to obtain color features and depth features. The classification result is obtained by concatenating the color features and depth features.

[0011] Furthermore, the training process of the lightweight multimodal recognition model includes:

[0012] 1) Construct a first network and a second network, and obtain a first dataset and a second dataset. The first dataset is a dataset composed of color images of the orbital region, each of which has corresponding category labeling information. The second dataset is a dataset composed of face depth images, each of which has corresponding category labeling information.

[0013] 2) Based on the first dataset and the second dataset, perform sparse training on the first network and the second network respectively;

[0014] 3) Prune and compress the first and second networks obtained in step 2) respectively;

[0015] 4) Train the first network and the second network obtained in step 3) normally respectively;

[0016] 5) Concatenate the feature vectors output by the first network and the second network obtained in step 4), and input the concatenated feature vector into a fully connected layer to construct a joint recognition model;

[0017] 6) Freeze the remaining parts of the joint recognition model before the fully connected layer, and simultaneously perform fusion training on the frozen joint recognition model based on the first dataset and the second dataset;

[0018] 7) Quantize and compress the joint recognition model obtained in step 6) to obtain the lightweight multimodal recognition model.

[0019] Furthermore, both the first and second networks are Resnet-34 networks.

[0020] Furthermore, the sparse training applies L1 regularization constraints to the BN layer scaling factor and convolution weight coefficients in the loss function.

[0021] Furthermore, the pruning compression includes structured pruning and unstructured pruning.

[0022] Furthermore, the quantization compression is INT8 quantization compression.

[0023] Furthermore, the number of training rounds for both normal training and fusion training is less than the number of training rounds for sparse training.

[0024] The present invention also provides a mobile terminal, comprising:

[0025] The first acquisition module is used to acquire a two-dimensional color image of a face to be identified, perform region segmentation on the two-dimensional color image of the face, and acquire a color image of the eye socket region.

[0026] The second acquisition module is used to acquire three-dimensional face data corresponding to the two-dimensional face color image, acquire a face depth image based on the three-dimensional face data, perform region segmentation on the face depth image, and acquire an orbital region depth image.

[0027] The recognition module is used to take the color image of the orbital region and the depth image of the orbital region as input to a lightweight multimodal recognition model to obtain the classification result of the recognition.

[0028] The lightweight multimodal recognition model extracts features from both the color image and the depth image of the orbital region, respectively, to obtain color features and depth features. The classification result is obtained by concatenating the color features and depth features.

[0029] Furthermore, the recognition module includes a module training unit, which trains and maintains the lightweight multimodal recognition model. The training process of the lightweight multimodal recognition model includes:

[0030] 1) Construct a first network and a second network, and obtain a first dataset and a second dataset. The first dataset is a dataset composed of color images of the orbital region, each of which has corresponding category labeling information. The second dataset is a dataset composed of face depth images, each of which has corresponding category labeling information.

[0031] 2) Based on the first dataset and the second dataset, perform sparse training on the first network and the second network respectively;

[0032] 3) Prune and compress the first and second networks obtained in step 2) respectively;

[0033] 4) Train the first network and the second network obtained in step 3) normally respectively;

[0034] 5) Concatenate the feature vectors output by the first network and the second network obtained in step 4), and input the concatenated feature vector into a fully connected layer to construct a joint recognition model;

[0035] 6) Freeze the remaining parts of the joint recognition model before the fully connected layer, and simultaneously perform fusion training on the frozen joint recognition model based on the first dataset and the second dataset;

[0036] 7) Quantize and compress the joint recognition model obtained in step 6) to obtain the lightweight multimodal recognition model.

[0037] The present invention also provides a computer-readable storage medium including one or more programs executable by one or more processors of an electronic device, the one or more programs including instructions for performing the multimodal orbital classification and recognition method as described above.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] 1. This invention can automatically obtain the orbit type recognition result by simply acquiring a face image, and then assess the possibility that the identified person has orbital disease, making it easy to use.

[0040] 2. This invention obtains recognition results based on the fusion of two-dimensional color facial images and three-dimensional facial data, effectively avoiding the limitations caused by insufficient recognition dimensions, achieving more accurate recognition, reducing recognition errors, and further reducing the occurrence of subsequent missed diagnoses and misdiagnoses.

[0041] 3. This invention uses sparse training, pruning and compression methods to optimize and compress the model, thereby reducing the model size and improving inference speed. It is closer to the user and more efficient, avoiding the situation where mobile devices such as smartphones cannot complete the evaluation due to limitations such as memory. This is beneficial for users to effectively assess their own eye condition, go to the hospital for timely treatment, and achieve early detection, early diagnosis and early treatment for patients with orbital diseases.

[0042] 4. This invention employs edge AI technology applied to mobile terminals, avoiding issues such as latency and privacy leaks that may arise from uploading to the cloud compared to traditional cloud-based AI technology. Furthermore, compared to traditional imaging-based orbital disease diagnostic equipment, this invention overcomes the problems of high investment costs and operational difficulties. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the lightweight multimodal recognition model of the present invention;

[0044] Figure 2 This is a schematic diagram of the weight coefficient distribution after sparse training and pruning in this invention;

[0045] Figure 3 This is a schematic diagram of INT8 quantization compression according to the present invention. Detailed Implementation

[0046] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0047] Existing orbital classification models are typically based on single 2D or 3D face images, resulting in significant detection errors. This invention employs a multimodal (2D + 3D) deep network for orbital classification to achieve more accurate classification and recognition. However, multimodal models also introduce more complex network structures and a larger number of model parameters. Mobile devices such as smartphones lack the computing power and memory to complete such multimodal interactive deep network inference tasks. Therefore, optimization of the constructed multimodal model is necessary. This invention uses sparse training, pruning compression, supplementary training, and quantization compression to optimize and compress the multimodal model, thereby reducing model size and improving inference speed, enabling the deployment and inference of the orbital classification AI model on mobile devices.

[0048] Example 1

[0049] This embodiment provides a multimodal orbital classification and recognition method based on edge AI. This method is applied to a mobile terminal and includes the following steps:

[0050] A two-dimensional color image of a face to be identified is obtained, and the two-dimensional color image of the face is segmented to obtain a color image of the eye socket region;

[0051] Obtain three-dimensional face data corresponding to the two-dimensional face color image, obtain a face depth image based on the three-dimensional face data, perform region segmentation on the face depth image, and obtain a depth image of the eye socket region;

[0052] The color image and depth image of the orbital region are used as input to a lightweight multimodal recognition model to obtain the classification result. Specifically, the lightweight multimodal recognition model extracts features from the color image and depth image of the orbital region respectively to obtain color features and depth features. The classification result is obtained based on the concatenation result of the color features and depth features.

[0053] Region segmentation of a 2D color face image specifically involves: extracting facial landmarks from the 2D color face image to obtain the key points of the left and right eye contours; and then segmenting the eye socket region after extracting the key points of the eye contours. Similarly, region segmentation can be performed on a face depth image.

[0054] In a specific implementation, the 3D face data can be acquired using specialized equipment, and the 3D face can be converted into a 2D depth map based on the 3D point cloud data.

[0055] In a specific implementation, after segmenting the orbital region of the image, image preprocessing and image enhancement are also performed to improve recognition accuracy.

[0056] In the above method, the training process of the lightweight multimodal recognition model is as follows: Figure 1 As shown, it includes the following steps:

[0057] 1) Construct the dataset.

[0058] Construct a first network and a second network, and obtain a first dataset and a second dataset. The first dataset is a dataset composed of color images of the orbital region, each of which has corresponding category labeling information. The second dataset is a dataset composed of face depth images, each of which has corresponding category labeling information.

[0059] In this embodiment, both the first network and the second network are Resnet-34 networks.

[0060] The human eye color images in the above dataset vary in aspect ratio and size, but they should all exactly include the eye socket area without excessive facial redundancy. After obtaining the dataset, it was randomly divided into a training set, a validation set, and a test set in a ratio of 7:2:1.

[0061] The above category labeling information can include orbital disease positive and orbital disease negative, or be labeled according to low, medium and high risk levels.

[0062] 2) Sparse training.

[0063] Based on the first dataset and the second dataset, sparse training is performed on the first network and the second network respectively to obtain the first network a1 and the second network b1.

[0064] To achieve pruning optimization and compression in the ResNet-34 network, 300 rounds of sparse training are required first. The ResNet-34 network for processing human eye color images and the ResNet-34 network for processing human eye depth images need to undergo their respective sparse training separately.

[0065] Sparse training, on the basis of normal training, applies L1 regularization constraints to the scaling factor of the BN layer and the convolution weight coefficients in the loss function. As a result, after a certain number of epochs of sparse training, the numerical distribution of most of the scaling factor of the BN layer and the weight coefficients of the convolution layer is concentrated around 0.

[0066] 3) Pruning.

[0067] The ResNet-34 network obtained after sparse training is pruned. The first network a1 and the second network b1 need to undergo their own pruning and compression processes separately.

[0068] Pruning is divided into structured pruning and unstructured pruning. In structured pruning, based on the scaling factor of the BN layer, the convolution-BN layer channels corresponding to BN layer scaling factors less than the threshold T1, as well as several convolutional filters associated with them, are deleted. In unstructured pruning, based on the magnitude of the convolution weight coefficients, the convolution weight coefficients in the convolution kernel whose absolute value is less than the threshold T2 are set to zero.

[0069] The threshold T1 is calculated based on the scaling factor distribution of the BN layer in network a1. a This ensures that 80% of the BN layer scaling factors are less than the threshold T1. a 20% of BN layer scaling factors are greater than the threshold T1 a Therefore, the values ​​of T1 and below in network a1 will be... a After deleting the convolution-BN layer channels corresponding to the BN layer scaling factor and the preceding and following convolutional filters, 80% structured pruning and sparsification of network a1 can be achieved.

[0070] The threshold T2 is calculated based on the distribution of convolution weight coefficients in network a1. a This ensures that 70% of the convolution weight coefficients are less than the threshold T1. a 30% of the convolution weights are greater than the threshold T2 a Therefore, in network a1, the absolute value less than the threshold T2 is... a Setting the convolution weight coefficients to zero can achieve 70% unstructured pruning and sparsification of network a1.

[0071] After structured pruning and unstructured pruning, network a2 is obtained.

[0072] Perform the same operation on network b1.

[0073] The threshold T1 is calculated based on the scaling factor distribution of the BN layer in network b1. b This ensures that 80% of the BN layer scaling factors are less than the threshold T1. b 20% of BN layer scaling factors are greater than the threshold T1 b Therefore, network b1 with values ​​less than the threshold T1 will be... b After deleting the convolution-BN layer channels corresponding to the BN layer scaling factor and the preceding and following convolutional filters, 80% structured pruning and network sparsification of network b1 can be achieved.

[0074] The threshold T2 is calculated based on the distribution of convolution weight coefficients in network b1. bThis ensures that 70% of the convolution weight coefficients are less than the threshold T1. b 30% of the convolution weights are greater than the threshold T2 b Therefore, in network b1, the absolute value less than the threshold T2 will be... b Setting the convolution weight coefficients to zero can achieve 70% unstructured pruning and sparsification of network b1.

[0075] After structured pruning and unstructured pruning, network b2 is obtained.

[0076] The weight coefficient distributions after sparse training and pruning are as follows: Figure 2 As shown.

[0077] 4) Supplementary training.

[0078] Since model pruning significantly impacts inference performance, 100 rounds of supplementary training (fine-tuning) are required for networks a2 and b2. This supplementary training follows the normal training method, without requiring additional L1 regularization constraints on the BN layer scaling factors and convolutional weight coefficients in the loss function. Weight coefficients set to zero or pruned convolutional layer filter channels remain unaffected or unchanged during supplementary training. After supplementary training, networks a3 and b3 are obtained.

[0079] 5) Integrated training.

[0080] The feature vectors output by the first and second networks obtained in step 4) are concatenated, and the concatenated feature vectors are input into a fully connected layer to construct a joint recognition model. The other parts before the fully connected layer in the joint recognition model are frozen, and the frozen joint recognition model is fused and trained based on the first and second datasets.

[0081] When constructing the joint recognition model, the fully connected layers in both the first and second networks are removed, and the feature vectors output by the GAP (Global Average Pooling) layers in both networks are concatenated to form a new feature vector. A fully connected layer for classification is then added, and the concatenated feature vector is input into this layer for classification. Thus, the first and second networks are combined into a multimodal model.

[0082] The fully connected layers of the multimodal model are trained using the same training and validation sets. Before training, all modules before the fully connected layers are frozen; that is, the parameters of the original model outside the fully connected layers are not modified during training. After 100 rounds of training, the trained multimodal joint recognition model is obtained.

[0083] 6) Quantization compression.

[0084] The joint recognition model obtained in step 5) is quantized and compressed to obtain the lightweight multimodal recognition model.

[0085] like Figure 3 As shown, quantization compression maps floating-point weight coefficients to INT8 type weight coefficients, a process known as INT8 quantization. To improve quantization accuracy, INT8 quantization typically employs saturation quantization. In saturation quantization, a truncation threshold |T| is first selected. Weight coefficients exceeding the threshold |T| are uniformly quantized to ±127, while weights within the threshold range are proportionally mapped to a specific INT8 value between ±127.

[0086] The lightweight multimodal recognition model obtained from the above steps can be deployed on mobile terminals to achieve edge AI computation for multimodal orbital classification.

[0087] The output of the above method is an assessment result, which can be divided into three levels: low, medium and high according to the degree of risk. This avoids absolute screening results, improves the reliability of the results, and allows doctors to conduct a secondary review as needed.

[0088] If the above methods are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0089] Example 2

[0090] This embodiment provides a mobile terminal, including a first acquisition module, a second acquisition module, and a recognition module. The first acquisition module is used to acquire a two-dimensional color image of a face to be recognized, and to perform region segmentation on the two-dimensional color image of the face to obtain a color image of the eye socket region. The second acquisition module is used to acquire three-dimensional face data corresponding to the two-dimensional color image of the face, to acquire a face depth image based on the three-dimensional face data, and to perform region segmentation on the face depth image to obtain a depth image of the eye socket region. The recognition module is used to use the color image of the eye socket region and the depth image of the eye socket region as input to a lightweight multimodal recognition model to obtain a classification result.

[0091] The recognition module includes a module training unit, which trains and maintains the lightweight multimodal recognition model. The training process of the lightweight multimodal recognition model includes:

[0092] 1) Construct a first network and a second network, and obtain a first dataset and a second dataset. The first dataset is a dataset composed of color images of the orbital region, each of which has corresponding category labeling information. The second dataset is a dataset composed of face depth images, each of which has corresponding category labeling information.

[0093] 2) Based on the first dataset and the second dataset, perform sparse training on the first network and the second network respectively;

[0094] 3) Prune and compress the first and second networks obtained in step 2) respectively;

[0095] 4) Train the first network and the second network obtained in step 3) normally respectively;

[0096] 5) Concatenate the feature vectors output by the first network and the second network obtained in step 4), and input the concatenated feature vector into a fully connected layer to construct a joint recognition model;

[0097] 6) Freeze the remaining parts of the joint recognition model before the fully connected layer, and simultaneously perform supplementary training on the frozen joint recognition model based on the first dataset and the second dataset;

[0098] 7) Quantize and compress the joint recognition model obtained in step 6) to obtain the lightweight multimodal recognition model.

[0099] The rest is the same as in Example 1.

[0100] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A multimodal orbital classification and recognition method based on edge AI, characterized in that, This method is applied to mobile terminals and includes the following steps: A two-dimensional color image of a face to be identified is obtained, and the two-dimensional color image of the face is segmented to obtain a color image of the eye socket region; Obtain three-dimensional face data corresponding to the two-dimensional face color image, obtain a face depth image based on the three-dimensional face data, perform region segmentation on the face depth image, and obtain a depth image of the eye socket region; The color image and depth image of the orbital region are used as input to a lightweight multimodal recognition model to obtain the classification result. The lightweight multimodal recognition model extracts features from both the color image and the depth image of the orbital region to obtain color features and depth features, respectively. The classification result is obtained by concatenating the color features and depth features. The training process of the lightweight multimodal recognition model includes: 1) Construct a first network and a second network, and obtain a first dataset and a second dataset. The first dataset is a dataset composed of color images of the orbital region, each of which has corresponding category labeling information. The second dataset is a dataset composed of face depth images, each of which has corresponding category labeling information. 2) Based on the first dataset and the second dataset, perform sparse training on the first network and the second network respectively; 3) Prune and compress the first and second networks obtained in step 2) respectively; 4) Train the first network and the second network obtained in step 3) normally respectively; 5) Concatenate the feature vectors output by the first and second networks obtained in step 4), and input the concatenated feature vector into a fully connected layer to construct a joint recognition model; 6) Freeze the remaining parts of the joint recognition model before the fully connected layer, and simultaneously perform fusion training on the frozen joint recognition model based on the first dataset and the second dataset; 7) Quantize and compress the joint recognition model obtained in step 6) to obtain the lightweight multimodal recognition model.

2. The multimodal orbital classification and recognition method based on edge AI according to claim 1, characterized in that, Both the first and second networks are Resnet-34 networks.

3. The multimodal orbital classification and recognition method based on edge AI according to claim 1, characterized in that, The sparse training applies L1 regularization constraints to the BN layer scaling factor and convolution weight coefficients in the loss function.

4. The multimodal orbital classification and recognition method based on edge AI according to claim 1, characterized in that, The pruning compression includes structured pruning and unstructured pruning.

5. The multimodal orbital classification and recognition method based on edge AI according to claim 1, characterized in that, The quantization compression is INT8 quantization compression.

6. The multimodal orbital classification and recognition method based on edge AI according to claim 1, characterized in that, The number of training rounds for both normal training and fusion training is less than the number of training rounds for sparse training.

7. A mobile terminal, characterized in that, include: The first acquisition module is used to acquire a two-dimensional color image of a face to be identified, perform region segmentation on the two-dimensional color image of the face, and acquire a color image of the eye socket region. The second acquisition module is used to acquire three-dimensional face data corresponding to the two-dimensional face color image, acquire a face depth image based on the three-dimensional face data, perform region segmentation on the face depth image, and acquire an orbital region depth image. The recognition module is used to take the color image of the orbital region and the depth image of the orbital region as input to a lightweight multimodal recognition model to obtain the classification result of the recognition. The lightweight multimodal recognition model extracts features from both the color image and the depth image of the orbital region to obtain color features and depth features, respectively. The classification result is obtained by concatenating the color features and depth features. The recognition module includes a module training unit, which trains and maintains the lightweight multimodal recognition model. The training process of the lightweight multimodal recognition model includes: 1) Construct a first network and a second network, and obtain a first dataset and a second dataset. The first dataset is a dataset composed of color images of the orbital region, each of which has corresponding category labeling information. The second dataset is a dataset composed of face depth images, each of which has corresponding category labeling information. 2) Based on the first dataset and the second dataset, perform sparse training on the first network and the second network respectively; 3) Prune and compress the first and second networks obtained in step 2) respectively; 4) Train the first network and the second network obtained in step 3) normally respectively; 5) Concatenate the feature vectors output by the first and second networks obtained in step 4), and input the concatenated feature vector into a fully connected layer to construct a joint recognition model; 6) Freeze the remaining parts of the joint recognition model before the fully connected layer, and simultaneously perform fusion training on the frozen joint recognition model based on the first dataset and the second dataset; 7) Quantize and compress the joint recognition model obtained in step 6) to obtain the lightweight multimodal recognition model.

8. A computer-readable storage medium, characterized in that, It includes one or more programs that are executed by one or more processors of an electronic device, the one or more programs including instructions for performing the multimodal orbital classification and recognition method as described in any one of claims 1-6.

Images