Smart contact lens fitting system with dual-vit and three-dimensional semantic grouping embeddings

Abstract

This invention provides an intelligent orthokeratology lens fitting system based on dual ViT and three-dimensional semantic grouping embedding. It introduces dual-modal input of corneal tangent and height maps, extracts features through a dual-stream independent ViT encoder, and fuses them patch-by-patch at the same location to achieve a complete representation of the corneal three-dimensional morphology. Simultaneously, a three-dimensional semantic grouping embedding method is designed to generate three semantic vectors for each parameter: numerical value, quantile, and clinical segmentation, fully leveraging the value of clinical information. The system employs a cross-modal bidirectional attention mechanism to establish fine-grained correlations between local image features and clinical parameters, generating high-quality multimodal joint representations. Based on this, it achieves accurate recommendation of core orthokeratology lens parameters and performs dual-task prediction of wearing effects. Furthermore, an interpretable doctor suggestion module is integrated, which fully presents the AI ​​decision-making logic through attention heatmaps, feature importance ranking, and natural language suggestions generated by a large language model, improving lens fitting efficiency and patient experience.

Description

Technical Field

[0001] This invention relates to the field of technology, and in particular to a smart orthokeratology lens fitting system with dual ViT and three-dimensional semantic grouping embedding, as well as a smart orthokeratology lens fitting method, electronic device and computer-readable storage medium based on the system. Background Technology

[0002] Orthokeratology (Ortho-k) lenses are currently the core non-surgical method for controlling the progression of myopia in adolescents, and the quality of their fitting directly determines the lens's effectiveness and long-term myopia control. Existing technologies, represented by patent CN202510740810.9, use corneal axial topography as the sole image input. A Vision Transformer (ViT) predicts the parameters of VST brand Ortho-k lenses. Then, by stitching and fusing candidate lens parameters with topography, refraction data, and user characteristics, the technology predicts the type of corneal difference map and the annual axial length growth rate after lens wearing, thus enabling Ortho-k lens-assisted fitting. Subsequent technologies have expanded this by incorporating nine additional clinical parameters and adding interpretable output and a three-level risk stratification module.

[0003] However, the aforementioned existing technologies have four major drawbacks that severely limit the accuracy and clinical usability of intelligent fitting of orthokeratology lenses: 1. Limited image modality and incomplete three-dimensional representation of corneal morphology: Existing technologies only use a single image type, such as corneal axial or tangential views, without introducing an elevation map that carries absolute deviation information of corneal morphology. This makes it impossible to fully represent the three-dimensional morphological features of the cornea and results in the loss of corneal height deviation information related to lens fitting height.

[0004] 2. The fusion method is crude and cannot establish fine-grained cross-modal associations: Existing technologies only use simple splicing and fusion of image features and structured clinical parameters. The model cannot learn the inherent clinical logic between local anatomical regions of the cornea and specific clinical parameters, and cannot establish fine-grained feature correspondences. Multimodal fusion is just a simple superposition of features, resulting in poor fusion quality.

[0005] 3. Limited semantics of parameters and insufficient utilization of clinical information: Existing technologies only perform single-dimensional linear embedding of clinical structured parameters, which cannot simultaneously express the three core semantic layers of parameters: absolute value, relative position in the same age group, and clinical grading meaning. This results in insufficient utilization of clinical information of the parameters, and the model cannot fit the decision-making logic of ophthalmology clinics.

[0006] 4. Lack of natural language clinical suggestions, resulting in insufficient interpretability and clinical usability: Existing systems only output structured numerical results, requiring doctors to interpret the decision-making logic of AI recommendations themselves, which places a heavy cognitive burden on them; at the same time, a complete interpretability chain from underlying visual features to high-level clinical semantics has not been constructed, which cannot meet the regulatory requirements and clinical implementation needs of medical AI. Summary of the Invention

[0007] To address the technical problems existing in the prior art, the present invention provides the following technical solution: On the one hand, a smart orthokeratology lens fitting system based on dual ViT and three-dimensional semantic grouping embedding is provided, including a dual-stream image coding and co-location fusion module, a three-dimensional semantic grouping embedding module, an image-parameter cross-modal bidirectional attention fusion module, and a recommendation head and effect prediction module, wherein: The dual-stream image encoding and co-location fusion module is used to receive corneal tangent maps and corneal height maps, extract tangent map feature sequences and height map feature sequences respectively through two visual transformer encoders with independent weights, perform co-location weighted fusion of image blocks based on the natural alignment characteristics of the two image spaces, and output the fused image feature sequence. The three-dimensional semantic grouping and embedding module is used to receive nine clinical structured parameters, divide the parameters into refractive group, biometric group and demographic group according to clinical semantics, perform three-dimensional parallel embedding of numerical embedding, quantile embedding and clinical segment embedding for each parameter, and output the structured parameter feature sequence after vector concatenation and linear projection. The image-parameter cross-modal bidirectional attention fusion module is communicatively connected to the dual-stream image coding and co-location fusion module and the three-dimensional semantic grouping embedding module. It is used to receive the fused image feature sequence and the structured parameter feature sequence, perform bidirectional cross-modal attention calculation of image-guided parameter attention and parameter-guided image attention, and output a multimodal joint representation vector. The recommendation head and effect prediction module are communicatively connected to the image-parameter cross-modal bidirectional attention fusion module. They are used to predict the core parameters of the orthokeratology lens based on the multimodal joint representation vector, complete the generation of the candidate lens parameter set, and perform dual-task binary classification prediction on the wearing effect of the candidate lenses, triggering the re-recommendation process.

[0008] Preferably, in the dual-stream image encoding and co-location fusion module, the two visual transformer encoders are a first encoder dedicated to tangent maps and a second encoder dedicated to height maps. Each encoder divides the input 224×224 resolution image into N non-overlapping image blocks. Each image block is linearly projected and mapped to a d-dimensional vector. After adding learnable positional encoding, features are extracted by a multi-layer self-attention module, and N×d-dimensional tangent map feature sequences and height map feature sequences are output respectively. The formula for weighted fusion at the same location for each image block is as follows:

[0009] in, For the fusion features of the i-th image patch, For the i-th image patch features of the tangent plot, For the features of the i-th image patch in the height map, and For learnable fusion weight scalars, satisfying .

[0010] Preferably, in the three-dimensional semantic grouping embedding module, the refractive group includes spherical power, cylindrical power, and uncorrected visual acuity; the biometric group includes axial length, flat corneal curvature K1, steep corneal curvature K2, and horizontally visible iris diameter HVID; and the demographic group includes age and sex. The numerical embedding is used to map the original scalar value of the parameter into a d / 3-dimensional vector through a linear transformation, preserving the precise absolute value information of the parameter; the quantile embedding is used to map the parameter value into a percentile index in the corresponding population stratification, and encodes it into a d / 3-dimensional vector through an embedding matrix lookup table, supplementing the relative position information of the parameter in the population; the clinical segmentation embedding is used to discretize the parameter value into a level index according to clinical standards, and encode it into a d / 3-dimensional vector through an embedding matrix lookup table, giving the parameter clinical grading semantics. After concatenating the three embedding vectors of the same parameter, they are mapped to d-dimensional parameter labels through a linear projection layer. The nine parameters are finally generated into a 9×d-dimensional structured parameter feature sequence.

[0011] Preferably, in the image-parameter cross-modal bidirectional attention fusion module, the basic calculation formula for scaled dot product attention is: in: Q is the query matrix, K is the key matrix, and V is the value matrix. The dimension of the key vector; The image-guided parameter attention uses the fused image feature sequence as the query end and the structured parameter feature sequence as the key end. The corresponding Q, K, and V matrices are generated through linear projection, and the attention weight matrix and the N×d-dimensional first output are calculated, so that the features of each image block are aggregated with relevant clinical parameter information. The parameters guide image attention, using structured parameter feature sequences as the query end and fused image feature sequences as the key end. The corresponding Q, K, and V matrices are generated through linear projection, and the attention weight matrix and a 9×d-dimensional second output are calculated, so that the model assigns higher weights to corneal regions related to clinical parameters. After the two outputs are spliced ​​and linearly projected, a d-dimensional multimodal joint representation vector is generated.

[0012] Preferably, the recommendation head and effect prediction module includes a lens parameter recommendation head and a dual-task prediction head for wearing effect, wherein: The lens parameter recommendation head is a three-layer fully connected regression network. It uses a multimodal joint representation vector as input to predict three core parameters of VST orthokeratology lenses: inner curvature radius ac1, total lens diameter, and toric astigmatism correction. It also completes the remaining parameters according to clinical rules based on corneal curvature and target correction power to generate a candidate lens parameter set. The dual-task prediction head for wearing effect is a dual-head classification network, consisting of a shared feature extraction backbone and two independent binary classification output heads. It is used to predict the corneal difference map adaptation type and the control effect of annual axial growth rate after wearing lenses. When the prediction probability of either task is lower than the 0.5 threshold, a re-recommendation process is triggered.

[0013] Preferably, it also includes a doctor suggestion module, which is communicatively connected to the recommendation head and effect prediction module and the image-parameter cross-modal bidirectional attention fusion module, including an attention heatmap generation unit, a feature importance calculation unit, and an ophthalmology-domain fine-tuned large language model; The attention heatmap generation unit is used to extract the parameter-guided image attention weight matrix and generate a corneal region attention heatmap. The feature importance calculation unit uses a gradient multiplication of input algorithm to calculate the importance score of each clinical parameter to the recommendation result and outputs a normalized Top-N ranking. The ophthalmology-based fine-tuned large language model is used to receive structured prompts composed of recommended lens parameters, effect prediction results, heatmap region descriptions, and feature importance rankings, and to generate natural language clinical suggestions for doctors.

[0014] Preferably, in the dual-stream image encoding and co-location fusion module, the visual transformer encoder can be replaced with DeiT, Swing Transformer, BEiT variants, or a hybrid architecture of CNN and Transformer; the two-image fusion method can be replaced with element-wise multiplication, gated fusion, or lightweight attention fusion; the input image can be expanded to include three or more image streams containing axial images and difference images.

[0015] Preferably, in the three-dimensional semantic grouping embedding module, the percentile calculation of quantile embedding can be statistically analyzed by age group, the clinical segmentation threshold can be adjusted according to the clinical center standard, the concatenation method of the three semantic vectors can be replaced by weighted summation, and the parameter grouping method can be replaced by automatic learning grouping based on data-driven correlation analysis.

[0016] Preferably, the difference map adaptation type prediction can be replaced with multi-class prediction of good centering / slight skewness / severe skewness / incomplete rings, and the annual growth rate prediction of axial length can be replaced with continuous regression prediction.

[0017] On the other hand, a method for intelligent selection of orthokeratology lenses based on dual ViT and three-dimensional semantic grouping embedding is provided, which is implemented based on the system described above and includes the following steps: Data Acquisition and Preprocessing: Corneal tangential and corneal height maps of patients were acquired, preprocessed to 224×224 pixel resolution and normalized; 9 clinical structured parameters of patients were acquired and semantically grouped and standardized according to refractive group, biometric group and demographic group. Dual-stream image coding and co-location feature fusion: Tangent map and height map are fed into two weighted visual transformer encoders to extract tangent map feature sequences and height map feature sequences. Co-location image block features are weighted and summed to generate a fused image feature sequence. Three-dimensional semantic grouping and embedding of clinical parameters: For the nine grouped clinical parameters, numerical embedding, quantile embedding and clinical segment embedding are performed respectively. The three embedding vectors of the same parameter are concatenated and linearly projected to generate parameter labels, and finally the structured parameter feature sequence is obtained. Cross-modal bidirectional attention fusion: The fused image feature sequence and the structured parameter feature sequence are input into the cross-modal bidirectional attention module, and image-guided parameter attention and parameter-guided image attention calculations are performed respectively to generate a multimodal joint representation vector; Intelligent recommendation of orthokeratology lens parameters: Input the multimodal joint representation vector into the fully connected regression recommendation head to predict the core parameters of orthokeratology lenses, and complete the remaining parameters according to clinical rules to generate a set of candidate lens parameters; Wearing effect prediction and re-recommendation: The candidate lens parameter set is encoded and spliced ​​with the multimodal joint representation, and then fed into a dual-head classification network to predict the corneal difference map adaptation type and axial elongation control effect after wearing the lens. If the prediction of either task fails to meet the standard, the re-recommendation process is triggered. Interpretable analysis and clinical suggestion generation: Extract cross-modal attention weights to generate corneal attention heatmaps, calculate the feature importance ranking of each clinical parameter, input structured information into the ophthalmology field fine-tuning large language model, generate natural language clinical suggestions, and complete the intelligent selection process for orthokeratology lenses.

[0018] On the other hand, an electronic device is provided, comprising: a processor; and a memory storing computer-readable instructions, which, when executed by the processor, implement the method described above.

[0019] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement the above method.

[0020] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: 1. Complementary fusion of two images enables complete representation of the three-dimensional morphology of the cornea. To address the shortcomings of existing technologies that rely on a single image modality, this invention introduces tangent and height maps as two independent image modalities into the orthokeratology (Ortho-k) lens fitting system for the first time. The tangent map provides information on the distribution of corneal curvature change rate, while the height map provides information on the absolute deviation of corneal morphology. The two complement each other in describing the three-dimensional morphology of the cornea from different physical dimensions, solving the problem of incomplete representation by a single image. The dual-stream independent ViT encoder ensures the semantic purity of the two types of image features, and the patch-by-patch same-location fusion strategy preserves the spatial correspondence of corneal anatomical regions while avoiding computational redundancy in cross-image global attention, thus improving feature representation capabilities while also considering computational efficiency.

[0021] 2. Three-dimensional semantic grouping embedding enables full utilization of clinical parameter information. To address the shortcomings of existing technologies in terms of the thin semantics of parameters, this invention designs a three-dimensional parallel embedding method that combines numerical embedding, quantile embedding, and clinical segmentation embedding. This method enables each clinical parameter to simultaneously carry three layers of semantic information: precise numerical value, relative position within the population, and clinical grading. The model can accurately distinguish between the numerical information of "axial length 26.8mm," the relative risk information of "at the same age P85," and the clinical semantic information of "belonging to the high myopia level," which is significantly superior to the single-dimensional linear embedding of existing technologies. Furthermore, by grouping parameters according to clinical semantics, the decoupling of semantic features is achieved, providing higher-quality structured input for subsequent cross-modal attention and improving the model's prediction accuracy from the feature source.

[0022] 3. Bidirectional cross-modal attention, fundamentally improving the quality of multimodal feature fusion. To address the shortcomings of existing technologies in terms of coarse fusion methods, this invention designs an image-parameter cross-modal bidirectional attention mechanism, replacing the simple feature splicing and fusion in existing technologies. This mechanism can establish fine-grained cross-modal associations such as "K2 steepness → corresponding abnormal region in the temporal paracentral height map," enabling the model to learn the inherent clinical logic between local corneal morphological features and clinical parameters, rather than simply superimposing features. The bidirectional attention mechanism realizes the aggregation of information from image regions to clinical parameters and the weighting of clinical parameters to corneal regions of interest, fundamentally improving the quality of multimodal joint representation at the fusion mechanism level, and providing more reliable feature support for lens parameter recommendation and effect prediction.

[0023] 4. LLM-driven natural language suggestions significantly improve the clinical usability of the system. To address the shortcomings of existing technologies that only output structured numerical values ​​and lack clinical interpretability, this invention integrates a finely tuned large language model from the ophthalmology field at the inference end. This model can transform structured recommendation results, prediction confidence, attention heatmaps, and feature importance rankings into natural language clinical suggestions that doctors can directly understand and use. This solves the problem of doctors having to interpret AI output themselves and the heavy cognitive burden in existing technologies, and significantly improves the clinical acceptability of the system.

[0024] 5. A complete and explainable chain, meeting the regulatory requirements for medical AI. This invention constructs a complete interpretability chain, from spatial attention (heatmap) of corneal images to contribution quantification of clinical parameters (feature importance ranking) and then to natural language output of high-level semantics (LLM clinical recommendations). It fully presents the decision-making logic of AI recommendations, solves the black box problem of medical AI, and meets the hard requirements of medical device AI regulation for algorithm transparency.

[0025] 6. Recommendation accuracy and control effect prediction performance are significantly improved. The system reduces the mean absolute error in predicting the core parameters of the lenses, has a high accuracy rate in predicting the type of lens fit using the difference map, and a significant accuracy rate in predicting the axial length control effect. At the same time, the re-recommendation mechanism can improve the success rate of lens fitting on the first try, greatly reduce the number of clinical trials, and improve the efficiency of orthokeratology lens fitting and the patient experience. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 System Overall Architecture Diagram – Demonstrating the overall data flow of dual-stream ViT encoding, co-location fusion, 3D semantic grouping embedding, cross-modal attention, recommendation head, effect prediction head, and doctor suggestion module. Figure 2 : 3D semantic grouping embedding module structure diagram - showing the three groups of 9 parameters, the three embedding sub-modules v_val / v_pct / v_cls and the splicing projection process; Figure 3 Image-parameter cross-modal bidirectional attention structure diagram—showing the Q / K / V calculation process of the image-guided parametric attention sublayer and the parameter-guided image attention sublayer; Figure 4 : A schematic diagram of merging tangent plot and height plot patch by patch at the same position—showing the spatial alignment relationship of the patches in the two plots and the merging operation; Figure 5 The doctor's suggestion module output diagram shows the overlay effect of the attention heatmap, the Top-N feature importance bar chart, and the natural language suggestion text style generated by LLM. Detailed Implementation

[0028] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0029] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0030] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0031] In this embodiment of the invention, sometimes a subscript such as W1 may be mistakenly written as a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0032] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0033] I. System Technology Introduction (a) Invention Abstract This invention aims to overcome the above-mentioned shortcomings and provide a cross-modal attention-based orthokeratology lens fitting system based on dual-stream ViT coding and three-dimensional semantic grouping embedding. Specific objectives include: (1) The corneal height map is introduced as the second image modality, which complements the tangent map. The features are extracted by independent ViT encoders and then fused at the same position patch by patch to comprehensively characterize the corneal curvature distribution and three-dimensional morphological deviation information. (2) Design a three-dimensional semantic grouping embedding method to divide the nine clinical parameters into refractive group, biometric group and demographic group according to clinical semantics. Each parameter generates three semantic vectors at the same time: numerical embedding, quantile embedding and clinical segment embedding. The three vectors are concatenated and linearly projected to obtain the parameter token, resulting in a more semantically rich structured feature sequence. (3) Design an image-parameter cross-modal bidirectional attention mechanism to replace the simple splicing and fusion of the original patent, establish a fine-grained correspondence between local image patch features and clinical parameter tokens, and generate a high-quality multimodal joint representation F_fused; (4) Based on F_fused, the key parameters of VST lenses (ac1, lens diameter, torus) are accurately recommended, and the wearing effect of candidate lenses is predicted in two tasks (difference map type Good / Bad, annual axial growth rate type Good / Bad). (5) After model inference, an interpretable doctor suggestion module is integrated. The key corneal regions are marked with attention heatmaps, the top-N influencing parameters are listed by feature importance, and the large language model (LLM) is called to convert the structured analysis results into natural language clinical suggestion text, thereby improving doctors' understanding and trust in AI recommendations.

[0034] The main innovative points of this invention are as follows: Innovation Point 1 (Core): Dual-stream independent ViT encoder + patch-by-patch same-position fusion For the first time, tangent graph and height graph are introduced as two independent image modalities into the OK mirror matching system. The two ViT encoders have independent weights to ensure the semantic purity of their respective features. Based on the natural alignment of the two graph spaces, patch-by-patch same-position fusion is used to replace global cross-graph attention, which balances information complementarity and computational efficiency.

[0035] Innovation Point Two (Core): Three-Dimensional Semantic Grouping Embedding Method To address the characteristics of nine clinical parameters in orthokeratology (Ortho-k) lens fitting scenarios, a three-dimensional parallel embedding method was designed, comprising numerical embedding (precise values), quantile embedding (relative position within the population), and clinical segmentation embedding (semantic representation of clinical grading). Semantic groups were further categorized into refractive, biometric, and demographic groups. The resulting 3D embedding, stitched together and projected, yielded a parameter token sequence with significantly richer semantic information, providing higher-quality structured input for subsequent cross-modal attention.

[0036] Innovation Point 3: Image-Parameter Cross-Modal Bidirectional Attention Mechanism Using fused image features F_img and 3D semantic parameter features F_struct as input, a bidirectional cross-modal attention layer is designed to establish a fine-grained correspondence between local image patches and clinical parameter tokens, replacing the simple concat fusion in two prior patents, and fundamentally improving the quality of multimodal joint representation from the fusion mechanism level.

[0037] Innovation Point 4: LLM-Driven Explainable Doctor Recommendation Module The model integrates a large language model (LLM) at the end of the inference process. The LLM automatically generates natural language clinical suggestions for doctors by taking the anatomical location description annotated by attention heatmap, the Top-N feature importance ranking and the dual-task prediction results as structured inputs. This forms a complete interpretable chain from low-level visual features to high-level semantic suggestions, which is the first application of AI-assisted selection of orthokeratology lenses.

[0038] (II) System Architecture Introduction like Figure 1 The diagram shown is the overall system architecture diagram, illustrating dual-stream ViT encoding, co-location fusion, 3D semantic grouping embedding, cross-modal attention, recommendation head, effect prediction head, and doctor suggestion module.

[0039] The present invention provides an intelligent selection system for orthokeratology lenses with dual ViT and three-dimensional semantic grouping embedding, which consists of four core modules. The network structure, computer mechanism and technical principle of each module are explained in detail below.

[0040] This system is an end-to-end intelligent orthokeratology lens fitting system, with its core consisting of four main modules: a dual-stream image encoding and co-location fusion module, a 3D semantic grouping and embedding module, an image-parameter cross-modal bidirectional attention fusion module, a recommendation head and effect prediction head, and a doctor's suggestion module. The system takes corneal tangent maps, height maps, and nine clinical structured parameters as input. After dual-stream visual feature extraction, 3D semantic embedding of clinical parameters, and cross-modal bidirectional attention fusion, it achieves accurate recommendation of core orthokeratology lens parameters and dual-task prediction of wearing effects. Furthermore, it generates interpretable natural language clinical suggestions through a domain-fine-tuned large language model, fully covering the entire intelligent orthokeratology lens fitting process.

[0041] Module 1: Dual-stream Image Coding and Co-location Fusion Module This module is the core of the system's visual feature extraction, addressing the shortcomings of existing technologies such as single image modality and incomplete corneal morphological representation. The working principle of this module is as follows: 1. Bimodal Image Input: Two complementary corneal topographic maps are input: a tangential map reflecting the rate of change of corneal curvature, and an elevation map reflecting the absolute deviation of the corneal morphology relative to a reference sphere. Both images are preprocessed to a resolution of 224×224 pixels and normalized. After processing, they are fed into two weighted, independent visual transformer encoders (a first encoder and a second encoder).

[0042] 2. Dual-Stream Independent ViT Encoder: Two visual transformer encoders with completely independent weights are set up: a first encoder dedicated to the tangent map and a second encoder dedicated to the height map. Each encoder performs a unified feature extraction process: each encoder divides the input image into N non-overlapping image patches. Each patch is mapped to a d-dimensional vector through a linear projection layer. After adding learnable positional encoding, deep feature extraction is completed through a multi-layer self-attention module, finally outputting a tangent map patch feature sequence. (Dimension N×d) and height tile feature sequences (Dimension N×d).

[0043] Specifically as follows: 3. Image patch fusion strategy: Since the tangent map and the height map come from the same examination of the same eye, the two maps are naturally aligned in space, and the image patch at position i corresponds to the same corneal anatomical region in the two maps.

[0044] Based on this characteristic, a fusion method of weighted summation at the same location for each image block is adopted, and the fusion formula is as follows: in, and For learnable fusion weight scalars, satisfying During model training, backpropagation is used for automatic optimization to learn the optimal fusion ratio between the two types of images. The final output is a fused image feature sequence. The dimension is N×d.

[0045] The following technical means can also be used in this module: The image encoder can be replaced with Transformer variants such as DeiT, Swing Transformer, BEiT, or a hybrid architecture of CNN and Transformer (such as CvT, ResNet50). The two graphs can be fused using element-wise multiplication, gated fusion, or lightweight attention fusion. Image input can be further supplemented with an axial map or difference map as a third image stream, based on tangent and height maps.

[0046] Module 2: 3D Semantic Grouping Embedding Module like Figure 2 The diagram shows the structure of the 3D semantic grouping embedding module—displaying the three groups of 9 parameters, the three embedding sub-modules v_val / v_pct / v_cls, and the splicing projection process.

[0047] This module is the core of the system's structured feature extraction, addressing the shortcomings of existing technologies such as weak semantic parameters and insufficient utilization of clinical information. The working principle of this module is as follows: 1. Clinical semantic grouping of parameters: The nine clinical parameters are divided into three groups according to ophthalmological clinical semantics, achieving feature decoupling of parameters with the same semantic meaning: Refractive group: spherical power, cylindrical power, and uncorrected visual acuity, reflecting the patient's refractive status and visual function; Biometrics group: axial length, flat corneal curvature K1, steep corneal curvature K2, and horizontally visible iris diameter HVID, reflecting the biological structural characteristics of the eye; Demographic group: age and sex, reflecting the individual background of patients and their risk characteristics for myopia progression.

[0048] 2. Three-dimensional parallel embedding submodule: Each parameter passes through three independent embedding submodules simultaneously, generating three d / 3-dimensional vectors carrying different semantics. The specific principle is as follows: (1) Numerical embedding ( ): to convert the original scalar value Learnable linear transformation matrix (Dimension 1×d / 3) and bias The mapping is to a d / 3 dimensional real vector, and the calculation formula is: Linear transformation encodes a single scalar value into a dense, continuous vector that can participate in attention calculations, fully preserving the precise absolute value information of the parameters and capturing subtle differences in axial length down to 0.01 mm. (2) Quantile embedding ( First, stratify the empirical distribution of each parameter in the training set by age group and gender, and then calculate the parameter values ​​for the current patient. It is mapped to its percentile integer index in the corresponding stratified population. (Value range 0~100); then through the learnable embedding matrix (Dimension 101 × d / 3) is used for table lookup encoding, and the calculation formula is: This embedding enables the model to perceive the relative position of parameters within the same age and sex population, supplementing the group comparison information that absolute values ​​cannot provide, such as an axial length of 26.8 mm corresponding to the 85th percentile of the same age population as a high-risk characteristic. (3) Clinical segmentation embedding ( According to ophthalmological clinical diagnostic criteria, the parameter values ​​are discretized into a finite number of levels and assigned integer indices. (For example, axial length is classified into three grades: normal, borderline elongated, and high myopia; spherical lens power is classified into three grades: low, moderate, and high myopia); through learnable embedding matrices (Dimension C × d / 3, where C is the total number of levels) is used for table lookup coding, and the calculation formula is: This embedding assigns clinical grading semantics to the parameters, enabling a significant feature distinction between the axial length of highly myopic eyes and normal axial lengths in the embedding space, thus helping the model learn feature representations consistent with clinical decision-making rules.

[0049] 3. Vector concatenation and linear projection: Concatenate three semantic vectors with the same parameter and then project them through a linear projection layer. (Dimension d×d) is mapped to a d-dimensional parameter token representation, and the calculation formula is: ,in This represents a vector concatenation operation. The projection layer is biased. Each of the nine parameters independently completes the above 3D embedding process, ultimately generating a structured feature sequence of length 9. The dimension is 9×d.

[0050] The following technical means can also be used in this module: The percentile calculation of quantile embedding can be stratified by age group; the clinical segmentation threshold can be adjusted according to the standards of different clinical centers; the concatenation method of the three semantic vectors can be replaced by weighted summation; The parameter grouping method can be automatically learned based on data-driven correlation analysis, replacing fixed clinical semantic grouping.

[0051] Module 3: Image-Parameter Cross-Modal Bidirectional Attention Fusion Module like Figure 3 The diagram shown is a cross-modal bidirectional attention structure diagram of image and parameter—demonstrating the Q / K / V calculation process of the image-guided parameter attention sublayer and the parameter-guided image attention sublayer.

[0052] The working principle of this module is as follows: This module uses fused image feature sequences. (N×d dimensional) and three-dimensional semantic structured feature sequences Using (9×d-dimensional) data as input, a bidirectional cross-modal attention mechanism is employed to establish a fine-grained correspondence between local image regions and clinical parameters. The core computational process is as follows: 1. Basic formula for attention calculation: Scaling the dot product attention as the basic calculation unit avoids the gradient vanishing problem caused by excessively large dot product results. The formula is as follows: in: (Query matrix) (Key matrix) The (value matrices) are obtained from the input features through learnable linear transformations. The dimension of the key vector. Scaling factor, divided by Its purpose is to prevent the gradient from vanishing due to an excessively large dot product result.

[0053] 2. Image-guided parameter attention sublayer: to fuse image features For the query end, structured parameter features For the key-value pairs, the corresponding Q, K, and V matrices are generated through linear projection: in , , This is the learnable projection matrix. The calculated attention weight matrix... The dimension is N×9, where This represents the attention weight of the i-th image patch to the j-th clinical parameter label, reflecting the correlation strength between the corneal anatomy region and the corresponding clinical parameter; the final output is... With a dimension of N×d, each image patch feature aggregates the clinical parameter information most relevant to it.

[0054] 3. Parameter-guided image attention sublayer: using structured parametric features For the query end, image features are fused. For the key-value pairs, the corresponding Q, K, and V matrices are generated through linear projection:

[0055] in , , This is the learnable projection matrix. The calculated attention weight matrix... The dimension is 9×N, where This represents the attention weight of the j-th clinical parameter to the i-th image patch, enabling the model to assign higher weights to the corneal region most relevant to the patient's clinical parameters during feature extraction; the final output... The dimension is 9×d.

[0056] 4. Multi-head Attention and Feature Output: A multi-head attention mechanism is employed, with h=8 attention heads set up in parallel to capture cross-modal correlations across different feature subspaces. The two attention outputs are then processed. and The vectors are concatenated and mapped to a d-dimensional vector via a linear projection layer, ultimately yielding the multimodal joint representation vector. (d-dimensional), serving as the unified input for subsequent recommendation heads and effect prediction heads.

[0057] In this module, the following techniques can also be used: bidirectional attention can be extended to multi-head cross-attention, or CLIP style contrast pre-training can be introduced to complete weight initialization.

[0058] Module Four: Recommended Head, Effect Prediction Head, and Doctor's Suggestions This module is the core of the system's output and interpretability, addressing the shortcomings of existing technologies such as the lack of natural language suggestions and insufficient clinical usability. It is divided into three sub-units: 1. Recommended lens parameters Network Structure: Multimodal Joint Representation As input, a three-layer fully connected regression network is used, with the specific structure as follows: fully connected layer → batch normalization layer → ReLU activation function → Dropout layer → fully connected layer. Finally, it predicts and outputs three core personalized parameters of VST orthokeratology lenses: inner curvature radius ac1 (unit: mm), total lens diameter (unit: mm), and toric astigmatism correction (unit: diopter).

[0059] Training and Inference: During the training phase, the mean squared error (MSE) loss function is used to supervise learning by comparing predicted values ​​with the true values ​​of historical expert prescription parameters. The Adam optimizer is used to iteratively update the network weights. During the validation phase, the recommendation accuracy is evaluated using mean absolute error (MAE) on an independent validation set. After the model converges, the input... The system can output three core prediction parameters; the remaining parameters (base curve radius of curvature BC, reduction in diopters, and outer curve radius of curvature ac2) are calculated and supplemented according to clinical rules based on the corneal curvature K value and the target correction power, together forming the candidate lens parameter set. .

[0060] 2. Dual-Task Prediction Head for Wearing Effects (Network Structure and Binary Classification of the Effect Prediction Head) Input processing: The candidate lens parameter set Encoded into a d-dimensional vector by the embedding layer, and jointly represented with multimodal modes. By concatenating the features, a 2d-dimensional joint feature vector is obtained.

[0061] Network Structure: A dual-head classification network is adopted, consisting of a shared feature extraction backbone and two independent binary classification output heads. The shared backbone consists of two fully connected layers → batch normalization layer → ReLU activation function, outputting d-dimensional shared features; each of the two output heads consists of a fully connected layer plus a Sigmoid activation function, performing two binary classification predictions respectively. Task A: Difference Plot Type Prediction, Output This indicates the probability of a good lens fit (a good fit is defined as a corneal difference map showing a standard bullseye shape centered and a complete and symmetrical steepening ring after wearing the lens). Task B: Prediction of annual axial length growth rate, output This indicates the probability that myopia control is effective (effective control is defined as an annual axial growth rate of less than 0.3 mm / year).

[0062] Training and Re-recommendation Mechanism: During the training phase, two output heads are supervised separately using a binary cross-entropy loss function, with the total loss being the sum of the losses from both tasks. During the inference phase, if the output probability of either task falls below a threshold of 0.5, a re-recommendation process is triggered. The candidate lens parameters are automatically adjusted, and predictions are re-made until both tasks predict well or the maximum number of re-recommendations is reached.

[0063] In this module, the following techniques can also be used: the difference map classification can be refined into multiple categories (well centered / mildly skewed / severely skewed / incomplete ring); the axial velocity threshold can be adjusted according to age group, or replaced with continuous regression prediction.

[0064] 3. Doctor's Recommendation Module (Reasoning Phase) This module constructs a complete interpretable chain from low-level features to high-level semantics, and consists of three core steps: (1) Attention heatmap generation: extracting parameters to guide the image attention weight matrix. (9×N dimensions), the average value of the 9 parameter dimensions is taken to obtain the comprehensive attention of each image patch (1×N dimensions), which is reshaped into a two-dimensional weight map corresponding to the spatial location of the image patch; it is upsampled to the original topographic map at a resolution of 224×224 pixels by bilinear interpolation, and after Gaussian smoothing, it is superimposed on the original tangent map with pseudo-color to generate a high-brightness heat map, which accurately marks the corneal area of ​​most interest to the model (such as the abnormal area of ​​the 3~5 mm annular band around the pericenter). (2) Feature importance calculation: the gradient multiplied by the input (Gradient×Input) algorithm is adopted, and the target output of the recommendation head is set to The d-dimensional label vector for the i-th parameter Calculate partial derivatives , and then with After multiplying each element by its element and taking the L2 norm, we obtain the importance score of the i-th parameter:

[0065] in This is the element-wise product operator. After normalizing the importance scores of all parameters to percentages, the Top-N rankings are output in descending order to quantitatively reflect the actual contribution of each parameter to the current recommendation result. (3) Domain-adapted large language model natural language suggestion generation: Model fine-tuning: A domain-supervised fine-tuning (SFT) strategy was adopted. Texts from ophthalmology clinical guidelines, orthokeratology lens fitting standards, and historical case reports were collected to supervise and fine-tune the open-source large language model, enabling it to understand ophthalmology terminology, generate clinical suggestions, and understand the semantics of the output of preceding modules.

[0066] Inference generation: Recommending lens parameter set The dual-task prediction results and confidence scores, Top-N feature importance rankings, and anatomical location descriptions of highlighted areas in the heatmap (mapping pixel coordinates to clinical anatomical descriptions such as paracentral temporal, nasal, superior, and inferior regions through corneal partitioning rules) are organized into structured prompts. These are then input into a domain-fine-tuned large language model to automatically generate natural language clinical advice text for doctors. The content includes the clinical basis for recommended parameters, explanations of expected wearing effects, and personalized follow-up suggestions.

[0067] In this module, the following technical means can also be used: the large language model that generates natural language suggestions can be replaced with any open source model (such as LLaMA, Qwen series) or a privately deployed model, and the Prompt template can be customized according to clinical needs.

[0068] The implementation of this system is divided into a model training phase and a clinical inference phase: 1. Model training phase (1) Dataset construction: Collect clinical orthokeratology lens fitting case data, including patients' corneal tangent diagrams, height diagrams, 9 clinical structured parameters, expert gold standard parameters for lens fitting, corneal difference map type annotation after lens wearing, and annual axial length growth rate annotation. Divide the data into training set, validation set and test set in a ratio of 8:1:1. Perform data augmentation on the training set data, including random image flipping, brightness fine adjustment and parameter Gaussian noise perturbation, to improve the model's generalization ability.

[0069] (2) Weight initialization: The two ViT encoders are initialized with Image Net pre-trained weights, the three-dimensional embedding matrix and the cross-modal attention projection matrix are initialized with Xavier uniform distribution, and the fully connected layer is initialized with He normal distribution.

[0070] (3) Multi-task joint training: A two-stage training strategy is adopted. In the first stage, the dual-stream image coding module and the three-dimensional semantic embedding module are pre-trained with lens parameter prediction as the supervised target. In the second stage, the entire system is jointly trained end-to-end. The total loss function is the sum of the MSE loss of lens parameter regression and the binary cross-entropy loss of dual-task classification. The Adam W optimizer is adopted, with the initial learning rate set to 1e-4, the batch size set to 32, and the training rounds set to 100 rounds. An early stopping mechanism is adopted to avoid overfitting. Training is terminated when the validation set loss does not decrease for 10 consecutive rounds. (4) Model validation and optimization: The model performance is validated on the test set. The core indicators include the mean absolute error of lens parameter prediction, the difference map classification accuracy, and the axial length control effect classification accuracy. Feature analysis is performed on cases with large model prediction deviations to optimize the embedding layer threshold and the number of attention heads, thereby improving the model's generalization ability on niche cases.

[0071] (5) Fine-tuning of the large language model: Construct a fine-tuning dataset of instructions for orthokeratology lens prescription, which includes 2,000 structured case inputs and corresponding clinical expert suggestions. Use the LoRA low-rank adaptation method to fine-tune the open-source large language model. The training rounds are set to 3 rounds and the learning rate is set to 2e-5. After completion, the weights are merged and deployed as an inference service.

[0072] 2. Clinical reasoning stage (1) Data input and preprocessing: collect corneal tangent and height maps of patients, preprocess them to 224×224 resolution and normalize them; enter 9 clinical structured parameters of patients, and complete numerical standardization and outlier filtering.

[0073] (2) Dual-stream image feature extraction and fusion: Tangent map and height map are respectively input into two independent ViT encoders to extract image patch feature sequences. The fused image feature sequences are obtained by weighted fusion at the same position in each patch (e.g., Figure 4 The diagram shown illustrates the merging of the tangent plot and the height plot at the same position, demonstrating the spatial alignment of patches in both plots and the merging operation.

[0074] (3) Three-dimensional semantic embedding of clinical parameters: The nine parameters were grouped into refractive group, biometric group and demographic group, and numerical embedding, quantile embedding and clinical segment embedding were completed respectively. After splicing and projection, a structured feature sequence was obtained.

[0075] (4) Cross-modal bidirectional attention fusion: Input image features and parameter features into the cross-modal attention module to complete bidirectional cross-attention calculation and generate a multimodal joint representation vector.

[0076] (5) Lens parameter recommendation and effect prediction: Input the joint characterization into the recommendation head to obtain three core lens parameters, complete the remaining parameters according to clinical rules, and generate a candidate lens parameter set; input the candidate parameters and joint characterization into the effect prediction head to complete the dual-task binary classification prediction. If the prediction result does not meet the standard, the re-recommendation process is triggered.

[0077] (6) Interpretable Analysis and Clinical Recommendation Generation: Generate attention heatmaps and feature importance rankings, input structured information into a domain-fine-tuned large language model, generate natural language clinical recommendations, and finally output complete lens recommendation plans, effect prediction results, and clinical recommendations to doctors. Figure 5 The image shows a schematic diagram of the doctor suggestion module output—demonstrating the overlay effect of the attention heatmap, the bar chart of the importance of Top-N features, and the style of the natural language suggestion text generated by LLM.

[0078] The implementation process of this system will be described in detail below, based on the system-based approach.

[0079] II. Intelligent Selection Method for Orthokeratology Lenses Based on Dual ViT and 3D Semantic Grouping Embedding This method is based on the above system and covers the entire process of intelligent fitting of orthokeratology lenses. The specific steps are as follows: Step 1: Data Acquisition and Preprocessing Corneal tangent and corneal height maps were collected from patients. Both types of images were preprocessed to a resolution of 224×224 pixels and normalized. At the same time, nine clinical structured parameters of patients were collected, and semantic grouping was performed according to refractive group, biometric group, and demographic group. Numerical standardization and outlier filtering were also performed.

[0080] Step 2: Dual-stream image coding and co-location feature fusion The preprocessed tangent map is fed into the first ViT encoder, and the height map is fed into the weighted independent second ViT encoder. The two encoders perform block segmentation, linear projection, position encoding, and multi-layer self-attention feature extraction on the input image, respectively, and output tangent map block feature sequences. With height patch feature sequence Based on the natural alignment of the two graph spaces, the features of image patches at the same location are weighted and fused to obtain a fused image feature sequence. .

[0081] Step 3: Embedding three-dimensional semantic grouping of clinical parameters For the nine grouped clinical parameters, a three-dimensional parallel embedding process was executed respectively: (1) Numerical embedding: The original scalar values ​​of the parameters are mapped to d / 3-dimensional dense vectors through linear transformation, while retaining the precise absolute value information of the parameters; (2) Quantile embedding: The parameter value is mapped to the percentile index in the corresponding population stratum, and encoded into a d / 3-dimensional vector by looking up the table through the embedding matrix to supplement the relative position information of the parameter in the population; (3) Clinical segmentation embedding: The parameter values ​​are discretized into grade indexes according to clinical standards, and encoded into d / 3-dimensional vectors by looking up the table through the embedding matrix, and the parameters are given clinical grade semantics; The three embedding vectors of the same parameter are concatenated and then linearly projected to generate d-dimensional parameter labels. The nine parameters ultimately form a structured feature sequence. .

[0082] Step 4: Image-parameter cross-modal bidirectional attention fusion Fuse image feature sequences With structured feature sequences The input cross-modal bidirectional attention module performs image-guided parameter attention calculation and parameter-guided image attention calculation respectively. A multi-head attention mechanism captures cross-modal associations in different subspaces. The two attention outputs are concatenated and linearly projected to generate a multi-modal joint representation vector. .

[0083] Step 5: Intelligent Recommendation of Orthokeratology Lens Parameters Multimodal joint representation vector Inputting a three-layer fully connected regression recommendation head, it predicts three core parameters of the VST orthokeratology lens: inner curvature radius ac1, total lens diameter, and annular astigmatism correction. Based on corneal curvature and target correction power, it completes other parameters such as base curve, reduction power, and outer curvature according to clinical rules, generating a candidate lens parameter set. .

[0084] Step 6: Dual-task prediction and re-recommendation of wearing effect Candidate lens parameter set Encoded and jointly characterized with multimodal features The images are stitched together and fed into a dual-head classification network to predict the corneal difference map adaptation type and the control effect of the annual axial growth rate after wearing the lenses. If the probability of good adaptation / effective control in either task is lower than the 0.5 threshold, the core parameters of the lenses are automatically adjusted and the prediction is repeated until both tasks meet the standard or the maximum number of re-recommendations is reached.

[0085] Step 7: Interpretability Analysis and Clinical Recommendation Generation Cross-modal attention weights are extracted to generate corneal region attention heatmaps. The feature importance ranking of each clinical parameter is calculated by multiplying the gradient by the input algorithm. The recommended parameters, effect prediction results, heatmap region descriptions, and feature importance rankings are organized into structured prompts, which are then input into a large language model fine-tuned in the ophthalmology field to generate natural language clinical suggestions for doctors, thus completing the entire intelligent selection process for orthokeratology lenses.

[0086] Example 1 This embodiment, based on a real clinical patient case, fully demonstrates the implementation process and detailed calculation process of this system. The patient's basic information is as follows: Patient: Female, 9 years old, uncorrected visual acuity 0.08; Refractive group parameters: spherical lens -5.25D, cylindrical lens -0.50D, uncorrected visual acuity 0.08; Biometric group parameters: axial length 27.2mm, K1=43.00D, K2=43.75D, HVID=11.6mm; Demographic group parameters: age 9 years old, sex female.

[0087] In this embodiment, the model feature dimension d=512, the number of image blocks N=196 (a 224×224 image is divided into 14×14 image blocks), the number of attention heads h=8, and the fusion weights are optimized after training. , .

[0088] Step 1: Dual-stream image input and feature fusion calculation 1. Image Preprocessing: Corneal tangent and height maps of the patient were acquired, uniformly scaled to 224×224 pixels, and normalized using Image Net mean and standard deviation. The formula is as follows: in , This yields the standardized input image.

[0089] 2. Dual-stream ViT Feature Extraction: The tangent map is fed into the first ViT encoder (ViT-Tan), and the height map is fed into the second ViT encoder (ViT-Hgt). Each encoder divides the image into 14×14=196 16×16 image blocks. Each image block is linearly projected into a 512-dimensional vector. After adding a 196-dimensional learnable positional encoding, features are extracted through a 12-layer Transformer self-attention module, finally outputting the tangent map feature sequence. (196×512 dimensional) and height map feature sequences (196×512 dimensions).

[0090] 3. Patch-by-patch fusion at the same location: The features of image patches at the same location in two images are weighted and summed. Taking the i-th image patch as an example, the fusion calculation is as follows: The fusion process is completed by traversing 196 image patches, ultimately yielding the fused image feature sequence. The dimensions are 196×512.

[0091] Step 2: Detailed computation of 3D semantic grouping embedding Three-dimensional embedding was performed on the nine parameters according to the refractive group, biometric group, and demographic group. Taking an axial length of 27.2 mm as an example, the detailed calculation process is as follows: 1. Numerical embedding: Original scalar value of axial length x=27.2, linear transformation matrix Dimension 1×170 (d / 3=512 / 3≈170), offset Given a 170-dimensional vector, the calculation yields: Output a 170-dimensional numerical embedding vector, fully preserving the precise numerical information of 27.2 mm.

[0092] 2. Quantile Embedding: Based on the empirical distribution of axial length in 9-year-old females in the training set, 27.2mm corresponds to the 94th percentile of the same age group, i.e., index p=94; embedding matrix With dimensions of 101×170, looking up the 94th row vector in the table yields the 170-dimensional quantile embedding vector. It carries relative positional information that "the axial length of the eye is significantly longer than that of its peers".

[0093] 3. Clinical Segmentation Embedding: According to clinical standards, an axial length > 26mm is classified as high myopia, corresponding to index c=2; Embedding Matrix With a dimension of 3×170, the second row vector is retrieved from the table to obtain a 170-dimensional clinical segment embedding vector. It carries the clinical classification semantics of "high myopia axial length".

[0094] 4. Concatenation and Linear Projection: Concatenate three 170-dimensional vectors into a 510-dimensional vector, pad with zeros to bring it to 512 dimensions, and then project it through a 512×512 linear projection layer. The calculation yields a 512-dimensional token vector representing the axial length parameters. Following the same process, the remaining 8 parameters are embedded in 3D, ultimately resulting in a 9×512-dimensional structured feature sequence. .

[0095] Step 3: Cross-modal bidirectional attention fusion computation 1. Image-guided parameter attention: with For the query end, As key values, Q1 (196×512), K1 (9×512), and V1 (9×512) are obtained through linear projection. The attention values ​​are then calculated using the scaling dot product attention formula. The attention weight matrix A1 (196×9) was calculated, in which the K2 (steep curvature) token and the temporal paracentral region image patch had the highest attention weight, reaching 0.72; the final output O1 (196×512 dimensions).

[0096] 2. Parameter-guided image attention: For the query end, As the key value, Q2 (9×512), K2 (196×512), and V2 (196×512) are obtained through linear projection. Similarly, the attention weight matrix A2 (9×196) is calculated. Among them, the axial length and age token have the highest attention weight for the 3~5mm annular image block in the paracentral region, with an average value of 0.68. The final output is O2 (9×512 dimensional).

[0097] 3. Multi-head fusion and output: The outputs of the eight attention heads are concatenated, and then O1 and O2 are flattened and concatenated. The result is mapped to a 512-dimensional multimodal joint representation vector through a linear projection layer. .

[0098] Step 4: Recommended Head and Effect Prediction Calculation 1. Recommended lens parameters: The input is a three-layer fully connected regression network. After forward propagation, the output is the predicted values ​​of three core parameters: inner arc curvature radius ac1 = 8.10 mm, lens diameter = 10.4 mm, and torus = 0.50 D.

[0099] 2. Clinical rule completion: Based on a flat corneal curvature K1 = 43.00D, calculate the base curve BC using the clinical formula: Simultaneously, parameters such as the reduction of -5.25D and the outer curvature radius ac2 are added according to the target correction power to generate a complete set of candidate lens parameters. .

[0100] 3. Effect prediction and re-recommendation: Encoded as a 512-dimensional vector, and The vectors are concatenated into a 1024-dimensional joint vector and fed into a dual-head classification network. Task A (Difference Plot Type) Prediction: Output That is, the probability of good adaptation is 81%, which is higher than the threshold of 0.5, and is judged as Good; Task B (Axial length growth rate) prediction: Output That is, the effective probability is controlled at 29%. If it is lower than the threshold of 0.5, it is judged as Bad and the re-recommendation process is triggered.

[0101] 4. Re-adjustment: The system automatically adjusted AC1 from 8.10mm to 8.05mm, recalculated and completed the parameters, and then re-entered the effect prediction head. Task B was updated to... Both tasks were rated as Good, and the candidate lens parameter set was finally determined.

[0102] Step 5: Doctor Recommendation Module Generation 1. Attention heatmap generation: The average of the nine parameters of the attention weight matrix A2 is taken to obtain the comprehensive attention of 196 image patches. The weight map is reshaped into a 14×14 two-dimensional weight map, upsampled to 224×224 resolution by bilinear interpolation, smoothed by Gaussian and superimposed on the original tangent map to generate a heatmap. The heatmap shows that the central area of ​​the temporal side (3~5mm) is significantly bright, followed by the nasal side area.

[0103] 2. Feature importance calculation: The gradient multiplication algorithm is used to calculate the importance scores of 9 parameters and normalize them to obtain the Top-5 ranking: ① Axial length (31%) ② Age (24%) ③ Spherical power (18%) ④ K2 (14%) ⑤ HVID (8%).

[0104] 3. Large Language Model Clinical Recommendation Generation: Recommendation parameters, dual-task prediction results, heatmap region descriptions, and feature importance rankings are organized into structured prompts, input into a finely tuned large language model for ophthalmology, and the final clinical recommendations are generated. "This 9-year-old patient has an axial length of 27.2mm, placing them in the 94th percentile for their age, indicating a high risk of myopia progression. This is the most significant influencing factor in this recommendation (weight 31%). Considering their young age (weight 24%), a follow-up appointment is strongly recommended within one month of prescription glasses. The focus should be on observing the corneal aberration map fit and changes in axial length. The recommended prescription is ac1=8.05mm, lens diameter 10.4mm, and torus curvature 0.50D. The aberration map is expected to be well-centered (81% probability), and axial length control is predicted to be effective (68% probability), but with moderate confidence. The doctor should make a comprehensive assessment based on the slit-lamp examination results. Attention heatmaps show the 3-5mm peritemporal ring as a high-attention area in the model; please focus on verifying the lens fit in this area. If the patient's annual axial length increase exceeds 0.3mm during follow-up, immediate follow-up and consideration of combined intervention with 0.05% low-concentration atropine are recommended."

[0105] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0106] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0107] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0108] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0109] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0110] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0111] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0112] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0113] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0114] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0115] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A contact lens intelligent fitting system with dual ViT and three-dimensional semantic grouping embedding, characterized in that, It includes a two-stream image coding and co-location fusion module, a 3D semantic grouping embedding module, an image-parameter cross-modal bidirectional attention fusion module, and a recommendation head and effect prediction module, wherein: The dual-stream image encoding and co-location fusion module is used to receive corneal tangent maps and corneal height maps, extract tangent map feature sequences and height map feature sequences respectively through two visual transformer encoders with independent weights, perform co-location weighted fusion of image blocks based on the natural alignment characteristics of the two image spaces, and output the fused image feature sequence. The three-dimensional semantic grouping and embedding module is used to receive nine clinical structured parameters, divide the parameters into refractive group, biometric group and demographic group according to clinical semantics, perform three-dimensional parallel embedding of numerical embedding, quantile embedding and clinical segment embedding for each parameter, and output the structured parameter feature sequence after vector concatenation and linear projection. The image-parameter cross-modal bidirectional attention fusion module is communicatively connected to the dual-stream image coding and co-location fusion module and the three-dimensional semantic grouping embedding module. It is used to receive the fused image feature sequence and the structured parameter feature sequence, perform bidirectional cross-modal attention calculation of image-guided parameter attention and parameter-guided image attention, and output a multimodal joint representation vector. The recommendation head and effect prediction module are communicatively connected to the image-parameter cross-modal bidirectional attention fusion module. They are used to predict the core parameters of the orthokeratology lens based on the multimodal joint representation vector, complete the generation of the candidate lens parameter set, and perform dual-task binary classification prediction on the wearing effect of the candidate lenses, triggering the re-recommendation process.

2. The system of claim 1, wherein, In the dual-stream image coding and co-location fusion module, the two visual transformer encoders are a first encoder dedicated to tangent maps and a second encoder dedicated to height maps. Each encoder divides the input 224×224 resolution image into N non-overlapping image blocks. Each image block is linearly projected and mapped into a d-dimensional vector. After adding learnable positional encoding, features are extracted by a multi-layer self-attention module, and N×d-dimensional tangent map feature sequences and height map feature sequences are output respectively. The formula of the same-position weighting fusion per image block is as follows: in, For the fusion features of the i-th image patch, For the i-th image patch features of the tangent plot, For the features of the i-th image patch in the height map, and For learnable fusion weight scalars, satisfying .

3. The system according to claim 1, characterized in that, In the three-dimensional semantic grouping embedding module, the refractive group includes spherical power, cylindrical power, and uncorrected visual acuity; the biometric group includes axial length, flat corneal curvature K1, steep corneal curvature K2, and corneal horizontal visible iris diameter HVID; and the demographic group includes age and sex. The numerical embedding is used to map the original scalar value of the parameter into a d / 3-dimensional vector through a linear transformation, preserving the precise absolute value information of the parameter; the quantile embedding is used to map the parameter value into a percentile index in the corresponding population stratification, and encodes it into a d / 3-dimensional vector through an embedding matrix lookup table, supplementing the relative position information of the parameter in the population; the clinical segmentation embedding is used to discretize the parameter value into a level index according to clinical standards, and encode it into a d / 3-dimensional vector through an embedding matrix lookup table, giving the parameter clinical grading semantics. After concatenating the three embedding vectors of the same parameter, they are mapped to d-dimensional parameter labels through a linear projection layer. The nine parameters are finally generated into a 9×d-dimensional structured parameter feature sequence.

4. The system according to claim 1, characterized in that, In the image-parameter cross-modal bidirectional attention fusion module, the basic calculation formula for scaled dot product attention is: in: Q is the query matrix, K is the key matrix, and V is the value matrix. The dimension of the key vector; The image-guided parameter attention uses the fused image feature sequence as the query end and the structured parameter feature sequence as the key end. The corresponding Q, K, and V matrices are generated through linear projection, and the attention weight matrix and the N×d-dimensional first output are calculated, so that the features of each image block are aggregated with relevant clinical parameter information. The parameters guide image attention, using structured parameter feature sequences as the query end and fused image feature sequences as the key end. The corresponding Q, K, and V matrices are generated through linear projection, and the attention weight matrix and a 9×d-dimensional second output are calculated, so that the model assigns higher weights to corneal regions related to clinical parameters. After the two outputs are spliced ​​and linearly projected, a d-dimensional multimodal joint representation vector is generated.

5. The system according to claim 1, characterized in that, The recommendation head and effect prediction module include a lens parameter recommendation head and a dual-task prediction head for wearing effect, wherein: The lens parameter recommendation head is a three-layer fully connected regression network. It uses a multimodal joint representation vector as input to predict three core parameters of VST orthokeratology lenses: inner curvature radius ac1, total lens diameter, and toric astigmatism correction. It also completes the remaining parameters according to clinical rules based on corneal curvature and target correction power to generate a candidate lens parameter set. The dual-task prediction head for wearing effect is a dual-head classification network, consisting of a shared feature extraction backbone and two independent binary classification output heads. It is used to predict the corneal difference map adaptation type and the control effect of annual axial growth rate after wearing lenses. When the prediction probability of either task is lower than the 0.5 threshold, a re-recommendation process is triggered.

6. The system according to claim 1, characterized in that, It also includes a doctor suggestion module, which is communicatively connected to the recommendation head and effect prediction module and the image-parameter cross-modal bidirectional attention fusion module, including an attention heatmap generation unit, a feature importance calculation unit, and an ophthalmology-domain fine-tuned large language model; The attention heatmap generation unit is used to extract the parameter-guided image attention weight matrix and generate a corneal region attention heatmap. The feature importance calculation unit uses a gradient multiplication of input algorithm to calculate the importance score of each clinical parameter to the recommendation result and outputs a normalized Top-N ranking. The ophthalmology-based fine-tuned large language model is used to receive structured prompts composed of recommended lens parameters, effect prediction results, heatmap region descriptions, and feature importance rankings, and to generate natural language clinical suggestions for doctors.

7. The system according to claim 2, characterized in that, In the dual-stream image encoding and co-location fusion module, the visual transformer encoder can be replaced with DeiT, Swing Transformer, BEiT variants, or a hybrid architecture of CNN and Transformer; the two-image fusion method can be replaced with element-wise multiplication, gated fusion, or lightweight attention fusion; the input image can be expanded to include three or more image streams containing axial and difference images.

8. The system according to claim 3, characterized in that, In the three-dimensional semantic grouping embedding module, the percentile calculation of quantile embedding can be statistically analyzed by age group, the clinical segmentation threshold can be adjusted according to the clinical center standard, the concatenation method of the three semantic vectors can be replaced by weighted summation, and the parameter grouping method can be replaced by automatic learning grouping based on data-driven correlation analysis.

9. The system according to claim 5, characterized in that, In the dual-task prediction head for wearing effect, the difference map adaptation type prediction can be replaced by multi-class prediction of good centering / mild deviation / severe deviation / incomplete ring, and the annual growth rate prediction of axial length can be replaced by continuous regression prediction.

10. A method for intelligent selection of orthokeratology lenses using dual ViT and three-dimensional semantic grouping embedding, characterized in that, The system implementation based on any one of claims 1-9 includes the following steps: Data Acquisition and Preprocessing: Corneal tangential and corneal height maps of patients were acquired, preprocessed to 224×224 pixel resolution and normalized; 9 clinical structured parameters of patients were acquired and semantically grouped and standardized according to refractive group, biometric group and demographic group. Dual-stream image coding and co-location feature fusion: Tangent map and height map are fed into two weighted visual transformer encoders to extract tangent map feature sequences and height map feature sequences. Co-location image block features are weighted and summed to generate a fused image feature sequence. Three-dimensional semantic grouping and embedding of clinical parameters: For the nine grouped clinical parameters, numerical embedding, quantile embedding and clinical segment embedding are performed respectively. The three embedding vectors of the same parameter are concatenated and linearly projected to generate parameter labels, and finally the structured parameter feature sequence is obtained. Cross-modal bidirectional attention fusion: The fused image feature sequence and the structured parameter feature sequence are input into the cross-modal bidirectional attention module, and image-guided parameter attention and parameter-guided image attention calculations are performed respectively to generate a multimodal joint representation vector; Intelligent recommendation of orthokeratology lens parameters: Input the multimodal joint representation vector into the fully connected regression recommendation head to predict the core parameters of orthokeratology lenses, and complete the remaining parameters according to clinical rules to generate a set of candidate lens parameters; Wearing effect prediction and re-recommendation: The candidate lens parameter set is encoded and spliced ​​with the multimodal joint representation, and then fed into a dual-head classification network to predict the corneal difference map adaptation type and axial elongation control effect after wearing the lens. If the prediction of either task fails to meet the standard, the re-recommendation process is triggered. Interpretable analysis and clinical suggestion generation: Extract cross-modal attention weights to generate corneal attention heatmaps, calculate the feature importance ranking of each clinical parameter, input structured information into the ophthalmology field fine-tuning large language model, generate natural language clinical suggestions, and complete the intelligent selection process for orthokeratology lenses.

Images