Myopia prediction system for teenagers based on multi-modal data
By constructing a four-dimensional multimodal data system and introducing the Mamba state space model, the problem of modeling the intrinsic correlation of adolescent myopia data was solved, enabling efficient myopia prediction and personalized intervention, and improving reasoning efficiency and decision interpretability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- THE FOURTH AFFILIATED HOSPITAL OF GUANGZHOU MEDICAL UNIV (GUANGZHOU ZENGCHENG DISTRICT PEOPLES HOSPITAL)
- Filing Date
- 2026-03-05
- Publication Date
- 2026-06-09
AI Technical Summary
Existing technologies cannot effectively model the intrinsic relationships in myopia-related data among adolescents. Traditional methods face problems such as memory explosion and high inference latency when processing long sequences, and lack the ability to dynamically adjust to the current state of individuals, making them difficult to deploy on mobile devices or in grassroots screening scenarios.
A four-dimensional multimodal data system is constructed, integrating vision information, ocular biological parameters, behavioral data, and overall health indicators. The Mamba state space model (SSM) and cross-attention weight mechanism are adopted to achieve adaptive fusion and complementary enhancement of cross-modal information, optimize time complexity, and intelligently focus on key risk factors based on individual status.
It significantly improves the reasoning efficiency of myopia prediction in adolescents, provides interpretable decision-making basis, and intuitively presents the behavioral-physiological-pathological correlation path through attention heatmap, supporting personalized intervention programs.
Smart Images
Figure CN122177437A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical and health information technology, specifically to a myopia prediction system for adolescents based on multimodal data. Background Technology
[0002] Myopia among adolescents has become a global public health challenge. Existing myopia prevention and control technologies suffer from the following problems: Data on myopia among adolescents comes from diverse sources, with significant differences in data structure, sampling frequency, and semantic granularity. Traditional methods (such as simple splicing or averaging fusion) cannot effectively model their inherent relationships. Myopia development is a slow, cumulative, and long-term process, requiring the analysis of continuous data over months or even years. Traditional Transformer-based models, due to their quadratic complexity, face issues of memory explosion and high inference latency when processing long sequences (such as T=180 days), making them difficult to deploy on mobile devices or in grassroots screening scenarios. Furthermore, existing fusion strategies typically have fixed weights, failing to dynamically adjust the focus based on an individual's current state and lacking interpretable identification of key risk drivers. Summary of the Invention
[0003] To address the above issues and overcome the shortcomings of existing technologies, this invention provides a multimodal data-based adolescent myopia prediction system. Addressing the significant differences in structure, sampling frequency, and semantic granularity among adolescent myopia-related data, which traditional methods cannot effectively model their inherent relationships, this solution integrates visual acuity information, ocular biological parameters, behavioral data, and overall health indicators to construct a four-dimensional multimodal data system encompassing visual physiology, biometry, behavioral environment, and overall health. This system dynamically maps images, temporal series, and structured data to a unified semantic space, achieving adaptive fusion and complementary enhancement of cross-modal information. Furthermore, it addresses the issue of traditional Transformer-based models being overly complex due to their quadratic nature. When processing long sequences, the model faces problems such as memory explosion and high inference latency. Existing fusion strategies usually have fixed weights and cannot dynamically adjust the focus based on the individual's current state. To address this, we introduce the Mamba State Space Model (SSM) to optimize the time complexity to a linear level. By relying on the dynamic allocation mechanism of cross-attention weights, the model can intelligently focus on key risk factors based on the individual's real-time state (such as sudden changes in axial length growth or a surge in eye strain). This not only significantly improves inference efficiency but also visualizes the "behavior-physiology-pathology" correlation path through attention heatmaps, providing doctors with interpretable decision-making basis and achieving a leap from "data fusion" to "cognitive enhancement".
[0004] The present invention provides a myopia prediction system for adolescents based on multimodal data. The system includes a multimodal data acquisition module, a data preprocessing module, a multimodal data fusion module, a myopia risk prediction module, and a visualization decision support module.
[0005] The multimodal data acquisition module collects vision information, ocular bioparameters, behavioral data, and physical health data of adolescents to obtain multimodal data;
[0006] The data preprocessing module performs interpolation resampling, data cleaning, and standardization operations on multimodal data with different sampling frequencies. It standardizes the image data in the multimodal data to a uniform resolution and constructs a time series to obtain a multimodal time series tensor. The multimodal time series tensor includes visual physiological modality, biometric modality, behavioral environmental modality, and whole-body health modality.
[0007] The multimodal data fusion module employs a cross-modal attention mechanism to fuse multimodal time series tensors, extract key feature data, and generate a multimodal fusion feature vector.
[0008] The myopia risk prediction module uses a deep learning prediction model to predict multimodal fusion feature vectors, generates risk level classifications, and obtains myopia development trend prediction results.
[0009] The visualization decision support module generates personalized intervention plans based on risk level classification and myopia development trend prediction results, visualizes the risks, and pushes early warning information.
[0010] Furthermore, the multimodal data acquisition module includes a vision information acquisition unit, an eye bioparameter acquisition unit, a behavioral data acquisition unit, and a physical health data acquisition unit;
[0011] The vision information acquisition unit collects various indicators for evaluating vision, including vision chart test results, fundus color photographs, OCT images, and corneal topography.
[0012] The ocular bioparameter acquisition unit collects data on the axial length, corneal curvature, anterior chamber depth, and retinal nerve fiber layer thickness of adolescent users;
[0013] The behavioral data collection unit collects data on adolescent users' screen time, outdoor activity time, electronic screen usage, and reading distance.
[0014] The physical health data collection unit collects data on adolescent users' height, weight, heart rate variability, sleep quality, and family history of myopia.
[0015] Furthermore, the multimodal data fusion module employs a cross-modal attention mechanism to fuse multimodal time series tensors, specifically including the following steps:
[0016] Step S1: Visual modality preprocessing. The ViT patch embedding module is used to flatten each frame of the visual physiological modality and biometric modality in the multimodal time series tensor into a sequence, and add temporal and spatial location codes to obtain the initial feature representation.
[0017] Step S2: Visual hierarchical feature extraction. The initial feature representation is processed using an independent ViT Mamba encoder. The ViT Mamba encoder consists of N stacked VM Blocks. The structural order of each VM Block is layer normalization, selective SSM, layer normalization and multilayer perceptron, resulting in L hierarchical visual feature sequences.
[0018] Step S3: Non-visual modality processing. The non-visual modality data in the multimodal time series tensor are concatenated and processed, and the dimension is unified through a linear projection layer. Temporal location encoding is added to obtain non-visual encoded features.
[0019] Step S4: Non-visual level feature extraction. Construct TM blocks using layer normalization, 1D selective SSM, layer normalization and multilayer perceptron. Construct a non-visual encoder using stacked TM blocks and extract features from the non-visual encoded features to obtain L levels of non-visual feature sequences.
[0020] Step S5: Cross-modal attention fusion. The hierarchical visual feature and non-visual feature sequences are fused using a cross-modal attention mechanism to obtain a multimodal fused feature vector.
[0021] Furthermore, in step S5, cross-modal attention fusion specifically includes the following steps:
[0022] Step S51: Intramodal feature enhancement. Use cross-attention mechanism to enhance the visual feature sequence and non-visual feature sequence of each level respectively to obtain the enhanced visual feature sequence and non-visual feature sequence.
[0023] Step S52: Correlation modeling, concatenating visual feature sequences and non-visual feature sequences, and processing them using an independent Mamba block to obtain cross-modal correlation-gated signals;
[0024] Step S53: Gated fusion, using a gating mechanism to process the cross-modal correlation gated signal and the enhanced visual feature sequence and non-visual feature sequence to obtain the gated visual feature sequence and non-visual feature sequence;
[0025] Step S54: Generate fusion features, calculate the weights of the cross-attention mechanism, dynamically weight the importance of each modality, combine the gated visual feature sequence and non-visual feature sequence with weighted aggregation of contextual information to obtain a multimodal fusion feature vector. The weight formula is as follows: ;
[0026] In the formula, Represents an exponential function. For learnable context vectors, Represents the learnable projection matrix. For modal indexing, Indicates the time step index. Representing modes In time The characteristic sequence, Representing modes In time The characteristic sequence, ( ) is the hyperbolic tangent activation function. This indicates the transpose operation. For attention weights.
[0027] The beneficial effects achieved by the present invention using the above solution are as follows:
[0028] (1) In view of the huge differences in structure, sampling frequency and semantic granularity of myopia-related data among adolescents, traditional methods cannot effectively model their internal relationship. This solution integrates vision information, ocular biological parameters, behavioral data and whole-body health indicators to construct a four-dimensional multimodal data system covering visual physiology, biometry, behavioral environment and whole-body health. It dynamically maps images, time series and structured data to a unified semantic space to achieve adaptive fusion and complementary enhancement of cross-modal information.
[0029] (2) To address the issues of memory explosion and high inference latency faced by traditional Transformer-based models when processing long sequences due to quadratic complexity, and the fact that existing fusion strategies usually have fixed weights and cannot dynamically adjust the focus based on the individual's current state, the Mamba State Space Model (SSM) is introduced to optimize the time complexity to the linear level. By relying on the dynamic allocation mechanism of cross-attention weights, the model can intelligently focus on key risk factors based on the individual's real-time state (such as sudden changes in axial length growth or a surge in eye load). This not only significantly improves inference efficiency, but also visually presents the "behavior-physiology-pathology" correlation path through attention heatmap visualization, providing doctors with interpretable decision-making basis and realizing the leap from "data fusion" to "cognitive enhancement". Attached Figure Description
[0030] Figure 1This is a schematic diagram of the adolescent myopia prediction system based on multimodal data proposed in this invention;
[0031] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. Detailed Implementation
[0032] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0033] Example 1, see Figure 1 The present invention provides a myopia prediction system for adolescents based on multimodal data. The system includes a multimodal data acquisition module, a data preprocessing module, a multimodal data fusion module, a myopia risk prediction module, and a visualization decision support module.
[0034] The multimodal data acquisition module collects vision information, ocular bioparameters, behavioral data, and physical health data of adolescents to obtain multimodal data;
[0035] The data preprocessing module performs interpolation resampling, data cleaning, and standardization operations on multimodal data with different sampling frequencies. It standardizes the image data in the multimodal data to a uniform resolution and constructs a time series to obtain a multimodal time series tensor. The multimodal time series tensor includes visual physiological modality, biometric modality, behavioral environmental modality, and whole-body health modality.
[0036] The multimodal data fusion module employs a cross-modal attention mechanism to fuse multimodal time series tensors, extract key feature data, and generate a multimodal fusion feature vector.
[0037] The myopia risk prediction module uses a deep learning prediction model to predict multimodal fusion feature vectors, generates risk level classifications, and obtains myopia development trend prediction results.
[0038] The visualization decision support module generates personalized intervention plans based on risk level classification and myopia development trend prediction results, visualizes the risks, and pushes early warning information.
[0039] Example 2, based on the above examples, describes a multimodal data acquisition module that includes a vision information acquisition unit, an eye bioparameter acquisition unit, a behavioral data acquisition unit, and a physical health data acquisition unit.
[0040] The vision information acquisition unit collects various indicators for evaluating vision, including vision chart test results, fundus color photographs, OCT images, and corneal topography.
[0041] The ocular bioparameter acquisition unit collects data on the axial length, corneal curvature, anterior chamber depth, and retinal nerve fiber layer thickness of adolescent users;
[0042] The behavioral data collection unit collects data on adolescent users' screen time, outdoor activity time, electronic screen usage, and reading distance.
[0043] The physical health data collection unit collects data on adolescent users' height, weight, heart rate variability, sleep quality, and family history of myopia.
[0044] Example 3, based on the above examples, describes a multimodal data fusion module that uses a cross-modal attention mechanism to fuse multimodal time series tensors, specifically including the following steps:
[0045] Step S1: Visual modality preprocessing. The ViT patch embedding module is used to flatten each frame of the visual physiological modality and biometric modality in the multimodal time series tensor into a sequence, and add temporal and spatial location codes to obtain the initial feature representation.
[0046] Step S2: Visual hierarchical feature extraction. The initial feature representation is processed using an independent ViT Mamba encoder. The ViT Mamba encoder consists of N stacked VM Blocks. The structural order of each VM Block is layer normalization, selective SSM, layer normalization and multilayer perceptron, resulting in 5 hierarchical visual feature sequences.
[0047] Step S3: Non-visual modality processing. The non-visual modality data in the multimodal time series tensor are concatenated and processed, and the dimension is unified through a linear projection layer. Temporal location encoding is added to obtain non-visual encoded features.
[0048] Step S4: Non-visual level feature extraction. TM blocks are constructed using layer normalization, 1D selective SSM, layer normalization, and multilayer perceptron. Stacked TM blocks are used to construct a non-visual encoder, and non-visual encoded features are extracted to obtain a 5-level non-visual feature sequence.
[0049] Step S5: Cross-modal attention fusion. Use a cross-modal attention mechanism to fuse hierarchical visual and non-visual feature sequences to obtain a multimodal fused feature vector.
[0050] By performing the aforementioned operations, this solution addresses the problem that traditional methods cannot effectively model the inherent relationships among adolescent myopia-related data due to the significant differences in structure, sampling frequency, and semantic granularity. This solution integrates visual information, ocular biological parameters, behavioral data, and overall health indicators to construct a four-dimensional multimodal data system covering visual physiology, biometry, behavioral environment, and overall health. It dynamically maps images, temporal data, and structured data to a unified semantic space, achieving adaptive fusion and complementary enhancement of cross-modal information.
[0051] Example 4, based on the above examples, includes the following steps in step S5 for cross-modal attention fusion:
[0052] Step S51: Intramodal feature enhancement. Use cross-attention mechanism to enhance the visual feature sequence and non-visual feature sequence of each level respectively to obtain the enhanced visual feature sequence and non-visual feature sequence.
[0053] Step S52: Correlation modeling. The visual feature sequence and the non-visual feature sequence are concatenated and processed using a separate Mamba block to obtain the cross-modal correlation-gated signal. The formula used is as follows: ;
[0054] In the formula, Representing a state-space model, It is a one-dimensional convolution function. This represents a multilayer perceptron. For concatenation functions, Represents a sequence of visual features. Represents non-visual feature sequences. To obtain the cross-modal correlation-gated signal;
[0055] Step S53: Gated fusion, using a gating mechanism to process the cross-modal correlation gated signal and the enhanced visual feature sequence and non-visual feature sequence to obtain the gated visual feature sequence and non-visual feature sequence;
[0056] Step S54: Generate fusion features, calculate the weights of the cross-attention mechanism, dynamically weight the importance of each modality, combine the gated visual feature sequence and non-visual feature sequence with weighted aggregation of contextual information to obtain a multimodal fusion feature vector. The weight formula is as follows: ;
[0057] In the formula, Represents an exponential function. For learnable context vectors, Represents the learnable projection matrix. For modal indexing, Indicates the time step index. Representing modes In time The characteristic sequence, Representing modes In time The characteristic sequence, ( ) is the hyperbolic tangent activation function. This indicates the transpose operation. For attention weights.
[0058] By performing the aforementioned operations, this paper addresses the problems of memory explosion and high inference latency faced by traditional Transformer-based models when processing long sequences due to quadratic complexity. Existing fusion strategies typically have fixed weights and cannot dynamically adjust the focus based on the individual's current state. The paper introduces the Mamba State Space Model (SSM) to optimize the time complexity to a linear level. Relying on the dynamic allocation mechanism of cross-attention weights, the model can intelligently focus on key risk factors based on the individual's real-time state (such as sudden changes in axial length growth or a surge in eye strain). This not only significantly improves inference efficiency but also visually presents the "behavior-physiology-pathology" correlation path through attention heatmap visualization, providing doctors with interpretable decision-making basis and achieving a leap from "data fusion" to "cognitive enhancement".
[0059] Example 5, based on the above examples, describes a myopia risk prediction module that uses a deep learning prediction model to predict multimodal fusion feature vectors.
[0060] Step M1: Regression prediction. Perform global average pooling and linear projection on the multimodal fusion feature vector to predict the myopia trend in the next six months and obtain the prediction results.
[0061] Step M2: Risk classification. A fully connected layer and Softmax are used to process the multimodal fusion feature vector, outputting three risk probabilities. The classification results are as follows:
[0062] Low risk: Ample hyperopic reserve (≥+1.00D), annual axial length increase ≤0.15mm, no family history of high myopia;
[0063] Medium risk: Insufficient hyperopic reserve (+0.50D to +0.75D), annual axial length increase of 0.15–0.25mm, or one parent is myopic;
[0064] High risk: Hyperopic reserve ≤ +0.50D or annual axial length increase ≥ 0.25mm or both parents have high myopia;
[0065] Step M3: Multi-task joint training, combining the loss functions of regression prediction and risk classification, calculates the total loss function, and uses the total loss function for training optimization;
[0066] Step M4: Risk level decision. Based on the output values of the three risk probabilities, the risk level with the highest probability value is selected.
[0067] Step M5: Confidence assessment. Set the confidence threshold and calculate the maximum probability value as the confidence level. If the confidence level is <0.8, the system marks it as "requires manual review" and pushes the final risk level of the prediction result to the ophthalmologist's workstation for manual evaluation.
[0068] Step M6: Dynamically update and continuously receive new data from adolescent users. A risk level reassessment will be automatically triggered when any of the following conditions are met:
[0069] The monthly increase in axial length is >0.03mm;
[0070] For seven consecutive days, the average daily close-range eye use exceeded 2.5 hours;
[0071] The rate of decline in farsightedness reserve exceeds the 90th percentile of its peers.
[0072] Example 6: Based on the above examples, the visualization decision support module generates a risk attribution heatmap based on dynamic cross-modal attention weight risk, and visualizes the results of risk level classification and myopia development trend prediction, outputting a structured report.
[0073] For example, if the high risk is mainly driven by "outdoor activity <1 hour / day in the past 30 days", then this interval is highlighted on the behavioral timeline; if axial elongation is strongly correlated with choroidal thinning shown by OCT, then an attention mask is superimposed on the fundus image.
[0074] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.
[0075] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
[0076] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.
Claims
1. A myopia prediction system for adolescents based on multi-modal data, characterized in that: The system includes a multimodal data acquisition module, a data preprocessing module, a multimodal data fusion module, a myopia risk prediction module, and a visualization decision support module; The multimodal data acquisition module collects vision information, ocular bioparameters, behavioral data, and physical health data of adolescents to obtain multimodal data; The data preprocessing module performs interpolation resampling, data cleaning, and standardization operations on multimodal data with different sampling frequencies. It standardizes the image data in the multimodal data to a uniform resolution and constructs a time series to obtain a multimodal time series tensor. The multimodal time series tensor includes visual physiological modality, biometric modality, behavioral environmental modality, and whole-body health modality. The multimodal data fusion module employs a cross-modal attention mechanism to fuse multimodal time series tensors, extract key feature data, and generate a multimodal fusion feature vector. The myopia risk prediction module uses a deep learning prediction model to predict multimodal fusion feature vectors, generates risk level classifications, and obtains myopia development trend prediction results. The visualization decision support module generates personalized intervention plans based on risk level classification and myopia development trend prediction results, visualizes the risks, and pushes early warning information.
2. The system for predicting myopia in teenagers based on multi-modal data according to claim 1, characterized in that: The multimodal data fusion module employs a cross-modal attention mechanism to fuse multimodal time series tensors, specifically including the following steps: Step S1: Visual modality preprocessing. The ViT patch embedding module is used to flatten each frame of the visual physiological modality and biometric modality in the multimodal time series tensor into a sequence, and add temporal and spatial location codes to obtain the initial feature representation. Step S2: Visual hierarchical feature extraction. The initial feature representation is processed using an independent ViT Mamba encoder. The ViT Mamba encoder consists of N stacked VM Blocks. The structural order of each VM Block is layer normalization, selective SSM, layer normalization and multilayer perceptron, resulting in L hierarchical visual feature sequences. Step S3: Non-visual modality processing. The non-visual modality data in the multimodal time series tensor are concatenated and processed, and the dimension is unified through a linear projection layer. Temporal location encoding is added to obtain non-visual encoded features. Step S4: Non-visual level feature extraction. Construct TM blocks using layer normalization, 1D selective SSM, layer normalization and multilayer perceptron. Construct a non-visual encoder using stacked TM blocks and extract features from the non-visual encoded features to obtain L levels of non-visual feature sequences. Step S5: Cross-modal attention fusion. Use a cross-modal attention mechanism to fuse hierarchical visual and non-visual feature sequences to obtain a multimodal fused feature vector.
3. The adolescent myopia prediction system based on multimodal data according to claim 2, characterized in that: In step S5, cross-modal attention fusion specifically includes the following steps: Step S51: Intramodal feature enhancement. Use cross-attention mechanism to enhance the visual feature sequence and non-visual feature sequence of each level respectively to obtain the enhanced visual feature sequence and non-visual feature sequence. Step S52: Correlation modeling, concatenating visual feature sequences and non-visual feature sequences, and processing them using an independent Mamba block to obtain cross-modal correlation-gated signals; Step S53: Gated fusion, using a gating mechanism to process the cross-modal correlation gated signal and the enhanced visual feature sequence and non-visual feature sequence to obtain the gated visual feature sequence and non-visual feature sequence; Step S54: Generate fusion features, calculate the weights of the cross-attention mechanism, dynamically weight the importance of each modality, combine the gated visual feature sequence and non-visual feature sequence to weighted aggregate contextual information, and obtain a multimodal fusion feature vector.