Refraction correction scheme generation method and system based on multi-modal data fusion

Abstract

The invention relates to a refraction correction scheme generation method and system based on multi-modal data fusion. According to the method, an eye movement track and multi-modal data are collected, a fixation stability index is calculated, motion artifacts are separated in combination with a generative adversarial network to eliminate data distortion caused by eyeball micro movement, multi-modal data fusion is achieved based on dynamic weight distribution and space-time alignment, and finally a correction scheme is generated through mechanical simulation optimization. Therefore, the robustness of data acquisition, the accuracy of a fusion result and the safety of a correction scheme are improved, the problems of data noise interference, insufficient multi-modal collaboration and poor biomechanical suitability caused by insufficient vision fixation capability of a traditional method are solved, and the whole-course adaptive optimization from data acquisition to scheme generation is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of ophthalmology, and particularly relates to a method and system for generating a refractive correction plan based on multi-modal data fusion. Background Art

[0002] In the field of refractive correction, the existing correction plan formulation mainly relies on the static measurement of single or a few eye parameters, such as corneal curvature, diopter, etc. However, in actual clinical practice, the eye state of patients is dynamically affected by various factors, such as the distortion of detection data caused by micro-eye movement, the parameter compatibility differences between different devices, and the diversity of individual eye use requirements.

[0003] When dealing with the detection data of patients with insufficient fixation ability, traditional methods often judge the data reliability through manual experience or require the patients to repeat the detection multiple times. This not only has low efficiency, but also may cause deviation of the correction plan due to the misuse of noise data.

[0004] In addition, most of the existing multi-modal data fusion technologies adopt fixed weights or simple splicing methods, and it is difficult to dynamically adapt to the influence of factors such as the fixation stability of different patients and the change of environmental light on the data quality, resulting in a significant difference between the fusion result and the real eye state.

[0005] In the link of generating the correction plan, the existing technology lacks a systematic design for the coordinated optimization of biomechanical characteristics and optical requirements, and may cause postoperative complications due to the mismatch between the cutting parameters and the corneal biomechanical response.

[0006] Therefore, there is an urgent need for a method for generating a refractive correction plan that can eliminate data noise in real time, dynamically fuse multi-modal features and accurately adapt to individual biomechanical characteristics. Summary of the Invention

[0007] Based on this, it is necessary to provide a method and system for generating a refractive correction plan based on multi-modal data fusion for the above technical problems.

[0008] In the first aspect, the present application provides a method for generating a refractive correction plan based on multi-modal data fusion, including:

[0009] S1: Collect the eye movement trajectory data, corneal topography data, whole-eye wavefront aberration data and pupil diameter data of the patient, and calculate the fixation stability index based on the eye movement trajectory data;

[0010] S2: Input the corneal topography data and the eye movement trajectory data into a pre-trained generative adversarial network to perform motion artifact separation processing on the corneal topography data to obtain real corneal morphology data;

[0011] S3: According to the eye movement trajectory data, perform spatio-temporal alignment processing on the full-eye wavefront aberration data to obtain spatio-temporally aligned wavefront aberration data;

[0012] S4: According to the fixation stability index, perform weight allocation processing on the real corneal morphology data, spatio-temporally aligned wavefront aberration data, and pupil diameter data to obtain fused multi-modal data;

[0013] S5: Generate initial cutting parameters based on the fused multi-modal data, and perform mechanical simulation optimization on the initial cutting parameters to obtain a refractive correction plan.

[0014] In a second aspect, the present application also provides a system for generating a refractive correction plan based on multi-modal data fusion, including:

[0015] A data acquisition module, configured to acquire the eye movement trajectory data, corneal topographic data, full-eye wavefront aberration data, and pupil diameter data of a patient, and calculate the fixation stability index based on the eye movement trajectory data;

[0016] An artifact separation module, configured to input the corneal topographic data and eye movement trajectory data into a pre-trained generative adversarial network, and perform motion artifact separation processing on the corneal topographic data to obtain real corneal morphology data;

[0017] A wavefront alignment module, configured to perform spatio-temporal alignment processing on the full-eye wavefront aberration data according to the eye movement trajectory data to obtain spatio-temporally aligned wavefront aberration data;

[0018] A data fusion module, configured to perform weight allocation processing on the real corneal morphology data, spatio-temporally aligned wavefront aberration data, and pupil diameter data according to the fixation stability index to obtain fused multi-modal data;

[0019] A plan generation module, configured to generate initial cutting parameters based on the fused multi-modal data, and perform mechanical simulation optimization on the initial cutting parameters to obtain a refractive correction plan.

[0020] In a third aspect, the present application also provides a computer device, including a memory and a processor, where the memory stores a computer program, and when the processor executes the computer program, it implements a method for generating a refractive correction plan based on multi-modal data fusion as described in the first aspect.

[0021] In a fourth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements a method for generating a refractive correction plan based on multi-modal data fusion as described in the first aspect.

[0022] The above method and system for generating a refractive correction plan based on multi-modal data fusion collect eye movement trajectories and multi-modal data, calculate the fixation stability index, use a generative adversarial network to separate motion artifacts to eliminate data distortion caused by micro-eye movements, and then achieve multi-modal data fusion based on dynamic weight allocation and spatio-temporal alignment. Finally, a correction plan is generated through mechanical simulation optimization, thereby improving the robustness of data collection, the accuracy of the fusion result, and the safety of the correction plan, solving the problems of data noise interference, insufficient multi-modal collaboration, and poor biomechanical adaptability caused by insufficient fixation ability in traditional methods, and realizing the whole-process adaptive optimization from data collection to plan generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required for use in the description of the embodiments or related technologies. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0024] Figure 1 It is a schematic flowchart of a method for generating a refractive correction plan based on multi-modal data fusion provided by the present invention;

[0025] Figure 2 It is a schematic structural diagram of a system for generating a refractive correction plan based on multi-modal data fusion provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0026] In order to make the purpose, technical solutions and advantages of the present application clearer, the following will further describe the present application in detail with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0027] Refer to Figure 1 , which shows a schematic flowchart of a method for generating a refractive correction plan based on multi-modal data fusion provided by the present application. The method includes the following steps:

[0028] S1: Collect the eye movement trajectory data, corneal topographic data, whole-eye wavefront aberration data, and pupil diameter data of the patient, and calculate the fixation stability index based on the eye movement trajectory data.

[0029] Specifically, when collecting eye movement trajectory data, a high-precision eye tracker can be used, with the infrared reflection technology as the core principle. By emitting infrared light of a specific wavelength to irradiate the patient's eyeball, and then using a high-frame-rate camera to capture the infrared light image reflected by the eyeball, the movement trajectory of the eyeball can be accurately tracked. The sampling frequency can be set above 250Hz to ensure the real-time and accuracy of the data.

[0030] When collecting corneal topographic data, a corneal topographer can be used, based on the slit-scan or Scheimpflug imaging principle. The slit-scan technique projects a slender slit light onto the corneal surface and then scans and images the reflected light from different angles to obtain the three-dimensional height information of the cornea; Scheimpflug imaging uses an inclined optical section to photograph the cornea and reconstructs the three-dimensional topographic map of the cornea through a series of two-dimensional images.

[0031] When collecting whole-eye wavefront aberration data, a wavefront aberrometer can be used, and its working principle is the Hartmann-Shack sensor. This sensor projects a laser beam into the eye. After the laser is reflected by the fundus, the light rays emerging from the pupil area are decomposed into multiple small light spots by a series of microlens arrays. By analyzing the position and shape changes of these light spots, the wavefront aberration situation of the eye can be deduced, covering low-order aberrations (such as myopia, hyperopia, astigmatism) and high-order aberrations (such as trefoil aberration, coma, etc.).

[0032] When collecting pupil diameter data, an infrared or visible light camera can be used to take images of the patient's pupil in a dark room or a specific lighting environment, and the diameter of the pupil can be accurately measured through an image processing algorithm. To ensure the measurement accuracy, preprocessing operations such as grayscale processing, binarization, and edge detection can be performed on the image, and then geometric measurement methods can be used to determine the edge pixel points of the pupil, and further calculate the actual diameter.

[0033] When calculating the fixation stability index, based on the collected eye movement trajectory data, multiple time points during the patient's fixation process can be selected to calculate the offset of the eye position. Specifically, statistical indicators such as standard deviation and root mean square error can be used to quantify the stability of the position offset. For example, the standard deviations of the position coordinate sequences of the eye in the horizontal and vertical directions within a certain period of time (such as 30 seconds) are calculated respectively, and then the fixation stability index is obtained by combining the standard deviation values in these two directions. The smaller the index, the better the fixation stability.

[0034] S2: Input the corneal topographic data and the eye movement trajectory data into a pre-trained generative adversarial network to perform motion artifact separation processing on the corneal topographic data and obtain the real corneal morphology data.

[0035] Specifically, the selected GAN (Generative Adversarial Network) consists of two parts: a generator and a discriminator. The generator aims to convert corneal topographic data with motion artifacts into real corneal morphology data without artifacts. Its network architecture can include multiple convolutional layers, deconvolutional layers, and residual blocks to effectively extract, map, and reconstruct image features. The discriminator is responsible for distinguishing whether the data output by the generator is real corneal topographic data without artifacts or the original data with artifacts. Its network structure can include convolutional layers, pooling layers, and fully connected layers to extract image features and perform binary classification judgments. In the pre-training stage, a large number of labeled corneal topographic data pairs (including corresponding data with and without artifacts) are used to train the GAN. By minimizing the difference between the data generated by the generator and the real data (such as using the mean squared error loss function) and the adversarial loss (the cross-entropy loss between the discriminator's discrimination result and the real label), the generator is enabled to have a powerful artifact removal ability, while the discriminator can accurately evaluate the authenticity of the data.

[0036] The collected corneal topographic data and eye movement trajectory data are used as inputs. The corneal topographic data is input in the form of an image matrix, and the gray value or color value of each pixel represents information such as the height and curvature of the corresponding position on the cornea; the eye movement trajectory data is used as auxiliary information to guide the generator to better understand the motion artifact patterns in the corneal topogram. Inside the GAN, the generator first extracts features from the input corneal topographic data, and at the same time combines the patient's eye movement characteristics reflected by the eye movement trajectory data. Through non-linear transformation, the motion artifacts are gradually separated from the corneal topogram, and finally real corneal morphology data is output. For example, when there are stripe-like artifacts caused by rapid eye movement during the collection of corneal topographic data, the generator can identify the direction and pattern of artifact generation based on the eye movement trajectory data, and use the sliding operation of the convolutional kernel and the correction mechanism of the residual connection to remove the stripe artifacts from the corneal topogram and restore the smooth and accurate real corneal morphology.

[0037] S3: According to the eye movement trajectory data, perform spatio-temporal alignment processing on the full-eye wavefront aberration data to obtain spatio-temporally aligned wavefront aberration data.

[0038] Specifically, the eye movement trajectory data and the full-eye wavefront aberration data are collected at different time points and coordinate systems, and spatio-temporal alignment is performed to ensure their consistency in time and space. In terms of time alignment, based on the time series of the eye movement trajectory data, interpolation or resampling is performed on the full-eye wavefront aberration data to make it have the same sampling rate and time points as the eye movement trajectory data, so as to ensure that the two can correspond in the time dimension. In terms of space alignment, according to the eye movement trajectory recorded in the eye movement trajectory data, the displacement and rotation of the full-eye wavefront aberration data relative to the corneal optical center during the acquisition process are calculated, and then coordinate transformation is performed on the wavefront aberration data to adjust it to the coordinate system that matches the corneal topographic data, achieving spatial alignment.

[0039] A spatio-temporal alignment algorithm based on interpolation can be adopted. For time alignment, linear interpolation or spline interpolation methods are used to map the wavefront aberration data from the original sampling time points to the time points consistent with the eye movement trajectory data. For example, if the sampling rate of the eye movement trajectory data is 250Hz and the sampling rate of the wavefront aberration data is 10Hz, the sampling rate of the wavefront aberration data can be increased to 250Hz through the interpolation algorithm to ensure that there is corresponding data at each time point for both. For space alignment, using an affine transformation or a projection transformation model, according to the displacement and rotation parameters calculated from the eye movement trajectory data, the coordinate points of the wavefront aberration data are adjusted. Specifically, the position coordinates of each light ray point in the wavefront aberration data are transformed through a transformation matrix to match the coordinate system in the corneal topographic data, thus obtaining the wavefront aberration data after spatio-temporal alignment.

[0040] S4: According to the fixation stability index, by performing weight allocation processing on the real corneal morphology data, spatio-temporally aligned wavefront aberration data, and pupil diameter data, fused multi-modal data is obtained.

[0041] Specifically, the weight allocation strategy is determined according to the magnitude of the fixation stability index. When the patient's fixation stability is good (the fixation stability index is low), it indicates that the influence of eye movement on data acquisition is small. At this time, the weight of eye movement-related data (such as eye movement trajectory data) in the fusion can be appropriately reduced, while increasing the weight of data that can better reflect the static optical characteristics of the eye, such as corneal topographic data and full-eye wavefront aberration data; conversely, if the patient's fixation stability is poor and the influence of eye movement on the data is large, the weight of the eye movement trajectory data and the corneal topographic data after motion artifact separation processing needs to be increased to more accurately reflect the dynamic influence of eye movement on the eye's optical state. At the same time, the pupil diameter data is combined to further adjust the weight because the change in pupil diameter will affect the amount of light entering the eye and the imaging quality, and the performance of the eye's optical characteristics is also different under different pupil sizes.

[0042] The multi-modal data can be fused in a linear weighted or non-linear weighted manner. For example, in linear weighting, different weight coefficients are assigned to the true corneal shape data, spatio-temporally aligned wavefront aberration data, eye movement trajectory data, and pupil diameter data respectively, and the sum of these weight coefficients is 1. The determination of the weight coefficients can be based on the mapping relationship between the fixation stability index and the weights, and this mapping relationship can be established through experimental statistics or machine learning methods. Assuming that the fixation stability index is positively correlated with the weight of the eye movement trajectory data and negatively correlated with the weights of the corneal topography data and wavefront aberration data, then according to the currently calculated fixation stability index value, the corresponding weight of the eye movement trajectory data is found from the mapping relationship, and then the weights of other data are determined by combining other prior knowledge or empirical formulas. Finally, the fused multi-modal data is obtained by weighted summation. For example, when the fixation stability index is 0.5 (assuming a full score of 1, indicating poor fixation stability), the weight of the eye movement trajectory data is set to 0.4, the weight of the true corneal shape data is 0.3, the weight of the spatio-temporally aligned wavefront aberration data is 0.2, and the weight of the pupil diameter data is 0.1. Then, each modal data is multiplied by its corresponding weight coefficient and added together to obtain the fused multi-modal data, which comprehensively reflects the overall optical state of the patient's eye under a specific fixation stability.

[0043] S5: Generate initial cutting parameters based on the fused multi-modal data, and perform mechanical simulation optimization on the initial cutting parameters to obtain a refractive correction plan.

[0044] Specifically, based on the fused multi-modal data, a pre-established refractive correction model is used to generate initial cutting parameters. This model can be trained based on a large amount of clinical data. By inputting the fused multi-modal data (including information such as corneal shape, wavefront aberration, eye movement characteristics, and pupil size), the model outputs initial parameters such as the corneal cutting depth, cutting area, and cutting shape. For example, for myopia correction, according to the relatively high refractive power in the central corneal area in the fused data and the low-order aberration components in the wavefront aberration, it is determined to perform appropriate cutting in a specific central corneal area. The cutting depth corresponds to the required reduction in refractive power, and the size and shape of the cutting area are determined by combining the pupil diameter and the visual demand area reflected by the eye movement trajectory to ensure good visual quality can be obtained within the patient's pupil area after cutting.

[0045] The process of mechanical simulation optimization can be as follows: Apply the initial cutting parameters to the finite element model of the cornea for mechanical simulation analysis. The finite element model of the cornea regards the cornea as a nonlinear biomechanical structure composed of multiple tiny elements, considering the anisotropy, inhomogeneity of the cornea, and its interaction with physiological environments such as intraocular pressure. During the simulation, simulate the healing process after corneal cutting surgery, including the stress and strain distribution in the cutting area, the rebound and remodeling of the cornea, etc. Calculate the biomechanical response parameters of the cornea after cutting, such as the maximum stress, maximum strain, and corneal thickness change. Evaluate whether the initial cutting parameters are reasonable based on these parameters. If potential risks such as excessive corneal stress concentration or overly thin remaining corneal thickness caused by cutting are found, adjust and optimize the cutting parameters, such as appropriately reducing the cutting depth and expanding the cutting area to disperse stress, until a combination of cutting parameters that meet the requirements of biomechanical safety and stability is obtained in the mechanical simulation, and finally determine it as the refractive correction plan to ensure that the surgery can effectively correct vision and guarantee the long-term biomechanical health of the cornea.

[0046] The above method for generating a refractive correction plan based on multimodal data fusion synchronously collects eye movement trajectories and multimodal data, calculates the fixation stability index, combines a generative adversarial network to separate motion artifacts to eliminate data distortion caused by eye micro-movement, then realizes multimodal data fusion based on dynamic weight allocation and spatio-temporal alignment, and finally generates a correction plan through mechanical simulation optimization, thereby improving the robustness of data collection, the accuracy of the fusion result, and the safety of the correction plan, solving the problems of data noise interference, insufficient multimodal collaboration, and poor biomechanical adaptability caused by insufficient fixation ability in traditional methods, and achieving full-process adaptive optimization from data collection to plan generation.

[0047] In an optional embodiment, the eye movement trajectory data includes the XYZ three-axis displacement sequences of the eyeball in three-dimensional space; calculating the fixation stability index based on the eye movement trajectory data includes the following steps: Combining the data collection period, calculate the mean value of the displacement amounts of adjacent frames in the eye movement trajectory data to obtain the fixation stability index; The calculation formula for the fixation stability index is:

[0048]

[0049] where FSI is the fixation stability index, T is the data collection period, N is the total number of frames of the eye movement trajectory data, {Δx i ,Δy i ,Δz i} is the XYZ three-axis displacement sequence, and i is the frame number sequence of the eye movement trajectory data.

[0050] Specifically, the eye movement trajectory data is collected by a high-precision eye tracker, with the infrared reflection technology as the core principle. Infrared light irradiates the patient's eyeball, and the camera captures the reflected light image to accurately track the movement trajectory of the eyeball in three-dimensional space. The data includes the displacement sequences of the XYZ three axes, reflecting the displacement changes of the eyeball in the X (horizontal), Y (vertical), and Z (front-back) directions, providing comprehensive eyeball movement information.

[0051] The fixation stability index (FSI) is calculated based on the eye movement trajectory data, reflecting the fixation stability of the patient. When calculating, the data acquisition period T and the total number of frames N are combined, and the average value of the displacement amounts between adjacent frames is calculated. Through the formula, the three-dimensional displacement change is quantified into a single stability index, and the formula is: Among them, the smaller the FSI value, the better the fixation stability.

[0052] The specific calculation steps and implementation means can be: Assume that the data acquisition period T is 30 seconds and the frame rate is 250 Hz, then the total number of frames N = 7500. For the displacement of each frame in the XYZ three axes, calculate the sum of the absolute values of the displacement amounts between adjacent frames, that is, ||Δx_i|| + ||Δy_i|| + ||Δz_i||, divide by T to obtain the displacement rate, and then average all the displacement rates to obtain the FSI. For example, if the calculated FSI value of a certain patient is 0.005, it means that during the fixation process, the displacement amount per unit time of the patient is relatively small on average, and the fixation stability is better.

[0053] To accurately calculate the FSI, the spatial resolution of the high-precision eye tracker can reach within 0.1 degree of visual angle, and the time resolution is not less than 250 Hz. When processing the data, a robust algorithm is used to preprocess the eye movement data, such as filtering to remove noise, interpolation to fill in missing values, etc., to improve the accuracy of the calculation.

[0054] In an optional embodiment, the S2 includes the following steps:

[0055] S21: Input the corneal topographic data and the eye movement trajectory data into the generator of the generative adversarial network to decompose the corneal topographic data, and obtain the preliminary corneal morphology data and the artifact component data; among them, the corneal topographic data is a linear superposition of the preliminary corneal morphology data and the artifact component data.

[0056] Specifically, input the corneal topographic data and the eye movement trajectory data into the generator of the generative adversarial network (GAN). The generator adopts an encoder-decoder structure. The encoder extracts data features through convolutional layers, and the decoder maps the features back to the corneal morphology space through deconvolutional layers. An attention mechanism is set inside the generator to focus on the key areas in the data and improve the decomposition accuracy.

[0057] Assume that the corneal topography data is a linear superposition of the preliminary corneal morphology data and the artifact component data, i.e., D noisy = D clean + D artifact . Among them, D noisy is the corneal topography data, D clean is the preliminary corneal morphology data, and D artifact is the artifact component data. By learning this linear relationship, the generator uses a large number of sample pairs in the training data to estimate the coefficient matrix of the linear superposition, thereby decomposing the noisy data into the preliminary corneal morphology data and the artifact component data. For example, during the training process, the generator learns the linear manifestation forms of different artifact types (such as motion artifacts, light interference artifacts, etc.) in the corneal topography data. When new noisy data is input, it can accurately separate these two parts of data.

[0058] S22: Input the preliminary corneal morphology data into the discriminator of the generative adversarial network. Based on the preset corneal curvature gradient distribution data of healthy people, evaluate the biomechanical compliance of the preliminary corneal morphology data and generate a biomechanical compliance score.

[0059] Specifically, input the preliminary corneal morphology data into the discriminator of the GAN. The discriminator contains multiple convolutional layers and fully connected layers for extracting high-level features of the data. Using the preset corneal curvature gradient distribution data of healthy people, these data are statistically analyzed and feature extracted to form a biomechanical feature model of healthy corneas. The discriminator evaluates its biomechanical compliance by comparing the feature differences between the preliminary corneal morphology data and the healthy people's data.

[0060] The discriminator calculates the similarity score of the preliminary corneal morphology data based on biomechanical features such as the corneal curvature gradient distribution of healthy people. For example, measurement methods such as cosine similarity or Euclidean distance are used to measure the closeness of the preliminary data and the healthy data in the feature space. At the same time, the rationality of biomechanical parameters such as the stress and strain of the cornea can also be considered to comprehensively generate a biomechanical compliance score. The scoring range can be set between 0 and 1. The higher the score, the more the preliminary corneal morphology data conforms to the biomechanical characteristics of healthy corneas and the closer it is to real and reliable data.

[0061] S23: Iteratively optimize the parameters of the generative adversarial network according to the biomechanical compliance score and the adversarial loss function of the generative adversarial network.

[0062] Specifically, according to the biomechanical compliance score and the adversarial loss function of the GAN, iteratively optimize the parameters of the generator and the discriminator. The adversarial loss function aims to make the data generated by the generator as close as possible to the real data, while enabling the discriminator to accurately distinguish between the generated data and the real data. Specifically, the adversarial loss can be expressed as L adv= -log(D(D clean )) - log(1 - D(G(D noisy ))), where D is the discriminator and G is the generator. Combining with the biomechanical compliance score S, the overall loss function can be designed as L = L adv + λ·(1 - S), where λ is the weight coefficient used to balance the contributions of the adversarial loss and the biomechanical evaluation.

[0063] The gradient descent method and its variants (such as the Adam optimizer) are used to update the network parameters. In each iteration, the gradients of the loss function with respect to the generator and discriminator parameters are calculated, and then the parameters are updated according to the backpropagation algorithm. For example, for the generator parameter θ G , the update formula is where η is the learning rate, is the gradient of the generator parameter θ G . Through continuous iterative optimization, the generator can generate corneal shape data that better conforms to biomechanical characteristics, and the discrimination ability of the discriminator is gradually improved, ultimately enhancing the performance and stability of the entire model.

[0064] S24: Loop S21 to S23. When the biomechanical compliance score reaches the preset threshold, the preliminary corneal shape data is taken as the real corneal shape data.

[0065] Specifically, loop S21 to S23 is executed, that is, data decomposition, biomechanical evaluation, and parameter optimization are continuously performed. In each loop, the generator decomposes the noisy corneal topographic data more accurately based on the updated parameters to obtain new preliminary corneal shape data; the discriminator then conducts a more precise biomechanical compliance evaluation according to the latest data features to generate a new score; subsequently, the network parameters are optimized again based on the score and the adversarial loss function. This loop process continues until the biomechanical compliance score reaches the preset threshold.

[0066] The preset threshold can be determined according to clinical requirements and experimental verification. For example, it is set to 0.9, indicating that the preliminary corneal shape data needs to reach a relatively high degree of biomechanical compliance. When the score reaches or exceeds this threshold, it is considered that the preliminary corneal shape data has sufficiently removed the artifact influence and conforms to the biomechanical characteristics of a healthy cornea, and can truly reflect the actual shape of the patient's cornea. Therefore, it is output as the final real corneal shape data.

[0067] In an alternative embodiment, S4 includes the following steps:

[0068] S41: Calculate the modal attenuation coefficient according to the fixation stability index. The modal attenuation coefficient is negatively correlated with the fixation stability index. Among them, the modal attenuation coefficient is used to characterize the influence degree of the patient's fixation ability on the reliability of each modal data, and the modal data are real corneal morphology data, spatio-temporally aligned wavefront aberration data, and pupil diameter data.

[0069] Specifically, the negative correlation between the modal attenuation coefficient and the fixation stability index means that the worse the patient's fixation stability (the higher the index), the smaller the modal attenuation coefficient. When calculating, a negative correlation model can be constructed based on the fixation stability index, such as linear regression or non-linear mapping. For example, through a large number of clinical data statistics, a linear relationship between the modal attenuation coefficient α and the fixation stability index FSI is established: α = a - b·FSI, where a and b are regression coefficients, which are determined by methods such as the least squares method to ensure a negative correlation between the two.

[0070] The modal attenuation coefficient is used to characterize the influence degree of the patient's fixation ability on the reliability of each modal data. For example, when the patient's fixation stability is poor, the reliability of the eye movement trajectory data decreases. At this time, the modal attenuation coefficient is small, and the weight of this modal data in the fusion is reduced through a weighting mechanism to avoid large deviations in the fusion result.

[0071] S42: Calculate the signal-to-noise ratio of each modal data respectively to generate the confidence level of each modal data.

[0072] Specifically, calculate the signal-to-noise ratio of each modal data respectively to generate the confidence level. The signal-to-noise ratio can be calculated by methods such as mean square error and variance analysis. For example, for the real corneal morphology data, calculate the mean square error between it and the ideal healthy corneal morphology data. The smaller the mean square error, the higher the signal-to-noise ratio and the confidence level. For the spatio-temporally aligned wavefront aberration data, analyze the stability of the aberration coefficients. If the high-order aberration coefficients fluctuate little, the signal-to-noise ratio is high and the confidence level is also high.

[0073] Convert the calculated signal-to-noise ratio to the confidence level. Through normalization processing, the signal-to-noise ratio can be mapped to the interval from 0 to 1 as the confidence level value. For example, use linear normalization: Confidence = (SNR - SNR_min) / (SNR_max - SNR_min), where SNR is the signal-to-noise ratio, and SNR_min and SNR_max are the minimum and maximum values of the signal-to-noise ratio respectively. The higher the confidence level value, the more reliable the quality of this modal data and the more accurately it can reflect the patient's eye condition.

[0074] S43: Based on the modal attenuation coefficient and the confidence level, perform weight assignment processing on each modal data through the Softmax function to obtain the weight of each modal data. The calculation formula for the weight assignment processing is:

[0075]

[0076] Among them, w k is the weight of the k-th modal data, Confidence k is the confidence of the k-th modal data, α is the modal attenuation coefficient; K is the number of types of modal data, and m is the serial number of the type of modal data.

[0077] Specifically, based on the modal attenuation coefficient and confidence, the Softmax function is used to assign weights to each modal data. The Softmax function converts the weighted confidence of each modal data (adjusted by the modal attenuation coefficient) into a probability distribution form of weights. The specific calculation formula is: Among them, w k is the weight of the k-th modal data, α is the modal attenuation coefficient, Confidence k is the confidence of the k-th modal data, and K is the number of types of modal data. The Softmax function ensures that the sum of all weights is 1, and each weight value is between 0 and 1.

[0078] The modal attenuation coefficient α adjusts the influence degree of the confidence. When α is large, the weights of the modal data with high confidence are further amplified, and the weights of the low confidence are suppressed; when α is small, the weight differences of each modal data are reduced, and it is closer to a uniform distribution. For example, if the patient's fixation stability is good (α is large), then higher weights are assigned to the modal data with high confidence to make full use of its reliability; if the fixation stability is poor (α is small), then the weights of each modal data are appropriately balanced to avoid having too much impact on the fusion result due to the low reliability of a certain modal data.

[0079] S44: Weighted fusion is performed on the modal data according to the weights to obtain fused multi-modal data.

[0080] Specifically, weighted fusion is performed on each modal data according to the calculated weights. For each data point or feature dimension, each modal data is multiplied by its corresponding weight, and then the sum is calculated to obtain the fused data. For example, assume there are true corneal shape data D1, spatio-temporal aligned wavefront aberration data D2, and pupil diameter data D3, and their weights are w1, w2, and w3 respectively, then the fused multi-modal data D fused is: D fused = w1·D1 + w2·D2 + w3·D3.

[0081] Through weighted fusion, the advantages of various modal data are integrated to obtain more comprehensive and accurate fused multi-modal data, providing a reliable basis for the generation of subsequent refractive correction plans. At the same time, the fusion results can also be verified and optimized. For example, the accuracy of the fused data in predicting the refractive correction effect can be evaluated through cross-validation. If deviations or unreasonable points are found in the fusion results, the weight allocation strategy can be further adjusted or the preprocessing methods of each modal data can be optimized to improve the quality and reliability of the fused data.

[0082] In an alternative embodiment, S3 includes the following steps:

[0083] S31: Segment the eye movement trajectory data into multiple time windows, and extract features from the eye movement trajectory data within each time window to generate an eye movement time series feature vector.

[0084] Specifically, the collected eye movement trajectory data is segmented according to a preset time window length. The selection of the time window length can comprehensively consider factors such as the data sampling rate, patient fixation stability, and computational complexity. For example, for eye movement trajectory data with a sampling rate of 250 Hz, if the time window length is selected as 0.1 seconds, each time window contains 25 data points. The time window can adopt a sliding window method, with a certain overlap between adjacent windows to capture the continuous features of the eye movement trajectory.

[0085] Extract features from the data within each time window to generate an eye movement time series feature vector. The feature extraction methods include calculating statistical features such as the mean, standard deviation, maximum value, and minimum value of the eye ball displacement, as well as dynamic features such as the speed, acceleration, and angular velocity of the eye ball movement. For example, calculate the mean displacement of the eye ball on the X, Y, and Z axes within each time window to reflect the average position of the eye ball during this time period; calculate the standard deviation of the displacement to reflect the stability of the eye ball displacement; calculate the speed and acceleration to reflect the dynamic characteristics of the eye ball movement. Combine these features into a vector, which is the eye movement time series feature vector, used to describe the time series features of the eye movement trajectory within each time window.

[0086] S32: Segment the full-eye wavefront aberration data according to the time window, and extract spatial features from the full-eye wavefront aberration data within each time window to generate a wavefront aberration feature vector.

[0087] Specifically, the full-eye wavefront aberration data is segmented according to the same time window as the eye movement trajectory data to ensure the consistency in the time dimension. This can ensure that the subsequent feature correlation calculations are carried out in the same time framework, improving the accuracy of spatio-temporal alignment.

[0088] Extract the spatial features of the wavefront aberration data within each time window to generate a wavefront aberration feature vector. The feature extraction methods include calculating information such as the amplitude and phase of the wavefront aberration at different spatial frequencies, as well as extracting the Zernike polynomial coefficients of the wavefront aberration. For example, by fitting the wavefront aberration data with Zernike polynomials, the Zernike coefficients of each order are obtained, and these coefficients can describe the spatial distribution characteristics of the wavefront aberration; the root mean square value of the wavefront aberration in different regions can also be calculated to reflect the aberration intensity of each region. Combine these spatial features into a vector, which is the wavefront aberration feature vector and is used to describe the spatial features of the wavefront aberration within each time window.

[0089] S33: Calculate the cosine similarity between the eye movement time series feature vector and the wavefront aberration feature vector to generate a spatio-temporal correlation matrix; and perform time phase adjustment on the full-eye wavefront aberration data according to the spatio-temporal correlation matrix to generate spatio-temporally aligned wavefront aberration data.

[0090] Specifically, calculate the cosine similarity between the generated eye movement time series feature vector and the wavefront aberration feature vector to generate a spatio-temporal correlation matrix. Cosine similarity is used to measure the similarity degree of two vectors in direction, and its calculation formula is: where a and b represent the eye movement time series feature vector and the wavefront aberration feature vector respectively, and θ is the angle between the two vectors. By calculating the cosine similarity between the eye movement time series feature vector and the wavefront aberration feature vector within each time window, a matrix is obtained, and the elements of the matrix represent the similarity degree between the two within the corresponding time window, that is, the spatio-temporal correlation matrix.

[0091] According to the spatio-temporal correlation matrix, perform time phase adjustment on the full-eye wavefront aberration data. The specific method is to find the time window corresponding to the wavefront aberration feature vector with the highest similarity to the eye movement time series feature vector, and use the wavefront aberration data within this time window as a reference to perform time alignment adjustment on the wavefront aberration data within other time windows. For example, if within a certain time window, the similarity between the eye movement time series feature vector and the wavefront aberration feature vector reaches the maximum value, it is considered that the two have the best spatio-temporal correlation at this time. Based on this time window, perform operations such as interpolation or shifting on the wavefront aberration data within other time windows to align their time phases with the eye movement trajectory data, and finally generate spatio-temporally aligned wavefront aberration data.

[0092] In an optional embodiment, performing time phase adjustment on the full-eye wavefront aberration data according to the spatio-temporal correlation matrix to generate spatio-temporally aligned wavefront aberration data includes the following steps:

[0093] S331: Normalize the spatio-temporal correlation matrix to generate a time window alignment coefficient.

[0094] Specifically, the spatio-temporal correlation matrix is normalized to map the element values in the matrix to a unified interval (usually [0,1]) to better reflect the relative correlation strength between time windows. The normalization method can adopt min-max normalization, and its formula is:

[0095]

[0096] where, normalized_similarity ij refers to the value obtained after normalizing the elements in the original spatio-temporal correlation matrix. Specifically, it represents the normalized similarity of the eye movement time series feature vector and the wavefront aberration feature vector between the i-th time window and the j-th time window. similarity ij is the similarity value between the i-th time window and the j-th time window in the original spatio-temporal correlation matrix, and min(similarity) and max(similarity) are the minimum and maximum similarity values in the entire matrix respectively. Through normalization, the time window alignment coefficient can be compared and applied on a unified scale.

[0097] The normalized spatio-temporal correlation matrix is the time window alignment coefficient matrix. Each element value in the matrix represents the spatio-temporal correlation degree between the corresponding time windows, and the larger the value, the stronger the correlation. For example, if an element value is 0.8, it means that the corresponding time windows have a high similarity in terms of eye movement trajectory and wavefront aberration characteristics, and the alignment relationship between these time windows can be preferentially considered in time phase adjustment.

[0098] S332: According to the time window alignment coefficient, perform time phase offset compensation on the full-eye wavefront aberration data to generate aligned full-eye wavefront aberration data.

[0099] Specifically, according to the time window alignment coefficient, perform time phase offset compensation on the full-eye wavefront aberration data. For the wavefront aberration data within each time window, calculate the required time offset amount according to its time window alignment coefficient with the eye movement trajectory data. The calculation of the time offset amount can be based on interpolation methods such as linear interpolation or spline interpolation to estimate the value of the wavefront aberration data at the target time point. The purpose of compensation is to align the wavefront aberration data with the eye movement trajectory data in time and eliminate the time phase difference caused by asynchronous data acquisition or patient eye movement.

[0100] The implementation of the compensation process can be as follows: First, determine the reference time window. Usually, the time window with the highest similarity to the eye movement trajectory data (i.e., the largest time window alignment coefficient) is selected as the reference time window. Then, for the wavefront aberration data within other time windows, calculate the corresponding time offset according to the time difference and alignment coefficient with the reference time window. For example, if a certain time window lags behind the reference time window by Δt time and the alignment coefficient is c, then according to the interpolation algorithm, the wavefront aberration data within this time window is shifted forward by Δt time to compensate for the time phase offset. After compensation, the aligned full-eye wavefront aberration data is generated to ensure its consistency with the eye movement trajectory data in the time dimension.

[0101] S333: Spatially superimpose the aligned full-eye wavefront aberration data and the true corneal morphology data to generate spatio-temporally aligned wavefront aberration data.

[0102] Specifically, spatially superimpose the aligned full-eye wavefront aberration data and the true corneal morphology data to generate spatio-temporally aligned wavefront aberration data. The method of spatial superposition is to fuse the two in the same corneal coordinate system. Specifically, it can be achieved by mapping the wavefront aberration data to the grid points of the corneal morphology data, or by converting the two to a unified three-dimensional space coordinate system for fusion. The significance of this is to combine the wavefront aberration information with the morphological structure of the cornea to obtain comprehensive data that includes both time alignment and spatial position information, more comprehensively reflecting the optical state of the patient's eye.

[0103] The generated spatio-temporally aligned wavefront aberration data provides an accurate data basis for subsequent multi-modal data fusion and the generation of refractive correction plans. In multi-modal data fusion, this data can better cooperate with other modal data (such as pupil diameter data, etc.) to improve the accuracy and reliability of the fusion results. In the generation of refractive correction plans, it can more accurately determine the ablation parameters, considering the combined effects of corneal morphology and wavefront aberration in space and time, thereby improving the correction effect and reducing the occurrence of postoperative complications.

[0104] In an alternative embodiment, S5 includes the following steps:

[0105] S51: Input the fused multi-modal data into a pre-trained Nomogram prediction model to generate the ablation depth and the optical zone diameter.

[0106] Specifically, input the fused multi-modal data into a pre-trained Nomogram prediction model, which comprehensively considers the influence of multi-modal information such as corneal morphology, wavefront aberration, eye movement trajectory, and pupil diameter on refractive correction parameters. The model outputs the preliminary ablation depth and the optical zone diameter. The ablation depth determines the amount of corneal tissue to be removed, and the optical zone diameter determines the area range on the cornea that needs to be corrected.

[0107] The nomogram prediction model is trained with a large amount of clinical data, which includes multimodal ocular data of different patients and the corresponding postoperative refractive status and correction effects. The model learns the complex non-linear relationship between multimodal data and refractive correction parameters. During prediction, based on the input fused multimodal data, the model generates the ablation depth and optical zone diameter that best match the current patient's ocular characteristics through internal mathematical mapping and probability calculation.

[0108] S52: Based on the ablation depth and optical zone diameter, calculate the stress distribution of the ablated cornea through a finite element simulation model to generate the maximum stress value.

[0109] Specifically, based on the patient's real corneal morphology data, a finite element model of the cornea is constructed. The cornea is regarded as a non-linear biomechanical structure composed of multiple tiny elements, considering the anisotropy, inhomogeneity of the cornea and its interaction with physiological environments such as intraocular pressure. In the model, the material properties of the cornea, such as Young's modulus, Poisson's ratio, etc., can be set in detail, and these parameters are determined according to a large amount of experimental data and clinical studies to ensure the accuracy and reliability of the model.

[0110] The stress distribution calculation process can be as follows: Apply the ablation depth and optical zone diameter generated by S51 to the finite element model to simulate the healing process after corneal ablation surgery. During the simulation, calculate the stress distribution of the ablated cornea, especially pay attention to the stress concentration phenomenon in the ablation area and its periphery. By solving complex biomechanical equations, obtain the stress values at different positions of the cornea, and the maximum stress value is an important indicator for evaluating the biomechanical safety of the cornea. For example, if the ablation depth is too large or the design of the optical zone diameter is unreasonable, it may cause the stress in some areas of the cornea to exceed its bearing limit, increasing the risk of postoperative complications.

[0111] S53: If the maximum stress value exceeds the preset safety threshold, perform gradient descent optimization on the ablation depth to update the ablation depth.

[0112] Specifically, if the maximum stress value calculated by S52 exceeds the preset safety threshold, it indicates that the current ablation depth may have an adverse effect on the biomechanical stability of the cornea and needs to be optimized. The gradient descent method is used to optimize the ablation depth. This method calculates the gradient of the loss function (here it is the difference between the maximum stress value and the safety threshold) with respect to the ablation depth, and gradually adjusts the value of the ablation depth to minimize the loss function. Specifically, update the ablation depth according to the negative direction of the gradient, and the update formula is: ablation depth = ablation depth - learning rate × gradient, where the learning rate is a parameter that controls the optimization step size and can be reasonably set according to the actual situation to ensure the stability and convergence of the optimization process.

[0113] During the optimization process, not only is the reduction of the maximum stress value considered to ensure biomechanical safety, but also the effect of refractive correction is taken into account. Therefore, the optimization process can seek a balance between biomechanical safety and the effect of refractive correction. For example, appropriately reducing the cutting depth can reduce stress, but it may also affect the correction effect. Therefore, through multiple iterations and comprehensive evaluations, the optimal value of the cutting depth that can meet both the requirements of biomechanical safety and achieve good refractive correction needs to be found.

[0114] S54: Loop S52 to S53. When the maximum stress value is less than or equal to the preset safety threshold, generate a refractive correction plan based on the cutting depth.

[0115] Specifically, loop S52 to S53, that is, continuously perform finite element simulation to calculate the stress distribution and optimize the cutting depth according to the stress value. In each loop, recalculate the stress distribution based on the updated cutting depth to obtain the new maximum stress value, and then judge again whether it exceeds the safety threshold. If it exceeds, continue to optimize the cutting depth; if it does not exceed, it is considered that the current cutting depth meets the requirements of biomechanical safety, and proceed to the next step to generate a refractive correction plan.

[0116] When the maximum stress value is less than or equal to the preset safety threshold, generate the final refractive correction plan based on the currently optimized cutting depth and the previously determined optical zone diameter. This plan details the specific parameters of corneal cutting, including the cutting depth, area range, shape, etc. These parameters will directly guide the implementation of refractive correction surgery, ensuring that the surgery can effectively correct the patient's vision problems, protect the long-term biomechanical health of the cornea, and reduce the risk of postoperative complications.

[0117] The above method for generating a refractive correction plan based on multi-modal data fusion collects multi-modal data such as eye movement trajectories, corneal topography, wavefront aberration, and pupil diameter, uses a generative adversarial network to strip the motion artifacts caused by eye micro-movements and reconstruct the real corneal morphology, combines dynamic weight allocation and spatio-temporal alignment technology to achieve adaptive fusion of multi-modal data, and at the same time introduces mechanical simulation to optimize and iteratively adjust the cutting parameters for stress, and finally generates a correction plan that adapts to the individual eye characteristics and eye use requirements, thus systematically solving the problems of data distortion, poor inter-modal cooperation, and insufficient biomechanical adaptation caused by insufficient fixation ability in traditional technologies, significantly improving data reliability, plan accuracy, and postoperative safety, and realizing a full-process closed-loop control from noise suppression, dynamic fusion to biomechanical optimization.

[0118] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are sequentially shown according to the indications of the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same moment, but can be executed at different moments. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0119] Based on the same inventive concept, an embodiment of the present application also provides a system for implementing the above-mentioned method for generating a refractive correction plan based on multimodal data fusion. The solution provided by this system to solve the problem is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the refractive correction plan generation system based on multimodal data fusion provided below can refer to the limitations on the method for generating a refractive correction plan based on multimodal data in the above text, and will not be repeated here.

[0120] In an exemplary embodiment, as Figure 2 shown, a refractive correction plan generation system 20 based on multimodal data fusion is provided, including:

[0121] A data acquisition module 21, configured to acquire the eye movement trajectory data, corneal topographic data, full-eye wavefront aberration data, and pupil diameter data of a patient, and calculate a fixation stability index based on the eye movement trajectory data.

[0122] An artifact separation module 22, configured to input the corneal topographic data and the eye movement trajectory data into a pre-trained generative adversarial network, perform motion artifact separation processing on the corneal topographic data, and obtain real corneal morphology data.

[0123] A wavefront alignment module 23, configured to perform spatio-temporal alignment processing on the full-eye wavefront aberration data according to the eye movement trajectory data, and obtain spatio-temporally aligned wavefront aberration data.

[0124] A data fusion module 24, configured to perform weight allocation processing on the real corneal morphology data, the spatio-temporally aligned wavefront aberration data, and the pupil diameter data according to the fixation stability index, and obtain fused multimodal data.

[0125] A plan generation module 25, configured to generate initial cutting parameters based on the fused multimodal data, and perform mechanical simulation optimization on the initial cutting parameters to obtain a refractive correction plan.

[0126] Optionally, the eye movement trajectory data includes the XYZ three-axis displacement sequences of the eyeball in a three-dimensional space. The data acquisition module 21 includes a fixation stability calculation unit 211, which is configured to calculate the fixation stability index by calculating the average value of the displacement amounts of adjacent frames in the eye movement trajectory data in combination with the data acquisition period; the calculation formula of the fixation stability index is:

[0127]

[0128] wherein, FSI is the fixation stability index, T is the data acquisition period, N is the total number of frames of the eye movement trajectory data, {Δx i , Δy i , Δz i} is the XYZ three-axis displacement sequence, and i is the frame number sequence of the eye movement trajectory data.

[0129] Optionally, the artifact separation module 22 includes:

[0130] A data decomposition unit 221, which is configured to input the corneal topography data and the eye movement trajectory data into the generator of the generative adversarial network, decompose the corneal topography data, and obtain preliminary corneal morphology data and artifact component data; wherein, the corneal topography data is a linear superposition of the preliminary corneal morphology data and the artifact component data.

[0131] A biomechanical evaluation unit 222, which is configured to input the preliminary corneal morphology data into the discriminator of the generative adversarial network, and generate a biomechanical compliance score by performing a biomechanical compliance evaluation on the preliminary corneal morphology data based on the preset corneal curvature gradient distribution data of healthy people.

[0132] A parameter optimization unit 223, which is configured to iteratively optimize the parameters of the generative adversarial network according to the biomechanical compliance score and the adversarial loss function of the generative adversarial network.

[0133] A first loop control unit 224, which is configured to loop through the steps performed by the data decomposition unit 221, the biomechanical evaluation unit 222, and the parameter optimization unit 223, and when the biomechanical compliance score reaches a preset threshold, use the preliminary corneal morphology data as the real corneal morphology data.

[0134] Optionally, the data fusion module 24 includes:

[0135] An attenuation coefficient calculation unit 241, which is configured to calculate a modal attenuation coefficient according to the fixation stability index, and the modal attenuation coefficient is negatively correlated with the fixation stability index; wherein, the modal attenuation coefficient is used to characterize the influence degree of the patient's fixation ability on the reliability of each modal data, and the modal data is the real corneal morphology data, the spatio-temporally aligned wavefront aberration data, and the pupil diameter data.

[0136] The confidence generation unit 242 is configured to calculate the signal-to-noise ratio for each modality data respectively and generate the confidence of each modality data.

[0137] The weight assignment unit 243 is configured to perform weight assignment processing on each modality data through the Softmax function based on the modality attenuation coefficient and the confidence, so as to obtain the weight of each modality data; the calculation formula for the weight assignment processing is:

[0138]

[0139] where, w k is the weight of the k-th modality data, Confidence k is the confidence of the k-th modality data, α is the modality attenuation coefficient; K is the number of types of modality data, and m is the type serial number of the modality data.

[0140] The fusion unit 244 is configured to perform weighted fusion on the modality data according to the weights to obtain the fused multi-modal data.

[0141] Optionally, the wavefront alignment module 23 includes:

[0142] The temporal feature extraction unit 231 is configured to divide the eye movement trajectory data into multiple time windows, and perform feature extraction on the eye movement trajectory data within each time window to generate an eye movement temporal feature vector.

[0143] The spatial feature extraction unit 232 is configured to divide the full-eye wavefront aberration data according to time windows, and perform spatial feature extraction on the full-eye wavefront aberration data within each time window to generate a wavefront aberration feature vector.

[0144] The similarity calculation and adjustment unit 233 is configured to calculate the cosine similarity between the eye movement temporal feature vector and the wavefront aberration feature vector to generate a spatio-temporal correlation matrix; and perform time phase adjustment on the full-eye wavefront aberration data according to the spatio-temporal correlation matrix to generate spatio-temporally aligned wavefront aberration data.

[0145] Optionally, the similarity calculation and adjustment unit 233 includes:

[0146] The normalization processing subunit 2331 is configured to perform normalization processing on the spatio-temporal correlation matrix to generate a time window alignment coefficient.

[0147] The time phase compensation subunit 2332 is configured to perform time phase offset compensation on the full-eye wavefront aberration data according to the time window alignment coefficient to generate the aligned full-eye wavefront aberration data.

[0148] The spatial superposition subunit 2333 is configured to perform spatial superposition on the aligned full-eye wavefront aberration data and the real corneal morphology data to generate spatio-temporally aligned wavefront aberration data.

[0149] Optionally, the solution generation module 25 includes:

[0150] A parameter generation unit 251, configured to input the fused multimodal data into a pre-trained Nomogram prediction model to generate a cutting depth and an optical zone diameter.

[0151] A stress calculation unit 252, configured to calculate the stress distribution of the cut cornea through a finite element simulation model based on the cutting depth and the optical zone diameter, and generate a maximum stress value.

[0152] An optimization unit 253, configured to perform gradient descent optimization on the cutting depth if the maximum stress value exceeds a preset safety threshold to update the cutting depth.

[0153] A second loop control unit 254, configured to loop through the steps performed by the stress calculation unit 252 and the optimization unit 253, and generate a refractive correction solution based on the cutting depth when the maximum stress value is less than or equal to the preset safety threshold.

[0154] An embodiment of the present application further provides a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps in the foregoing method embodiments are implemented.

[0155] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the foregoing method embodiments are implemented.

[0156] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are only illustrative. The components described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present disclosure solution. Those of ordinary skill in the art can understand and implement it without creative work.

[0157] The above embodiments only represent several implementation manners of the embodiments of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the application embodiments. It should be noted that for those of ordinary skill in the art, without departing from the concept of the embodiments of the present application, several deformations and improvements can still be made, and these all belong to the protection scope of the embodiments of the present application.

Claims

1. A method for generating a refractive correction plan based on multi-modal data fusion, characterized in that, The method includes: S1: Collect the eye movement trajectory data, corneal topography data, whole-eye wavefront aberration data, and pupil diameter data of the patient, and calculate the fixation stability index based on the eye movement trajectory data; S2: Input the corneal topography data and the eye movement trajectory data into a pre-trained generative adversarial network, perform motion artifact separation processing on the corneal topography data, and obtain the true corneal morphology data; S3: According to the eye movement trajectory data, perform spatio-temporal alignment processing on the whole-eye wavefront aberration data to obtain spatio-temporally aligned wavefront aberration data; S4: According to the fixation stability index, perform weight allocation processing on the true corneal morphology data, the spatio-temporally aligned wavefront aberration data, and the pupil diameter data to obtain fused multimodal data; S5: Generate initial cutting parameters based on the fused multimodal data, and perform mechanical simulation optimization on the initial cutting parameters to obtain a refractive correction plan.

2. The method according to claim 1, wherein The eye movement trajectory data includes the XYZ three-axis displacement sequences of the eyeball in three-dimensional space; The calculation of the fixation stability index based on the eye movement trajectory data includes: combining the data acquisition period, calculating the mean value of the displacement amounts of adjacent frames in the eye movement trajectory data to obtain the fixation stability index; the calculation formula of the fixation stability index is: wherein, FSI is the fixation stability index, T is the data acquisition period, N is the total number of frames of the eye movement trajectory data, {Δx i , Δy i , Δz i} is the XYZ three-axis displacement sequence, and i is the frame number sequence of the eye movement trajectory data.

3. The method according to claim 1, characterized in that, The S2 includes: S21: Input the corneal topography data and the eye movement trajectory data into the generator of the generative adversarial network, decompose the corneal topography data to obtain preliminary corneal morphology data and artifact component data; wherein, the corneal topography data is a linear superposition of the preliminary corneal morphology data and the artifact component data; S22: Input the preliminary corneal morphology data into the discriminator of the generative adversarial network, and perform biomechanical compliance evaluation on the preliminary corneal morphology data based on the preset corneal curvature gradient distribution data of healthy people to generate a biomechanical compliance score; S23: According to the biomechanical compliance score and the adversarial loss function of the generative adversarial network, iteratively optimize the parameters of the generative adversarial network; S24: Loop S21 to S23. When the biomechanical compliance score reaches the preset threshold, use the preliminary corneal morphology data as the true corneal morphology data.

4. The method according to claim 1, wherein The S4 includes: S41: Calculate the modal attenuation coefficient according to the fixation stability index, and the modal attenuation coefficient is negatively correlated with the fixation stability index; wherein, the modal attenuation coefficient is used to characterize the influence degree of the patient's fixation ability on the reliability of each modal data, and the modal data is the true corneal morphology data, the spatio-temporally aligned wavefront aberration data, and the pupil diameter data; S42: Calculate the signal-to-noise ratio of each modal data respectively to generate the confidence level of each modal data; S43: Based on the modal attenuation coefficient and the confidence level, perform weight allocation processing on each modal data through the Softmax function to obtain the weights of each modal data; the calculation formula of the weight allocation processing is: where w k is the weight of the k-th modal data, Confidence k is the confidence of the k-th modal data, α is the modal decay coefficient; K is the number of types of the modal data, and m is the type serial number of the modal data; S44: Weightedly fuse the modal data according to the weights to obtain the fused multi-modal data.

5. The method according to claim 1, wherein The S3 includes: S31: Segment the eye movement trajectory data into multiple time windows, and extract features from the eye movement trajectory data within each time window to generate an eye movement time-series feature vector; S32: Segment the full-eye wavefront aberration data according to the time window, and extract spatial features from the full-eye wavefront aberration data within each time window to generate a wavefront aberration feature vector; S33: Calculate the cosine similarity between the eye movement time-series feature vector and the wavefront aberration feature vector to generate a spatio-temporal correlation matrix; and adjust the time phase of the full-eye wavefront aberration data according to the spatio-temporal correlation matrix to generate the spatio-temporally aligned wavefront aberration data.

6. The method according to claim 5, characterized in that, The adjusting the time phase of the full-eye wavefront aberration data according to the spatio-temporal correlation matrix to generate the spatio-temporally aligned wavefront aberration data includes: S331: Normalize the spatio-temporal correlation matrix to generate a time window alignment coefficient; S332: Compensate for the time phase shift of the full-eye wavefront aberration data according to the time window alignment coefficient to generate the aligned full-eye wavefront aberration data; S333: Spatially superimpose the aligned full-eye wavefront aberration data and the true corneal shape data to generate the spatio-temporally aligned wavefront aberration data.

7. The method according to any one of claims 1 to 6, characterized in that, The S5 includes: S51: Input the fused multi-modal data into a pre-trained Nomogram prediction model to generate the ablation depth and the optical zone diameter; S52: Based on the ablation depth and the optical zone diameter, calculate the stress distribution of the ablated cornea through a finite element simulation model to generate the maximum stress value; S53: If the maximum stress value exceeds a preset safety threshold, optimize the ablation depth by gradient descent to update the ablation depth; S54: Loop S52 to S53. When the maximum stress value is less than or equal to the preset safety threshold, generate the refractive correction plan based on the ablation depth.

8. A refractive correction plan generation system based on multimodal data fusion, characterized in that, The system includes: A data acquisition module, configured to acquire the eye movement trajectory data, corneal topography data, full-eye wavefront aberration data, and pupil diameter data of a patient, and calculate a fixation stability index based on the eye movement trajectory data; An artifact separation module, configured to input the corneal topography data and the eye movement trajectory data into a pre-trained generative adversarial network to perform motion artifact separation processing on the corneal topography data to obtain the true corneal shape data; A wavefront alignment module, configured to perform spatio-temporal alignment processing on the full-eye wavefront aberration data according to the eye movement trajectory data to obtain the spatio-temporally aligned wavefront aberration data; A data fusion module, configured to obtain the fused multi-modal data by performing weight allocation processing on the true corneal shape data, the spatio-temporally aligned wavefront aberration data, and the pupil diameter data according to the fixation stability index; A plan generation module, configured to generate initial ablation parameters based on the fused multi-modal data, and perform mechanical simulation optimization on the initial ablation parameters to obtain a refractive correction plan.

9. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the method described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the method described in any one of claims 1 to 7 is implemented.

Images