Cornea space-time biomechanical feature extraction method and device based on attention mechanism

Abstract

The invention discloses a cornea space-time biomechanical feature extraction method and device based on an attention mechanism, and relates to the technical field of cornea biomechanical feature recognition, and the method comprises the steps: obtaining a target cornea video; the target cornea video comprises a plurality of frames of cornea images; a trained cornea video classification model is adopted to obtain a classification result corresponding to the target cornea video, the trained cornea video classification model comprises a partitioning module, a linear embedding module, a fusion module and an output module, and the fusion module comprises an attention unit; the attention unit introduces 3D relative position codes in the self-attention calculation process; the attention unit comprises a window multi-head self-attention layer based on a 3D sliding window and a displacement window multi-head self-attention layer based on the 3D sliding window, the dynamic space-time characteristics of global biomechanics in the whole deformation process of the cornea can be obtained, and accurate data support is provided for clinical corneal anomaly recognition.

Description

Technical Field

[0001] The present application relates to the technical field of corneal biometric identification, and in particular to a method and device for extracting spatiotemporal biomechanical features of the cornea based on an attention mechanism. Background Art

[0002] The cornea is a crucial optical tissue of the eye, and its biomechanical properties directly influence its morphological stability, refractive state, and response to mechanical forces (such as intraocular pressure). Under normal circumstances, the cornea maintains stable deformation under external mechanical forces and rapidly recovers upon force removal. However, in certain ophthalmic diseases, the corneal biomechanical response undergoes significant changes, making corneal biomechanics research a key area of research in ophthalmology, refractive surgery, and visual science. For example, keratoconus (KC), a corneal lesion characterized by central or paracentral corneal dilation and a progressive, conical protrusion, is a major cause of blindness. Early detection of keratoconus is crucial for slowing or halting disease progression. However, due to the subtle early symptoms, traditional ophthalmic examinations, such as slit-lamp microscopy and corneal topography, often fail to detect early keratoconus (Early KC). However, changes in corneal biomechanics are becoming increasingly important in early identification. In recent years, rapid advances in corneal biomechanical testing technologies have made it possible to study the dynamic mechanical response of the cornea under external forces. However, existing studies have focused on changes in a single or limited number of locations and time points during corneal deformation, ignoring the global dynamic response and failing to capture the global biomechanical spatiotemporal abnormalities of the cornea. Therefore, there is an urgent need to develop a method and device that can comprehensively capture the global dynamic spatiotemporal biomechanical properties of the cornea under external forces to address the shortcomings of existing detection methods and provide accurate data support for clinical corneal abnormality identification. Summary of the Invention

[0003] The purpose of this application is to provide a method and device for extracting corneal spatiotemporal biomechanical features based on an attention mechanism, which can obtain the dynamic spatiotemporal characteristics of global biomechanics during the entire corneal deformation process, and provide accurate data support for clinical corneal abnormality identification.

[0004] To achieve the above objectives, this application provides the following solutions:

[0005] In a first aspect, the present application provides a method for extracting spatiotemporal biomechanical features of the cornea based on an attention mechanism, comprising:

[0006] Acquire a target cornea video; the target cornea video includes a plurality of frames of cornea images;

[0007] Inputting the target cornea video into a trained cornea video classification model to obtain a classification result corresponding to the target cornea video; the trained cornea video classification model includes a block module, a linear embedding module, a fusion module and an output module;

[0008] The blocking module is used to block the cornea image in the target cornea video to obtain a plurality of image blocks;

[0009] The linear embedding module is used to: adjust the channel dimension of the image block to obtain the target feature vector;

[0010] The fusion module includes several attention units and patch merging units with cross-connections, and the last unit is an attention unit; the attention unit introduces 3D relative position encoding in the self-attention calculation process; the attention unit includes a window multi-head self-attention layer based on a 3D sliding window and a shift window multi-head self-attention layer based on a 3D sliding window; the window multi-head self-attention layer based on a 3D sliding window is used to extract local spatiotemporal features within a local window; the shift window multi-head self-attention layer based on a 3D sliding window is used to introduce global dependencies when the window slides and extract global spatiotemporal information; the attention unit outputs spatiotemporal weighted features; the patch merging unit is used to perform filling processing and channel dimension splicing on the spatiotemporal weighted features to obtain a fused feature vector; the last attention unit outputs the final spatiotemporal features;

[0011] The output module is used to obtain a classification result corresponding to the target cornea video according to the final spatiotemporal features.

[0012] In a second aspect, the present application provides a device for extracting spatiotemporal biomechanical features of the cornea based on an attention mechanism, comprising:

[0013] A target cornea video acquisition module is used to acquire a target cornea video; the target cornea video includes a plurality of frames of cornea images;

[0014] A corneal video classification module is used to input the target corneal video into a trained corneal video classification model to obtain a classification result corresponding to the target corneal video; the trained corneal video classification model includes a block module, a linear embedding module, a fusion module and an output module;

[0015] The blocking module is used to block the cornea image in the target cornea video to obtain a plurality of image blocks;

[0016] The linear embedding module is used to: adjust the channel dimension of the image block to obtain the target feature vector;

[0017] The fusion module includes several attention units and patch merging units with cross-connections, and the last unit is an attention unit; the attention unit introduces 3D relative position encoding in the self-attention calculation process; the attention unit includes a window multi-head self-attention layer based on a 3D sliding window and a shift window multi-head self-attention layer based on a 3D sliding window; the window multi-head self-attention layer based on a 3D sliding window is used to extract local spatiotemporal features within a local window; the shift window multi-head self-attention layer based on a 3D sliding window is used to introduce global dependencies when the window slides and extract global spatiotemporal information; the attention unit outputs spatiotemporal weighted features; the patch merging unit is used to perform filling processing and channel dimension splicing on the spatiotemporal weighted features to obtain a fused feature vector; the last attention unit outputs the final spatiotemporal features;

[0018] The output module is used to obtain a classification result corresponding to the target cornea video according to the final spatiotemporal features.

[0019] In a third aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned method for extracting corneal spatiotemporal biomechanical features based on the attention mechanism.

[0020] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned corneal spatiotemporal biomechanical feature extraction method based on the attention mechanism.

[0021] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0022] The present application provides a method and device for extracting spatiotemporal biomechanical features of the cornea based on an attention mechanism. By introducing 3D relative position encoding in the self-attention calculation process of the attention unit in the fusion module, the self-attention calculation is enhanced. Based on the dynamic video of the corneal force deformation, more subtle biomechanical characteristics can be detected in the early stage, providing accurate data support for clinical corneal abnormality identification. The attention unit in the fusion module is constructed by a window multi-head self-attention layer based on a 3D sliding window and a shifted window multi-head self-attention layer based on a 3D sliding window. Fine-grained features are extracted within the local window, and global connections are established during the window sliding process to enhance the recognition ability of corneal biomechanical characteristics. The 3D sliding window attention mechanism is adopted to localize the self-attention calculation, reduce the computational complexity, and ensure efficient spatiotemporal information interaction, effectively reduce the computational cost, and improve the reasoning speed (corneal video classification speed). BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 This is a diagram of the application environment of a method for extracting spatiotemporal biomechanical features of the cornea based on an attention mechanism in one embodiment of the present application.

[0025] Figure 2 A flowchart of a method for extracting spatiotemporal biomechanical features of the cornea based on an attention mechanism is provided in accordance with an embodiment of the present application.

[0026] Figure 3 A schematic diagram of the trained corneal video classification model architecture provided in one embodiment of the present application.

[0027] Figure 4 A process diagram of a block segmentation module provided in one embodiment of the present application.

[0028] Figure 5 A schematic diagram of the self-attention subunit structure provided in one embodiment of the present application.

[0029] Figure 6 A schematic diagram of a 3D sliding window mechanism provided in one embodiment of the present application.

[0030] Figure 7 A schematic diagram of the functional modules of a device for extracting spatiotemporal biomechanical features of the cornea based on an attention mechanism is provided as another embodiment of the present application.

[0031] Figure 8 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0033] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0034] The method for extracting spatiotemporal biomechanical features of the cornea based on the attention mechanism provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown in FIG. , the terminal 102 communicates with the server 104 via a network. The data storage system can store data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the target cornea video to be processed to the server 104. After the server 104 receives the target cornea video to be processed, the server 104 inputs the target cornea video to the trained cornea video classification model to obtain a classification result label corresponding to the target cornea video. The server 104 can feed back the obtained classification result label for the target cornea video to the terminal 102. In addition, in some embodiments, the corneal spatiotemporal biomechanical feature extraction method based on the attention mechanism can also be implemented separately by the server 104 or the terminal 102. For example, the terminal 102 can directly perform video labeling on the target cornea video to be processed, or the server 104 can obtain the target cornea video to be processed from the data storage system and perform video labeling on the target cornea video to be processed.

[0035] Terminal 102 may include, but is not limited to, various desktop computers, laptops, smartphones, tablet computers, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, and smart car devices. Portable wearable devices may include smart watches, smart bracelets, and head-mounted devices. Server 104 may be implemented as a standalone server or a server cluster consisting of multiple servers, or may be a cloud server.

[0036] In an exemplary embodiment, Figure 2 As shown, a method for extracting spatiotemporal biomechanical features of the cornea based on an attention mechanism is provided. The method is executed by a computer device, specifically a computer device such as a terminal or a server, or a terminal and a server. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used for explanation, including the following steps 201 to 202.

[0037] Step 201: Acquire a target cornea video; the target cornea video includes several frames of cornea images.

[0038] Step 202: Input the target cornea video into a trained cornea video classification model to obtain a classification result corresponding to the target cornea video.

[0039] Implement the above-mentioned steps 201 to 202. By introducing 3D relative position encoding in the self-attention calculation process of the attention unit in the fusion module, the self-attention calculation is enhanced. Based on the dynamic video of corneal force-induced deformation, more subtle biomechanical characteristics can be detected at an early stage, providing accurate data support for clinical corneal abnormality recognition, and can be used to improve the level of personalized eye health assessment and the formulation of intervention strategies. The attention unit in the fusion module is constructed by a window multi-head self-attention layer based on a 3D sliding window and a shifted window multi-head self-attention layer based on a 3D sliding window, which extracts fine-grained features within a local window and establishes global connections during the window sliding process, enhancing the recognition ability of corneal biomechanical characteristics. The 3D sliding window attention mechanism is adopted to localize the self-attention calculation, reduce the computational complexity, and ensure efficient spatio-temporal information interaction, effectively reducing the computational cost and improving the inference speed (corneal classification speed).

[0040] In another exemplary embodiment of the present application, the above step 201 is used for video data preprocessing, and may include the following steps 301 to 304.

[0041] Step 301: Obtain the original corneal video.

[0042] Step 302: Use an unsupervised shared model to extract corneal contour points and write them into a txt file. After reading the video data, use the unsupervised shared model to extract corneal contour points and write them into a txt file. The format of each line in the txt file is as follows: x y frame layer, where: x represents the abscissa, and the value range is 0 < x < W (W is the video width); y represents the ordinate, and the value range is 0 < y < H (H is the video height); frame represents the frame number, and the value range is 0 < frame < N (N is the total number of frames of the original corneal video to be processed); layer represents the layer to which the cornea belongs, and the value is 0 or 1, where 0 represents the upper corneal layer and 1 represents the lower corneal layer. The resolution of the original corneal video is W×H, and the original corneal video has a total of N frames. Parse the corneal area from the txt file and extract the corneal area in the original corneal video, and finally synthesize a new video.

[0043] Step 303: Parse the txt file to construct a corneal area mask.

[0044] Initialize two dictionaries upper_layer and lower_layer to store the upper and lower surface contours of the cornea for each frame respectively. upper_layer[frame] is an array of length W, storing the y coordinates of the upper surface at each abscissa x. lower_layer[frame] is an array of length W, storing the y coordinates of the lower surface at each abscissa x. Initially, all values are set to -1, indicating that there is no data in this column.

[0045] Read the txt file line by line. For each line of data, if layer = 0, it means that the point belongs to the upper surface of the cornea, and the vertical coordinate y is stored in upper_layer[frame][x]. If layer = 1, it means that the point belongs to the lower surface of the cornea, and the vertical coordinate y is stored in lower_layer[frame][x].

[0046] After parsing is completed, traverse the upper_layer and lower_layer, and in each frame frame, construct the mask image mask[frame] according to the following rules: for each column coordinate x, upper_layer[frame][x]!=-1 and lower_layer[frame][x]!=-1, then fill the area between the upper and lower layers mask[upper_layer[frame][x]:lower_layer[frame][x], x]=255, and keep the other areas 0, that is, the background is black.

[0047] After the above steps, mask_dict is constructed, where mask_dict[frame] is the binary mask of the cornea area corresponding to the frame.

[0048] Step 304: Read the video application mask frame by frame to obtain the target cornea video.

[0049] Read a video in avi format and obtain its resolution (W×H), frame rate (fps), and total number of frames (N). First, initialize cv2.VideoWriter, use XVID encoding, and set fps to maintain the smoothness of the original video. Next, read the video frame by frame. For each frame (frame_idx), read the original frame (frame), which has a shape of H×W×3. Read the corresponding mask (mask_dict[frame_idx]), which has a shape of H×W. Finally, extract the cornea region using a bitwise_and operation: frame = frame⊙mask. ⊙ represents an element-by-element multiplication operation. For regions with mask 255, the original pixel values are retained. For regions with mask 0, the original pixel values are set to 0. Write frame_{new} to VideoWriter. After traversing all N frames, release cv2.VideoCapture and cv2.VideoWriter to complete the cornea video processing and obtain the target cornea video.

[0050] The corneal video classification model in this application adopts the Transformer architecture. Based on the Transformer architecture, the images in the corneal video are divided into non-overlapping blocks. Using sliding window attention, hierarchical structure, and spatiotemporal feature extraction, it not only extracts visual information between video frames in space, but also captures the dynamic changes of the video in the temporal dimension, achieving efficient extraction of spatial and temporal features in the video. It performs well in tasks such as video classification and action recognition, and is currently the most cutting-edge model in video detection problems. Based on the Transformer architecture, it directly analyzes the entire video of the cornea deforming after being subjected to force, detects the dynamic spatiotemporal changes of the corneal biomechanical response, and obtains the dynamic spatiotemporal characteristics of the global biomechanics of the cornea during the entire deformation process. It has been applied to the early keratoconus recognition scenario and achieved good results.

[0051] We collect video data of corneal deformation (target corneal video) and use the inherent spatiotemporal locality of the target corneal video to expand the scope of local attention calculation from the spatial domain to the spatiotemporal domain, capturing the spatial-temporal characteristics of the entire process of corneal deformation under force. We introduce local inductive bias, use spatiotemporal factorization and non-overlapping windows to learn the locality, hierarchy and translation invariance of the video.

[0052] The trained corneal video classification model includes a segmentation module, a linear embedding module, a fusion module and an output module. The architecture of the trained corneal video classification model is as follows: Figure 3 shown.

[0053] The block module is used to block the cornea image in the target cornea video to obtain a number of image blocks. Figure 3 In the Patch Partition module,

[0054] The collected corneal deformation video (target cornea video) is used as input data. Each target cornea video is 31.88ms long and contains 139 frames. The width of each cornea image frame is 576 and the height is 200. The frame interval is set to 2 and the clip len is set to 70, that is, one frame is collected every two frames. Therefore, the entire target cornea video can collect 70 frames of cornea images, that is, T is 70, H is 200, and W is 576.

[0055] In the Patch Partition module, the target cornea video (the target cornea video after the filtered frame) is divided into non-overlapping image blocks (patches) by dividing the target cornea video into blocks. The patch size is (2, 8, 8), which represent the dimensions of time, height and width respectively. tokens. Each token is 384-dimensional (2×8×8×3=384), which is the value of the three RGB channels of all pixels in a patch. Figure 3 For example, the input of the model is a target cornea video with a size of T×H×W×3, which will become a vector (i.e. image block), such as Figure 4 As shown, the size of 3Dtokens is: T×H×W=2×8×8.

[0056] The linear embedding module is used to adjust the channel dimension of the image block to obtain the target feature vector. Figure 3 The Linear Embedding module in .

[0057] The linear embedding module includes a first filling unit, a first splicing unit and a first linear unit connected in sequence; the first filling unit is used to perform 3D filling processing on each image block respectively to obtain a feature vector corresponding to each image block; the first splicing unit is used to splice the feature vectors corresponding to all image blocks to obtain a spliced feature vector; the first linear unit is used to perform a linear transformation on the spliced feature vector to obtain a target feature vector.

[0058] The above image block ( The vector of ) is input into the Linear Embedding module, and the first filling unit of the LinearEmbedding module (the first filling unit refers to Figure 3 The padding in the Linear Embedding module in the 3D padding process is performed on the input to obtain a feature vector, which is then passed through the first splicing unit (the first splicing unit refers to Figure 3 2×2cat in the LinearEmbedding module), perform feature concatenation to obtain the concatenated feature vector, and pass it through the first linear unit (the first linear unit refers to Figure 3 The linear transformation in the LinearEmbedding module is performed to map it to a fixed-dimensional feature space to ensure that the boundary information is not lost, thereby obtaining the target feature vector.

[0059] The fusion module includes several cross-connected attention units and patch merging units, the last unit of which is the attention unit; the attention unit introduces 3D relative position encoding (3D Relative Position Encoding) in the self-attention calculation process; the attention unit includes a window multi-head self-attention (W-MSA) layer based on a 3D sliding window and a shifted-window multi-head self-attention (SW-MSA) layer based on a 3D sliding window; the window multi-head self-attention layer based on a 3D sliding window is used to extract local spatiotemporal features within a local window; the shifted-window multi-head self-attention layer based on a 3D sliding window is used to introduce global dependencies when the window slides and extract global spatiotemporal information; the attention unit outputs spatiotemporal weighted features; the patch merging unit is used to fill the spatiotemporal weighted features and perform channel dimension splicing to obtain a fused feature vector; the last attention unit outputs the final spatiotemporal features.

[0060] The above attention unit refers to Figure 3 The Video Swin Transformer module in the video swintransformer mainly includes the W-MSA layer based on the 3D sliding window, the SW-MSA layer based on the 3D sliding window, and the MLP (multilayer perceptron) module. Its specific structure is as follows Figure 5 As shown. The W-MSA layer based on 3D sliding window refers to Figure 5 In the 3DW-MSA, the SW-MSA layer based on the 3D sliding window refers to Figure 5 3D SW-MSA in.

[0061] The attention unit consists of two self-attention sub-units, which are Figure 5VideoSwinTransformerBlock in. Each self-attention subunit includes a first unit, a second unit, a third unit and a fourth unit; the first unit includes a first normalization layer (LayerNorm, LN), a window multi-head self-attention layer based on a 3D sliding window, a first regularization layer (Dropout / DropPath) and a first addition layer, and the first addition layer is used to add the output of the first regularization layer to the input of the first unit; the second unit includes a second normalization layer, a first MLP layer, a second regularization layer and a second addition layer, and the second addition layer is used to add the output of the second regularization layer to the input of the second unit; the third unit includes a third normalization layer, a shifted window multi-head self-attention layer based on a 3D sliding window, a third regularization layer and a third addition layer, and the third addition layer is used to add the output of the third regularization layer to the input of the third unit; the fourth unit includes a fourth normalization layer, a second MLP layer, a fourth regularization layer and a fourth addition layer, and the fourth addition layer is used to add the output of the fourth regularization layer to the input of the fourth unit.

[0062] First, the target cornea video is divided into T×H×W tokens in the Patch Partition module, where T is the time dimension (number of frames), H is the height, and W is the width. The window size is defined as P×M×M, where P=4 and M=4. Self-attention is calculated within the window, that is, focusing on the local area in the spatial and temporal dimensions of the target cornea video sequence. The window is then moved to overlap with the adjacent window of the previous layer. This overlapping or moving window strategy enables the cornea video classification model to capture the interaction between different regions of the target cornea video, ensuring that attention is calculated over the entire spatiotemporal dimensions of the target cornea video, which helps to learn more global relationships, improve the representation ability of complex video data, and reduce the computational burden brought by global operations on the entire target cornea video.

[0063] In the self-attention calculation process, the introduction of 3D relative position encoding can better understand the spatial-temporal relationship between different frames in the video. The self-attention calculation process of the attention unit is calculated as follows:

[0064]

[0065] Among them, Attention(Q,K,V) is the attention weight, Q is the query vector, K is the key vector, V is the value vector, softmax is the normalization function, d k is the scaling factor.

[0066] Q, K, and V are obtained by linearly transforming the input matrix X. The formulas are as follows:

[0067] Q=XW Q (2);

[0068] K=XW K (3);

[0069] V=XW V (4);

[0070] Among them, W Q 、W K and W V It is a parameter matrix. The input matrix X is multiplied by the three parameter matrices respectively, which is equivalent to a linear transformation to obtain the query vector Q, key vector K, and value vector V respectively.

[0071] The Video Swin Transformer module uses two structures: 3D W-MSA (window MSA) for local information exchange within a window, and 3D SW-MSA (shifted-window MSA) for global information exchange between windows. The two structures are used in pairs. The output of the self-attention subunit is represented as follows:

[0072]

[0073] in, is the output of the first unit, i.e., the 3D(S)W-MSA module output of the lth Video Swin Transformer Block; 3DW-MSA represents a window multi-head self-attention layer based on a 3D sliding window. 3DW-MSA represents the use of conventional window multi-head self-attention for information exchange within the window (local); LN represents layer normalization; z l-1 represents the input of the first unit; z l is the output of the second unit, that is, the output of the MLP module of the lth Video Swin Transformer Block; FFN represents the FFN network in the multi-layer perceptron; is the output of the third unit; 3DSW-MSA represents a shift window multi-head self-attention layer based on a 3D sliding window, and 3DSW-MSA represents a multi-head self-attention layer using a shift window for information exchange between windows (global); z l+1 is the output of the fourth unit.

[0074] The target feature vector obtained by the linear embedding module undergoes layer normalization (Layer Norm) to ensure stable training. It then passes through the 3D W-MSA layer to calculate self-attention within the local window. This calculated self-attention is a weighted representation of the local spatiotemporal features. Regularization (Dropout / DropPath) is then performed to reduce the risk of overfitting. The calculated result is used as a residual block, normalized (Layer Norm), and passed to the MLP. Regularization (Dropout / DropPath) is performed again, and the residual is concatenated and normalized before entering the 3D SW-MSA layer. A shifted window strategy is used to enhance feature interaction.

[0075] The specific process involves localizing self-attention computations using a 3D SW-MSA layer, reducing computational complexity. Global dependencies are introduced during window sliding, maintaining the model's ability to capture local features. This allows the model to extract local spatiotemporal information within smaller time segments. Furthermore, by shifting the window, cross-window connections are established. The temporal correlation between video frames captures both local and global spatial and temporal dependencies across different time frames, enhancing video representation.

[0076] By using uniformly distributed non-overlapping windows, not only local attention is calculated in the spatial dimension, but also global attention is calculated in the temporal dimension. Figure 6 The data then undergoes regularization and residual connections in the same order, passes through the MLP layer again for feature transformation, and is finally passed to the next layer through a residual connection. The first, second, third, or third addition layers mentioned above are all residual connections.

[0077] The patch merging unit refers to Figure 3 The patch merging unit includes a second filling unit, a second splicing unit and a third linear unit connected in sequence. The second filling unit refers to Figure 3 In the patchmerging, the second splicing unit refers to Figure 3 The 2×2cat in the patchmerging module, the third linear unit refers to Figure 3 Linear in the patchmerging module.

[0078] According to the above attention unit ( Figure 3The spatiotemporal weighted features obtained by the Video SwinTransformer in are padded (to ensure that the boundary information is not lost in subsequent operations) to obtain the padded feature vector. The specific process is: add zero-value padding at the boundary of the spatial dimension so that the feature map size can be divided by the subsequent 2×2 window. 2x2cat concatenates adjacent 2×2 patches in the channel direction, thereby reducing the spatial dimension (that is, reducing the width and height) while increasing the number of channels. The specific process is: the features in each 2×2 spatial region are concatenated along the channel axis to generate a new feature tensor whose spatial resolution is halved. The number of channels is increased to 4 times the original number and linear processing is performed (the concatenated data is mapped to adjust the channel dimension and extract features) to obtain a fused feature vector. The specific process is: the concatenated high-dimensional features are mapped down to the target dimension through a linear layer, the number of channels is adjusted to the target dimension (for example, from 4 times the number of channels to 2 times the number of channels), and a more compact feature representation is extracted. This module does not perform merging operations on the time dimension, that is, the output time dimension remains unchanged. This part is used to change the shape of the feature, which can be regarded as a pooling operation in the CNN model, and it will also change the dim of the feature. The whole process imitates the downsampling process in the CNN module, so that the model can generate features for different scales. The shallow layer can see small objects, while the deep layer focuses on large objects.

[0079] like Figure 3 As shown in FIG. 4 , the fusion module includes four attention units and three patch merging units, and the fusion module includes a first attention unit, a first patch merging unit, a second attention unit, a second patch merging unit, a third attention unit, a third patch merging unit, and a fourth attention unit connected in sequence.

[0080] According to the first fused feature vector obtained by the above-mentioned first patch merging unit, the enhanced spatiotemporal feature is obtained again through the second attention unit (Video SwinTransformer). The enhanced spatiotemporal feature is then processed by patchmerging in the second patch merging unit to obtain a downsampled feature vector. The downsampled feature vector is again passed through the third attention unit (Video SwinTransformer) to obtain a deep spatiotemporal feature representation. The deep spatiotemporal feature representation is then processed by patchmerging in the third patch merging unit to obtain a multi-scale feature vector. The multi-scale feature vector is again passed through the fourth attention unit (Video SwinTransformer) to obtain the final spatiotemporal feature.

[0081] An output module is used to obtain a classification result corresponding to the target cornea video according to the final spatiotemporal features. The output module includes a layer normalization layer, a global pooling layer, and a dense layer connected in sequence.

[0082] The final spatiotemporal features output by the fusion module are normalized and fused using LayerNormalization to produce a comprehensive feature vector. This comprehensive feature vector is then pooled using the GlobalPooling layer to produce a global feature vector. Based on this global feature vector, classification mapping is performed using the Dense Layer layer to obtain the classification result corresponding to the target cornea video. The classification results include Normal (Normal), Early Keratoconus (EarlyKC), and Cornea (KC).

[0083] Before step 202, a training process of a corneal video classification model is also included, specifically including: obtaining a sample set; the sample set includes several sample corneal videos and a sample classification result corresponding to each sample corneal video; using the sample set to train the corneal video classification model to obtain a trained corneal video classification model.

[0084] In the overall model configuration, Recognizer3D is used as the model type, its backbone is SwinTransformer3D, the configuration parameters include patch_size = (2, 8, 8), embed_dim = 96, depths = [2, 2, 6, 2], num_heads = [3, 6, 12, 24], window_size = (8, 7, 7), mlp_ratio = 4.0, qkv_bias = True, qk_scale is set to None, drop_rate and attn_drop_rate are both 0.0, drop_path_rate is 0.2, and patch_norm is enabled. The classification head uses I3DHead with in_channels set to 768, num_classes set to 3, spatial_type set to avg, and dropout_ratio set to 0.5. The model training uses the SGD (Stochastic Gradient Descent) optimizer, and its update formula is:

[0085]

[0086] Among them, w t+1 is the model parameter obtained at time step t+1, w t is the model parameter obtained at time step t, xi is the i-th sample cornea video, y i is the classification result of the i-th sample cornea video, Is in (x i ,y i ) is the gradient of the loss function L with respect to the model parameters w; η is the learning rate, which controls the update step size.

[0087] The present application also provides an application scenario, which applies the above-mentioned corneal spatiotemporal biomechanical feature extraction method based on the attention mechanism. Specifically: the corneal spatiotemporal biomechanical feature extraction method based on the attention mechanism provided in this embodiment can be applied in the corneal classification scenario. The corneal classification scenario includes a content production link and a corneal classification link; the target corneal video to be processed enters the corneal classification link from the content production link, and the corresponding classification result is obtained through human-machine collaboration. The corneal spatiotemporal biomechanical feature extraction method based on the attention mechanism provided in this embodiment belongs to the machine labeling link in the corneal classification link. Specifically, in the corneal classification link process for videos, the target corneal video can be input into the trained corneal video classification model to obtain the classification result label corresponding to the target corneal video.

[0088] This application has the following advantages:

[0089] 1. Efficient spatiotemporal feature extraction, low computational complexity, and high inference efficiency: The Video SwinTransformer module extracts fine-grained features within a local window while establishing global connections during the window sliding process, enhancing the ability to identify corneal biomechanical properties. A 3D sliding window attention mechanism localizes self-attention calculations, reducing computational complexity while ensuring efficient spatiotemporal information exchange, effectively reducing computational costs and improving inference speed.

[0090] 2. Accurately capture early-stage biomechanical abnormalities of keratoconus: Traditional methods mostly rely on corneal morphological features. The method provided in this application benefits from 3D relative position encoding enhanced self-attention calculation and is based on corneal biomechanical dynamic video. It can detect more subtle biomechanical abnormalities in the early stages and provide accurate data support for clinical corneal abnormality identification.

[0091] 3. Enhanced spatial-temporal dependency modeling capabilities: Through the 3D relative position encoding self-attention calculation scheme, the model's ability to model the spatial-temporal relationship between different time frames is enhanced, which is more consistent with the physical characteristics of corneal deformation videos.

[0092] 4. Applicable to various corneal detection scenarios: It can be widely used in local corneal biomechanical assessment, providing data support for preoperative screening for refractive surgery and other corneal condition identification, and has broad clinical application value.

[0093] Based on the same inventive concept, embodiments of the present application also provide an attention-based corneal spatiotemporal biomechanical feature extraction device for implementing the aforementioned attention-based corneal spatiotemporal biomechanical feature extraction method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more embodiments of the attention-based corneal spatiotemporal biomechanical feature extraction device provided below can be found in the limitations of the attention-based corneal spatiotemporal biomechanical feature extraction method described above and will not be repeated here.

[0094] In an exemplary embodiment, Figure 7 As shown, a corneal spatiotemporal biomechanical feature extraction device based on an attention mechanism is provided, which includes the following modules.

[0095] The target cornea video acquisition module T1 is used to acquire the target cornea video; the target cornea video includes several frames of cornea images.

[0096] The corneal video classification module T2 is used to input the target corneal video into the trained corneal video classification model to obtain the classification result corresponding to the target corneal video; the trained corneal video classification model includes a blocking module, a linear embedding module, a fusion module and an output module.

[0097] The blocking module is used to block the cornea image in the target cornea video to obtain a plurality of image blocks.

[0098] The linear embedding module is used to adjust the channel dimension of the image block to obtain the target feature vector.

[0099] The fusion module includes several cross-connected attention units and patch merging units, the last unit of which is an attention unit; the attention unit introduces 3D relative position encoding in the self-attention calculation process; the attention unit includes a window multi-head self-attention layer based on a 3D sliding window and a shift window multi-head self-attention layer based on a 3D sliding window; the window multi-head self-attention layer based on a 3D sliding window is used to extract local spatiotemporal features within a local window; the shift window multi-head self-attention layer based on a 3D sliding window is used to introduce global dependencies when the window slides and extract global spatiotemporal information; the attention unit outputs spatiotemporal weighted features; the patch merging unit is used to fill the spatiotemporal weighted features and perform channel dimension splicing to obtain a fused feature vector; the last attention unit outputs the final spatiotemporal features.

[0100] The output module is used to obtain a classification result corresponding to the target cornea video according to the final spatiotemporal features.

[0101] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store corneal video label processing data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for extracting corneal spatiotemporal biomechanical features based on an attention mechanism is implemented.

[0102] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0103] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0104] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0105] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0106] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0107] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0108] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0109] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for extracting spatiotemporal biomechanical features of the cornea based on an attention mechanism, characterized in that: The corneal spatiotemporal biomechanical feature extraction method based on the attention mechanism includes: Acquire a target cornea video; the target cornea video includes a plurality of frames of cornea images; Inputting the target cornea video into a trained cornea video classification model to obtain a classification result corresponding to the target cornea video; the trained cornea video classification model includes a block module, a linear embedding module, a fusion module and an output module; The blocking module is used to block the cornea image in the target cornea video to obtain a plurality of image blocks; The linear embedding module is used to: adjust the channel dimension of the image block to obtain the target feature vector; The fusion module includes several attention units and patch merging units with cross-connections, and the last unit is an attention unit; the attention unit introduces 3D relative position encoding in the self-attention calculation process; the attention unit includes a window multi-head self-attention layer based on a 3D sliding window and a shift window multi-head self-attention layer based on a 3D sliding window; the window multi-head self-attention layer based on a 3D sliding window is used to extract local spatiotemporal features within a local window; the shift window multi-head self-attention layer based on a 3D sliding window is used to introduce global dependencies when the window slides and extract global spatiotemporal information; the attention unit outputs spatiotemporal weighted features; the patch merging unit is used to perform filling processing and channel dimension splicing on the spatiotemporal weighted features to obtain a fused feature vector; the last attention unit outputs the final spatiotemporal features; The output module is used to obtain a classification result corresponding to the target cornea video according to the final spatiotemporal features.

2. The method for extracting spatiotemporal biomechanical features of the cornea based on the attention mechanism according to claim 1, characterized in that: The linear embedding module includes a first filling unit, a first splicing unit and a first linear unit connected in sequence; the first filling unit is used to perform 3D filling processing on each image block respectively to obtain a feature vector corresponding to each image block; the first splicing unit is used to splice the feature vectors corresponding to all image blocks to obtain a spliced feature vector; the first linear unit is used to perform a linear transformation on the spliced feature vector to obtain a target feature vector.

3. The method for extracting spatiotemporal biomechanical features of the cornea based on the attention mechanism according to claim 1, characterized in that: The calculation formula of the self-attention calculation process of the attention unit is as follows: Among them, Attention(Q,K,V) is the attention weight, Q is the query vector, K is the key vector, V is the value vector, softmax is the normalization function, d k is the scaling factor.

4. The method for extracting spatiotemporal biomechanical features of the cornea based on the attention mechanism according to claim 1, characterized in that: The attention unit includes two self-attention sub-units; each self-attention sub-unit includes a first unit, a second unit, a third unit and a fourth unit; the first unit includes a first normalization layer, a window multi-head self-attention layer based on a 3D sliding window, a first regularization layer and a first addition layer, and the first addition layer is used to add the output of the first regularization layer and the input of the first unit; the second unit includes a second normalization layer, a first MLP layer, a second regularization layer and a second addition layer, and the second addition layer is used to add the output of the second regularization layer and the input of the second unit; the third unit includes a third normalization layer, a shifted window multi-head self-attention layer based on a 3D sliding window, a third regularization layer and a third addition layer, and the third addition layer is used to add the output of the third regularization layer and the input of the third unit; the fourth unit includes a fourth normalization layer, a second MLP layer, a fourth regularization layer and a fourth addition layer, and the fourth addition layer is used to add the output of the fourth regularization layer and the input of the fourth unit.

5. The method for extracting spatiotemporal biomechanical features of the cornea based on the attention mechanism according to claim 4, characterized in that: The output of the self-attention subunit is expressed as follows: in, is the output of the first unit, 3DW-MSA represents the window multi-head self-attention layer based on 3D sliding window, LN represents layer normalization, z l-1 represents the input of the first unit, z l is the output of the second unit, FFN represents the FFN network in the multilayer perceptron, is the output of the third unit, 3DSW-MSA represents the shift window multi-head self-attention layer based on 3D sliding window, z l +1 is the output of the fourth unit.

6. The method for extracting spatiotemporal biomechanical features of cornea based on attention mechanism according to claim 1, characterized in that: Acquire the target cornea video, including: Obtain original corneal video; Use the unsupervised shared model to extract corneal contour points and write them into a txt file; Parse the txt file to construct the corneal area mask; The video is read frame by frame and the mask is applied to obtain the target cornea video.

7. The method for extracting spatiotemporal biomechanical features of cornea based on attention mechanism according to claim 1, characterized in that: Before inputting the target cornea video into the trained cornea video classification model to obtain the classification result corresponding to the target cornea video, the cornea spatiotemporal biomechanical feature extraction method based on the attention mechanism further includes: Acquire a sample set; the sample set includes a plurality of sample cornea videos and a sample classification result corresponding to each sample cornea video; The sample set is used to train a cornea video classification model to obtain a trained cornea video classification model.

8. A device for extracting spatiotemporal biomechanical features of cornea based on attention mechanism, characterized in that: The corneal spatiotemporal biomechanical feature extraction device based on the attention mechanism includes: A target cornea video acquisition module is used to acquire a target cornea video; the target cornea video includes a plurality of frames of cornea images; A corneal video classification module is used to input the target corneal video into a trained corneal video classification model to obtain a classification result corresponding to the target corneal video; the trained corneal video classification model includes a block module, a linear embedding module, a fusion module and an output module; The blocking module is used to block the cornea image in the target cornea video to obtain a plurality of image blocks; The linear embedding module is used to: adjust the channel dimension of the image block to obtain the target feature vector; The fusion module includes several attention units and patch merging units with cross-connections, and the last unit is an attention unit; the attention unit introduces 3D relative position encoding in the self-attention calculation process; the attention unit includes a window multi-head self-attention layer based on a 3D sliding window and a shift window multi-head self-attention layer based on a 3D sliding window; the window multi-head self-attention layer based on a 3D sliding window is used to extract local spatiotemporal features within a local window; the shift window multi-head self-attention layer based on a 3D sliding window is used to introduce global dependencies when the window slides and extract global spatiotemporal information; the attention unit outputs spatiotemporal weighted features; the patch merging unit is used to perform filling processing and channel dimension splicing on the spatiotemporal weighted features to obtain a fused feature vector; the last attention unit outputs the final spatiotemporal features; The output module is used to obtain a classification result corresponding to the target cornea video according to the final spatiotemporal features.

9. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the corneal spatiotemporal biomechanical feature extraction method based on the attention mechanism according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for extracting spatiotemporal biomechanical features of the cornea based on the attention mechanism described in any one of claims 1 to 7 is implemented.

Images