A Classification Method and System for Nystagmus Videos with Spatiotemporal Feature Fusion

Abstract

The present invention discloses a nystagmus video classification method and system based on spatio-temporal feature fusion, belonging to the technical field of video classification in medical image processing. It includes encoding and mapping the nystagmus video, obtaining spatial dimension features and temporal dimension features through a Spatial Transformer module and a Temporal Transformer module; passing them through a spatio-temporal feature fusion module to obtain fused spatio-temporal features; then inputting them into the Temporal-Spatial Transformer to continue feature extraction to obtain a feature map; and finally obtaining a classification result through a fully connected layer and a classification function. The present invention adopts a novel spatio-temporal feature fusion Transformer framework, without the need for manual feature extraction, and can well extract features of nystagmus videos, improving the classification accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of video classification in medical image processing, and particularly relates to a method and system for classifying nystagmus videos by fusing spatio-temporal features. Background Art

[0002] Benign paroxysmal positional vertigo (BPPV) has a high incidence rate and shows an increasing trend year by year. Untreated BPPV will seriously affect the daily life of patients. The characteristic analysis of nystagmus in BPPV patients is crucial for the diagnosis of BPPV. In clinical diagnosis and treatment, nystagmus information is generally obtained through positional tests. Video nystagmography (VNG) can significantly improve the detection rate of BPPV nystagmus. During diagnosis, medical staff need to analyze the characteristics of nystagmus direction, frequency, intensity, duration, and strength changes in VNG videos one by one, which requires high requirements for doctors and is very susceptible to the influence of doctors' experience, subjective judgment, and fatigue level. With the rise of computer technology and deep learning, compared with the manual diagnosis method, computer-aided diagnosis is objective, efficient, and has high clinical application value.

[0003] Existing nystagmus video classification methods include nystagmus feature extraction for auxiliary diagnosis and machine learning-based nystagmus video classification methods. Researchers in this field have done a lot of research work based on the above methods. Slama et al. in "Features extraction for medical characterization of nystagmus" extracted the position and radius of the pupil based on a robust segmentation method for pupil fitting, used the circular Hough transform to extract the ellipse center to characterize the movement of the pupil, and then calculated the frequency changes of nystagmus in the fast and slow phases as additional features used in video analysis. Finally, a fisher linear discriminator was used to analyze and select significant features. Charoenpong et al. in "Pupil extraction system for Nystagmus diagnosis by using K-mean clustering and Mahalanobis distance technique" chose to perform technical noise elimination. Based on the darkness of the pupil, they used the K-means clustering algorithm to segment the black pixels, and after eliminating the noise data through the Mahalanobis distance technique, they obtained the final extraction result. Slama et al. in "Automated approach for vestibular disorder diagnosis based on clinical VNG feature selection and fuzzy clustering" obtained the movement trajectory of the eyeball through pupil localization, and calculated the time and frequency changes of nystagmus in two phases (fast and slow). Finally, the principal component analysis method (PCA) was used to select relevant components from all the calculated features, and then the FCM clustering algorithm was used to classify vestibular diseases. Sahli et al. in "Machine learning based approach for vestibular disorder diagnostic in video nystagmography" used the Hough transform to estimate the pupil movement vector to obtain the movement curve of the pupil. Then, the time and frequency features were calculated based on the rotation angle change of the pupil movement, and the support vector machine technique was used to classify it into normal and case, achieving a classification accuracy of over 90%.In "A new neural network method for peripheral vestibular disorder recognition using VNG parameter optimisation", Sahli et al. extracted features through Fisher linear discriminant analysis and sent them into a neural network, which consisted of four input layers followed by a hidden layer and an output layer. The two output nodes of the output layer could distinguish between patients with vestibular disorders and normal testers. In "Developing a Diagnostic Decision Support System for Benign Paroxysmal Positional Vertigo Using a Deep-Learning Model", Lim et al. obtained the movement vectors of the eyeballs in the horizontal, vertical, and axial directions for video nystagmograms using edge detection, binarization, Hough transform, and iris template matching, formed a 3×10 feature matrix, and input it into a convolutional neural network to obtain the classification results. This method could distinguish typical nystagmus in the horizontal, vertical, and axial directions, but was more sensitive to the judgment error of axial rotation.

[0004] As described above, the existing methods extract features using manual experience, which have strong subjectivity and limitations, can only provide low-precision eye movement information, and inputting manual features into a neural network can only distinguish between patients and the normal population or classify simple nystagmus patterns. Therefore, it is of great practical significance to develop an eye movement video method with strong feature extraction ability that can classify various compound nystagmus. Summary of the Invention

[0005] In view of the above problems, considering the relevant characteristics of sequence data in video nystagmograms, the present invention proposes a deep learning method based on the Transformer framework. The Transformer framework was initially proposed for feature extraction of data with sequence characteristics. The picture sequence with temporal information obtained by decomposing the eye movement video frames is directly used as the input of the algorithm, allowing the network to extract features by itself without additional feature engineering, which not only improves efficiency but also solves the problems of subjectivity and limitations in manual feature extraction.

[0006] To achieve the above object, the present invention proposes an eye movement video classification method for spatio-temporal feature fusion, specifically including:

[0007] (1) Obtain the nystagmus video information, and construct a nystagmus video classification model including a Spatial Transformer module, a Temporal Transformer module, a spatio-temporal feature fusion module, and a Temporal-Spatial Transformer module;

[0008] (2) Perform encoding mapping on the video information respectively, and obtain spatial dimension features and temporal dimension features through the Spatial Transformer module and the Temporal Transformer module;

[0009] (3) Pass the spatial dimension features and the temporal dimension through the spatio-temporal feature fusion module to obtain fused spatio-temporal features;

[0010] The spatio-temporal feature fusion module includes: a Piont-Wise convolution module, a GeLu activation function module, and a Sigmoid activation function module; the specific process includes:

[0011] (3.1) Input the spatial dimension features and the temporal dimension features into the Piont-Wise convolution module respectively for a linear transformation;

[0012] (3.2) Pass the linear transformation results through the GeLu activation function module respectively to perform non-linear activation on them;

[0013] (3.3) The non-linear activation results are respectively passed through the Piont-Wise convolution module for another linear transformation;

[0014] (3.4) After adding the two linear transformation results, obtain an attention weight map through the Sigmoid activation function;

[0015] Introduce a relative position representation encoding that changes with network training in the calculation stage of the attention weight map, and the calculation formula is as follows:

[0016]

[0017] Where: PE

[0014] ,

[0013] , , , ,

[0016] ,

[0015] ,

[0019] , ,

[0018] , ,

[0017] , K , , , represents the result after the trainable relative position representation encoding passes through the relative position index;

[0018] (3.5) Multiply the attention weight map with the spatial dimension features and the temporal dimension features respectively to obtain the recalibrated features, and finally add them to obtain the fused spatio-temporal features;

[0019] (4) Input the fused spatio-temporal features into the Temporal-Spatial Transformer to continue feature extraction and obtain a feature map;

[0020] (5) Obtain a classification result by passing the feature map through a fully connected layer and a classification function;

[0021] (6) Based on this classification result and the true label, through the constraint of a loss function, iteratively train the nystagmus video classification model until the loss function converges or reaches a preset number of iterations, and save the current nystagmus video classification model as the final recognition model;

[0022] (7) Input the nystagmus video information to be classified into the final recognition model to obtain the category of the nystagmus video information to be classified.

[0023] Further, the step (2) includes:

[0024] (1.1) Perform equal-amplitude sampling on the video information in the time dimension, and after sampling, input it into the SpatialTransformer to obtain the spatial dimension features;

[0025] (1.2) Perform equal-amplitude sampling on the video information in the spatial dimension, and after sampling, input it into the TemporalTransformer to obtain the time dimension features.

[0026] Further, in the step (6), Accuracy, Precision, and Recall are used as evaluation indicators to obtain the classification result;

[0027]

[0028] Among them, TP, FP, TN, and FN represent the numbers of true positives, false positives, true negatives, and false negatives respectively.

[0029] Further, the loss function used in the step (6) is the adaptive cross-entropy loss, and its formula is as follows:

[0030]

[0031] Among them: represents the weight coefficient of the number of samples in each category accounting for the total number of samples, which is determined by the following formula:

[0032]

[0033] Among them, N i represents the number of samples of the i-th category in the dataset, and the total number of sample categories is K.

[0034] The present invention also provides a nystagmus video classification system with spatio-temporal feature fusion, including:

[0035] An acquisition module: used to acquire nystagmus video information and construct a nystagmus video classification model including a Spatial Transformer module, a Temporal Transformer module, a spatio-temporal feature fusion module, and a Temporal-Spatial Transformer module;

[0036] An encoding module: used to perform encoding mapping on the video information respectively, and obtain spatial dimension features and temporal dimension features through the Spatial Transformer module and the Temporal Transformer module;

[0037] A fusion module: used to pass the spatial dimension features and the temporal dimension features through the spatio-temporal feature fusion module to obtain fused spatio-temporal features;

[0038] The spatio-temporal feature fusion module includes: a Piont-Wise convolution module, a GeLu activation function module, and a Sigmoid activation function module; the specific process includes:

[0039] (1) Input the spatial dimension features and the temporal dimension features into the Piont-Wise convolution module respectively for a linear transformation;

[0040] (2) Pass the linear transformation results through the GeLu activation function module respectively to perform non-linear activation on them;

[0041] (3) The non-linear activation results are respectively passed through the Piont-Wise convolution module for another linear transformation;

[0042] (4) After adding the two linear transformation results, an attention weight map is obtained through the Sigmoid activation function;

[0043] A relative position representation encoding that changes with network training is introduced in the calculation stage of the attention weight map, and the calculation formula is as follows:

[0044]

[0045] where: PE K represents the result after the trainable relative position representation encoding passes through the relative position index;

[0046] (5) Multiply the attention weight map with the spatial dimension features and the temporal dimension features respectively to obtain the recalibrated features, and finally add them to obtain the fused spatio-temporal features;

[0047] Extraction module: used to input the fused spatio-temporal features into the Temporal-Spatial Transformer for further feature extraction to obtain a feature map;

[0048] Recognition module: used to obtain a classification result by passing the feature map through a fully connected layer and a classification function;

[0049] Judgment module: used to iteratively train the nystagmus video classification model based on the classification result and the ground truth label through the constraint of a loss function until the loss function converges or reaches a preset number of iterations, and save the current nystagmus video classification model as the final recognition model;

[0050] Verification module: used to input the information of the nystagmus video to be classified into the final recognition model to obtain the category of the information of the nystagmus video to be classified.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] The present invention adopts a novel spatio-temporal feature fusion Transformer framework, which does not require manual feature extraction and can well extract features from nystagmus videos, improving the classification accuracy. Comprehensive experiments on a clinical nystagmus dataset show that the present invention has good detection effects on mainstream evaluation indicators. Description of the Drawings

[0053] Figure 1 is a schematic flowchart of the method for classifying nystagmus videos with spatio-temporal feature fusion according to an embodiment of the present invention.

[0054] Figure 2 is a network structure diagram of the method for classifying nystagmus videos with spatio-temporal feature fusion according to an embodiment of the present invention.

[0055] Figure 3 Schematic diagram of the spatio-temporal feature fusion module according to an embodiment of the present invention.

[0056] Figure 4 is the equal-amplitude sampling in the time dimension according to an embodiment of the present invention.

[0057] Figure 5 is the uniform sampling in the space dimension according to an embodiment of the present invention.

[0058] Figure 6 is the relationship between Loss (left) and Accuracy (right) of different loss functions and the number of iterations according to an embodiment of the present invention. Detailed Embodiments

[0059] The present invention will be described in detail below with reference to the drawings and specific embodiments.

[0060] This embodiment provides a method for classifying nystagmus videos by fusing spatio-temporal features, as Figure 1 , Figure 2 shown. The specific implementation details are as follows:

[0061] S101. Obtain nystagmus video information, and construct a nystagmus video classification model including a Spatial Transformer module, a Temporal Transformer module, a spatio-temporal feature fusion module, and a Temporal-Spatial Transformer module;

[0062] In this embodiment, the nystagmus video information is obtained in advance, and the acquisition method is not limited, and a classification model is constructed.

[0063] S102. Perform encoding mapping on the video information respectively, and obtain spatial dimension features and temporal dimension features through the Spatial Transformer module and the Temporal Transformer module;

[0064] In this embodiment, two different video data encoding and sampling methods are used, and the data size is T×C×H×W.

[0065] The first way to encode and embed the input video is to perform equal-amplitude sampling in the temporal dimension on the continuous frame sequence of the video. As Figure 4 shown, this method encodes the Patches in each frame, and combines the Patch encodings generated by all frames into a sequence and then inputs it into the network. For each Patch, its size is C×h×w, and the number distribution of Patches in the length and width directions is For such continuous T-frame images, T×n h ×n w image blocks can be generated, and then these image blocks are encoded and input into the Transformer framework.

[0066] The second way to encode the input video is to perform spatial dimension slicing on the entire continuous frame sequence while keeping the time information intact. As Figure 5 shown, the size of each Patch generated by uniform sampling in the spatial dimension is T×C×h×w, and the number distribution of Patches in the length and width directions is In this case, for continuous T-frame images, n h ×n w image blocks can be generated.

[0067] For the two-dimensional image obtained by equal-amplitude sampling in the time dimension, input it into the Spatial Transformer, and perform feature extraction based on its sufficient spatial detail features. For the Patch obtained by uniform sampling in the spatial dimension, input it into the Temporal Transformer. Since it retains the complete time information, the feature vector obtained after feature extraction has rich inter-frame timing information.

[0068] After the two different sampling methods pass through the two Transformers respectively, their outputs are the spatial dimension features and the time dimension features respectively.

[0069] S103. Pass the spatial dimension features and the time dimension features through the spatio-temporal feature fusion module to obtain the fused spatio-temporal features;

[0070] As Figure 3 shown, the spatio-temporal feature fusion module first performs a linear transformation on the two input features. In order to reduce the number of parameters and computational complexity of the module, Point-Wise convolution is used instead of the fully connected layer for the transformation. Then, in order to increase the non-linear expression ability of the module and better obtain the attention weights for the next step, the GeLu activation function is used to perform non-linear activation on it. After the GeLu non-linear activation, a linear transformation is performed respectively. After the outputs are added and then passed through the Sigmoid activation function, the attention weight maps for the two features are obtained, which are multiplied by the input X and Y respectively to obtain the re-calibrated features. Finally, after adding them, the output of the ATFF module is obtained.

[0071] S104. Input the fused spatio-temporal features into the Temporal-Spatial Transformer to continue feature extraction and obtain a feature map;

[0072] The fused output features are finally modeled by the Temporal-Spatial Transformer to obtain a feature map.

[0073] S105. Pass the feature map through the fully connected layer and the classification function to obtain the classification result;

[0074] Pass the feature map through the classification layer again to obtain the class output.

[0075] S106. Based on this classification result and the true label, through the constraint of the loss function, iteratively train the nystagmus video classification model until the loss function converges or reaches the preset number of iterations, and save the current nystagmus video classification model as the final recognition model;

[0076] This embodiment uses the WCE Loss function (Weighted CrossEntropyLoss) provided by the Pytorch framework, and a modulation coefficient is added on the basis of the cross-entropy loss function for optimizing the training of the model. The formula for the adaptive cross-entropy loss used is as follows:

[0077]

[0078] represents the weight coefficient of the number of samples in each category accounting for the total number of samples in the population. The coefficients 0.25 and the bias 0.375 are to ensure that the weights assigned by this coefficient can reflect the quantitative relationship of the samples and are not too extreme. It is determined by the following formula:

[0079]

[0080] Among them, N i represents the number of samples of the i-th category in the dataset, and the total number of sample categories is K.

[0081] Figure 6 shows the relationship between Loss (left) and Accuracy (right) of different loss functions in this embodiment and the number of iterations.

[0082] The present invention adopts a multi-class evaluation strategy of One vs Rest, that is, if the model classifies a certain data sample into a certain category, then the other categories are incorrect for this sample. Accuracy, Precision, and Recall are used as evaluation indicators to comprehensively evaluate the model. The calculation formulas for each indicator are as follows:

[0083]

[0084] Among them, TP, FP, TN, and FN represent the numbers of true positives, false positives, true negatives, and false negatives respectively.

[0085] S107. Input the nystagmus video information to be classified into the final recognition model to obtain the category of the nystagmus video information to be classified.

[0086] The model of the present invention is implemented based on PyTorch. The machine's graphics card is NVIDIA GEForce RTX3090 (24G), the memory is 64.0GB, and the CPU is Intel(R) Core(TM) i7-6950X CPU @ 3.00GHz. The number of iterations is 200. The learning rate for the first 20 iterations is 10e-4, the learning rate for the 21st - 100th iterations is 10e-5, and the learning rate for the 101st - 200th iterations is 10e-6. The weight decay rate is 1e-5, and the batch size is 16. There is no pre-training or other pre-tasks. For each iteration of training, a model test is performed, and the model parameters with the best results are saved. Finally, it is verified on the validation set.

[0087] The present invention conducts sufficient experiments on the clinical nystagmus dataset to verify the method proposed in the present invention. For a fair comparison with other methods, this method is used to conduct comparative experiments with the mainstream time series models RNN, LSTM, Bi-LSTM, and Vit (using the sampling method of the present invention). The results are shown in Table 1:

[0088] Table 1 Performance of different methods on the nystagmus dataset

[0089]

[0090] The ablation experiment also verifies the effectiveness of each module proposed in the present invention. The results are shown in Table 2:

[0091] Table 2 Influence of different modules on the model

[0092]

[0093]

[0094] Figure 5 It indicates that the improved loss function of the present invention also correspondingly produces positive effects.

[0095] In summary, the method proposed in the present invention has certain superiority compared with other methods, with accurate classification results, high efficiency, and great application prospects.

[0096] Another embodiment of the present invention also proposes a nystagmus video classification system for spatio-temporal feature fusion, including:

[0097] An acquisition module: used to acquire nystagmus video information and construct a nystagmus video classification model including a Spatial Transformer module, a Temporal Transformer module, a spatio-temporal feature fusion module, and a Temporal-Spatial Transformer module;

[0098] Encoding module: used to perform encoding mapping on the video information respectively, and obtain spatial-dimensional features and temporal-dimensional features through the SpatialTransformer module and the Temporal Transformer module;

[0099] Fusion module: used to obtain fused spatio-temporal features by passing the spatial-dimensional features and temporal-dimensional features through the spatio-temporal feature fusion module;

[0100] Extraction module: used to input the fused spatio-temporal features into the Temporal-Spatial Transformer to continue feature extraction and obtain a feature map;

[0101] Recognition module: used to obtain a classification result by passing the feature map through a fully connected layer and a classification function;

[0102] Judgment module: used to iteratively train the nystagmus video classification model based on the classification result and the true label through the constraint of a loss function until the loss function converges or reaches a preset number of iterations, and save the current nystagmus video classification model as the final recognition model;

[0103] Verification module: used to input the nystagmus video information to be classified into the final recognition model to obtain the category of the nystagmus video information to be classified.

[0104] The preferred specific embodiments of the present invention have been described in detail above. It should be understood that those of ordinary skill in the art can make many modifications and variations according to the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by those skilled in the art in the technical field of the present invention based on the concept of the present invention through logical analysis, reasoning or limited experiments on the basis of the prior art should be within the protection scope determined by the claims.

Claims

1. A method for classifying nystagmus videos by fusing spatio-temporal features, characterized in that, The classification method specifically includes the following steps: (1) Obtain the nystagmus video information, and construct a nystagmus video classification model including a Spatial Transformer module, a Temporal Transformer module, a spatio-temporal feature fusion module, and a Temporal-Spatial Transformer module; (2) Perform encoding mapping on the video information respectively, and obtain the spatial dimension features and the temporal dimension features through the Spatial Transformer module and the Temporal Transformer module; (3) Pass the spatial dimension features and the temporal dimension features through the spatio-temporal feature fusion module to obtain the fused spatio-temporal features; The spatio-temporal feature fusion module includes: a Piont-Wise convolution module, a GeLu activation function module, and a Sigmoid activation function module; the specific process includes: (3.1) Input the spatial dimension features and the temporal dimension features into the Piont-Wise convolution module respectively for a linear transformation; (3.2) Pass the linear transformation results through the GeLu activation function module respectively to perform non-linear activation on them; (3.3) The non-linear activation results are respectively passed through the Piont-Wise convolution module for another linear transformation; (3.4) After adding the two linear transformation results, obtain the attention weight map through the Sigmoid activation function; Introduce a relative position representation encoding that changes with network training in the calculation stage of the attention weight map, and the calculation formula is as follows: Wherein: PE K represents the result of the trainable relative position representation encoding after relative position indexing; (3.5) Multiply the attention weight map by the spatial dimension features and the temporal dimension features respectively to obtain the recalibrated features, and finally add them to obtain the fused spatio-temporal features; (4) Input the fused spatio-temporal features into the Temporal-Spatial Transformer to continue feature extraction and obtain the feature map; (5) Obtain the classification result by passing the feature map through the fully connected layer and the classification function; (6) Based on the classification result and the true label, through the constraint of the loss function, iteratively train the nystagmus video classification model until the loss function converges or reaches the preset number of iterations, and save the current nystagmus video classification model as the final recognition model; (7) Input the nystagmus video information to be classified into the final recognition model to obtain the category of the nystagmus video information to be classified.

2. The method for classifying nystagmus videos by fusing spatio-temporal features according to claim 1, wherein, The step (2) includes: (1.1) Perform equal-amplitude sampling on the video information in the temporal dimension, and input the sampled data into the Spatial Transformer to obtain the spatial dimension features; (1.2) Perform equal-amplitude sampling on the video information in the spatial dimension, and input the sampled data into the Temporal Transformer to obtain the temporal dimension features.

3. A nystagmus video classification method with spatio-temporal feature fusion according to claim 1, characterized in that In the step (6), Accuracy, Precision, and Recall are used as evaluation indicators to obtain the classification result; Among them, TP, FP, TN, and FN respectively represent the numbers of true positives, false positives, true negatives, and false negatives.

4. A nystagmus video classification method based on spatio-temporal feature fusion according to claim 1, characterized in that The loss function used in step (6) is the adaptive cross-entropy loss, and its formula is as follows: Wherein: represents the weight coefficient of the number of samples in each category accounting for the total number of samples, which is determined by the following formula: Among them, N i represents the number of samples of the i-th category in the dataset, and the total number of sample categories is K.

5. A nystagmus video classification system with spatio-temporal feature fusion, characterized in that, Including: Acquisition module: used to acquire the nystagmus video information and construct a nystagmus video classification model including a Spatial Transformer module, a Temporal Transformer module, a spatio-temporal feature fusion module, and a Temporal-Spatial Transformer module; Encoding module: used to perform encoding mapping on the video information respectively, and obtain spatial dimension features and temporal dimension features through the Spatial Transformer module and the Temporal Transformer module; Fusion module: used to pass the spatial dimension features and the temporal dimension features through the spatio-temporal feature fusion module to obtain fused spatio-temporal features; The spatio-temporal feature fusion module includes: a Piont-Wise convolution module, a GeLu activation function module, and a Sigmoid activation function module; the specific process includes: (1) Input the spatial dimension features and the temporal dimension features into the Piont-Wise convolution module respectively for a linear transformation; (2) Pass the linear transformation results through the GeLu activation function module respectively to perform non-linear activation on them; (3) The non-linear activation results are respectively passed through the Piont-Wise convolution module for another linear transformation; (4) After adding the two linear transformation results, an attention weight map is obtained through the Sigmoid activation function; A relative position representation encoding that changes with network training is introduced in the calculation stage of the attention weight map, and the calculation formula is as follows: Wherein: PE K represents the result of the trainable relative position representation encoding after relative position indexing; (5) Multiply the attention weight map by the spatial dimension features and the temporal dimension features respectively to obtain the recalibrated features, and finally add them to obtain the fused spatio-temporal features; Extraction module: used to input the fused spatio-temporal features into the Temporal-Spatial Transformer to continue feature extraction and obtain a feature map; Recognition module: used to obtain a classification result by passing the feature map through a fully connected layer and a classification function; Judgment module: used to iteratively train the nystagmus video classification model based on the classification result and the true label through the constraint of the loss function until the loss function converges or reaches the preset number of iterations, and save the current nystagmus video classification model as the final recognition model; Verification module: used to input the nystagmus video information to be classified into the final recognition model to obtain the category of the nystagmus video information to be classified.

Images