A Multi-View Oscillation Classification Method and System Based on Time Series Prediction

By employing a multi-view nystagmus classification method based on time-series prediction, and utilizing neural networks with neural frequent differential equations and an improved multi-head attention mechanism, the method solves the problems of feature extraction difficulties and scene dependence in existing nystagmus classification techniques, achieving efficient and accurate nystagmus classification and improving the accuracy and ease of diagnosis.

CN117765603BActive Publication Date: 2026-11-17SHANGHAI UNIV OF ENG SCI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310436982.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-04-21
Publication Date
2026-11-17
Estimated Expiration
2043-04-21

AI Technical Summary

Technical Problem

Existing nystagmus classification methods based on machine learning and deep learning are insufficient in feature extraction, making it difficult to accurately identify multiple types of complex nystagmus. Furthermore, they rely on insufficient prior knowledge of the scene, resulting in large diagnostic errors.

Method used

A multi-view nystagmus classification method based on time series prediction is adopted. The pupil center position is extracted by combining neural network algorithms with neural frequent differential equations and improved multi-head attention mechanism, Hough circle transform and contour detection algorithm, and transformed into Gram feature map. The cosine similarity function is used for feature extraction and classification.

Benefits of technology

It achieves accurate classification of multiple types of complex nystagmus, reduces computational load and overhead, improves the robustness and generalization ability of the model, and simplifies the clinical diagnosis process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117765603B_ABST
    Figure CN117765603B_ABST
Patent Text Reader

Abstract

This invention relates to a multi-view nystagmus classification method and system based on time series prediction. The method includes the following steps: S1, reading the input video nystagmus image; S2, preprocessing the video nystagmus image obtained in step S1 to obtain preprocessed video nystagmus image data; S3, using a neural network algorithm to extract features from the preprocessed video nystagmus image data in step S2; wherein the neural network algorithm combines an improved multi-head attention mechanism and neural frequent differential equations, and the improved multi-head attention mechanism is a linearized improvement; S4, classifying nystagmus based on the features extracted in step S3. Compared with the prior art, this invention has advantages such as strong feature extraction capability, high accuracy in nystagmus type classification, and high value in medical clinical applications.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to a nystagmus classification method, and more particularly to a multi-view nystagmus classification method and system based on time series prediction. Background Technology

[0002] Dizziness and vertigo are common and typical clinical symptoms of many diseases. In my country's large and aging society, the number of patients with dizziness and vertigo is increasing daily. In bedside examinations for dizziness and vertigo, nystagmus is the most sensitive and specific sign of vestibular disorders. Measuring nystagmus patterns through clinically acquired eye movement videos can provide valuable reference for the diagnosis of dizziness and vertigo. Clinically, video nystagmography (VNG) is the gold standard for diagnosing benign paroxysmal positional vertigo (BPPV). It requires medical professionals to examine the direction, frequency, intensity, duration, and intensity changes of nystagmus on VNG videos. This is a tedious process heavily influenced by the physician's experience and prone to errors. Existing machine learning-based or deep learning-based methods have many shortcomings, mainly focusing on weak feature extraction capabilities, difficulty in extracting axial rotation vector features of the eyeball, and limitations where some algorithms can only identify horizontal and vertical nystagmus or perform binary classification.

[0003] In linear algebra, a Gram matrix can be represented as a Hermitian matrix of vectors in an inner product space, its value determined by the inner product of its elements. Gram matrices have wide applications in mathematics, control system theory, machine learning, and other fields. To a certain extent, Gram matrices can measure the similarity between two vectors and are used in machine learning tasks such as image style transfer and sequential data classification (e.g., time series, sound events). Based on the properties of Gram matrices, they can measure the relationships between points in time-series data.

[0004] Recent research has confirmed the relationship between neural networks and differential equations. For example, networks with residual mechanisms can be represented as discrete ordinary differential equations. Neural ordinary differential equations are continuous models obtained by setting the discretization step to zero. Neural ordinary differential equations connect deep learning and dynamic systems, and have great potential in many fields such as constructing normalized flows and modeling continuous temporal information.

[0005] Currently, medical imaging is widely used in clinical practice, becoming an indispensable tool for disease diagnosis. Nystagmus video classification can be viewed as an action recognition problem within the field of video understanding. 3D Convolutional Neural Networks (3D-CNNs) are a classic algorithm for solving action recognition problems, capturing motion information in video streams more effectively through 3D convolutions in spatial and temporal dimensions. In fact, in many cases, action recognition relies on prior knowledge provided by the scene; however, VNG videos often have simple scenes, similar targets, and relatively small eye movements, making it difficult to provide effective prior knowledge for training.

[0006] Therefore, how to accurately extract the features of video nystagmus and accurately classify multiple types of complex nystagmus has become a technical problem that needs to be solved. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a multi-view nystagmus classification method and system based on time series prediction.

[0008] The objective of this invention can be achieved through the following technical solutions:

[0009] A multi-view nystagmus classification method based on time series prediction includes the following steps:

[0010] S1. Read the input video nystagmogram;

[0011] S2. Preprocess the video nystagmus obtained in step S1 to obtain preprocessed video nystagmus data.

[0012] S3. Feature extraction is performed on the preprocessed video nystagmus data in step S2 using a neural network algorithm; wherein the neural network algorithm combines an improved multi-head attention mechanism and neural frequent differential equations, and the improved multi-head attention mechanism is a linearized improvement.

[0013] S4. Classify nystagmus based on the features extracted in step S3.

[0014] Furthermore, the video nystagmogram is in standard audio-video interleaved format or Dynamic Image Experts Group format.

[0015] Furthermore, the preprocessing of the video nystagmus image obtained in step S1 includes the following steps:

[0016] Convert color video nystagmography to grayscale;

[0017] The pupil center position in the video nystagmogram is obtained using Hough circle transform and contour detection algorithms.

[0018] Record pupil coordinates in chronological order and convert the pupil coordinates into time series data;

[0019] The time series data is normalized and converted into Gram feature maps.

[0020] Furthermore, the neural network algorithm combined with the neural ordinary differential equation specifically involves treating the neural network as a stochastic continuous system, using a neural ordinary differential equation solver to dynamically model the hidden state of the neural network under the condition of continuous time, and using the ReLU function as the activation function.

[0021] Furthermore, the Softmax function in the conventional multi-head self-attention mechanism is replaced with a decomposable kernel function, and the cosine similarity function is selected as the kernel function to complete the low-rank linearization improvement of the multi-head attention mechanism.

[0022] A multi-view nystagmus classification system based on time series prediction includes a video database construction subsystem for reading input video nystagmus maps and preprocessing them;

[0023] The nystagmus prediction subsystem is used to extract features from preprocessed video nystagmus data through a neural network algorithm that incorporates an improved multi-head attention mechanism and neural frequent differential equations, thereby classifying nystagmus.

[0024] The video database construction subsystem and the nystagmus prediction subsystem are connected.

[0025] Furthermore, the video database construction subsystem includes:

[0026] The data input unit is used to input video nystagmus images;

[0027] The data reading unit is used to read the input video nystagmus image;

[0028] The data preprocessing unit is used to preprocess the input video nystagmus.

[0029] The data input unit, the first data reading unit, and the data preprocessing unit are connected in sequence.

[0030] Furthermore, the nystagmus prediction subsystem includes:

[0031] The neural network unit is used to extract features from preprocessed video nystagmography data using a neural network algorithm that improves the multi-head attention mechanism and neural frequent differential equations.

[0032] An attention mechanism unit is used to further extract features from video nystagmography data by improving the multi-head attention mechanism;

[0033] A prediction unit is used to classify nystagmus based on the extracted features;

[0034] The neural network unit, the improved multi-head attention mechanism unit, and the prediction unit are connected in sequence; the data preprocessing unit is connected to the neural network unit.

[0035] Furthermore, the data preprocessing unit performs data preprocessing on the input video nystagmus map, including the following steps:

[0036] Convert color video nystagmography to grayscale;

[0037] The pupil center position in the video nystagmogram is obtained using Hough circle transform and contour detection algorithms.

[0038] Record pupil coordinates in chronological order and convert the pupil coordinates into time series data;

[0039] The time series data is normalized and converted into Gram feature maps.

[0040] Furthermore, in the neural network unit, the neural network algorithm combined with the neural ordinary differential equation is specifically as follows: the neural network is regarded as a stochastic continuous system, and the neural ordinary differential equation solver is used to perform dynamic modeling of the hidden state of the neural network under the condition of continuous time, with the activation function being the ReLU function;

[0041] The improved multi-head attention mechanism specifically involves replacing the Softmax function in the conventional multi-head self-attention mechanism with a decomposable kernel function, and selecting the cosine similarity function as the kernel function to achieve a low-rank linearization improvement to the multi-head attention mechanism.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] 1. This invention utilizes a neural network based on neural frequent differential equations (NDEs) to extract features, overcoming the subjectivity and limitations of manual feature extraction. The improved neural network uses a NDE solver to directly model the network's hidden states dynamically under continuous time conditions, and uses the adjoint method instead of the traditional backpropagation algorithm, achieving high memory efficiency. The neural network modeled using NDEs has reusable parameters, significantly reducing the total number of model parameters and increasing model robustness. Furthermore, the number of time steps in the NDEs can be dynamically adjusted according to the required accuracy and computational cost of the specific task. Finally, the network containing NDEs can serve as a generative model, acting as a decoder in time series generation models, exhibiting strong generalization ability.

[0044] 2. This invention improves the multi-head attention mechanism by replacing the normalized exponential function (softmax function) with a decomposable kernel function, which can be freely chosen based on different tasks. In this invention, a cosine similarity function is chosen, reducing the overall computational cost of feature extraction.

[0045] 3. This invention can accurately classify various types of complex nystagmus in clinical diagnosis. It is simple, efficient and quick to operate, and has high value in clinical applications. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0047] Figure 2 This is a schematic diagram of the system structure of the present invention;

[0048] Figure 3 This is a schematic diagram illustrating the principle of the present invention.

[0049] The labels in the diagram indicate:

[0050] 1-Video database construction subsystem, 2-Nystagmus prediction subsystem, 3-Data input unit, 4-First data reading unit, 5-Data preprocessing unit, 6-Neural network unit, 7-Multi-head attention mechanism unit, 8-Prediction unit. Detailed Implementation

[0051] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0052] To address the shortcomings of existing technologies, this invention transforms VNG video data into time-series data, providing a new solution for VNG video classification tasks.

[0053] 3D convolutional networks can extract motion information between consecutive frames in nystagmus videos, thereby improving state prediction and achieving excellent performance in many tasks such as video understanding. For nystagmus video (VNG) classification, it is essentially a video understanding task, with key elements being the spatiotemporal nature, features, and motion patterns of the video. In fact, in many cases, action recognition relies on prior knowledge provided by the scene; however, VNG videos have simple scenes, similar targets, and small eye movement amplitudes, making it difficult to provide effective prior knowledge for training. Therefore, this invention extracts pupil position, transforms video data into temporal data, and then transforms the temporal data into Gram feature maps, significantly reducing the complexity of the problem. A neural network with frequent differential equations is then used for classification, while an improved multi-head attention mechanism is introduced. Compared to conventional multi-head self-attention mechanisms, the improved multi-head attention mechanism significantly reduces computational overhead by replacing the normalized exponential function (softmax function) with a decomposable kernel function.

[0054] like Figure 1 As shown, this invention provides a multi-view nystagmus classification method and system based on time series prediction, comprising the following steps:

[0055] Step S1: Read the input video nystagmus image;

[0056] Step S2: Preprocess the video nystagmus obtained in step S1 to obtain preprocessed video nystagmus data;

[0057] Step S3: Use a neural network algorithm to extract features from the preprocessed video nystagmus data in step S2; wherein the neural network algorithm combines an improved multi-head attention mechanism and neural frequent differential equations, and the improved multi-head attention mechanism is a linearized improvement.

[0058] Step S4: Classify nystagmus based on the features extracted in step S3.

[0059] The video nystagmogram should be in standard Audio Video Interleaved (AVI) or Moving Pictures Experts Group (MPEG-4) format.

[0060] In this embodiment, the preprocessing method is as follows: the video nystagmus image is converted into grayscale, the Hough circle transform algorithm and contour detection algorithm are used to obtain the pupil center position in the video nystagmus image, the pupil center position is saved in coordinate form in strict chronological order to form a time series, and then the time series is normalized and converted into a Gram feature map.

[0061] The improved neural network algorithm for ordinary differential equations (NDEs) specifically involves treating the neural network as a stochastic continuous system, using an NDE solver to dynamically model the hidden states under continuous time conditions and solve for the hidden values. The activation function is changed to the ReLU function. In this embodiment, the NDE algorithm is built using the Python programming language and the PyTorch library.

[0062] This invention incorporates an improved multi-head attention mechanism into a neural network with frequent differential equations, which helps to extract video features more effectively. The improved multi-head attention mechanism includes replacing the Softmax function in the conventional multi-head self-attention mechanism with a decomposable kernel function, which can be freely chosen according to different tasks. In this embodiment, a cosine similarity function is chosen as the kernel function, thereby achieving a low-rank linearization improvement to the multi-head attention mechanism and saving computational resources.

[0063] like Figure 2 As shown, a multi-view nystagmus classification method and system based on time series prediction includes a video database construction subsystem 1 and a nystagmus prediction subsystem 2 that are interconnected.

[0064] The video database construction subsystem 1 is used to read the input video nystagmus and preprocess it. The video database construction subsystem 1 includes a data input unit 3, a data reading unit 4, and a data preprocessing unit 5 connected in sequence. The data input unit 3 is used to input the video nystagmus. The data reading unit 4 is used to read the input video nystagmus. The data preprocessing unit 5 is used to preprocess the input video nystagmus.

[0065] The nystagmus prediction subsystem 2 is used to extract features from preprocessed video nystagmus data and predict the type of nystagmus by introducing a neural network algorithm with an improved multi-head attention mechanism and neural frequent differential equations.

[0066] The nystagmus prediction subsystem 2 includes a neural network unit 6, an attention mechanism unit 7, and a prediction unit 8 connected in sequence. The neural network unit 6 is used to extract features from the preprocessed video nystagmus data using a neural network algorithm based on an improved multi-head attention mechanism and neural frequent differential equations. The improved multi-head attention mechanism unit 7 is used to further extract features from the video nystagmus data using an improved multi-head attention mechanism. The prediction unit 8 is used to predict the type of nystagmus based on the extracted features.

[0067] The data preprocessing unit 5 and the neural network unit 6 are connected, linking the video database construction subsystem 1 and the nystagmus prediction subsystem 2 together.

[0068] like Figure 3 As shown, this invention first preprocesses the video nystagmus, then extracts features from the processed frame images by combining an improved multi-head attention mechanism and a neural network algorithm based on neural frequent differential equations, and predicts the type of video nystagmus, outputting the prediction results. This method can accurately classify multiple types of complex nystagmus, and has strong recognition capabilities for each type. It is simple to operate, efficient and fast, and has high value in clinical applications.

[0069] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A multi-view nystagmus classification method based on time series prediction, characterized in that, Includes the following steps: S1. Read the input video nystagmus image; S2. Preprocess the video nystagmus obtained in step S1 to obtain preprocessed video nystagmus data. S3. Use a neural network algorithm to extract features from the preprocessed video nystagmography data in step S2; The neural network algorithm described herein combines an improved multi-head attention mechanism with neural ordinary differential equations, and the improved multi-head attention mechanism is a linearized improvement. S4. Classify nystagmus based on the features extracted in step S3; The neural network algorithm combined with the neural ordinary differential equation specifically involves treating the neural network as a stochastic continuous system and using a neural ordinary differential equation solver to dynamically model the hidden state of the neural network under the condition of continuous time, with the activation function being the ReLU function. The Softmax function in the conventional multi-head self-attention mechanism is replaced with a decomposable kernel function, and the cosine similarity function is selected as the kernel function to complete the low-rank linearization improvement of the multi-head attention mechanism.

2. The multi-view nystagmus classification method based on time series prediction according to claim 1, characterized in that, The video nystagmogram is in standard audio-video interleaved format or Dynamic Image Experts Group format.

3. The multi-view nystagmus classification method based on time series prediction according to claim 1, characterized in that, The preprocessing of the video nystagmus image obtained in step S1 includes the following steps: Convert color video nystagmography to grayscale; The pupil center position in the video nystagmogram is obtained using Hough circle transform and contour detection algorithms. Record pupil coordinates in chronological order and convert the pupil coordinates into time series data; The time series data is normalized and converted into Gram feature maps.

4. A system for implementing multi-view nystagmus classification based on time series prediction as described in any one of claims 1-3, characterized in that, It includes a video database construction subsystem (1) for reading input video nystagmus maps and preprocessing them; The nystagmus prediction subsystem (2) is used to extract features from preprocessed video nystagmus data by introducing an improved multi-head attention mechanism and a neural network algorithm based on the regular differential equation of the neural network, thereby classifying nystagmus. The video database construction subsystem (1) and the nystagmus prediction subsystem (2) are connected.

5. The system according to claim 4, characterized in that, The video database construction subsystem (1) includes: Data input unit (3) is used to input video nystagmus; The data reading unit (4) is used to read the input video nystagmus image; The data preprocessing unit (5) is used to preprocess the input video nystagmus. The data input unit (3), the first data reading unit (4), and the data preprocessing unit (5) are connected in sequence.

6. The system according to claim 5, characterized in that, The nystagmus prediction subsystem (2) includes: The neural network unit (6) is used to extract features from the preprocessed video nystagmography data by using a neural network algorithm that improves the multi-head attention mechanism and the neural ordinary differential equation. Attention mechanism unit (7) is used to further extract features from video nystagmography data by improving the multi-head attention mechanism; Prediction unit (8) is used to classify nystagmus based on the extracted features; The neural network unit (6), the improved multi-head attention mechanism unit (7), and the prediction unit (8) are connected in sequence; the data preprocessing unit (5) and the neural network unit (6) are connected in sequence.

7. The system according to claim 5, characterized in that, The data preprocessing unit (5) performs data preprocessing on the input video nystagmus, including the following steps: Convert color video nystagmography to grayscale; The pupil center position in the video nystagmogram is obtained using Hough circle transform and contour detection algorithms. Record pupil coordinates in chronological order and convert the pupil coordinates into time series data; The time series data is normalized and converted into Gram feature maps.

8. The system according to claim 6, characterized in that, In the neural network unit (6), the neural network algorithm combined with the neural ordinary differential equation is as follows: the neural network is regarded as a stochastic continuous system, and the neural ordinary differential equation solver is used to perform dynamic modeling of the hidden state of the neural network under the condition of continuous time, and the activation function is the ReLU function. The improved multi-head attention mechanism specifically involves replacing the Softmax function in the conventional multi-head self-attention mechanism with a decomposable kernel function, and selecting the cosine similarity function as the kernel function to achieve a low-rank linearization improvement to the multi-head attention mechanism.