Vestibular function detection system and data processing method, electronic equipment and program product

Abstract

The present invention discloses a vestibular function testing system and data processing method, electronic device, and program product. The method includes: obtaining a video of a subject's eye movements during a vestibular function test; extracting the subject's eye movement trajectory from the eye movement video; the eye movement trajectory including one or more of a horizontal eye movement trajectory, a vertical eye movement trajectory, and a torsional eye movement trajectory; obtaining the subject's auxiliary testing data; the auxiliary testing data including the subject's body position data during the vestibular function test and / or nystagmus characteristic parameters obtained from the eye movement trajectory; inputting the eye movement trajectory and the auxiliary testing data into a trained neural network model, and outputting the subject's vestibular function data through the neural network model; wherein the eye movement trajectory and the auxiliary testing data are associated through time synchronization. Compared to only considering eye movement video or eye movement trajectory data, the addition of body position data or nystagmus characteristic parameters in this application solution makes the assessed vestibular function data more accurate.

Description

Technical Field

[0001] The present invention relates to the field of medical technology, and in particular to a vestibular function detection system and data processing method, electronic equipment and program product. Background Art

[0002] Vestibular dysfunction is a common clinical manifestation of balance disorder, and its detection and evaluation are of great significance for the diagnosis and treatment of patients. Vestibular function can be evaluated by recording and analyzing the patient's eye movement trajectory through eye movement video. For example, the nystagmus view currently used in clinical practice, however, the information provided by eye movement videos or trajectories has great limitations and cannot intuitively reflect the core conclusions of the examination. Moreover, because eye movement trajectories contain a large amount of complex information, and even much of it is additional information irrelevant to the diagnosis, it is difficult to accurately reflect the patient's actual vestibular function status through manual observation or simple analysis alone. At the same time, vestibular function testing is often accompanied by multiple factors such as changes in body position, and analysis of a single data source may not provide a comprehensive basis for diagnosis. Summary of the Invention

[0003] In order to solve the above technical problems, the present invention provides a vestibular function detection system and data processing method, electronic equipment, and computer program product.

[0004] In a first aspect, the present application discloses a method for processing vestibular function test data, comprising: obtaining an eye movement video of a subject in a vestibular function test; extracting the subject's eye movement trajectory from the eye movement video; the eye movement trajectory comprising at least one of the following: a horizontal eye movement trajectory, a vertical eye movement trajectory, and a torsional eye movement trajectory; obtaining auxiliary test data of the subject; the auxiliary test data comprising the subject's body position data in the vestibular function test and / or eye movement feature data obtained from the eye movement trajectory; inputting the eye movement trajectory and the auxiliary test data into a pre-trained neural network model, and outputting the subject's vestibular function data through recognition by the neural network model; and comparing the eye movement trajectory and the auxiliary test data. Preferably, the eye movement trajectory is associated with the auxiliary detection data through time synchronization; the neural network model includes: an auxiliary data processing network, which is used to process the input auxiliary detection data and extract auxiliary features; an eye movement data processing network, which is used to process the input eye movement trajectory and extract eye movement trajectory features; a feature fusion layer, which is used to fuse the auxiliary features output by the auxiliary data processing network and the eye movement trajectory features output by the eye movement data processing network to obtain comprehensive feature data related to vestibular function; and an output layer, which is used to generate corresponding vestibular function data based on the output data of the feature fusion layer.

[0005] In some embodiments, the auxiliary detection data input into the neural network model includes at least eye movement feature data, which is nystagmus feature parameters; obtaining the nystagmus feature parameters includes: identifying nystagmus information in eye movement trajectories in various directions, and obtaining corresponding nystagmus feature parameters; the nystagmus feature parameters include: any one or more of the nystagmus slow phase angular velocity, nystagmus direction, nystagmus time, and nystagmus change trend.

[0006] In some embodiments, identifying nystagmus information in the eye movement trajectory characteristics in each direction specifically includes: obtaining the motion characteristics of each data point in the eye movement trajectory in each direction; judging whether there is periodic eye movement in the eye movement trajectory based on the motion characteristics; if there is periodic eye movement in the eye movement trajectory, identifying the periodic eye movement in the eye movement trajectory as nystagmus.

[0007] In some embodiments, identifying nystagmus information in eye movement trajectories in various directions specifically includes: based on each data point in the eye movement trajectory in each direction, comparing the position change trend of each data point with the previous data point, and finding the data point with a changed change trend as a key data point; according to all the key data points found, dividing the eye movement data in that direction into N data segments; N is a positive integer greater than 1; calculating the slope of each data segment, and according to the calculated slope, identifying a first data segment with a slope greater than a first set value and a second data segment with a slope less than a second set value; wherein the first set value is greater than or equal to the second set value; traversing all data segments, screening out target eye movement data containing alternating first and second data segments as nystagmus; wherein the first data segment is a fast phase of nystagmus, used to indicate the direction of nystagmus; and the second data segment is a slow phase of nystagmus, used to indicate the amplitude of nystagmus.

[0008] In some embodiments, the auxiliary detection data input into the neural network model includes the subject's body position data or nystagmus characteristic parameters; the auxiliary data processing network of the neural network model includes a body position data processing network or a nystagmus characteristic processing network; wherein: the body position data processing network is used to process the input body position data and extract body position features; the nystagmus characteristic processing network is used to receive the input nystagmus characteristic parameters.

[0009] In some embodiments, the eye movement feature data input into the neural network model includes body position data and nystagmus feature parameters; the auxiliary data processing network of the neural network model includes: a body position data processing network and a nystagmus feature processing network; wherein: the body position data processing network is used to process the input body position data and extract body position features; the nystagmus feature processing network is used to receive input nystagmus feature parameters; the feature fusion layer is used to fuse the feature data output by the body position data processing network, the eye movement data processing network and the nystagmus feature processing network; the feature fusion layer is also used to use a self-attention mechanism to adjust the weights of different features based on the fused feature vector; the output layer is used to identify and output the vestibular function data of the subject based on the data processed by the self-attention mechanism.

[0010] In some embodiments, the feature fusion layer includes a first fusion layer and a second fusion layer; wherein: if the feature fusion layer adopts the first fusion architecture, then: the first fusion layer is used to perform feature fusion on the posture features output by the posture data processing network and the eye movement trajectory features output by the eye movement data processing network; the second fusion layer is used to fuse the nystagmus feature parameters output by the nystagmus feature processing network with the data output by the first fusion layer, and adopt a self-attention mechanism to dynamically adjust the weights of different features; if the feature fusion layer adopts the second fusion architecture, then: the first fusion layer is used to perform feature fusion on the eye movement trajectory features output by the eye movement data processing network and the nystagmus feature parameters output by the nystagmus feature processing network; the second fusion layer is used to perform feature fusion on the posture features output by the posture data processing network and the data output by the first fusion layer, and adopt a self-attention mechanism to dynamically adjust the weights of different features.

[0011] In some embodiments, the vestibular function data output by the neural network model is the nystagmus information of the subject in each specific body position during the vestibular function test; the vestibular function test data processing method also includes: performing data optimization processing and structured text processing on the identified nystagmus information in each specific body position to generate vestibular function text data.

[0012] In some embodiments, data optimization processing includes any one or more of the following: based on preset screening rules, removing redundant information irrelevant to vestibular function diagnosis; based on preset feature thresholds, eliminating invalid data that does not conform to nystagmus characteristics; generating corresponding classification descriptions based on the duration of the nystagmus latency in each specific body position identified, combined with several set duration stages of the nystagmus latency; generating corresponding classification descriptions based on the duration of the nystagmus in each specific body position identified, combined with several set duration stages of the nystagmus duration; generating corresponding classification descriptions based on the slow phase angular velocity of the nystagmus in each specific body position identified, combined with a set minimum threshold for the slow phase angular velocity of the nystagmus; and merging nystagmus information of the same type.

[0013] In a second aspect of the present application, an electronic device is disclosed, comprising a screen, a memory, one or more data processors, and one or more programs; wherein the one or more programs are stored in the memory; when the one or more data processors execute the one or more programs, the electronic device implements any of the above-mentioned vestibular function detection data processing methods.

[0014] In a third aspect of the present application, a storage medium is disclosed, which stores computer instructions. When the computer instructions are executed by a data processor, the data processor executes the steps of any one of the above-mentioned methods for processing vestibular function test data.

[0015] In a fourth aspect of the present application, a computer program product is disclosed. When the computer program product is run on a computer, the computer is caused to execute any one of the steps of the vestibular function detection data processing method described above.

[0016] In a fifth aspect of the present application, a vestibular function detection system is disclosed, comprising: an eye movement shooting module, an eye movement trajectory acquisition module, an auxiliary data acquisition module, and an intelligent recognition module; wherein: the eye movement shooting module is equipped with at least one camera for collecting eye movement videos of subjects during vestibular function tests; the eye movement trajectory acquisition module is communicatively connected to the eye movement shooting module, for receiving eye movement videos transmitted by the eye movement shooting module and extracting the subject's eye movement trajectory therefrom; the eye movement trajectory comprises at least one of the following: horizontal direction eye movement trajectory, vertical direction eye movement trajectory, torsional direction eye movement trajectory; the auxiliary data acquisition module is used to obtain auxiliary detection data of the subject; the auxiliary detection data comprises the subject's body position data and / or nystagmus characteristic parameters; the auxiliary data acquisition module comprises a body position acquisition submodule and / or an eye movement processing submodule; wherein: the body position acquisition submodule is used to obtain the subject's body position data in vestibular function testing; the eye movement processing submodule is communicatively connected to the eye movement trajectory acquisition module, for The invention relates to a smart recognition module for receiving eye movement trajectories transmitted by the eye movement trajectory acquisition module and obtaining corresponding nystagmus feature parameters from the eye movement trajectories; an intelligent recognition module having a pre-trained neural network model built in, and the intelligent recognition module being communicated with the eye movement trajectory acquisition module and the auxiliary data acquisition module respectively, and being used to input the eye movement trajectories and auxiliary detection data into the neural network model to obtain vestibular function data of the subject; wherein the neural network model comprises: an auxiliary data processing network for processing the input auxiliary detection data and extracting auxiliary features; an eye movement data processing network for processing the input eye movement trajectories and extracting eye movement trajectory features; a feature fusion layer for fusing the auxiliary features output by the auxiliary data processing network and the eye movement trajectory features output by the eye movement data processing network to obtain comprehensive feature data related to vestibular function; and an output layer for generating corresponding vestibular function data according to the output data of the feature fusion layer.

[0017] In some embodiments, the eye movement capture module is set in the eye movement acquisition device, and the eye movement acquisition device adopts any product form among wearable eye movement collector, head-mounted virtual reality or augmented reality device, and non-wearable capture device.

[0018] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0019] 1. This application identifies nystagmus information in specific body positions based on eye movement data and auxiliary detection data (body position data and / or nystagmus characteristic parameters). Compared with only considering eye movement video or eye movement trajectory data, the addition of body position data or nystagmus characteristic parameters makes the vestibular function data evaluated more accurate.

[0020] 2. The vestibular function detection system of the present application, combined with artificial intelligence technology, realizes the fusion and recognition of the input auxiliary detection data and eye movement trajectory data through a trained neural network model. In particular, the neural network model mainly adopts a model architecture that can process time series data, so that it can effectively obtain the correlation and before-and-after dependencies of time series data (such as body position data and eye movement trajectory data), capture the complex patterns in the body position data and eye movement trajectory data, and thus make better predictions; better, in terms of model input, in addition to body position data and eye movement data in various directions, the corresponding nystagmus characteristic parameters obtained based on the eye movement trajectory data in various directions are also input, especially the slow phase angular velocity of nystagmus, thereby greatly improving the accuracy of the output vestibular function data.

[0021] 3. The vestibular function data output by the vestibular function detection system of the present application can be text information. Compared with traditional parameter data types, the output vestibular function text information is more easy to understand and removes a lot of complicated information irrelevant to the diagnosis. It only presents the core diagnostic information of characteristic nystagmus in a specific body position. In particular, it is presented in the form of text, so that medical workers who do not need professional training can understand it directly and can quickly obtain key clues for vestibular function diagnosis without having to compare and check complex curves. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The preferred embodiments will be described below in a clear and understandable manner with reference to the accompanying drawings to further illustrate the above-mentioned characteristics, technical features, advantages and implementation methods of the present invention.

[0023] Figure 1 is a structural block diagram of an embodiment of the vestibular function detection system of the present application;

[0024] Figure 2 is a flowchart of an embodiment of a method for processing vestibular function test data of the present application;

[0025] Figure 3 is a schematic diagram showing body position data in one embodiment of the present application;

[0026] Figure 4 is a schematic diagram of the architecture of a neural network model in one embodiment of the present application;

[0027] Figure 5 This is a schematic diagram showing the eye movement trajectory and head position data of a subject in one embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the specific embodiments of the present invention will be described below with reference to the accompanying drawings. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings and other embodiments can be obtained based on these drawings without inventive work.

[0029] To simplify the drawings, only portions relevant to the invention are schematically depicted in each figure; they do not represent the actual structure of the product. Furthermore, to simplify the drawings and facilitate understanding, in some figures, only one component with the same structure or function is schematically depicted or labeled. In this document, "one" not only means "only one" but also "more than one."

[0030] It should be further understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0031] The present invention is described herein in an exemplary manner with reference to a computer system architecture and exemplary processes executed by the computer system. In one or more embodiments, the functions described herein may be implemented via computer system instructions. These computer program instructions may be loaded directly onto the internal data storage of a computing device (e.g., the internal data storage of a smartphone or laptop computing device). Alternatively, these computer program instructions may be stored on a portable computer-readable medium (e.g., a flash drive) and subsequently loaded onto the computing device so that the instructions can be executed. In other embodiments, these computer program instructions may be embodied in the hardware of the computing device rather than in its software. Computer program instructions may also be embodied in a combination of hardware and software. Furthermore, in this disclosure, when reference is made to a computing device "configured to," "arranged to," and / or "configured and arranged to" perform a specific function (e.g., a data acquisition / data processing device being configured and arranged to perform a specific function), it should be understood that, in one or more embodiments of the present invention, this means that the computing device is specifically programmed to perform the specific function (e.g., the data acquisition / data processing device is specifically programmed to perform the specific function).

[0032] This description describes in general terms the computer program required for analyzing information from a vestibular function test. Any competent programmer in the field of information technology can use the description presented in this article to develop a system.

[0033] For the sake of brevity, traditional computer system components, traditional data networks, and traditional software coding will not be described in detail herein. In addition, it should be understood that the connecting lines shown in the block diagrams included herein are intended to represent the functional relationships and / or operational couplings between the various components. Except as explicitly described, it should be understood that many alternative or additional functional relationships and / or physical connections may be incorporated into actual applications of the system.

[0034] Vestibular function testing evaluates the vestibular function of the semicircular canals, otoliths, or central vestibular system by observing and analyzing the vestibular system's response to movement, changes in body position, and balance control. There are many methods for vestibular function testing, such as the Dix-Hallpike test, video head impulse test (vHIT), Roll Test, Romberg Test, etc., to help diagnose vestibular-related diseases (such as vertigo, balance disorders, benign paroxysmal positional vertigo, etc.) and formulate treatment plans. Let's take otolithiasis testing as an example. Currently, clinical practice relies more on nystagmus imaging to assist in the diagnosis and treatment of BPPV. By recording the nystagmus during the entire examination process through video, manual reduction and rotation chair reduction can be achieved by having the patient wear an infrared video goggles with a gyroscope to realize the function of nystagmus imaging.

[0035] The information provided by the nystagmus view currently used in clinical practice is severely limited and cannot directly reflect the core conclusions of the examination. Although the nystagmus view can provide complete video playback and analyze the eye movement trajectories obtained from video analysis, this information is extremely complex and contains a lot of additional information irrelevant to the diagnosis. The examination and treatment of a BPPV patient may involve more than ten examinations and treatment maneuvers, hundreds of branching maneuvers, and dozens of minutes of examination data. If the operator needs to review the specific situation of the subject, they still need to fully replay the nystagmus information, curves, or videos for each body position. However, the core diagnostic information that doctors are most concerned about cannot be directly obtained by reading the result data. Even doctors who have received complex professional training need to review complex curves and videos before reaching a final diagnosis.

[0036] The present application provides a method and system for processing vestibular function test data, which adopts artificial intelligence combined with comprehensive consideration of multi-source data to make vestibular function assessment more intelligent and accurate.

[0037] The vestibular function detection system of this application is as follows Figure 1 As shown, it includes the following functional modules:

[0038] The eye movement capture module 10 includes at least one camera for capturing eye movement videos of subjects during vestibular function testing;

[0039] The eye movement trajectory acquisition module 20 is in communication with the eye movement capture module 10 and extracts eye movement trajectory data based on the eye movement video captured by the camera. The eye movement trajectory data includes eye movement trajectories in at least one of the horizontal, vertical, and torsional directions. Of course, the eye movement trajectory acquisition module 20 that obtains eye movement trajectories based on eye movement videos can be integrated into any device, such as an eye movement acquisition device, a data processing device, or a cloud server, and processed by the data processor of the device. For example, after the camera of the eye tracking device captures the eye movement video, the data processor of the eye tracking device extracts eye movement trajectory features from the eye movement video.

[0040] Auxiliary data acquisition module 30 is used to acquire auxiliary detection data of the subject, wherein the auxiliary detection data includes the subject's body position data and / or nystagmus characteristic parameters; this module includes at least one of the following:

[0041] The posture acquisition submodule 31 is used to obtain the posture data of the subject in the vestibular function test; preferably, it includes: a posture acquisition unit and a posture processing unit; wherein the posture acquisition unit is used to acquire the posture data of the subject, such as acquiring the subject's head position data through a gyroscope or an inertial measurement unit; and the posture processing unit pre-processes the posture data acquired by the posture acquisition unit, such as denoising, data reduction, coordinate conversion, etc., to obtain posture data that meets the input of the subsequent neural network model; if the acquired posture data is video data, image processing is required to extract the posture change trajectory of the subject from the posture image video frame, that is, what we finally input into the neural network model can be one-dimensional data or two-dimensional data, but not three-dimensional video data.

[0042] The eye movement processing submodule 32 is used to identify nystagmus in the subject's eye movement trajectory and obtain corresponding nystagmus characteristic parameters.

[0043] The intelligent recognition module 40 has a pre-trained neural network model 41 built in. The intelligent recognition module is respectively connected to the eye movement trajectory acquisition module 20 and the data acquisition module 30 for inputting the eye movement trajectory and the auxiliary detection data into the neural network model 41 to obtain the vestibular function data of the subject; preferably, the eye movement trajectory and the auxiliary detection data are temporally correlated. Specifically, the intelligent recognition module 40 identifies the nystagmus conditions in each specific body position through the neural network model 41, and then obtains the vestibular function data of the subject. The specific body position refers to the specified body position in the vestibular function test, and the specific body position will be different for different test items. In addition, even for the same test item, there may be multiple different specific body positions, which depend on the vestibular function test item and can also be directly identified by the neural network model (specific body positions need to be marked for learning during model training).

[0044] Among the above modules, the eye movement trajectory acquisition module 20, the eye movement processing submodule 32, the body position processing unit, the intelligent recognition module 40, etc. are all data processing function modules, and can realize corresponding functions by executing corresponding instructions through one or more data processors.

[0045] In addition, based on the above, the system can also include the following data processing function modules:

[0046] The pre-processing sub-module is used to perform data cleaning, data conversion (such as formatting) and other operations on the collected eye movement data or body position data;

[0047] The data synchronization submodule is used to synchronize the timestamps of eye movement trajectories and auxiliary detection data;

[0048] The post-processing submodule is used to receive the output results of the neural network model and perform post-processing on them, such as converting the output results into structured data for subsequent display or storage.

[0049] In this embodiment, each data processing functional module can be integrated into at least one data processor, meaning that data processing and model inference are performed within the same processor, thereby simplifying the system architecture and reducing the complexity of module interactions. Of course, it is more preferable to separate the intelligent recognition module with a built-in neural network model and the other data processing functional modules and implement them through different processors. One data processor is responsible for data preprocessing (such as synchronization, cleaning, and formatting) and post-processing (such as result display and storage), serving as the logical module responsible for general data operations in the system. Another data processor, which also integrates a neural network model, focuses on complex feature extraction, fusion, and prediction tasks, serving as a dedicated analysis and inference module. Together, the two processors process vestibular function test data.

[0050] Furthermore, the vestibular function test data processing system also includes:

[0051] Display device: used to display data processing status, including eye movement trajectory curves, vestibular function data of the subject (such as nystagmus information in specific body positions, vestibular function assessment results, rehabilitation training recommendations, etc.), etc.

[0052] The storage module is used to save the results of data processing, such as the evaluation report generated by the final analysis of the model.

[0053] In the above system embodiments, each module (including submodules) can be set or integrated into the same or different devices, and the functions of each functional module can be realized by configuring the data processor of the device where each functional module is located accordingly. Generally, the modules of the vestibular function detection system can basically be set in the acquisition device (such as eye movement acquisition device, body position acquisition device), data processing terminal device and / or server; the following are several exemplary descriptions:

[0054] An eye movement capture module can be set in the eye movement acquisition device to capture the subject's eye movement video;

[0055] The eye movement trajectory acquisition module can be set in any device among the eye movement acquisition device, the data processing terminal device, and the server;

[0056] The body position acquisition submodule includes a body position acquisition unit that can be arranged in a body position acquisition device or integrated in an eye movement acquisition device to collect the subject's body position data; and a body position processing unit that can be arranged in a data processing terminal device or a server.

[0057] The eye movement processing submodule can be set in any device among the eye movement acquisition device, the data processing terminal device, and the server;

[0058] The intelligent recognition module can be set in any device among the eye movement acquisition device, data processing terminal device, and server.

[0059] It is worth noting that the above-mentioned body position acquisition device can be a device independent of the eye movement acquisition device. For example, the body position acquisition device uses a device that drives the subject's body position change; the body position acquisition device and the eye movement acquisition device can also be combined into one, and the two are integrated into the same device. For example, through a head-mounted device with a built-in gyroscope and camera, the subject's head movement information and eye movement video can be collected respectively; for example, through an external adjustable camera, the camera is set near the subject through a bracket to shoot the subject's body position change video and eye movement video during the vestibular function test, and then the video data is processed to obtain corresponding body position data (not video data, but body position trajectory data in all directions) and eye movement trajectory data.

[0060] The aforementioned data processing terminal device can be a computer, tablet computer, mobile phone, medical terminal or platform device, etc. The server can be a local server or a cloud server.

[0061] Preferably, in one example, the vestibular function testing system consists of an eye tracker, a proximal data processing terminal device, and a cloud server. The eye tracker is equipped with at least one camera and at least one motion sensor (such as a gyroscope) to collect eye movement video and head position change data, and transmit them to the proximal data processing terminal device. The proximal data processing terminal device processes the received data, such as extracting eye movement trajectory data in various directions from the eye movement video and pre-processing the head position data to conform to the input format of the neural network model before sending it to the cloud server. The cloud server uses the neural network model to intelligently identify nystagmus information in each specific body position (specified body position), obtains the subject's vestibular function data, and feeds it back to the proximal data processing terminal device.

[0062] Of course, if the input source of the model also includes nystagmus characteristic parameters, then the step of obtaining the nystagmus characteristic parameters can be set at the proximal data processing terminal device or at the cloud server end, which is not limited in this embodiment.

[0063] In this system embodiment, the proximal data processing terminal, located near the data collection source, processes the collected data, reducing irrelevant information, minimizing the data volume, and focusing on the information content. This saves transmission bandwidth, speeds up transmission, and improves transmission quality. Furthermore, the eye movement tracks transmitted to the cloud server do not contain private information such as pupil images, effectively preventing privacy leaks and ensuring the security of personal data.

[0064] The proximal data processing terminal device sits between the data acquisition end (eye tracker) and the cloud server, serving as a data processing and transfer agent. In terms of hardware, it typically includes data storage capabilities, a processing unit, and a network communication module. For example, in a typical medical testing scenario, the proximal data processing terminal device can be a moderately configured medical workstation computer. It synchronizes data with the eye tracker in real time via a wired or wireless connection. After the patient completes the test, the workstation computer rapidly processes the data, converting the raw eye movement video into eye trajectory data in all directions, pre-processing the head position data, and then transmitting the processed results to the cloud server via a wireless network for in-depth analysis. This approach not only meets the needs of primary healthcare institutions for rapid local data processing, but also leverages the powerful computing power of the cloud to achieve more accurate vestibular function assessments, while ensuring efficient and secure data transmission. Of course, larger medical institutions can also utilize a local server to rapidly assess vestibular function without connecting to an external network.

[0065] Another embodiment of the present application provides a vestibular function testing system, comprising: at least one data processor (e.g., a data processor of a computing device such as a mobile device); at least one body position acquisition submodule and at least one eye movement capture module (including at least one camera); the body position acquisition submodule and the eye movement capture module are operably coupled to the data processor, and the data processor is configured to execute the steps of a vestibular function testing data processing method, specifically including:

[0066] Receive the subject's body position data and eye movement video during vestibular function testing; specifically, for example, receive the eye movement video collected by the eye movement capture module and the body position data collected by the body position acquisition submodule;

[0067] Extracting eye movement data of the subject from the eye movement video, the eye movement data including an eye movement trajectory of the pupil in at least one of the three directions of horizontal, vertical, and torsional directions;

[0068] The eye movement trajectory and body position data are input into a trained neural network model, and the vestibular function data of the subject is output through the neural network model; wherein the eye movement trajectory and body position data are associated through time synchronization.

[0069] In this embodiment, the output vestibular function data is processed based on body position data and eye movement data. Compared to conventional methods that predict vestibular function data based solely on eye movement videos or eye movement trajectories, this embodiment closely integrates the subject's body position information during vestibular function testing. The integration of body position information greatly improves the accuracy of vestibular function assessment. Taking benign paroxysmal positional vertigo as an example, when conducting a positional test, the subject's body position needs to be changed based on the test type (such as the Dix-Hallpike test). The doctor observes whether there is characteristic nystagmus in a specific position. For example, when diagnosing the canalolithic type of right posterior semicircular canal BPPV, it is necessary to observe an upward-jumping nystagmus with the upper pole of the eye twisting to the right in the right posterior Dix-Hallpike position before a diagnosis can be made.

[0070] In this embodiment, the body position acquisition submodule is mainly used to collect the body position data of the subject; the eye movement capture module includes at least one camera for collecting the subject's eye movement video; the data processor is configured to receive the eye movement video and extract the eye movement trajectory of the subject's pupil in at least one of the three directions of horizontal, vertical, and torsional directions; the data processor or another data processor is configured to receive the body position data, and input the body position data and the extracted eye movement trajectory into a trained neural network model, identify the nystagmus information under each specific body position through the neural network model, and output vestibular function examination information.

[0071] In one example of a body position acquisition submodule, the submodule includes at least one of an accelerometer configured to detect linear acceleration and a gyroscope configured to detect angular velocity. For example, in an illustrative embodiment, the body position acquisition submodule may include a 3-axis accelerometer, a 3-axis gyroscope, and a 3-axis magnetometer to enable the use of a motion fusion algorithm. Of course, posture sensors such as linear accelerometers or gyroscopes can generally be integrated into the head-mounted device to facilitate the acquisition of changes in the subject's head position.

[0072] In another exemplary embodiment, the body position acquisition submodule is a body position change conversion device, such as a BPPV vertigo diagnosis and treatment swivel chair. During the detection or resetting process, the BPPV vertigo diagnosis and treatment swivel chair will transmit the head position change data of the subject at each time point in the process to the data processor for body position recognition, etc. For example, Figure 3 The figure shows the body position (head position) data curve of a subject sitting on a BPPV swivel chair, including the trajectory data of the subject's head position on the Pitch axis and the trajectory data on the Yaw axis.

[0073] In another exemplary embodiment, the body position acquisition submodule is a camera, which can be set near the subject through a fixed bracket or other equipment to capture images of the subject's body position changes during the test, and then perform image processing to obtain the subject's body position data corresponding to each video frame (corresponding to different time points).

[0074] In one embodiment, a device integrated with an eye movement capture module (having an eye movement capture function) may take any of the following product forms:

[0075] A wearable eye-tracking recorder, which has the eye-tracking capture module built in and can be conveniently worn on the subject's head so that the eye-tracking capture module can capture the subject's eyes. It can communicate with an external device via a wired or wireless communication protocol (such as Bluetooth or Wi-Fi) to transmit the collected eye-tracking video or processed eye-tracking trajectory to the device where the subsequent data processing module is located;

[0076] Head-mounted virtual reality (VR) / augmented reality (AR) devices integrate eye movement acquisition capabilities, using their built-in sensor arrays and camera combinations to capture user eye movements and communicate with external independent data processing devices through the device's own high-speed internal bus;

[0077] The eye movement acquisition device is fixed on the detection seat headrest position, and uses a multi-angle adjustable bracket to fix the camera to accommodate subjects of different heights and sitting postures. It directly transmits data to the nearest desktop computer as the device where the data processing module is located via a wireless network connection or a wired network connection.

[0078] The mobile device with a data processor is selected from the group consisting of: (i) a smartphone, (ii) a tablet computing device, (iii) a laptop computing device, (iv) a smartwatch, and (v) a head-mounted display. For example, in an illustrative embodiment, the body position acquisition submodule and / or the eye movement capture module may be a built-in position sensor and / or camera of a video nystagmus device. A video nystagmus device is generally wearable in front of a user's eyes and has a built-in camera that can capture video of the user's eye movements. It also has an integrated position sensor or motion sensor (such as a gyroscope) that can capture body position data related to the user's head movement. In another illustrative embodiment, another type of computing device is used instead of a mobile computing device. For example, the other type of computing device may be a desktop computing device, a server computing device, or a small personal computer. In yet another illustrative embodiment, the user's body position change data can be obtained by using chair rotation change data output by a control unit that controls the movement of the chair in which the user sits, and the user's eye video data can be captured by the camera of the video nystagmus device worn on the user's head.

[0079] In an illustrative embodiment, after collecting a video of a subject's eye movements during a vestibular function test, a data processor (an eye movement trajectory acquisition module) is used to extract eye movement trajectories from the video to obtain eye movement trajectory data of the subject's pupil in the horizontal, vertical, and torsional directions. Figure 5 The diagram shows the eye movement trajectories of the subject's pupils in the horizontal, vertical, and torsional directions, as well as the head position on the Pitch and Yaw axes during the left-posterior and right-anterior Dix-Hallpike test. The horizontal axis is time, the vertical axis of the eye movement trajectory is the eye movement amplitude, and the vertical axis of the head movement trajectory is the head movement amplitude.

[0080] Regarding data preprocessing of eye movement trajectory and auxiliary detection data, the auxiliary detection data and eye movement trajectory can be synchronized using timestamps before input into the model. For example, if the auxiliary detection data is body position data, both the acquired body position data and eye movement data contain time information. To more accurately assess and judge the subject's vestibular function, the body position data and eye movement data can be temporally correlated. This allows the neural network model to obtain correlated features of the body position data and eye movement data during recognition processing, such as the characteristics of eye movement changes caused by body position changes. Generally, the body position data and eye movement data are temporally correlated before being input into the neural network model for recognition processing. Of course, this temporal correlation step can also be implemented within the neural network model. For example, after receiving the input body position data and eye movement data, the neural network model performs preprocessing on these two types of data through a data preprocessing layer, mainly including time alignment and normalization operations. The data from each sensor is treated as an independent input channel to ensure temporal synchronization and consistent data formats across different sensors. This is the foundation for ensuring subsequent processing.

[0081] Based on the same technical concept, another embodiment of the present application provides a method for processing vestibular function detection data, such as Figure 2 As shown, the following steps are included:

[0082] S101, obtaining the subject's eye movement video during vestibular function testing;

[0083] S102, extracting the subject's eye movement trajectory from the eye movement video, where the eye movement trajectory includes at least one of the following: a horizontal eye movement trajectory, a vertical eye movement trajectory, and a torsional eye movement trajectory;

[0084] S103, obtaining auxiliary test data of the subject, including the subject's body position data in the vestibular function test and / or nystagmus characteristic parameters obtained from the eye movement trajectory;

[0085] S104, inputting the eye movement trajectory and the auxiliary detection data into a pre-trained neural network model, and outputting the vestibular function data of the subject through recognition by the neural network model.

[0086] In this embodiment, each track point in the eye movement trajectory data contains eye position information and associated time information (for example, the horizontal eye movement trajectory data contains the sampling time point information and the horizontal position information of the eye pupil at that time point); similarly, the auxiliary detection data also contains time information, thereby facilitating the subsequent time synchronization or association of different types of data.

[0087] Preferably, the auxiliary detection data is structured parameter data and / or static image data. The auxiliary detection data does not include video data, that is, it does not include dynamic image sequences represented in the form of continuous image frames. Structured parameter data includes, but is not limited to, sequence data or parameter data, such as body position sequences, head movement trajectory sequences, and nystagmus characteristic parameters; while static image data includes, for example, body position trajectory maps, head movement trajectory maps, and nystagmus characteristic maps, but does not include dynamic video data composed of continuous image frames.

[0088] Preferably, before the eye movement trajectory and auxiliary detection data are input into the model, data preprocessing is performed: the eye movement trajectory and the auxiliary detection data are synchronously associated; further, the eye movement trajectory and the auxiliary detection data are both time series data.

[0089] In the above embodiment, the acquired auxiliary detection data may be body position data and / or nystagmus characteristic parameters. Taking the auxiliary detection data including body position data and nystagmus characteristic parameters as an example, another embodiment of the present application provides a method for processing vestibular function detection data, including the following steps:

[0090] 1. Steps for obtaining raw data; including:

[0091] S201, obtaining the subject's eye movement video during vestibular function testing;

[0092] S202, obtaining the subject's body position data during the vestibular function test;

[0093] The above two steps are mainly about acquiring the raw data of vestibular function testing. This acquisition can be done by directly receiving eye movement videos and body position data collected by external acquisition devices, or by actively controlling external acquisition devices (such as body position acquisition devices and eye movement acquisition devices).

[0094] 2. Preliminary data processing steps, including:

[0095] 2.1 Eye movement data acquisition:

[0096] S203, extracting the subject's eye movement trajectory from the eye movement video, including at least one of the three items: horizontal eye movement trajectory, vertical eye movement trajectory, and torsional eye movement trajectory;

[0097] 2.2 Auxiliary detection data acquisition:

[0098] S204, analyzing and processing the subject's eye movement trajectory to obtain corresponding nystagmus characteristic parameters; specifically, the nystagmus characteristic parameters include: any one or more of: nystagmus slow phase angular velocity, nystagmus direction, nystagmus time, and nystagmus change trend.

[0099] S205 , pre-process the subject's body position data to obtain body position data that meets the requirements of the neural network model input; for example, the acquired body position data is subjected to denoising, data simplification, and coordinate conversion. Of course, if the acquired body position data is video data, image processing is also required to extract the subject's body position change trajectory from the body position image video frames. In other words, the final input to the neural network model can be one-dimensional or two-dimensional data, but not three-dimensional video data.

[0100] 3. Intelligent identification steps; specifically including:

[0101] S206: Input the eye movement trajectory and auxiliary detection data (in this embodiment, nystagmus characteristic parameters and body position data) into the trained neural network model, and use the neural network model to identify and output the subject's vestibular function data. Preferably, the eye movement trajectory and auxiliary detection data are temporally correlated.

[0102] In the above method embodiment, the auxiliary detection data includes nystagmus characteristic parameters and body position data, which, along with the eye movement trajectory data, serve as inputs to the neural network model, which then outputs vestibular function data after model recognition. Of course, the auxiliary detection data may also be body position data or nystagmus characteristic parameters.

[0103] Regarding the input data combination of the neural network model:

[0104] In the above system embodiment or method embodiment, the auxiliary detection data includes body position data in vestibular function testing and / or nystagmus characteristics obtained from the eye movement trajectory. Therefore, the data combination of the input model includes at least the following three schemes:

[0105] Solution 1: The input data of the model includes: eye movement trajectory + body position data; preferably, the body position data is head position data (data on the change in the subject's head position).

[0106] This solution focuses on analyzing the correlation between body position and eye movements, specifically when changes in body position cause changes in eye movement (such as nystagmus). It is particularly suitable for detecting conditions that cause dizziness due to changes in body position. Positional tests such as the Dix-Hallpike test and the Roll test involve moving the patient's head to a specific position (i.e., a specific body position) to observe whether nystagmus occurs. This allows for analyzing the correlation between nystagmus and body position changes, providing doctors with key diagnostic information for conditions such as otolithiasis.

[0107] Solution 2: The input data of the model includes: eye movement trajectory + nystagmus characteristic parameters;

[0108] This solution focuses on the details of eye movement characteristics - nystagmus characteristic parameters. On the one hand, the eye movement trajectory provides comprehensive eye movement information. On the other hand, the nystagmus characteristic parameters extracted from the eye movement trajectory can provide an important basis for vestibular function evaluation.

[0109] Solution 3: The model's input data includes: eye movement trajectory, body position data, and nystagmus characteristic parameters. This solution integrates these three data sets to provide a more comprehensive and accurate assessment of the subject's vestibular function.

[0110] The above-mentioned eye movement trajectory mainly includes the subject's horizontal eye movement trajectory data, and / or vertical eye movement trajectory data, and / or torsional eye movement trajectory data; generally, the eye movement data input to the model includes horizontal and vertical eye movement trajectory data, and preferably, further includes torsional eye movement trajectory data, thereby improving the comprehensiveness of vestibular function diagnosis and providing more key data for vestibular function diagnosis.

[0111] Both Options 2 and 3 above use nystagmus characteristic parameters as one of the model inputs. These parameters include any one or more of the following: slow-phase angular velocity, direction, duration (duration or onset), latency, and trend of nystagmus. These parameters are crucial for the diagnosis or assessment of vestibular function. Typical nystagmus exhibits characteristics of fluctuating intensity and delayed onset. For example, in the canalolithic form of right posterior semicircular canal BPPV, nystagmus typically appears within 40 seconds of head positioning and disappears within 1 minute. The intensity of the nystagmus changes from weak to strong and then back to weak, with a latency of less than 40 seconds. Therefore, combining these nystagmus characteristic parameters with body position data can significantly improve the accuracy of vestibular function assessment and / or vestibular rehabilitation training recommendations based on vestibular function data. Preferably, the more nystagmus characteristic parameters are selected, the better. It is best to select 1-2 nystagmus characteristic parameters. For example, the addition of the input factor of the slow phase angular velocity of nystagmus can greatly improve the accuracy of vestibular function diagnosis or evaluation, and thus improve the accuracy of vestibular function rehabilitation training program recommendations.

[0112] Compared to conventional methods of assessing vestibular function solely based on eye movement trajectories or eye movement videos, although eye movement trajectories can capture eye movement patterns and provide a preliminary assessment of vestibular function, a single data source cannot provide an accurate assessment. The addition of auxiliary detection data in this application can effectively compensate for this shortcoming. The selection of auxiliary detection data is also unique. The more types of auxiliary detection data, the better. The more types, the more complex the model will be for subsequent processing, which poses a great challenge to model training and accuracy improvement. In this case, body position data was selected as one of the auxiliary detection data, mainly because body position data is complementary to eye movement trajectories, can provide richer vestibular function information, and help the model more accurately capture the causal relationship between body position changes and nystagmus. Eye movement trajectories alone cannot reflect the specific body position changes that induce nystagmus, and it is difficult to associate them with the cause of vestibular dysfunction (for example, it is difficult to distinguish abnormalities in the bilateral semicircular canals based on eye movement trajectories alone), thus compensating for the limitations of eye movement trajectories.

[0113] Why are nystagmus characteristic parameters used as another option for auxiliary detection data? The main reason is that nystagmus characteristic parameters deeply quantify and supplement the dynamic characteristics of eye movement trajectories, and can more accurately reflect the functional status of the vestibular system. Let's take the slow phase angular velocity of nystagmus as an example. The slow phase angular velocity of nystagmus is the core parameter of nystagmus characteristics, representing the speed of the eyeball in slow phase movement. Compared with the path information provided by the eye movement trajectory, the slow phase angular velocity of nystagmus quantifies the dynamic intensity of eye movement and is an important supplement to the eye movement trajectory data. The slow phase angular velocity of nystagmus is the core information extracted from the eye movement trajectory, avoiding information redundancy and noise interference. Of course, the selection of nystagmus characteristic parameters is not the more the better. It is better to select the slow phase angular velocity of nystagmus in the nystagmus characteristic parameters and combine it with the eye movement trajectory as the model input.

[0114] In summary, this application uses eye movement trajectories combined with body position data and / or nystagmus characteristic parameters as model input. On the one hand, eye movement trajectories preserve the detailed information of the original data, allowing the model to access global information. On the other hand, body position data and / or nystagmus characteristic parameters supplement this information, thereby improving the model's accuracy. Furthermore, by introducing a neural network model, automated feature extraction, fusion, and evaluation of multimodal data are achieved, offering greater accuracy and comprehensiveness than traditional solutions that rely on rules and statistical methods. Its end-to-end processing significantly improves efficiency and robustness, while supporting flexible expansion to meet diverse clinical diagnostic and research needs and provide real-time, intuitive evaluation results.

[0115] Regarding the identification of nystagmus and the acquisition of nystagmus characteristic parameters:

[0116] The acquisition of nystagmus characteristic parameters is mainly achieved by identifying the nystagmus contained in the eye movement trajectory in each direction, and then obtaining the corresponding nystagmus characteristic parameters. The identification of nystagmus can be achieved through software algorithms or artificial intelligence, which is not limited in this application. Figure 5 As shown in the figure, after nystagmus is identified based on eye movement trajectories in various directions, the identified nystagmus can be marked on the eye movement trajectory graph. For example, in the horizontal eye movement data diagram, if nystagmus occurs in the eye movement trajectory, the part of the eye movement trajectory where nystagmus occurs will be marked (a short line near the trajectory in the figure indicates nystagmus).

[0117] In an illustrative example, nystagmus identification and acquisition of nystagmus characteristic parameters include the following steps:

[0118] Obtain the motion characteristics of each data point in the eye movement data in each direction;

[0119] Determine whether there are periodic eye movements in the eye movement data based on movement characteristics;

[0120] If there are periodic eye movements in the eye movement data, the periodic eye movements in the eye movement data are identified as nystagmus;

[0121] The eye movement trajectory of the data segment identified as nystagmus is analyzed to determine the slow phase and fast phase of nystagmus, and then obtain the nystagmus characteristic parameters.

[0122] Periodic motion generally refers to regular, repetitive movements. A key characteristic of periodic motion is that it repeats over time. In nystagmus, the eye moves in a specific direction for a period of time (the slow phase), followed by a rapid return to normal position (the fast phase), before repeating the pattern again. Thus, the fast and slow phases alternate.

[0123] In the above embodiment, nystagmus identification primarily focuses on identifying periodic eye movements from eye movement trajectories. Nystagmus identification criteria include: periodic eye movements, specifically alternating fast and slow phases; preferably, the duration of the periodic eye movements is greater than a first time threshold; and / or the duration of the slow phase falls within a preset time range; and / or the acceleration of the slow phase falls within a preset acceleration interval. The slow phase has a relatively long duration and a relatively slow movement speed, while the fast phase has a shorter duration and a faster movement speed. Therefore, the combination of the time window and the velocity curve effectively distinguishes the two.

[0124] In another illustrative example, the data processor is configured to execute the following instructions to obtain nystagmus characteristic parameters:

[0125] Preprocess the eye movement data in each direction of the eye movement data, such as denoising and / or streamlining (extracting eye movement data points at equal intervals to simplify data processing);

[0126] Traverse each data point in the preprocessed eye movement trajectory data, compare the position change trend of each data point with the previous data point, and find the data point with a changed change trend as the key data point;

[0127] Based on all the key data points found, the eye movement trajectory data is divided into N data segments; each data segment starts from one key data point and ends at the next key data point; N is a positive integer greater than 1;

[0128] Calculating a slope of each data segment, and identifying, based on the calculated slopes, a first data segment having a slope greater than a first set value and a second data segment having a slope less than a second set value; wherein the first set value is greater than or equal to the second set value;

[0129] All data segments are traversed to select the first and second data segments that meet the nystagmus identification criteria as nystagmus. Among them, the first data segment in each data segment marked as nystagmus is the fast phase of nystagmus, indicating the direction of nystagmus; the second data segment is the slow phase of nystagmus, indicating the intensity of nystagmus, that is, the angular velocity of the slow phase of nystagmus.

[0130] In the above example, based on the target eye movement trajectory data after preprocessing the eye movement data, by comparing the position change trends of the previous and subsequent data points, the data points with changed position change trends are found as key data points. After all key data points are found, the adjacent key data points are connected to form line segments. The slope of each line segment is calculated to identify the first data segment and the second data segment. Preferably, after calculating the slope of each data segment, the data segment with a slope greater than the set first slope is identified as the first data segment; the data segment with a slope less than the second slope is identified as the second data segment; and then the first data segment and the second data segment that meet the nystagmus identification conditions are screened out and marked as nystagmus. Regarding the nystagmus identification conditions, as mentioned above, for example, the first data segment and the second data segment appear alternately; and / or the duration of the second data segment meets the preset time range.

[0131] In another illustrative embodiment, in addition to identifying nystagmus contained in the eye movement data in each direction, further, body position recognition processing is performed based on the collected time series body position data to obtain the specific body position and its start and end time points in the vestibular function test, and then the eye movement trajectory feature data in the corresponding time period is obtained based on the start and end time points of the specific body position to determine whether nystagmus exists in the eye movement trajectory feature data. If so, other nystagmus feature parameters under the specific body position are further obtained, including nystagmus latency, nystagmus trend changes, nystagmus duration, etc.

[0132] About the model architecture of the neural network model:

[0133] The specific network architecture of the neural network model may vary based on the data input to the model, but the overall structural framework is similar and may include: input layer (inputting external data into the network), feature extraction layer (extracting features from the input data), feature fusion layer (fusing different types of features) / feature interaction layer (capturing the correlation between different types of data / features), and output layer (outputting the final classification results or generated text information). Among them:

[0134] The neural network model of the present application can have an input layer with a single input channel, or more preferably with multiple input channels. If it is a single input channel, the input data are combined, such as combining body position data with eye movement trajectory, to form a comprehensive input format that includes body position and eye movement trajectory. Splicing or creating a structured input vector can be used. If the input layer uses multiple input channels, for example, if body position data in two directions and eye movement trajectory in three directions are used as model input, the input layer can use five input channels, with one input channel used to receive body position data / eye movement trajectory data in one direction.

[0135] After receiving input data through multiple input channels, the feature extraction layer extracts features from the data input through the multiple input channels, and the feature fusion layer performs feature fusion processing. In this application, the feature extraction layer includes an auxiliary data processing network and an eye movement data processing network as an example. The neural network model includes:

[0136] Auxiliary data processing network, used to process the input auxiliary detection data and extract auxiliary features;

[0137] Eye movement data processing network, used to process the input eye movement trajectory and extract eye movement trajectory features;

[0138] The feature fusion layer is used to fuse the auxiliary features output by the auxiliary data processing network and the eye movement trajectory features output by the eye movement data processing network to obtain comprehensive feature data related to vestibular function;

[0139] The output layer is used to generate corresponding vestibular function data based on the output data of the feature fusion layer.

[0140] Since the auxiliary detection data includes body position data and / or nystagmus characteristic parameters, the corresponding auxiliary data processing network also includes a body position data processing network and / or a nystagmus characteristic processing network; wherein:

[0141] Body posture data processing network, used to process the input body posture data and extract body posture features;

[0142] The nystagmus feature processing network is used to receive input nystagmus feature parameters.

[0143] Given that the auxiliary detection data contains different data, the input data of the model will also be different. Below we explain the neural network model architecture under different input schemes:

[0144] Taking the above-mentioned solution 1 (where the input data is body position data and eye movement trajectory) as an example, the feature extraction layer of the neural network model in this embodiment includes a body position data processing network and an eye movement data processing network; wherein:

[0145] Body posture data processing network, used to process the input body posture data and extract body posture features;

[0146] Eye movement data processing network, used to process the input eye movement trajectory and extract eye movement trajectory features;

[0147] The feature fusion layer is used to fuse and associate the body position features output by the body position data processing network and the eye movement trajectory features output by the eye movement data processing network to obtain comprehensive feature data related to vestibular function;

[0148] The output layer is used to generate corresponding vestibular function data based on the output data of the feature fusion layer.

[0149] More preferably, in an exemplary embodiment, the eye movement data processing network includes at least two sub-networks, each sub-network is used to receive and process eye movement trajectory data in one direction; specifically, for example, the eye movement data includes eye movement trajectories of the subject's pupil in three directions: horizontal eye movement trajectory, vertical eye movement trajectory, and torsional eye movement trajectory; then the eye movement trajectories in these three directions are respectively subjected to feature extraction processing by the three sub-networks of the eye movement data processing network.

[0150] In another exemplary embodiment, the body position data processing network includes at least two sub-networks, each sub-network is used to receive and process body position data in one direction; specifically, for example, the body position data of the subject's head position on the Pitch axis is subjected to feature extraction processing by one sub-network of the body position data processing network, and the body position data of the subject's head position on the Yaw axis is subjected to feature extraction processing by another sub-network of the body position data processing network.

[0151] In another embodiment of the present application, as described in Solution 3, the input of the neural network model includes three data types: eye movement trajectory data in all directions, body position data, and nystagmus characteristic parameters; wherein, the eye movement trajectory data and body position data are obtained by preliminary processing of the collected raw data, while the nystagmus characteristic parameters are higher-dimensional feature data further extracted from the eye movement trajectory data. Therefore, when using these three types of data as input, the architecture of the neural network model can be further adjusted and optimized. Specifically, the neural network model includes three network structures (these three network structures can be set to different structures according to the characteristics of the data they need to process); the body position data processing network and the eye movement data processing network can be described with reference to the previous embodiments and will not be repeated here. The nystagmus characteristic processing network is used to receive the input nystagmus characteristic parameters; then, the features output by each network are fused through the feature fusion layer, and finally the vestibular function data is output through the output layer. The feature fusion layer in this embodiment can also be implemented in different forms. The following are several examples:

[0152] (1) Example 1 of feature fusion layer

[0153] like Figure 4 As shown, the posture features in each direction output by the posture data processing network, the eye movement trajectory features in each direction output by the eye movement data processing network, and the nystagmus feature parameters in each direction output by the nystagmus feature processing network are integrated to obtain a fused overall feature vector. Finally, the vestibular function data is obtained based on the fused overall feature vector through the output layer.

[0154] (2) Feature fusion layer example 2

[0155] In this example, the feature fusion layer includes a first fusion layer and a second fusion layer. The first fusion layer is used to fuse the body posture features extracted by the body posture data processing network and the eye movement trajectory features extracted by the eye movement data processing network to obtain preliminary comprehensive feature data. This preliminary comprehensive feature data is then fused with nystagmus feature parameters in the second fusion layer. Preferably, a self-attention mechanism is also employed in the feature fusion layer. Specifically, the self-attention mechanism can be used in the first and / or second fusion layers to set weights for different features, thereby improving the accuracy of the final evaluation data.

[0156] In this example, multi-level feature fusion is used to gradually fuse low-level features (such as eye movement trajectories and body position data) with high-level features (such as nystagmus feature parameters) to achieve a more complete feature representation to support the judgment and assessment of vestibular function preparation.

[0157] The goal of multi-level feature fusion is to enable the model to gradually extract more representative high-level features from low-level raw data by integrating features at different levels. Specifically, in this example, the steps to implement multi-level feature fusion include the following:

[0158] S1, low-level feature extraction:

[0159] Extracting eye movement trajectory feature data from the eye movement trajectory data through a body position data processing network (such as a convolutional neural network and / or a recurrent neural network); extracting dynamic features of body position changes from the body position data through an eye movement data processing network (such as an LSTM or RNN) to obtain body position feature data;

[0160] S2, mid-level feature construction

[0161] In the middle layer, the low-level features are preliminarily integrated through the first fusion layer; specifically, the eye movement trajectory features in each direction of the eye movement trajectory and the body position features of the body position data are integrated to form an overall middle-level feature representation.

[0162] S3, high-level feature introduction

[0163] The nystagmus feature parameters are introduced into the model as high-level features. In the feature fusion layer, the high-level features are further fused with the middle-level features obtained in the previous step through the second fusion layer. Furthermore, a weighted fusion method can be used to enable the model to automatically adjust the importance of features according to the specific situation of the input feature data.

[0164] S4, based on the final fused feature data, passes through the fully connected layer and outputs the vestibular function assessment results.

[0165] (3) Feature fusion layer example 3

[0166] Similarly, in this example, the feature fusion layer also contains a first fusion layer and a second fusion layer, wherein: the first fusion layer is used to perform feature fusion on the eye movement trajectory features extracted by the eye movement data processing network and the nystagmus feature parameters output by the nystagmus feature processing network; the second fusion layer further fuses the data output by the first fusion layer with the posture feature data output by the posture data processing network, and finally performs evaluation and analysis based on the data fused by the second fusion layer to obtain the vestibular function data of the subject.

[0167] In this example, eye trajectory features are first extracted based on eye movement trajectories, and then fused with nystagmus feature parameters. This fully utilizes the spatiotemporal characteristics of the eye movement trajectory data, while combining them with nystagmus feature parameters to obtain a more complete eye movement feature representation. Since nystagmus feature parameters are closely related to eye movement trajectories, fusing them first helps extract more diagnostically valuable eye movement features. Fusing eye movement features first and then with body position features allows for hierarchical feature processing, reducing interference between different features and facilitating the model's extraction of unique features for each direction.

[0168] Preferably, on this basis, a self-attention mechanism is further provided in the first and / or second fusion layer, through which different weights of each feature in the fused data are set, so that the final recognition output data is more accurate.

[0169] The neural network model architecture in the above embodiment uses a feature fusion layer to perform feature fusion. In another embodiment of the present application, a feature interaction layer is used in the neural network model. The specific implementation steps are as follows:

[0170] Multiple input channels: Receive data from multiple input channels, such as eye movement data in three directions (horizontal, vertical, and torsion) and body position data in two directions (pitch and yaw).

[0171] Independent feature extraction: Preliminary feature extraction is performed on the eye movement trajectory data and body position data input by each input channel. For example, the LSTM layer is used to extract time series features, or the CNN layer is used to extract spatiotemporal features. The eye movement and body position data in each direction are extracted separately to obtain feature representations.

[0172] Feature Interaction Layer: Applying a self-attention mechanism or bilinear interaction, the extracted eye movement and body position features are interactively processed to generate new interaction features. A preferred method is bilinear interaction (multiplication): Each eye movement direction feature is element-wise multiplied with the body position feature to generate interaction features. A self-attention mechanism is then applied to the interaction features, dynamically adjusting the weight of each interaction feature to ensure the model focuses more on diagnostically meaningful feature combinations.

[0173] Feature fusion and output: The interacted features are finally fused through the feature fusion layer, input into the fully connected layer, and the vestibular function evaluation results are output.

[0174] Preferably, if the data input to the model also includes nystagmus feature parameters, then after the body position data processing network outputs body position feature data, the eye movement data processing network outputs eye movement trajectory feature data, and the nystagmus feature processing network outputs nystagmus feature parameters, the body position feature data, the eye movement trajectory feature data, and the nystagmus feature parameters are subjected to feature interaction through the feature interaction layer.

[0175] The core function of the feature interaction layer is to directly generate new interaction features between nystagmus and / or eye movement and body position features, capturing their interrelationships and dependencies. Through element-wise operations (such as multiplication) or self-attention mechanisms, the interaction layer dynamically adjusts and combines different features to generate more diagnostically meaningful combined features. Because the feature interaction layer reveals relationships between different features, it makes the model easier to understand, such as the impact of specific body positions on eye movement trajectories.

[0176] Although the feature interaction layer has provided a certain feature fusion effect, retaining the feature fusion layer can still further improve the performance of the model in some complex vestibular function assessment tasks for the following reasons:

[0177] Further integration of multi-level features: If the relationship between eye movement features and body position features is very complex, the interaction features generated by the feature interaction layer may not be sufficient to capture all the information. In this case, additional feature fusion layers can further integrate the multi-level information of the interaction features.

[0178] Introducing higher-level feature representation: After feature interaction, the feature fusion layer can be used as a higher-level processing to integrate the combination results of different interacting features to generate a more representative comprehensive feature.

[0179] Therefore, the architecture of feature interaction layer + feature fusion layer can be adopted, and the features generated by the feature interaction layer are further integrated with the body position and eye movement data, so that the model can capture more comprehensive contextual information.

[0180] The above mainly describes the main architecture of the neural network model of this application. In actual application, different models can be adopted according to actual conditions. Generally, eye movement trajectory data and auxiliary detection data contain time information, so time association or alignment can be performed later. Taking scheme one as an example, the posture data and eye movement trajectory data in the input data are both time series data. Therefore, a model suitable for processing time series data can be used, such as a long short-term memory network, a combination of a convolutional neural network and LSTM, a Transformer model, etc.

[0181] Taking Solution 3 as an example, if the input data includes: horizontal eye movement trajectory, vertical eye movement trajectory, pitch axis head position data, yaw axis head position data, horizontal nystagmus slow phase angular velocity data, and vertical nystagmus slow phase angular velocity data; the network architecture of the neural network model of this embodiment is as follows:

[0182] (1) Input layer, responsible for receiving all types of input data. We can use six input channels to receive the six types of input data mentioned above. Preferably, these input data need to be preprocessed, such as standardization or normalization, time window segmentation, etc. Each input can be a sequence of time series data (for example, eye movement trajectory and nystagmus information are sequences that change over time, and head position data are coordinate information of the time series);

[0183] (2) Feature extraction layer, which is mainly used to extract spatial and temporal features from the input data. It usually uses a convolutional layer or other network structure. The following is an example:

[0184] Eye movement trajectory data processing: For horizontal and vertical eye movement trajectory data, LSTM networks are used to capture changes in eye movements over time and extract eye movement trajectory features. Independent LSTM networks can be used for processing in each direction.

[0185] Head position data processing: For pitch and yaw axis head position data, LSTM or CNN can also be used to capture the changes in head position data over time and extract the corresponding head position features;

[0186] Nystagmus information processing: Because nystagmus information is actually higher-level feature information extracted based on eye movement trajectories, the received nystagmus information can be directly processed into the subsequent feature fusion layer without further extraction. Of course, nystagmus features can also be further extracted through CNN, or lightweight mapping processing can be performed through a layer of MLP, which is not limited in this embodiment.

[0187] Preferably, a self-attention mechanism can also be introduced in the feature extraction layer to help the model focus on key moments, such as determining the change or maintenance stage of the head position according to needs.

[0188] (3) Feature Fusion Layer: This layer fuses the features extracted by each network in the feature extraction layer, combines different types of data features, and captures the relationship between them to generate comprehensive feature data. Preferably, a self-attention mechanism can also be introduced in the feature fusion layer to dynamically adjust the weights of different features.

[0189] Specifically, we fuse features extracted from different inputs (eye movement trajectory, head position, nystagmus, etc.) and assign a dynamic weight to each feature through an attention mechanism. The attention mechanism weights different features to ensure that the model can flexibly adjust its focus on different inputs based on task requirements. For example, nystagmus feature parameters are critical for BPPV diagnosis. Therefore, if applied to the BPPV field, nystagmus feature parameters can be given a higher weight.

[0190] (4) Output layer: Based on the comprehensive feature data output by the feature fusion layer, identify and output the vestibular function test results.

[0191] The vestibular function classification data output by each of the above-described example models includes, but is not limited to, one or more nystagmus parameters indicating nystagmus, one or more diagnostic parameters indicating vestibular function, and / or one or more recommended parameters indicating vestibular rehabilitation training. The specific parameters output by the neural network model can be used to indicate or characterize corresponding vestibular function information.

[0192] Of course, if you want the model to output text information rather than a simple classification result, then the neural network model you need to use cannot simply use the above model example, but a neural network model that can generate coherent text. For example, use an encoder-decoder architecture with an attention mechanism, which can convert the model input data into detailed diagnostic text. Taking the sequence-to-sequence (Seq2Seq) model plus the attention mechanism as an example, the Seq2Seq model usually consists of two parts: an encoder and a decoder. The encoder processes the input data and the decoder generates the output text. Adding an attention mechanism allows the model to focus more on the relevant parts of the input sequence when generating each word, thereby improving the relevance and accuracy of the generated text. Specifically, the architecture example is as follows:

[0193] Sequence-to-sequence (Seq2Seq) model with attention mechanism, including:

[0194] Input layer: accepts pre-processed gyroscope and eye movement data, which may be converted into a format more suitable for model processing through feature extraction.

[0195] Encoder: Uses multiple layers of LSTM or GRU units to process time series input. The encoder learns to understand the context of the input sequence and compresses this understanding into a fixed-length context vector.

[0196] Attention Mechanism: When computing the output of the decoder at each step, the attention mechanism dynamically selects information about the parts of the input that are most relevant to the current output. This helps the model better understand which parts of the input data are most important for generating the current output vocabulary.

[0197] Decoder: Also uses LSTM or GRU units, using the encoder's context vector and the input information weighted by the attention mechanism to gradually generate each word of the diagnostic text.

[0198] Output layer: Usually a fully connected layer, which uses the softmax function to output the vocabulary probability of each step and selects the word with the highest probability as the output of the current step.

[0199] In addition to the aforementioned model architecture, the enhanced Transformer model can also be used for text generation. Because the Transformer model excels in processing sequence data, particularly in natural language processing, it is also a good choice for generating diagnostic text. The Transformer is entirely based on an attention mechanism and lacks recurrent layers, making it highly effective at parallel processing and capturing long-distance dependencies.

[0200] Enhanced Transformer model architecture example:

[0201] Input layer: Same as above, with preprocessed gyroscope and eye movement data as input.

[0202] Positional encoding: adds position information because the Transformer does not naturally handle the time series nature of the data.

[0203] Transformer encoder: Consists of multiple self-attention layers and a feed-forward network that processes input data and encodes comprehensive contextual information of the input sequence.

[0204] Transformer decoder: It also consists of multiple layers, but in addition to processing the output of the encoder, it also predicts the next word, and each prediction is based on the previous output.

[0205] Output layer: Generates the final diagnosis text, each time selecting the word with the highest probability from the decoder output.

[0206] Whether using a Seq2Seq plus attention model or a Transformer model, the key lies in effectively converting gyroscope and eye movement data into a form the model can understand, and then training the model to accurately generate text information reflecting a BPPV diagnosis. This may require a large amount of labeled data to train the model, as well as meticulous parameter tuning and optimization.

[0207] Two neural network model combination examples:

[0208] The first neural network model mainly uses a network model capable of processing time series, such as a long short-term memory network, a combination of a convolutional neural network and LSTM, a Transformer model, or any other model architecture. It is used to identify and output nystagmus information in specific body positions based on the input body position data and eye movement data.

[0209] The second neural network model mainly adopts a model capable of text processing, such as an existing language model, etc., which is used to output text information of vestibular function data based on the nystagmus information in each specific body position output by the first neural network model.

[0210] Compared with the solution of using only one model architecture to achieve text output, the combination of two neural network models in this example greatly reduces the complexity of the neural network model and the difficulty of data processing. It also reduces the difficulty of model training and improves the accuracy of model output.

[0211] The vestibular function data output by this embodiment is vestibular function text information, including but not limited to the identified nystagmus text data in each specific body position, and / or vestibular function diagnosis or evaluation text data, and / or vestibular rehabilitation training program recommendation text data. The neural network model used in this embodiment has the function of generating corresponding vestibular function text information based on input data. For example, the aforementioned sequence-to-sequence model plus attention mechanism model, enhanced version of the Transformer model, etc., of course, two neural network models can also be used in combination for processing, the latter is specifically used for text processing to simplify the complexity of the model. In view of the fact that the input data of the model contains multiple types, therefore, if multiple input channels are used to receive different types of input data, then the feature extraction of each input data and the subsequent feature fusion within the model can also refer to the previous embodiment, which will not be repeated here.

[0212] Specifically, if the trained neural network model hopes to output key nystagmus text information in order to provide doctors with a reference for vestibular function diagnosis, then during the training period, in the medical record information used as a training sample, the vestibular function examination information annotated is the characteristic nystagmus text information of this vestibular function examination.

[0213] In an illustrative embodiment, during training, the neural network model uses a large amount of different medical records to form a training sample data set, wherein a complete medical record information should include auxiliary detection data (such as posture data and / or nystagmus characteristic parameters) when the subject undergoes at least one vestibular function examination, eye movement trajectory data, and the result information of the subject's vestibular function examination given by the doctor.

[0214] Another embodiment of the present application, based on any of the above embodiments, further includes:

[0215] Performing body position recognition processing on the body position data to obtain each specific body position and its start and end time points in the vestibular function test;

[0216] According to the start and end time points of each specific body posture, the corresponding time period of the input data of the neural network model is marked; specifically, for example, if the start and end time points of the specific body posture A are detected to be the a1--a2 time period, the data with time points in the a1--a2 time period in the body posture data, and / or eye movement trajectory data, and / or nystagmus characteristic parameter data are marked, so that the subsequent neural network model can focus on the data in this time period during recognition processing and increase the recognition weight of this time period.

[0217] Of course, the above-mentioned recognition and processing of body position data can also be implemented in a variety of ways, which are not limited to this. In one exemplary implementation, a trained body position recognition neural network model is used to recognize the body position data and output the specific body position information contained in the body position data. In another exemplary implementation, the specific body position and its start and end time can be obtained by the following steps:

[0218] Extract the coordinates of each data point from the time series body position data;

[0219] Based on the coordinate data of all data points, find the consecutive data points with the same coordinates and divide them into a body position;

[0220] Obtaining the duration of the body position maintenance, and selecting the body position maintenance duration greater than a set threshold as the target body position maintenance;

[0221] Obtain the start and end time points of the target's posture, as well as the corresponding coordinate data;

[0222] According to the coordinate data of the target body posture, combined with the corresponding reference relationship between each specific body posture type and the coordinates, the specific body posture type corresponding to each target body posture is identified.

[0223] In another exemplary embodiment, if the head position data is collected by a built-in gyroscope in a head-mounted device, head position data collected by the gyroscope is processed to identify the subject's head position changes throughout the vestibular function test. Taking the right posterior Dix-Hallpike test as an example, the subject's body position (head position) changes are as follows:

[0224] Initially, the subject sits on the examination bed (sitting position). The physician holds the subject's head with both hands and rotates it 45 degrees to the right, maintaining this position (hereinafter referred to as the right-turned 45-degree position). The subject is then quickly positioned supine, with the head hanging back over the bed at an angle of 30 degrees to the horizontal. This position (hereinafter referred to as the hanging-supine position) is maintained for 30 seconds. The subject then slowly returns to a sitting position.

[0225] Throughout the entire experiment, the subject's head is wearing a headset with a built-in gyroscope. The initial position serves as the initial posture. As the subject's body position changes during the experiment, the head position data collected by the gyroscope also changes. By recognizing and processing this series of collected head position data, the subject's specific head position changes can be identified. Many existing solutions for position / posture recognition based on gyroscope data are available, which will not be detailed here.

[0226] In addition to identifying and determining specific body positions based on collected body position data, another illustrative embodiment for identifying specific body positions can identify specific body positions by filtering data segments where the body position remains unchanged for a set period of time. For example, in the right posterior Dix-Hallpike test, a specific body position primarily refers to a position that remains unchanged for a period of time after a position change. During the test, the patient's head remains unchanged after rotating 45 degrees to the right. Therefore, this 45-degree right rotation position can be considered a specific body position. More importantly, in the hanging supine position (where the head hangs back over the bed at a 30-degree angle to the horizontal plane, and the head position is maintained at 45 degrees), the doctor observes the patient for dizziness and nystagmus. The patient must maintain this position for a certain period of time. Therefore, this hanging supine position is considered a specific body position. As can be seen, the above-mentioned specific body positions are necessarily those where the body position remains unchanged, and changes in movement do not constitute specific body positions. Therefore, specific body positions can be identified by filtering for positions where the position remains unchanged for a set period of time.

[0227] Regarding the identification and determination of specific body positions, in another illustrative embodiment, in addition to the aforementioned specific body position screening based on body position holding time, further screening can be performed in conjunction with body position recognition results. Specifically, the body position data is identified to determine the body position type corresponding to each data point. Based on the starting time period corresponding to the body position where the body position holding time reaches a set time, the body position type corresponding to this time period is searched. If this body position type also belongs to the specific body position type, the body position corresponding to this time period is further determined to be a specific body position.

[0228] Preferably, after identifying each body position of the body position data, especially identifying each specific body position, the test type of this vestibular function test is identified based on the time sequence of the identified body positions / specific body positions.

[0229] In another example, a video nystagmus recorder, because it has a built-in posture sensor and a camera for capturing the user's eyes, can serve as a body position acquisition submodule to collect body position data and also capture the user's eye movement videos to obtain eye movement trajectories. Body position information is correlated and synchronized with eye movement trajectory data to obtain eye movement trajectories for each body position. Based on the eye movement trajectory corresponding to the start and end time points of the body position, characteristic nystagmus data for that specific body position can be calculated. This characteristic nystagmus data includes nystagmus type, nystagmus intensity, nystagmus latency, nystagmus duration, and nystagmus trend. Compared to the previous embodiment, which identified nystagmus only based on eye movement trajectories, this example uses the start and end time points of a specific body position to quickly identify whether nystagmus exists within that time period. If so, further nystagmus characteristic parameters are determined, thereby significantly reducing the amount of data processing and improving data processing efficiency and accuracy.

[0230] About data optimization processing:

[0231] In another embodiment of the present application, based on any of the above embodiments, the vestibular function data is nystagmus information in each specific body position in the current vestibular function test, and after the nystagmus information in each specific body position is identified by the neural network model or other algorithm, the method further includes:

[0232] The nystagmus information identified in each specific body position is subjected to any one or more of the following data optimization processes: information simplification, specialized expression processing, and structured text processing, to generate vestibular function text data. Data optimization processing steps include, but are not limited to, any one or more of the following: information simplification (e.g., simplifying data representation, filtering invalid information, merging valid information, etc.), specialized expression processing (e.g., feature visualization processing), and structured text processing.

[0233] In one illustrative embodiment, the data processor is configured to perform any one or more of the following steps to simplify the data representation:

[0234] (1) Based on the duration of the nystagmus latency in the nystagmus information in each specific body position, a classification description is performed in combination with several set nystagmus latency stages. Specifically, the time value is linked to the diagnosis, and it is no longer presented directly as a value. Instead, it is modified according to the latency duration to include no latency, less than 10 seconds, less than 40 seconds, and more than 40 seconds.

[0235] (2) Based on the duration of nystagmus in the nystagmus information under each specific body position, a classification description is performed in combination with several time stages of the set nystagmus duration; for example, the nystagmus duration is divided into: less than half a minute, less than one minute, and more than one minute.

[0236] In one illustrative embodiment, at least one data processor is configured to perform invalid information filtering by performing any one or more of the following steps:

[0237] (1) Nystagmus with an intensity lower than a set threshold is uniformly described as no significant nystagmus;

[0238] (2) Nystagmus that persisted throughout the test was uniformly described as spontaneous nystagmus;

[0239] (3) Among the nystagmus recognition results based on the eye movement data in the horizontal, vertical and torsional directions, only the nystagmus recognition results in the corresponding directions where the nystagmus intensity reaches the set threshold are retained.

[0240] In an illustrative embodiment, at least one data processing is configured to perform feature visualization processing by performing the following steps: for nystagmus identified from horizontal eye movement trajectory data, a professional description of the direction of nystagmus is provided based on the current specific body position and the relationship between the direction of nystagmus and the ground.

[0241] Specifically, in the diagnosis of horizontal semicircular canal BPPV, the nystagmus is usually described as geotropic or apogeotropic, rather than horizontal leftward or rightward, based on its direction relative to the ground. This makes it easier to associate with the diagnosis. In the right lateral decubitus position, horizontal rightward nystagmus becomes geotropic, and horizontal leftward nystagmus becomes apogeotropic. In the left lateral decubitus position, the reverse is true.

[0242] In the Dix-Hallpike examination, if typical nystagmus is present when lying down and when sitting up, it is described as the opposite direction of the nystagmus rather than the direction of the nystagmus itself.

[0243] In one illustrative embodiment, at least one data processing module is configured to perform efficient information merging by performing any one or more of the following steps:

[0244] (1) The nystagmus induced by the same diagnostic factor in different body positions is combined and expressed. For example, in the supine position and the right posterior Dix-Hallpike position, an upward nystagmus with torsion to the right is observed, with a latency of less than 10 seconds and a duration of less than half a minute.

[0245] (2) All diagnostic elements of the same diagnosis should be combined. For example, if a roll test shows leftward nystagmus in the left lateral decubitus position and rightward nystagmus in the right lateral decubitus position, the combined description should be "bilateral geotropic nystagmus on the roll test" and other features should be described later.

[0246] (3) When there are multiple diagnostic factors with different diagnoses, they should be described in order of nystagmus intensity. For example, a right posterior roll to right horizontal test will have an upward nystagmus with torsional right nystagmus, while a right horizontal roll test will have horizontal nystagmus to the left and right.

[0247] (4) Combine and describe the nystagmus information detected in the same position at different times during the complete process of vestibular function testing. Specifically, establish a connection between the entire examination process. For example, if a specific nystagmus is observed during the first roll test, describe it. On the second test, directly describe whether the nystagmus has changed in characteristics, rather than repeating the characteristics. If there is no change in nystagmus, combine the descriptions of multiple tests.

[0248] (5) Establish a correlation between diagnosis and treatment, and combine information on nystagmus characteristics before and after diagnosis and treatment. Specifically, if a patient undergoes a complete diagnosis process, then treatment, and finally a review, when combining the available information, summarize the information before treatment, summarize the characteristics during treatment, and compare the characteristics after treatment with the characteristics before treatment.

[0249] In one illustrative embodiment, structured text processing includes:

[0250] Generate a segmented report of nystagmus for each position based on the set test type and position sequence;

[0251] According to the segmented report of nystagmus corresponding to each body position and combined with the vestibular function assessment standards, a vestibular function assessment text is generated.

[0252] In this embodiment, the data optimization process described above outputs the key information required for vestibular function diagnosis while filtering out invalid data. By highlighting the core information most relevant to the diagnosis and filtering out the vast amount of complex and invalid information, the test results are clearer and more conducive to the doctor's judgment. This, in turn, improves the doctor's diagnostic efficiency and treatment accuracy, effectively reducing the possibility of missed diagnoses and misdiagnoses.

[0253] The vestibular function data processing method embodiment of the present application corresponds to the vestibular function data processing system embodiment, and the technical details of the two can be referenced to each other. To reduce repetition, they will not be described in detail.

[0254] Another embodiment of the present application provides an electronic device, including a screen, a memory, one or more data processors, and one or more programs; wherein the one or more programs are stored in the memory; when the one or more data processors execute the one or more programs, the electronic device implements the vestibular function detection data processing method of the present application. For details, please refer to the steps of the vestibular function detection data processing method of any of the above embodiments.

[0255] Another embodiment of the present application discloses a storage medium storing computer-executable instructions. When executed by a processor, the computer-executable instructions cause the processor to perform the vestibular function data processing method of any of the aforementioned method embodiments or the steps that the data processor is configured to perform in any of the system embodiments. To avoid repetition, these instructions are not further described here.

[0256] Another embodiment of the present application further discloses a computer program product. When the computer program product is run on a computer, the computer executes the vestibular function detection data processing method of any embodiment of the present application.

[0257] It should be noted that the above embodiments can be freely combined as needed. The above are only preferred embodiments of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principles of the present invention, and such improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for processing vestibular function test data, characterized in that: include: Obtain the subject's eye movement video during vestibular function testing; Extracting the subject's eye movement trajectory from the eye movement video; the eye movement trajectory includes at least one of the following: a horizontal eye movement trajectory, a vertical eye movement trajectory, and a torsional eye movement trajectory; Acquiring auxiliary detection data of the subject; the auxiliary detection data including nystagmus characteristic parameters acquired from the eye movement trajectory; The eye movement trajectory and the auxiliary detection data are input into a pre-trained neural network model, and the vestibular function data of the subject is recognized and output by the neural network model; the neural network model includes: An auxiliary data processing network, used for processing the input auxiliary detection data and extracting auxiliary features; Eye movement data processing network, used to process the input eye movement trajectory, capture the changes of eye movement over time, and extract the features of eye movement trajectory; a feature fusion layer for fusing the auxiliary features output by the auxiliary data processing network and the eye movement trajectory features output by the eye movement data processing network to obtain comprehensive feature data related to vestibular function; The output layer is used to generate corresponding vestibular function data based on the output data of the feature fusion layer.

2. The vestibular function test data processing method according to claim 1, characterized in that: The auxiliary detection data also includes: the body position data of the subject in the vestibular function test; and the vestibular function test data processing method also includes a preprocessing step: associating the eye movement trajectory with the auxiliary detection data through time synchronization.

3. The vestibular function test data processing method according to claim 2, characterized in that: When the feature fusion layer performs feature fusion on the eye movement trajectory and the auxiliary features, it includes merging different types of data features and capturing the relationship between them.

4. The vestibular function test data processing method according to claim 1, characterized in that: The acquisition of the nystagmus characteristic parameters includes: Identify nystagmus information in eye movement trajectories in various directions and obtain corresponding nystagmus characteristic parameters; the nystagmus characteristic parameters include: any one or more of the following: nystagmus slow phase angular velocity, nystagmus direction, nystagmus time, and nystagmus change trend.

5. The vestibular function test data processing method according to claim 4, characterized in that: The identification of nystagmus information in the eye movement trajectory characteristics in each direction specifically includes: Obtain the motion characteristics of each data point in the eye movement trajectory in each direction; determining whether there is periodic eye movement in the eye movement trajectory based on the movement characteristics; If periodic eye movements exist in the eye movement trajectory, the periodic eye movements in the eye movement trajectory are identified as nystagmus.

6. The vestibular function test data processing method according to claim 4, characterized in that: The identification of nystagmus information in eye movement trajectories in various directions specifically includes: Based on each data point in the eye movement trajectory in each direction, compare the position change trend of each data point with the previous data point, and find the data point with a changed change trend as the key data point; Based on all the key data points found, the eye movement data in this direction is divided into N data segments; N is a positive integer greater than 1; Calculating a slope of each data segment, and identifying, based on the calculated slopes, a first data segment having a slope greater than a first set value and a second data segment having a slope less than a second set value; wherein the first set value is greater than or equal to the second set value; All data segments are traversed to select target eye movement data including alternating first and second data segments as nystagmus; wherein the first data segment is a fast phase of nystagmus, used to indicate the direction of the nystagmus; and the second data segment is a slow phase of nystagmus, used to indicate the amplitude of the nystagmus.

7. The vestibular function test data processing method according to claim 2 or 3, characterized in that: The auxiliary data processing network of the neural network model includes a body position data processing network and a nystagmus feature processing network, wherein: The body position data processing network is used to process the input body position data and extract body position features; The nystagmus feature processing network is used to receive input nystagmus feature parameters.

8. The vestibular function test data processing method according to claim 7, characterized in that: The feature fusion layer is used to fuse the feature data output by the body position data processing network, the eye movement data processing network, and the nystagmus feature processing network; and is also used to adjust the weights of different features using a self-attention mechanism based on the fused feature vector; The output layer is used to identify and output the vestibular function data of the subject based on the data processed by the self-attention mechanism.

9. The vestibular function test data processing method according to claim 8, characterized in that: The feature fusion layer includes a first fusion layer and a second fusion layer; wherein: If the feature fusion layer adopts the first fusion architecture, then: The first fusion layer is used to perform feature fusion on the body posture features output by the body posture data processing network and the eye movement trajectory features output by the eye movement data processing network; The second fusion layer is used to fuse the nystagmus feature parameters output by the nystagmus feature processing network with the data output by the first fusion layer, and dynamically adjust the weights of different features using a self-attention mechanism; If the feature fusion layer adopts the second fusion architecture, then: The first fusion layer is used to perform feature fusion on the eye movement trajectory features output by the eye movement data processing network and the nystagmus feature parameters output by the nystagmus feature processing network; The second fusion layer is used to perform feature fusion on the posture features output by the posture data processing network and the data output by the first fusion layer, and adopt a self-attention mechanism to dynamically adjust the weights of different features.

10. The vestibular function test data processing method according to claim 2 or 3, characterized in that: The vestibular function data output by the neural network model is the nystagmus information of the subject in each specific body position during the vestibular function test; The vestibular function detection data processing method further includes: Data optimization and structured text processing are performed on the nystagmus information in each specific body position to generate vestibular function text data.

11. The vestibular function test data processing method according to claim 10, characterized in that: The data optimization process includes any one or more of the following: Based on preset screening rules, redundant information irrelevant to vestibular function diagnosis is removed; Based on the preset feature threshold, invalid data that does not meet the nystagmus characteristics are eliminated; Generate a corresponding classification description based on the duration of the nystagmus latency period in each specific body position identified and combined with several set duration stages of the nystagmus latency period; Generate a corresponding classification description based on the duration of nystagmus in each specific position identified and several stages of nystagmus duration set; Generate a corresponding classification description based on the identified slow phase angular velocity of nystagmus in each specific body position and the set minimum threshold of the slow phase angular velocity of nystagmus; The same type of nystagmus information is merged and processed.

12. An electronic device comprising a screen, a memory, one or more data processors, and one or more programs; wherein, The one or more programs are stored in the memory; it is characterized in that when the one or more data processors execute the one or more programs, the electronic device implements the vestibular function detection data processing method according to any one of claims 1 to 11.

13. A computer program product, characterized in that When the computer program product is run on a computer, the computer is enabled to execute the vestibular function testing data processing method according to any one of claims 1 to 11.

14. A vestibular function detection system, characterized in that: include: Eye movement capture module, eye movement trajectory acquisition module, auxiliary data acquisition module, intelligent recognition module; among which: The eye movement capture module is equipped with at least one camera for capturing eye movement videos of the subject during the vestibular function test; an eye movement trajectory acquisition module, communicatively connected to the eye movement capture module, configured to receive the eye movement video transmitted by the eye movement capture module and extract the eye movement trajectory of the subject therefrom; the eye movement trajectory comprising at least one of the following: a horizontal eye movement trajectory, a vertical eye movement trajectory, and a torsional eye movement trajectory; The auxiliary data acquisition module is used to acquire auxiliary detection data of the subject; the auxiliary detection data includes nystagmus characteristic parameters; the auxiliary data acquisition module includes an eye movement processing submodule; wherein: The eye movement processing submodule is in communication with the eye movement trajectory acquisition module, and is configured to receive the eye movement trajectory transmitted by the eye movement trajectory acquisition module and acquire corresponding nystagmus characteristic parameters from the eye movement trajectory; The intelligent recognition module has a built-in pre-trained neural network model, and is respectively connected to the eye movement trajectory acquisition module and the auxiliary data acquisition module for inputting the eye movement trajectory and the auxiliary detection data into the neural network model to obtain the vestibular function data of the subject; wherein: the neural network model includes: An auxiliary data processing network, used for processing the input auxiliary detection data and extracting auxiliary features; Eye movement data processing network, used to process the input eye movement trajectory, capture the changes of eye movement over time, and extract the features of eye movement trajectory; a feature fusion layer, configured to fuse the auxiliary features output by the auxiliary data processing network and the eye movement trajectory features output by the eye movement data processing network to obtain comprehensive feature data related to vestibular function; The output layer is used to generate corresponding vestibular function data based on the output data of the feature fusion layer.

15. The vestibular function detection system according to claim 14, characterized in that: The auxiliary detection data also includes the body position data of the subject; the auxiliary data acquisition module also includes a body position acquisition submodule; The body position acquisition submodule is used to obtain the body position data of the subject in the vestibular function test; The vestibular function detection system also includes: A preprocessing module is used to associate the eye movement trajectory with the auxiliary detection data through time synchronization.

16. The vestibular function testing system according to claim 14, characterized in that: The eye movement capture module is arranged in an eye movement acquisition device, and the eye movement acquisition device adopts any product form among a wearable eye movement acquisition instrument, a head-mounted virtual reality or augmented reality device, and a non-wearable capture device.

Images