Vestibular function detection system, data processing method, electronic equipment and program product

Abstract

The invention discloses a vestibular function detection system, a data processing method, electronic equipment and a program product. The method comprises the following steps: acquiring an eye movement video of a subject in vestibular function detection; extracting an eye movement track of the subject from the eye movement video; the eye movement track comprises one or more of a horizontal direction eye movement track, a vertical direction eye movement track and a twisting direction eye movement track; acquiring auxiliary detection data of the subject; the auxiliary detection data comprises body position data of the subject in vestibular function detection and / or nystagmus characteristic parameters obtained from eye movement tracks; inputting the eye movement track and the auxiliary detection data into a trained neural network model, and identifying and outputting vestibular function data of the subject through the neural network model; wherein the eye movement track is associated with the auxiliary detection data through time synchronization. Compared with only considering eye movement video or eye movement track data, body position data or nystagmus characteristic parameters, the scheme of the application enables the evaluated vestibular function data to be more accurate.

Description

Technical Field

[0001] The present invention relates to the field of medical technology, and in particular to a vestibular function detection system and a data processing method, an electronic device and a program product. Background Art

[0002] Vestibular dysfunction is a common manifestation of balance disorder in clinical practice, and its detection and evaluation are of great significance for the diagnosis and treatment of patients. Vestibular function can be evaluated by recording and analyzing the patient's eye movement trajectory through eye movement video. For example, the nystagmus view currently used in clinical practice, however, the information provided by eye movement videos or trajectories has great limitations and cannot intuitively reflect the core conclusions of the examination. And because the eye movement trajectory contains a lot of complex information, and even a lot of additional information that is irrelevant to the diagnosis, it is difficult to accurately reflect the patient's actual vestibular function status through manual observation or simple analysis alone. At the same time, vestibular function testing is often accompanied by multiple factors such as changes in body position, and analysis of a single data source may not provide a comprehensive basis for diagnosis. Summary of the invention

[0003] In order to solve the above technical problems, the present invention provides a vestibular function detection system and data processing method, electronic equipment, and computer program product.

[0004] In a first aspect, the present application discloses a method for processing vestibular function test data, comprising: obtaining an eye movement video of a subject in a vestibular function test; extracting the subject's eye movement trajectory from the eye movement video; the eye movement trajectory comprising at least one of the following: a horizontal eye movement trajectory, a vertical eye movement trajectory, and a torsional eye movement trajectory; obtaining the subject's auxiliary test data; the auxiliary test data comprising the subject's body position data in the vestibular function test and / or eye movement feature data obtained from the eye movement trajectory; inputting the eye movement trajectory and the auxiliary test data into a pre-trained neural network model, and outputting the subject's vestibular function data through the neural network model recognition; and comparing the eye movement trajectory and the auxiliary test data. Preferably, the eye movement trajectory is associated with the auxiliary detection data through time synchronization; the neural network model includes: an auxiliary data processing network, which is used to process the input auxiliary detection data and extract auxiliary features; an eye movement data processing network, which is used to process the input eye movement trajectory and extract eye movement trajectory features; a feature fusion layer, which is used to fuse the auxiliary features output by the auxiliary data processing network and the eye movement trajectory features output by the eye movement data processing network to obtain comprehensive feature data related to vestibular function; an output layer, which is used to generate corresponding vestibular function data according to the output data of the feature fusion layer.

[0005] In some embodiments, the auxiliary detection data input into the neural network model includes at least eye movement feature data, which is nystagmus feature parameters; the acquisition of nystagmus feature parameters includes: identifying nystagmus information in eye movement trajectories in various directions, and obtaining corresponding nystagmus feature parameters; nystagmus feature parameters include: any one or more of nystagmus slow phase angular velocity, nystagmus direction, nystagmus time, and nystagmus change trend.

[0006] In some embodiments, identifying nystagmus information in the characteristics of eye movement trajectories in each direction specifically includes: obtaining motion characteristics of each data point in the eye movement trajectory in each direction; judging whether there are periodic eye movements in the eye movement trajectory based on the motion characteristics; if there are periodic eye movements in the eye movement trajectory, identifying the periodic eye movements in the eye movement trajectory as nystagmus.

[0007] In some embodiments, identifying nystagmus information in eye movement trajectories in various directions specifically includes: based on each data point in the eye movement trajectory in each direction, comparing the position change trend of each data point with that of the previous data point, and finding the data point whose change trend changes as the key data point; according to all the key data points found, dividing the eye movement data in this direction into N data segments; N is a positive integer greater than 1; calculating the slope of each data segment, and according to the calculated slope, identifying a first data segment whose slope is greater than a first set value and a second data segment whose slope is less than a second set value; wherein the first set value is greater than or equal to the second set value; traversing all data segments, screening out target eye movement data containing alternating first data segments and second data segments as nystagmus; wherein the first data segment is a fast phase of nystagmus, used to indicate the direction of nystagmus; and the second data segment is a slow phase of nystagmus, used to indicate the amplitude of nystagmus.

[0008] In some embodiments, the auxiliary detection data input into the neural network model includes the subject's body position data or nystagmus characteristic parameters; the auxiliary data processing network of the neural network model includes a body position data processing network or a nystagmus characteristic processing network; wherein: the body position data processing network is used to process the input body position data and extract body position features; the nystagmus characteristic processing network is used to receive the input nystagmus characteristic parameters.

[0009] In some embodiments, the eye movement feature data input into the neural network model includes body position data and nystagmus feature parameters; the auxiliary data processing network of the neural network model includes: a body position data processing network and a nystagmus feature processing network; wherein: the body position data processing network is used to process the input body position data and extract body position features; the nystagmus feature processing network is used to receive input nystagmus feature parameters; the feature fusion layer is used to perform feature fusion on the feature data output by the body position data processing network, the eye movement data processing network and the nystagmus feature processing network; the feature fusion layer is also used to adjust the weights of different features based on the fused feature vector using a self-attention mechanism; the output layer is used to identify and output the vestibular function data of the subject based on the data processed by the self-attention mechanism.

[0010] In some embodiments, the feature fusion layer includes a first fusion layer and a second fusion layer; wherein: if the feature fusion layer adopts the first fusion architecture, then: the first fusion layer is used to perform feature fusion on the body position features output by the body position data processing network and the eye movement trajectory features output by the eye movement data processing network; the second fusion layer is used to fuse the nystagmus feature parameters output by the nystagmus feature processing network with the data output by the first fusion layer, and adopt a self-attention mechanism to dynamically adjust the weights of different features; if the feature fusion layer adopts the second fusion architecture, then: the first fusion layer is used to perform feature fusion on the eye movement trajectory features output by the eye movement data processing network and the nystagmus feature parameters output by the nystagmus feature processing network; the second fusion layer is used to perform feature fusion on the body position features output by the body position data processing network and the data output by the first fusion layer, and adopt a self-attention mechanism to dynamically adjust the weights of different features.

[0011] In some embodiments, the vestibular function data output by the neural network model is the nystagmus information of the subject in each specific body position during the vestibular function test; the vestibular function test data processing method also includes: performing data optimization processing and structured text processing on the identified nystagmus information in each specific body position to generate vestibular function text data.

[0012] In some embodiments, data optimization processing includes any one or more of the following: based on preset screening rules, removing redundant information irrelevant to vestibular function diagnosis; based on preset feature thresholds, eliminating invalid data that does not conform to nystagmus characteristics; generating corresponding classification descriptions based on the duration of nystagmus latency in each specific body position identified, combined with several set duration stages of nystagmus latency; generating corresponding classification descriptions based on the duration of nystagmus in each specific body position identified, combined with several set duration stages of nystagmus duration; generating corresponding classification descriptions based on the slow phase angular velocity of nystagmus in each specific body position identified, combined with a set minimum threshold for the slow phase angular velocity of nystagmus; merging nystagmus information of the same type.

[0013] In a second aspect of the present application, an electronic device is disclosed, comprising a screen, a memory, one or more data processors, and one or more programs; wherein the one or more programs are stored in the memory; when the one or more data processors execute the one or more programs, the electronic device implements any of the above-mentioned vestibular function detection data processing methods.

[0014] In a third aspect of the present application, a storage medium is disclosed, which stores computer instructions. When the computer instructions are executed by a data processor, the data processor executes the steps of any one of the above-mentioned methods for processing vestibular function detection data.

[0015] In a fourth aspect of the present application, a computer program product is disclosed. When the computer program product is run on a computer, the computer is enabled to execute any one of the steps of the vestibular function detection data processing method described above.

[0016] In a fifth aspect of the present application, a vestibular function detection system is disclosed, including: an eye movement shooting module, an eye movement trajectory acquisition module, an auxiliary data acquisition module, and an intelligent recognition module; wherein: the eye movement shooting module is equipped with at least one camera for collecting eye movement videos of subjects during vestibular function tests; the eye movement trajectory acquisition module is communicatively connected to the eye movement shooting module, and is used to receive eye movement videos transmitted by the eye movement shooting module and extract eye movement trajectories of subjects therefrom; the eye movement trajectories include at least one of the following: horizontal eye movement trajectories, vertical eye movement trajectories, and torsional eye movement trajectories; the auxiliary data acquisition module is used to obtain auxiliary detection data of the subjects; the auxiliary detection data includes the subject's body position data and / or nystagmus characteristic parameters; the auxiliary data acquisition module includes a body position acquisition submodule and / or an eye movement processing submodule; wherein: the body position acquisition submodule is used to obtain the subject's body position data in vestibular function detection; the eye movement processing submodule is communicatively connected to the eye movement trajectory acquisition module, and is used to The invention relates to a smart recognition module for receiving the eye movement trajectory transmitted by the eye movement trajectory acquisition module and obtaining the corresponding nystagmus characteristic parameters from the eye movement trajectory; an intelligent recognition module having a pre-trained neural network model built therein, the intelligent recognition module being respectively connected to the eye movement trajectory acquisition module and the auxiliary data acquisition module in communication, and being used to input the eye movement trajectory and the auxiliary detection data into the neural network model to obtain the vestibular function data of the subject; wherein the neural network model comprises: an auxiliary data processing network for processing the input auxiliary detection data and extracting auxiliary features; an eye movement data processing network for processing the input eye movement trajectory and extracting eye movement trajectory features; a feature fusion layer for fusing the auxiliary features output by the auxiliary data processing network and the eye movement trajectory features output by the eye movement data processing network to obtain comprehensive feature data related to the vestibular function; and an output layer for generating corresponding vestibular function data according to the output data of the feature fusion layer.

[0017] In some embodiments, the eye movement capture module is disposed in an eye movement acquisition device, and the eye movement acquisition device adopts any product form of a wearable eye movement acquisition instrument, a head-mounted virtual reality or augmented reality device, or a non-wearable capture device.

[0018] Compared with the prior art, the present invention has at least one of the following beneficial effects: 1. This application identifies nystagmus information in each specific body position based on eye movement data and auxiliary detection data (body position data and / or nystagmus characteristic parameters). Compared with only considering eye movement video or eye movement trajectory data, the addition of body position data or nystagmus characteristic parameters makes the vestibular function data evaluated more accurate.

[0019] 2. The vestibular function detection system of the present application, combined with artificial intelligence technology, realizes the fusion and recognition of the input auxiliary detection data and eye movement trajectory data through a trained neural network model. In particular, the neural network model mainly adopts a model architecture that can process time series data, so that it can effectively obtain the correlation and causal dependency of time series data (such as body position data and eye movement trajectory data), capture the complex patterns in the body position data and eye movement trajectory data, and make better predictions; better, in terms of model input, in addition to body position data and eye movement data in various directions, the corresponding nystagmus characteristic parameters obtained based on the eye movement trajectory data in various directions are also input, especially the slow phase angular velocity of nystagmus, thereby greatly improving the accuracy of the output vestibular function data.

[0020] 3. The vestibular function data output by the vestibular function detection system of the present application may be text information. Compared with traditional parameter data types, the output vestibular function text information is more easy to understand and removes a lot of complicated information irrelevant to the diagnosis, and only presents the core diagnostic information of characteristic nystagmus in a specific body position. In particular, it is presented in the form of text, so that medical workers who do not need professional training can understand it directly and can quickly obtain key clues for vestibular function diagnosis without comparing and checking complex curves. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The preferred implementation modes will be described below in a clear and understandable manner with reference to the accompanying drawings to further illustrate the above-mentioned characteristics, technical features, advantages and implementation methods of the present invention.

[0022] Figure 1 is a structural block diagram of an embodiment of the vestibular function detection system of the present application; Figure 2 is a flow chart of an embodiment of a method for processing vestibular function detection data of the present application; Figure 3is a schematic diagram showing body position data in one embodiment of the present application; Figure 4 is a schematic diagram of the architecture of a neural network model in one embodiment of the present application; Figure 5 It is a schematic diagram showing the eye movement trajectory and head position data of a subject in one embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the specific implementation methods of the present invention will be described below with reference to the accompanying drawings. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings and other implementation methods can be obtained based on these drawings without creative work.

[0024] In order to simplify the drawings, only the parts related to the invention are schematically shown in each figure, and they do not represent the actual structure of the product. In addition, in order to simplify the drawings and facilitate understanding, in some figures, only one of the parts with the same structure or function is schematically drawn or marked. In this article, "one" not only means "only one", but also means "more than one".

[0025] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0026] The present invention is described in an exemplary manner with reference to a computer system architecture and an exemplary process performed by a computer system. In one or more embodiments, the functions described herein may be implemented by computer system instructions. These computer program instructions may be directly loaded onto an internal data storage device of a computing device (e.g., an internal data storage device of a smart phone or laptop computing device). Alternatively, these computer program instructions may be stored on a portable computer-readable medium (e.g., a flash drive, etc.) and then subsequently loaded onto a computing device so that the instructions may be executed thereby. In other embodiments, these computer program instructions may be embodied in the hardware of the computing device rather than in its software. Computer program instructions may also be embodied in a combination of both hardware and software. In addition, in the present disclosure, when referring to a computing device that is "configured to", "arranged to" and / or "configured and arranged to" perform a specific function (e.g., a data acquisition / data processing device) is configured and arranged to perform a specific function, it should be understood that in one or more embodiments of the present invention, this means that the computing device is specifically programmed to perform a specific function (e.g., a data acquisition / data processing device is specifically programmed to perform a specific function).

[0027] This description describes in general form the computer program required for the analysis of vestibular function test information. Any competent programmer in the field of information technology can use the description set forth in this article to develop a system.

[0028] For the sake of brevity, traditional computer system components, traditional data networks, and traditional software coding will not be described in detail here. In addition, it should be understood that the connecting lines shown in the block diagrams included herein are intended to represent the functional relationships and / or operational couplings between the various components. Except as explicitly described, it should be understood that many alternative or additional functional relationships and / or physical connections can be incorporated into the actual application of the system.

[0029] Vestibular function testing evaluates the vestibular function of the semicircular canals, otoliths, or central vestibular system by observing and analyzing the vestibular system's response to movement, position changes, and balance control. There are many methods for testing vestibular function, such as the Dix-Hallpike test, video head impulse test (vHIT), Roll Test, Romberg Test, etc., to help diagnose vestibular-related diseases (such as vertigo, balance disorders, benign paroxysmal positional vertigo, etc.) and develop treatment plans. Let's take otolithiasis testing as an example. Currently, clinical practice relies more on nystagmus viewing to assist in the diagnosis and treatment of BPPV. The nystagmus during the complete examination is recorded by video. Manual reduction and rotation chair reduction can be achieved by having the patient wear an infrared video goggles with a gyroscope to achieve the function of nystagmus viewing.

[0030] The information provided by the nystagmus view currently used in clinical practice has great limitations and cannot directly reflect the core conclusions of the examination. Although the nystagmus view can provide complete video playback and analyze the eye movement trajectory obtained based on video analysis, this information is very complex and contains a lot of additional information that is irrelevant to the diagnosis. The examination and treatment of a BPPV patient may involve more than ten examinations and treatment actions, hundreds of branch actions, and dozens of minutes of examination data. If the operator needs to review the specific situation of the subject, it is still necessary to fully replay the nystagmus information, curves or videos of each body position. The core diagnostic information that doctors are most concerned about cannot be obtained directly by reading the result data. Even doctors who have undergone complex professional training need to view complex curves and videos before they can obtain the final diagnostic conclusion.

[0031] The present application provides a method and system for processing vestibular function test data, which uses artificial intelligence combined with comprehensive consideration of multi-source data to make vestibular function assessment more intelligent and accurate.

[0032] The vestibular function detection system of the present application is as follows Figure 1 As shown, it includes the following functional modules: The eye movement shooting module 10 includes at least one camera for collecting eye movement videos of the subject in the vestibular function test; The eye movement trajectory acquisition module 20 is in communication connection with the eye movement shooting module 10, and extracts eye movement trajectory data based on the eye movement video captured by the camera, and the eye movement trajectory data includes eye movement trajectory in at least one direction of horizontal direction, vertical direction and twisting direction; of course, the eye movement trajectory acquisition module 20 for acquiring eye movement trajectory based on eye movement video can be integrated into any device such as an eye movement acquisition device, a data processing device, or a cloud server, and is processed and implemented by the data processor of the device. For example, after the camera of the eye movement tracking device captures the eye movement video, the data processor of the eye movement tracking device extracts eye movement trajectory features from the eye movement video.

[0033] The auxiliary data acquisition module 30 is used to acquire auxiliary detection data of the subject, wherein the auxiliary detection data includes the body position data and / or nystagmus characteristic parameters of the subject; the module includes at least one of the following: The body position acquisition submodule 31 is used to obtain the body position data of the subject in the vestibular function test; preferably, it includes: a body position acquisition unit and a body position processing unit; wherein the body position acquisition unit is used to acquire the body position data of the subject, such as acquiring the head position data of the subject through a gyroscope or an inertial measurement unit; and the body position processing unit pre-processes the body position data acquired by the body position acquisition unit, such as denoising, data reduction, coordinate conversion, etc., to obtain body position data that meets the subsequent neural network model input; if the acquired body position data is video data, image processing is required to extract the subject's body position change trajectory from the body position image video frame, that is, what we finally input into the neural network model can be one-dimensional data or two-dimensional data, but not three-dimensional video data.

[0034] The eye movement processing submodule 32 is used to identify nystagmus in the eye movement trajectory of the subject and obtain corresponding nystagmus characteristic parameters.

[0035] The intelligent recognition module 40 is equipped with a pre-trained neural network model 41. The intelligent recognition module is respectively connected to the eye movement trajectory acquisition module 20 and the data acquisition module 30 for inputting the eye movement trajectory and the auxiliary detection data into the neural network model 41 to obtain the vestibular function data of the subject; preferably, the eye movement trajectory is temporally associated with the auxiliary detection data. Specifically, the intelligent recognition module 40 identifies the nystagmus in each specific body position through the neural network model 41, and then obtains the vestibular function data of the subject. The specific body position refers to the specified body position in the vestibular function test. The specific body position will be different for different test items. In addition, even for the same test item, there may be multiple different specific body positions, which depends on the vestibular function test item and can also be directly identified by the neural network model (when training the model, the specific body position needs to be marked for learning).

[0036] Among the above modules, the eye movement trajectory acquisition module 20, the eye movement processing submodule 32, the body position processing unit, the intelligent recognition module 40, etc. are all data processing function modules, and can realize corresponding functions by executing corresponding instructions through one or more data processors.

[0037] In addition, based on the above, the system can also include the following data processing function modules: The pre-processing sub-module is used to perform data cleaning, data conversion (such as formatting) and other operations on the collected eye movement trajectory data or body position data; A data synchronization submodule is used to synchronize the timestamps of the eye movement trajectory and the auxiliary detection data; The post-processing submodule is used to receive the output results of the neural network model and perform post-processing on them, such as converting the output results into structured data for subsequent display or storage.

[0038] In this embodiment, each data processing functional module can be integrated into at least one data processor, that is, data processing and model reasoning are completed in the same processor, thereby simplifying the system architecture and reducing the complexity of module interaction. Of course, it is better to separate the intelligent recognition module with a built-in neural network model and other data processing functional modules through different processors for execution and implementation, with one data processor responsible for data preprocessing (such as synchronization, cleaning, formatting) and post-processing (such as result display, storage, etc.), which is the logical module responsible for general data operations in the system; and another data processor integrated with a neural network model focuses on complex feature extraction, fusion and prediction tasks, and is a dedicated analysis and reasoning module; the two together realize the processing of vestibular function detection data.

[0039] Furthermore, the vestibular function detection data processing system also includes: Display device: used to display data processing status, including eye movement trajectory curves, vestibular function data of subjects (such as nystagmus information in specific body positions, vestibular function assessment results, rehabilitation training recommendation plans, etc.), etc.

[0040] The storage module is used to save the results of data processing, such as the evaluation report generated by the final analysis of the model.

[0041] In the above system embodiment, each module (including submodules) can be set or integrated in the same or different devices, and the function of each functional module can be realized by configuring the data processor of the device where each functional module is located. Generally, each module of the vestibular function detection system can be basically set in an acquisition device (such as an eye movement acquisition device, a body position acquisition device), a data processing terminal device and / or a server; the following are several exemplary descriptions: An eye movement capture module can be set in the eye movement acquisition device to collect the eye movement video of the subject; The eye movement trajectory acquisition module can be set in any device among the eye movement acquisition device, the data processing terminal device, and the server; The body position acquisition submodule, wherein the body position acquisition unit can be set in the body position acquisition device or integrated in the eye movement acquisition device to collect the body position data of the subject; the body position processing unit can be set in the data processing terminal device or the server.

[0042] The eye movement processing submodule can be set in any device among the eye movement acquisition device, the data processing terminal device, and the server; The intelligent recognition module can be set in any device among the eye movement acquisition device, the data processing terminal device and the server.

[0043] It is worth noting that the above-mentioned body position acquisition device can be a device independent of the eye movement acquisition device. For example, the body position acquisition device uses a device that drives the subject's body position change. The body position acquisition device and the eye movement acquisition device can also be combined into one, and the two are integrated into the same device. For example, through a head-mounted device with a built-in gyroscope and camera, the subject's head movement information and eye movement video can be collected respectively. For example, through an external adjustable camera, the camera is set near the subject through a bracket to shoot the subject's body position change video and eye movement video during the vestibular function test, and then the video data is processed to obtain the corresponding body position data (not video data, but body position trajectory data in all directions) and eye movement trajectory data.

[0044] The above-mentioned data processing terminal device can be a terminal device with data processing function, such as a computer, a tablet computer, a mobile phone, a medical terminal or a platform device. The server can be a local server or a cloud server.

[0045] Preferably, in one example, the vestibular function detection system is composed of an eye tracker, a proximal data processing terminal device and a cloud server. The eye tracker is equipped with at least one camera and at least one motion sensor (such as a gyroscope) for collecting eye movement video and head position change data, and transmitting them to the proximal data processing terminal device. The proximal data processing terminal device processes the received data, such as extracting eye movement trajectory data in each direction from the eye movement video, pre-processing the head position data, and sending it to the cloud server after making it conform to the input format of the neural network model. The cloud server intelligently identifies the nystagmus information in each specific body position (specified body position) through the neural network model, obtains the vestibular function data of the subject and feeds it back to the proximal data processing terminal device.

[0046] Of course, if the input source of the model also includes nystagmus characteristic parameters, the step of obtaining the nystagmus characteristic parameters can be set in the proximal data processing terminal device or in the cloud server, which is not limited in this embodiment.

[0047] In this system embodiment, the proximal data processing terminal is close to the data collection source and can process the collected data, reduce irrelevant information, reduce the amount of data, focus on information content, save transmission bandwidth, speed up transmission speed, and improve transmission quality. At the same time, the eye movement trajectory transmitted to the cloud server does not contain privacy information such as pupil images, which effectively prevents privacy leakage and ensures personal data security.

[0048] The proximal data processing terminal device is located between the data acquisition end (eye tracker) and the cloud server, and plays a role in data processing and transfer. From the perspective of hardware composition, it usually has a certain data storage capacity, arithmetic processing unit and network communication module. Taking the common medical testing scenario as an example, the proximal data processing terminal device can be a medical workstation computer with moderate configuration. It synchronizes data with the eye tracker in real time through wired or wireless connection. When the patient completes the test, the workstation computer quickly processes the data, converts the original eye movement video into eye movement trajectory data in all directions, pre-processes the head position change data, and then sends the processing results to the cloud server through the wireless network for in-depth analysis. In this way, it can not only meet the needs of grassroots medical units for rapid processing of local data, but also use the powerful computing power of the cloud to achieve more accurate vestibular function assessment, while ensuring the efficiency and security of data transmission. Of course, for large medical institutions, local servers can also be used to achieve rapid vestibular function assessment without connecting to the external network.

[0049] Another embodiment of the present application provides a vestibular function detection system, including: at least one data processor (for example, a data processor of a computing device such as a mobile device); at least one body position acquisition submodule and at least one eye movement shooting module (including at least one camera); the body position acquisition submodule and the eye movement shooting module are operably coupled to the data processor, and the data processor can be configured to perform the steps of the vestibular function detection data processing method; specifically including: Receiving the subject's body position data and eye movement video in vestibular function testing; specifically, for example, receiving the eye movement video collected by the eye movement shooting module and the body position data collected by the body position acquisition submodule; Based on the eye movement video, extracting the subject's eye movement data therefrom, the eye movement data including the eye movement trajectory of the pupil in at least one of the three directions of horizontal direction, vertical direction and torsional direction; The eye movement trajectory and body position data are input into a trained neural network model, and the vestibular function data of the subject is output through the neural network model recognition; wherein the eye movement trajectory and body position data are associated through time synchronization.

[0050] In this embodiment, the output vestibular function data is processed based on body position data and eye movement data. Compared with the conventional method of predicting vestibular function data only based on eye movement video or eye movement trajectory, this embodiment closely combines the body position information of the subject during vestibular function testing. The fusion of body position information greatly improves the accuracy of vestibular function assessment. Let's take benign paroxysmal positional vertigo as an example. When conducting a position test, it is necessary to change the subject's body position based on the test type (such as the Dix-Hallpike test), and the doctor observes whether there is characteristic nystagmus in a specific position. For example, when diagnosing the right posterior semicircular canal BPPV canalolithic type, it is necessary to observe the nystagmus of the upper pole of the eyeball twisting to the right in the right posterior Dix-Hallpike position before it can be determined.

[0051] In this embodiment, the body position acquisition submodule is mainly used to collect the body position data of the subject; the eye movement shooting module includes at least one camera for collecting the eye movement video of the subject; the data processor is configured to receive the eye movement video and extract the eye movement trajectory of the subject's pupil in at least one of the three directions of horizontal, vertical and torsion; the data processor or another data processor is configured to receive the body position data, and input the body position data and the extracted eye movement trajectory into the trained neural network model, identify the nystagmus information in each specific body position through the neural network model, and output the vestibular function examination information.

[0052] In an example of a posture acquisition submodule, the posture acquisition submodule includes at least one of an accelerometer configured to detect linear acceleration and a gyroscope configured to detect angular velocity. For example, in an illustrative embodiment, the posture acquisition submodule may include a 3-axis accelerometer, a 3-axis gyroscope, and a 3-axis magnetometer to implement the use of a motion fusion algorithm. Of course, posture sensors such as linear accelerometers or gyroscopes can generally be integrated into the head-mounted device to facilitate the acquisition of changes in the subject's head position.

[0053] In another exemplary embodiment, the body position acquisition submodule is a body position change conversion device, such as a BPPV vertigo diagnosis and treatment swivel chair. During the detection or resetting process, the BPPV vertigo diagnosis and treatment swivel chair will transmit the head position change data of the subject at each time point in the process to the data processor for body position recognition, etc. For example, Figure 3 The figure shows the body position (head position) data curve of a subject sitting on a BPPV swivel chair, including the trajectory data of the subject's head position on the Pitch axis and the trajectory data on the Yaw axis.

[0054] In another exemplary embodiment, the body position acquisition submodule is a camera, which can be set near the subject by means of a fixed bracket or other equipment to capture images of the subject's body position changes during the test, and then perform image processing to obtain the subject's body position data corresponding to each video frame (corresponding to different time points).

[0055] In one embodiment, a device integrated with an eye movement capture module (having an eye movement capture function) may adopt any of the following product forms: A wearable eye movement acquisition device, which has the eye movement capture module built in and is convenient for the subject to wear on the head so that the eye movement capture module can capture the subject's eyes, and can communicate with external devices through wired or wireless communication protocols (such as Bluetooth or Wi-Fi, etc.) to transmit the collected eye movement video or processed eye movement trajectory to the device where the subsequent data processing module is located; Head-mounted virtual reality (VR) / augmented reality (AR) devices integrate eye movement acquisition functions, using their built-in sensor arrays and camera combinations to capture user eye movements and communicate with external independent data processing devices through the device's own high-speed internal bus; The eye movement acquisition device is fixed on the detection seat headrest position, and uses a multi-angle adjustable bracket to fix the camera to accommodate subjects of different heights and sitting postures. It directly transmits data to the nearest desktop computer as the device where the data processing module is located via a wireless network connection or a wired network connection.

[0056] The mobile device with a data processor is selected from the group consisting of: (i) a smart phone, (ii) a tablet computing device, (iii) a laptop computing device, (iv) a smart watch, and (v) a head-mounted display. For example, in an illustrative embodiment, the body position acquisition submodule and / or the eye movement capture module can be a built-in position sensor and / or camera of a video nystagmus. The video nystagmus is generally wearable in front of the user's eyes, has a built-in camera, can collect the user's eye movement video, and is also integrated with a position sensor or a motion sensor (such as a gyroscope) to collect the user's head movement body position data. In another illustrative embodiment, another type of computing device is used instead of a mobile computing device. For example, the other type of computing device can be a desktop computing device, a server computing device, or a small personal computer. In yet another illustrative embodiment, the user's body position change data can be obtained by controlling the swivel chair change data output by a control unit that controls the movement of the swivel chair in which the user sits, and the user's eye video data can be collected by the camera of the video nystagmus worn on the user's head.

[0057] In an illustrative embodiment, after collecting the eye movement video of the subject during the vestibular function test, a data processor (eye movement trajectory acquisition module) is used to extract the eye movement trajectory of the video to obtain the eye movement trajectory data of the subject's pupil in the horizontal direction, vertical direction, and torsional direction. Figure 5 The figure shows the eye movement trajectories of the subject's pupils in the horizontal, vertical and torsional directions and the schematic diagram of the head position on the Pitch axis and the Yaw axis during the left-posterior and right-anterior Dix-Hallpike test, where the horizontal axis is time, the vertical axis of the eye movement trajectory is the eye movement amplitude, and the vertical axis of the head movement trajectory is the head movement amplitude.

[0058] Regarding the data preprocessing of eye movement trajectory and auxiliary detection data, before the model is input, the auxiliary detection data and eye movement trajectory can be synchronized by timestamp. Let's take the auxiliary detection data as body position data as an example. The acquired body position data and eye movement data are data containing time information. In order to more accurately evaluate and judge the vestibular function of the subject, the body position data and eye movement data can be associated through time synchronization, so that the neural network model can obtain the associated features of body position data and eye movement data during recognition processing, such as the eye movement change characteristics caused by body position changes. Generally, the body position data and eye movement data are first time-synchronized and associated, and then input into the neural network model for recognition processing. Of course, the time synchronization association step can be implemented inside the neural network model. For example, after receiving the input body position data and eye movement data, the neural network model preprocesses these two types of data through the data preprocessing layer, mainly including time alignment and normalization operations. The data of each sensor is used as an independent input channel to ensure that the data is synchronized in time and the data format of different sensors is consistent. This is the basis for ensuring subsequent processing.

[0059] Based on the same technical concept, another embodiment of the present application provides a method for processing vestibular function detection data, such as Figure 2 As shown, the following steps are included: S101, obtaining the eye movement video of the subject during the vestibular function test; S102, extracting the subject's eye movement trajectory from the eye movement video, where the eye movement trajectory includes at least one of the following: a horizontal eye movement trajectory, a vertical eye movement trajectory, and a torsional eye movement trajectory; S103, obtaining auxiliary test data of the subject, including the subject's body position data in the vestibular function test and / or nystagmus characteristic parameters obtained from the eye movement trajectory; S104, inputting the eye movement trajectory and the auxiliary detection data into a pre-trained neural network model, and outputting the vestibular function data of the subject through recognition by the neural network model.

[0060] In this embodiment, each track point in the eye movement trajectory data contains eye position information and associated time information (for example, the horizontal eye movement trajectory data contains the sampling time point information and the position information of the pupil in the horizontal direction at that time point); similarly, the auxiliary detection data also contains time information, so as to facilitate the subsequent time synchronization or association of different types of data.

[0061] Preferably, the data type of the auxiliary detection data is structured parameter data and / or static image data, and the auxiliary detection data does not include video data, that is, does not include a dynamic image sequence represented by a continuous image frame. The structured parameter data includes but is not limited to sequence data or parameter data, such as a body position sequence, a head movement trajectory sequence, nystagmus characteristic parameters, etc.; and the static image data, such as a body position trajectory map, a head movement trajectory map, a nystagmus characteristic map, etc., does not include dynamic video data composed of continuous image frames.

[0062] Preferably, before the eye movement trajectory and the auxiliary detection data are input into the model, data preprocessing is also performed: the eye movement trajectory is synchronously associated with the auxiliary detection data; further, the eye movement trajectory and the auxiliary detection data are both time series data.

[0063] In the above embodiment, the auxiliary detection data obtained may be body position data and / or nystagmus characteristic parameters; taking the auxiliary detection data including body position data and nystagmus characteristic parameters as an example, another embodiment of the present application provides a method for processing vestibular function detection data including the following steps: 1. Steps for obtaining raw data; including: S201, obtaining the eye movement video of the subject in the vestibular function test; S202, obtaining body position data of the subject in the vestibular function test; The above two steps are mainly about acquiring the raw data of vestibular function detection, which can be acquired by directly receiving eye movement video and body position data collected by external acquisition equipment, or by actively controlling external acquisition equipment (such as body position acquisition equipment and eye movement acquisition equipment).

[0064] 2. Preliminary data processing steps, including: 2.1 Eye movement data acquisition: S203, extracting the subject's eye movement trajectory from the eye movement video, including at least one of the three items of horizontal eye movement trajectory, vertical eye movement trajectory, and torsional eye movement trajectory; 2.2 Auxiliary detection data acquisition: S204, analyzing and processing the eye movement trajectory of the subject to obtain corresponding nystagmus characteristic parameters; specifically, the nystagmus characteristic parameters include: any one or more of nystagmus slow phase angular velocity, nystagmus direction, nystagmus time, and nystagmus change trend.

[0065] S205, preprocessing the subject's body position data to obtain body position data that meets the neural network model input; for example, denoising, data simplification, coordinate conversion, etc. are performed on the acquired body position data. Of course, if the acquired body position data is video data, image processing is also required to extract the subject's body position change trajectory from the body position image video frame, that is, what we finally input to the neural network model can be one-dimensional data or two-dimensional data, but not three-dimensional video data.

[0066] 3. Intelligent identification steps; specifically including: S206, input the eye movement trajectory and auxiliary detection data (in this embodiment, nystagmus characteristic parameters and body position data) into the trained neural network model, and output the vestibular function data of the subject through the neural network model recognition. Preferably, the eye movement trajectory and the auxiliary detection data are temporally correlated; In the above method embodiment, the auxiliary detection data includes nystagmus characteristic parameters and body position data, which together with the eye movement trajectory data are used as the input of the neural network model, and the vestibular function data is output after the model recognizes it. Of course, the auxiliary detection data can also be body position data or nystagmus characteristic parameters.

[0067] Regarding the input data combination of the neural network model: In the above system embodiment or method embodiment, the auxiliary detection data includes body position data in vestibular function detection and / or nystagmus characteristics obtained from the eye movement trajectory. Therefore, the data combination of the input model includes at least the following three schemes: Solution 1: The input data of the model includes: eye movement trajectory + body position data; preferably, the body position data is head position data (data on the change of the subject's head position).

[0068] This solution focuses on the correlation analysis between body position and eye movement, and can focus on the situation where changes in body position cause changes in eye movement (such as nystagmus), and is particularly suitable for detecting certain symptoms that induce dizziness due to changes in body position. For example, positional tests such as the Dix-Hallpike test and the Roll test, by moving the patient's head to a specific position (i.e. a specific body position) to observe whether nystagmus occurs, can analyze the correlation between nystagmus and changes in body position, thereby providing doctors with key information for the diagnosis of diseases such as otolithiasis.

[0069] Solution 2: The input data of the model includes: eye movement trajectory + nystagmus characteristic parameters; This solution focuses on the details of eye movement characteristics - nystagmus characteristic parameters. On the one hand, the eye movement trajectory provides comprehensive eye movement information. On the other hand, the nystagmus characteristic parameters extracted from the eye movement trajectory can provide an important basis for vestibular function evaluation.

[0070] Solution 3: The input data of the model include: eye movement trajectory + body position data + nystagmus characteristic parameters. This solution integrates these three data, so as to more comprehensively and accurately evaluate the vestibular function of the subjects.

[0071] The above-mentioned eye movement trajectory mainly includes the subject's horizontal eye movement trajectory data, and / or vertical eye movement trajectory data, and / or torsional eye movement trajectory data; generally, the eye movement data input to the model includes horizontal and vertical eye movement trajectory data, and preferably, further includes torsional eye movement trajectory data, thereby improving the comprehensiveness of vestibular function diagnosis and providing more key data for vestibular function diagnosis.

[0072] In both Scheme 2 and Scheme 3 above, nystagmus characteristic parameters are used as one of the input data of the model. Nystagmus characteristic parameters include any one or more of the following: slow phase angular velocity of nystagmus, direction of nystagmus, nystagmus time (nystagmus duration or start time), nystagmus latency, and nystagmus change trend. These nystagmus characteristic parameters are very important for the diagnosis or evaluation of vestibular function. In general, typical nystagmus has the characteristics of strong and weak changes and delayed appearance. For example, in the right posterior semicircular canal BPPV canalolithic type, nystagmus usually appears within 40 seconds after the head is in place and disappears within 1 minute after appearance, that is, the intensity of nystagmus changes from weak to strong and then from strong to weak, and there will be a latency period of less than 40 seconds. Therefore, based on the body position data, combined with these nystagmus characteristic parameters, the accuracy of vestibular function assessment and / or vestibular rehabilitation training program recommendations in vestibular function data can be greatly improved. Preferably, the more nystagmus characteristic parameters are selected, the better. It is better to select 1-2 nystagmus characteristic parameters. For example, the addition of the input factor of the slow phase angular velocity of nystagmus greatly improves the accuracy of vestibular function diagnosis or evaluation, and thus improves the accuracy of vestibular function rehabilitation training program recommendations.

[0073] Compared with the conventional single reliance on eye movement trajectory or eye movement video to evaluate vestibular function, although eye movement trajectory can obtain the law of eye movement and then preliminarily evaluate vestibular function, a single data source cannot provide accurate evaluation, and the addition of auxiliary detection data in this application can effectively make up for this deficiency. In the selection of auxiliary detection data, it is also unique. The more types of auxiliary detection data, the better. The more types, the more complex the model will be, which poses great challenges to the training and accuracy of the model. In this case, body position data was selected as one of the auxiliary detection data, mainly because body position data and eye movement trajectory are complementary, which can provide richer vestibular function information and help the model more accurately capture the causal relationship between body position changes and nystagmus. The eye movement trajectory alone cannot reflect the specific body position changes that induce nystagmus, and it is difficult to associate it with the cause of vestibular dysfunction (for example, it is difficult to distinguish abnormalities of the bilateral semicircular canals based on eye movement trajectory data alone), thereby making up for the limitations of eye movement trajectory.

[0074] Why are nystagmus characteristic parameters used as another choice for auxiliary detection data? The main reason is that nystagmus characteristic parameters deeply quantify and supplement the dynamic characteristics of eye movement trajectories, and can more accurately reflect the functional status of the vestibular system. Let's take the slow phase angular velocity of nystagmus as an example. The slow phase angular velocity of nystagmus is the core parameter of nystagmus characteristics, which represents the speed of the eyeball in slow phase movement. Compared with the path information provided by the eye movement trajectory, the slow phase angular velocity of nystagmus quantifies the dynamic intensity of eye movement, which is an important supplement to the eye movement trajectory data. The slow phase angular velocity of nystagmus is the core information extracted from the eye movement trajectory, avoiding information redundancy and noise interference. Of course, the selection of nystagmus characteristic parameters is not the more the better. It is better to select the slow phase angular velocity of nystagmus in the nystagmus characteristic parameters and combine it with the eye movement trajectory as the model input.

[0075] In summary, this application uses eye movement trajectory + body position data and / or nystagmus characteristic parameters as model input. On the one hand, the detailed information of the original data is retained through the eye movement trajectory, so that the model can access global information. On the other hand, the information is supplemented by body position data and / or nystagmus characteristic parameters, thereby improving the accuracy of the model. In addition, by introducing a neural network model, automatic feature extraction, fusion and evaluation of multimodal data are realized, which has higher accuracy and comprehensiveness than traditional solutions that rely on rules and statistical methods. Its end-to-end processing flow significantly improves efficiency and robustness, while supporting flexible expansion to meet diverse clinical diagnosis and research needs, and can provide real-time and intuitive evaluation results.

[0076] Regarding the identification of nystagmus and the acquisition of nystagmus characteristic parameters: The acquisition of nystagmus characteristic parameters is mainly through identifying the nystagmus contained in the eye movement trajectory in each direction, and then obtaining the corresponding nystagmus characteristic parameters. The identification of nystagmus can be achieved through software algorithms or artificial intelligence, which is not limited in this application. Figure 5 As shown, after nystagmus recognition is performed based on the eye movement trajectories in each direction, the recognized nystagmus can be marked on the eye movement trajectory diagram. For example, in the horizontal eye movement trajectory data diagram, if nystagmus occurs in the eye movement trajectory, the part of the eye movement trajectory where nystagmus occurs will be marked (a short line near the trajectory in the figure indicates nystagmus).

[0077] In an illustrative example, nystagmus identification and acquisition of nystagmus characteristic parameters include the following steps: Obtain the motion characteristics of each data point in the eye movement data in each direction; Determine whether there is periodic eye movement in the eye movement data based on the movement characteristics; If there are periodic eye movements in the eye movement data, the periodic eye movements in the eye movement data are identified as nystagmus; The eye movement trajectory of the data segment identified as nystagmus is analyzed to determine the slow phase and the fast phase of nystagmus, and then obtain the characteristic parameters of nystagmus.

[0078] Periodic motion is generally defined as a regular, repetitive motion. A key characteristic of periodic motion is that it repeats over a period of time. In nystagmus, the eye moves slowly in a particular direction for a period of time (slow phase), then quickly returns to its original position (fast phase), and then repeats the pattern again. That is, the fast phase and the slow phase alternate.

[0079] In the above embodiment, when nystagmus is identified, periodic eye movements are mainly identified from the eye movement trajectory; the nystagmus identification conditions include: periodic eye movements, specifically, the fast phase of nystagmus and the slow phase of nystagmus appear alternately; more preferably, the duration of the periodic eye movement is greater than the first time threshold; and / or the duration of the slow phase of nystagmus meets the preset time range; and / or the acceleration of the slow phase of nystagmus meets the preset acceleration interval. The time period of the slow phase is relatively long, and the movement speed is slow, while the time period of the fast phase is short, and the movement speed is fast. Therefore, the combination of the time window and the speed curve can well distinguish the two.

[0080] In another illustrative example, the data processor is configured to execute the following instructions to obtain nystagmus characteristic parameters: Preprocess the eye movement trajectory data in each direction in the eye movement data, such as denoising and / or streamlining (extracting eye movement trajectory data points at equal intervals to simplify the amount of data processing); Traverse each data point in the preprocessed eye movement trajectory data, compare the position change trend of each data point with the previous data point, and find the data point with a changed change trend as the key data point; According to all the key data points found, the eye movement trajectory data is divided into N data segments; each data segment starts from a key data point and ends at the next key data point; N is a positive integer greater than 1; Calculating the slope of each data segment, and identifying, based on the calculated slope, a first data segment having a slope greater than a first set value and a second data segment having a slope less than a second set value; wherein the first set value is greater than or equal to the second set value; All data segments are traversed, and the first data segment and the second data segment that meet the nystagmus identification conditions are screened out as nystagmus; wherein the first data segment in each data segment marked as nystagmus is the fast phase of nystagmus, indicating the direction of nystagmus; the second data segment is the slow phase of nystagmus, indicating the intensity of nystagmus, that is, the angular velocity of the slow phase of nystagmus.

[0081] In the above example, based on the target eye movement trajectory data after preprocessing the eye movement data, by comparing the position change trends of the previous and next data points, the data points with changed position change trends are found as key data points. After all key data points are found, the adjacent key data points are connected to form line segments, and the slope of each line segment is calculated to identify the first data segment and the second data segment; preferably, after calculating the slope of each data segment, the data segment with a slope greater than the set first slope is identified as the first data segment; the data segment with a slope less than the second slope is identified as the second data segment; and then the first data segment and the second data segment that meet the nystagmus identification conditions are screened out and marked as nystagmus. Regarding the nystagmus identification conditions, as before, for example, the first data segment and the second data segment appear alternately; and / or the duration of the second data segment meets the preset time range.

[0082] In another illustrative embodiment, in addition to identifying nystagmus contained in the eye movement data in each direction, further, body position recognition processing is performed based on the collected time series body position data to obtain the specific body position and its start and end time points in the vestibular function test, and then the eye movement trajectory feature data in the corresponding time period is obtained based on the start and end time points of the specific body position to determine whether nystagmus exists in the eye movement trajectory feature data. If so, other nystagmus feature parameters in the specific body position are further obtained, including nystagmus latency, nystagmus trend changes, nystagmus duration, etc.

[0083] About the model architecture of the neural network model: The specific network architecture of the neural network model may vary based on the different data input to the model, but the overall structure is similar, including: input layer (inputting external data into the network), feature extraction layer (extracting features from the input data), feature fusion layer (fusing different types of features) / feature interaction layer (capturing the correlation between different types of data / features), and output layer (outputting the final classification results or generated text information); among them: The input layer of the neural network model of the present application can be a single input channel, and preferably multiple input channels are used. If it is a single input channel, the input data are combined, such as combining the body position data with the eye movement trajectory, to form a comprehensive input format including the body position and the eye movement trajectory. Splicing or creating a structured input vector can be used. If the input layer uses multiple input channels, for example, if the body position data in two directions and the eye movement trajectory in three directions are used as model inputs, the input layer can use five input channels, and one input channel is used to receive the body position data / eye movement trajectory data in one direction.

[0084] After receiving each input data through multiple input channels, the feature extraction layer extracts features from the data input through the multiple input channels, and the feature fusion layer performs feature fusion processing. In this application, the feature extraction layer includes an auxiliary data processing network and an eye movement data processing network as an example, and the neural network model includes: An auxiliary data processing network is used to process the input auxiliary detection data and extract auxiliary features; An eye movement data processing network is used to process the input eye movement trajectory and extract the eye movement trajectory features; A feature fusion layer is used to fuse the auxiliary features output by the auxiliary data processing network and the eye movement trajectory features output by the eye movement data processing network to obtain comprehensive feature data related to vestibular function; The output layer is used to generate corresponding vestibular function data according to the output data of the feature fusion layer.

[0085] Since the auxiliary detection data includes body position data and / or nystagmus characteristic parameters, the corresponding auxiliary data processing network also includes a body position data processing network and / or a nystagmus characteristic processing network; wherein: A body position data processing network is used to process the input body position data and extract body position features; The nystagmus feature processing network is used to receive input nystagmus feature parameters.

[0086] Given that the auxiliary detection data contains different data, the input data of the model will also be different. Below we explain the neural network model architecture under different input schemes: Let's take the above-mentioned solution 1 (input data is body position data and eye movement trajectory) as an example. The feature extraction layer of the neural network model in this embodiment includes a body position data processing network and an eye movement data processing network; wherein: A body position data processing network is used to process the input body position data and extract body position features; An eye movement data processing network is used to process the input eye movement trajectory and extract the eye movement trajectory features; The feature fusion layer is used to fuse and associate the body position features output by the body position data processing network and the eye movement trajectory features output by the eye movement data processing network to obtain comprehensive feature data related to vestibular function; The output layer is used to generate corresponding vestibular function data according to the output data of the feature fusion layer.

[0087] More preferably, in an exemplary embodiment, the eye movement data processing network includes at least two sub-networks, each sub-network is used to receive and process eye movement trajectory data in one direction; specifically, for example, the eye movement data includes the eye movement trajectories of the subject's pupil in three directions: horizontal eye movement trajectory, vertical eye movement trajectory, and torsional eye movement trajectory; then the eye movement trajectories in these three directions are respectively processed for feature extraction through the three sub-networks of the eye movement data processing network.

[0088] In another exemplary embodiment, the body position data processing network includes at least two sub-networks, each sub-network is used to receive and process body position data in one direction; specifically, for example, the body position data of the subject's head on the Pitch axis is subjected to feature extraction processing by one sub-network of the body position data processing network, and the body position data of the subject's head on the Yaw axis is subjected to feature extraction processing by another sub-network of the body position data processing network.

[0089] In another embodiment of the present application, as described in Scheme 3, the input of the neural network model includes three data types: eye movement trajectory data in all directions, body position data, and nystagmus characteristic parameters; wherein, the eye movement trajectory data and body position data are obtained by preliminary processing from the collected original data, and the nystagmus characteristic parameters are higher-dimensional feature data further extracted from the eye movement trajectory data, so when these three types of data are used for input, the architecture of the neural network model can be further adjusted and optimized. Specifically, the neural network model includes three network structures (these three network structures can be set to different structures according to the characteristics of the data they need to process); the body position data processing network and the eye movement data processing network can refer to the previous embodiments for explanation, and will not be repeated here, and the nystagmus characteristic processing network is used to receive the input nystagmus characteristic parameters; then, the features of the output of each network are fused through the feature fusion layer, and finally the vestibular function data is output through the output layer. The feature fusion layer in this embodiment can also be implemented in different forms, and the following are several examples: (1) Example 1 of feature fusion layer like Figure 4 As shown, the posture features in each direction output by the posture data processing network, the eye movement trajectory features in each direction output by the eye movement data processing network, and the nystagmus feature parameters in each direction output by the nystagmus feature processing network are integrated to obtain a fused overall feature vector, and finally the vestibular function data is obtained based on the fused overall feature vector through the output layer.

[0090] (2) Feature fusion layer example 2 In this example, the feature fusion layer includes a first fusion layer and a second fusion layer, wherein: the first fusion layer is used to fuse the body position features extracted by the body position data processing network and the eye movement trajectory features extracted by the eye movement data processing network to obtain preliminary comprehensive feature data; and then the preliminary comprehensive feature data is fused with the nystagmus feature parameters through the second fusion layer. Preferably, a self-attention mechanism is also used in the feature fusion layer. Specifically, the weights of different features can be set by setting the self-attention mechanism in the first fusion layer and / or the second fusion layer to improve the accuracy of the final evaluation data.

[0091] In this example, multi-level feature fusion is used to gradually fuse low-level features (such as eye movement trajectory and body position data) with high-level features (such as nystagmus feature parameters) to achieve a more complete feature representation to support the preparation judgment and evaluation of vestibular function.

[0092] The goal of multi-level feature fusion is to enable the model to gradually extract more representative high-level features from low-level raw data through the integration of features at different levels. Specifically, in this example, the specific steps to implement multi-level feature fusion include the following: S1, low-level feature extraction: Extracting eye movement trajectory feature data from the eye movement trajectory data through a body position data processing network (such as a convolutional neural network and / or a recurrent neural network); extracting dynamic features of body position changes from the body position data through an eye movement data processing network (such as an LSTM or RNN) to obtain body position feature data; S2, mid-level feature construction In the middle layer, the low-level features are preliminarily integrated through the first fusion layer; specifically, the eye movement trajectory features in each direction of the eye movement trajectory and the body position features of the body position data are integrated to form an overall middle-level feature representation.

[0093] S3, high-level feature introduction The nystagmus feature parameters are introduced into the model as high-level features. In the feature fusion layer, the high-level features are further fused with the middle-level features obtained in the previous step through the second fusion layer. Furthermore, a weighted fusion method can be used to enable the model to automatically adjust the importance of the features according to the specific circumstances of the input feature data.

[0094] S4, based on the final fused feature data, passes through the fully connected layer and outputs the vestibular function assessment results.

[0095] (3) Feature fusion layer example 3 Similarly, in this example, the feature fusion layer also contains a first fusion layer and a second fusion layer, wherein: the first fusion layer is used to perform feature fusion on the eye movement trajectory features extracted by the eye movement data processing network and the nystagmus feature parameters output by the nystagmus feature processing network; the second fusion layer further fuses the data output by the first fusion layer with the posture feature data output by the posture data processing network, and finally performs evaluation and analysis based on the data fused and output by the second fusion layer to obtain the vestibular function data of the subject.

[0096] In this example, eye movement trajectory features are first extracted based on eye movement trajectory, and then fused with nystagmus feature parameters. This can fully utilize the spatiotemporal features of eye movement trajectory data, and combine nystagmus feature parameters to obtain a more complete eye movement trajectory feature expression. Since nystagmus feature parameters are closely related to eye movement trajectory, fusing them first helps to extract eye movement trajectory features with more diagnostic value. After fusing the eye movement trajectory features first and then fusing them with the body position features, features can be processed in layers, reducing interference between different features, which is more conducive to the model extracting unique features in each direction.

[0097] Preferably, on this basis, a self-attention mechanism is also provided in the first and / or second fusion layer, and different weights of each feature in the fused data are set through the self-attention mechanism, so that the final recognition output data is more accurate.

[0098] The neural network model architecture in the above embodiment uses a feature fusion layer to perform feature fusion. In another embodiment of the present application, a feature interaction layer is used in the neural network model. The specific implementation steps are as follows: Multiple input channels: Receive data from multiple input channels, such as eye movement data in three directions (horizontal, vertical, and torsion) and body position data in two directions (Pitch axis and Yaw axis).

[0099] Independent feature extraction: Preliminary feature extraction is performed on the eye movement trajectory data and body position data input by each input channel. For example, the LSTM layer is used to extract time series features, or the CNN layer is used to extract spatiotemporal features. The eye movement and body position data in each direction are extracted separately to obtain feature representation.

[0100] Feature interaction layer: Apply the self-attention mechanism or bilinear interaction to interactively process the extracted eye movement and body position features to generate new interactive features. Preferably, bilinear interaction (Multiply) can be used: the feature of each eye movement direction is element-wise multiplied with the body position feature to generate interactive features. Then apply the self-attention mechanism to the interactive features, dynamically adjust the weight of each interactive feature, so that the model pays more attention to the feature combination with diagnostic significance.

[0101] Feature fusion and output: The interacted features are finally fused through the feature fusion layer, input into the fully connected layer, and the vestibular function evaluation results are output.

[0102] Preferably, if the data input to the model also includes nystagmus feature parameters, then after the body position data processing network outputs the body position feature data, the eye movement data processing network outputs the eye movement trajectory feature data, and the nystagmus feature processing network outputs the nystagmus feature parameters, the body position feature data, the eye movement trajectory feature data, and the nystagmus feature parameters are subjected to feature interaction through the feature interaction layer.

[0103] The core function of the feature interaction layer is to directly generate new interaction features between nystagmus and / or eye movement and body position features to capture their mutual relationships and dependencies. Through element-level operations (such as multiplication) or self-attention mechanisms, the interaction layer can dynamically adjust and combine different features to generate more diagnostically meaningful combined features. Because the feature interaction layer can reveal the relationship between different features, it makes the model easier to understand, such as the impact of a specific body position on eye movement trajectory.

[0104] Although the feature interaction layer has provided a certain feature fusion effect, retaining the feature fusion layer can still further improve the performance of the model in some complex vestibular function assessment tasks for the following reasons: Further integration of multi-level features: If the relationship between eye movement features and body position features is very complex, the interaction features generated by the feature interaction layer may not be sufficient to capture all the information. In this case, additional feature fusion layers can further integrate the multi-level information of the interaction features.

[0105] Introducing higher-level feature representation: After feature interaction, the feature fusion layer can be used as a higher-level processing to integrate the combination results of different interacting features to generate a more representative comprehensive feature.

[0106] Therefore, the architecture of feature interaction layer + feature fusion layer can be adopted, and the features generated by the feature interaction layer can be further fused with the body position and eye movement data, so that the model can capture more comprehensive contextual information.

[0107] The above mainly describes the main architecture of the neural network model of the present application. In actual application, different models can be used according to actual conditions. Generally, eye movement trajectory data and auxiliary detection data contain time information, so time association or alignment can be performed later. Taking scheme one as an example, the body position data and eye movement trajectory data in the input data are both time series data. Therefore, a model suitable for processing time series data can be used, such as a long short-term memory network, a combination of a convolutional neural network and LSTM, a Transformer model, etc.

[0108] Let's take Solution 3 as an example. If the input data includes: horizontal eye movement trajectory, vertical eye movement trajectory, pitch axis head position data, yaw axis head position data, horizontal nystagmus slow phase angular velocity data, vertical nystagmus slow phase angular velocity data; the network architecture of the neural network model of this embodiment is as follows: (1) Input layer, which is responsible for receiving all types of input data. We can use six input channels to receive the above six types of input data. Preferably, these input data need to be preprocessed, such as standardization or normalization, time window segmentation, etc. Each input can be a sequence of time series data (for example, eye movement trajectory and nystagmus information are sequences that change over time, and head position data is the coordinate information of the time series); (2) Feature extraction layer, which is mainly used to extract spatial and temporal features from the input data. Usually, a convolutional layer or other network structure can be used. The following is an example: Eye movement trajectory data processing: For horizontal and vertical eye movement trajectory data, LSTM networks are used to capture the changes in eye movements over time and extract eye movement trajectory features. An independent LSTM network can be used for processing in each direction.

[0109] Head position data processing: For the pitch axis head position data and the yaw axis head position data, LSTM or CNN can also be used to capture the changes of head position data over time and extract the corresponding head position features; Nystagmus information processing: because nystagmus information is actually a higher-level feature information extracted based on the eye movement trajectory, the received nystagmus information does not need to be further extracted and processed, and directly enters the subsequent feature fusion layer; of course, nystagmus features can also be further extracted through CNN, or lightweight mapping processing can be performed through a layer of MLP, which is not limited in this embodiment.

[0110] Preferably, at the feature extraction layer, a self-attention mechanism can also be introduced to help the model focus on key moments, such as determining the change or holding stage of the head position according to needs.

[0111] (3) Feature fusion layer: The features extracted by each network in the above feature extraction layer are fused, different types of data features are combined and the relationship between them is captured to generate comprehensive feature data. Preferably, a self-attention mechanism can also be introduced in the feature fusion layer to dynamically adjust the weights of different features.

[0112] Specifically, we fuse the features extracted from different inputs (eye movement trajectory, head position, nystagmus, etc.) and assign a dynamic weight to each feature through the attention mechanism. The attention mechanism ensures that the model can flexibly adjust the degree of attention to different inputs according to task requirements by weighting different features. For example, nystagmus feature parameters are critical for BPPV diagnosis. Therefore, if applied to the field of BPPV, nystagmus feature parameters can be given a higher weight.

[0113] (4) Output layer: Based on the comprehensive feature data output by the feature fusion layer, identify and output the vestibular function test results.

[0114] The classification data of vestibular function outputted by the above-mentioned example models include, but are not limited to: one or more nystagmus parameters for indicating nystagmus conditions, and / or one or more diagnostic parameters for indicating vestibular function, and / or one or more recommended parameters for indicating vestibular rehabilitation training. The specific parameters outputted by the neural network model can be used to indicate or characterize the corresponding vestibular function information.

[0115] Of course, if you want the model to output text information rather than a simple classification result, then the neural network model you need to use cannot simply use the above model example, but a neural network model that can generate coherent text. For example, use an encoder-decoder architecture with an attention mechanism, which can convert the model input data into detailed diagnostic text. Taking the sequence-to-sequence (Seq2Seq) model plus an attention mechanism as an example, the Seq2Seq model usually consists of two parts: an encoder and a decoder. The encoder processes the input data and the decoder generates the output text. The addition of an attention mechanism allows the model to focus more on the relevant parts of the input sequence when generating each word, thereby improving the relevance and accuracy of the generated text. Specifically, the architecture example is as follows: Sequence to Sequence (Seq2Seq) model with attention mechanism, including: Input layer: accepts pre-processed gyroscope and eye movement data, which may be converted into a format more suitable for model processing through feature extraction.

[0116] Encoder: Uses multiple layers of LSTM or GRU units to process time series input. The encoder learns to understand the context of the input sequence and compresses this understanding into a fixed-length context vector.

[0117] Attention Mechanism: When calculating the output of the decoder at each step, the attention mechanism dynamically selects information about the parts of the input that are most relevant to the current output. This helps the model better understand which parts of the input data are most important for generating the current output vocabulary.

[0118] Decoder: Also uses LSTM or GRU units, using the encoder's context vector and the input information weighted by the attention mechanism to gradually generate each word of the diagnostic text.

[0119] Output layer: Usually a fully connected layer, which uses the softmax function to output the vocabulary probability of each step and selects the word with the highest probability as the output of the current step.

[0120] In addition to the above model architecture, the model for generating text can also use the enhanced Transformer model. Since the Transformer model performs well in processing sequence data, especially in the field of natural language processing, it is also a good choice for generating diagnostic text. Transformer is completely based on the attention mechanism and has no loop layer, which makes it very effective in parallel processing and capturing long-distance dependencies.

[0121] Enhanced Transformer model architecture example: Input layer: Same as above, with preprocessed gyroscope and eye movement data as input.

[0122] Positional encoding: Adding positional information, since Transformer does not naturally handle the time series nature of the data.

[0123] Transformer encoder: It consists of multiple self-attention layers and feed-forward networks, which processes the input data and encodes the comprehensive contextual information of the input sequence.

[0124] Transformer decoder: It also consists of multiple layers, but in addition to processing the output of the encoder, it also predicts the next word, and each prediction is based on the previous output.

[0125] Output layer: Generates the final diagnosis text, each time selecting the word with the highest probability from the decoder output.

[0126] Whether using a Seq2Seq plus attention model or a Transformer model, the key is how to effectively convert the gyroscope and eye movement data into a form that the model can understand, and through training, enable the model to accurately generate text information reflecting the BPPV diagnosis. This may require a large amount of labeled data to train the model, as well as careful parameter adjustment and optimization.

[0127] Example of combining two neural network models: The first neural network model mainly adopts a network model that can process time series, such as a long short-term memory network, a combination of a convolutional neural network and LSTM, a Transformer model, or any other model architecture, which is used to identify and output nystagmus information in each specific body position based on the input body position data and eye movement data; The second neural network model mainly adopts a model capable of text processing, such as an existing language model, etc., which is used to output text information of vestibular function data according to the nystagmus information in each specific body position output by the first neural network model.

[0128] Compared with the solution of using only one model architecture to achieve text output, the combination of two neural network models in this example greatly reduces the complexity of the neural network model, the difficulty of data processing, and the difficulty of model training, thereby improving the accuracy of model output.

[0129] The vestibular function data output by this embodiment is vestibular function text information, including but not limited to the nystagmus text data identified in each specific body position, and / or vestibular function diagnosis or evaluation text data, and / or vestibular rehabilitation training program recommendation text data. The neural network model used in this embodiment has the function of generating corresponding vestibular function text information based on input data. For example, the aforementioned sequence-to-sequence model plus the model of the attention mechanism, the enhanced version of the Transformer model, etc., of course, two neural network models can also be used in combination for processing, the latter is specifically used for text processing to simplify the model complexity. In view of the fact that the input data of the model contains multiple types, therefore, if multiple input channels are used to receive different types of input data, the feature extraction of each input data and the later feature fusion within the model can also refer to the previous embodiment, which will not be repeated here.

[0130] Specifically, if the trained neural network model hopes to output key nystagmus text information in order to provide doctors with a reference for vestibular function diagnosis, then during the training period, in the medical record information used as a training sample, the vestibular function examination information annotated is the characteristic nystagmus text information of this vestibular function examination.

[0131] In an illustrative embodiment, during training, the neural network model uses a large amount of different medical record information to form a training sample data set, wherein a complete medical record information should include auxiliary detection data (such as body position data and / or nystagmus characteristic parameters), eye movement trajectory data when the subject undergoes at least one vestibular function examination, and the result information of the subject's vestibular function examination given by the doctor.

[0132] Another embodiment of the present application, based on any of the above embodiments, before inputting the body position data into the neural network model, further includes: Performing body position recognition processing on the body position data to obtain each specific body position and its start and end time points in the vestibular function test; According to the start and end time points of each specific body position, the corresponding time period of the input data of the neural network model is marked; specifically, for example, if the start and end time points of the specific body position A are detected to be the a1--a2 time period, the data with time points in the a1--a2 time period in the body position data, and / or eye movement trajectory data, and / or nystagmus characteristic parameter data are marked, so that the subsequent neural network model can focus on the data in this time period during recognition processing and increase the recognition weight of this time period.

[0133] Of course, the above-mentioned recognition and processing of body position data can also be implemented in a variety of ways, which are not limited to this. In an exemplary implementation, a trained body position recognition neural network model is used to recognize the body position data, and the specific body position information contained in the body position data is output. In another exemplary implementation, the specific body position and its start and end time can be obtained by the following steps: Extract the coordinates of each data point from the time series body position data; Based on the coordinate data of all data points, continuous data points with the same coordinates are found and divided into a maintained position; Obtaining the duration of holding the body position, and selecting the body position with a duration greater than a set threshold as the target body position; Obtain the start and end time points of the target's posture, as well as the corresponding coordinate data; According to the coordinate data of the target body position, combined with the corresponding reference relationship between each specific body position type and the coordinates, the specific body position type corresponding to each target body position is identified.

[0134] In another exemplary embodiment, if the head position posture change data (head position data) is collected by the built-in gyroscope of the head-mounted device, the head position data collected by the gyroscope is processed for head position posture recognition to identify the head position changes of the subject during the entire vestibular function test. Taking the right rear Dix-hallpike test as an example, the subject's body position (head position) change process is as follows: At the beginning, the subject sits on the examination bed (sitting position), the doctor holds the subject's head with both hands, rotates it 45 degrees to the right, and keeps this position unchanged (hereinafter referred to as the right 45-degree position); then quickly changes the position to the supine position, and the head hangs back outside the bed, 30 degrees from the horizontal plane, and the head position is kept at 45 degrees, and this position (hereinafter referred to as the hanging-supine position) should be maintained for 30 seconds. Then the subject slowly returns to the sitting position.

[0135] During the whole test, the subject's head is wearing a head-mounted device with a built-in gyroscope, with the initial posture as the initial position. Then, as the body position changes during the test, the head position data collected by the gyroscope also changes. By identifying and processing the series of collected head position data, many solutions for position / posture recognition based on gyroscope data in the prior art are not described here. The specific head position changes of the subject can be identified.

[0136] In addition to performing position recognition based on the collected position data to obtain a specific position, in another illustrative embodiment of the recognition and judgment of a specific position, the position corresponding to the data segment in which the position remains unchanged for a set time can be selected as a specific position. For example, in the right rear Dix-hallpike test, the specific position mainly refers to the position that remains unchanged for a period of time after the position changes. During the above test, the patient's head rotates 45 degrees to the right and the position remains unchanged. Therefore, the 45-degree right turn position can be regarded as a specific position. More importantly, the doctor observes whether the subject has dizziness and nystagmus in the hanging-supine position (the head hangs backward outside the bed, 30 degrees from the horizontal plane, and the head position is also maintained at 45 degrees). The subject needs to maintain this position for a certain period of time. Therefore, the hanging-supine position is a specific position. We can see that the above-mentioned specific position must be a position in which the position remains unchanged, and the changing movements will not belong to the specific position. Therefore, the specific position can be selected by selecting the position in which the position is maintained for a set time.

[0137] Regarding the identification and judgment of specific body positions, in another illustrative embodiment, in addition to the above-mentioned specific body position screening by body position holding time, further screening can be performed in combination with body position recognition results. Specifically, the body position data is identified to obtain the body position type corresponding to each data point, and based on the starting time period corresponding to the body position when the body position holding time reaches the set time, the body position type corresponding to the time period is searched, and if the body position type also belongs to the specific body position type, the body position corresponding to the time period is further determined to be the specific body position.

[0138] Preferably, after identifying each body position of the body position data, especially identifying each specific body position, the test type of this vestibular function test is identified based on the time sequence of the identified body positions / specific body positions.

[0139] In another example, for example, a video nystagmus meter, because it has a built-in posture sensor and a camera for photographing the user's eyes, it can be used as a body position acquisition submodule to collect body position data, and can also collect the user's eye movement video to obtain the eye movement trajectory. The body position information is associated and synchronized with the eye movement trajectory data to obtain the eye movement trajectory in each body position, and based on the eye movement trajectory corresponding to the start and end time points of the body position, the characteristic nystagmus data in the specific body position can be calculated, and the characteristic nystagmus data includes nystagmus type, nystagmus intensity, nystagmus latency, nystagmus duration, and nystagmus trend. Compared with the aforementioned embodiment in which nystagmus is identified only based on the eye movement trajectory, in this example, based on the start and end time points of the specific body position, it is possible to quickly identify whether nystagmus exists in the time period, and if so, further determine the characteristic parameters of nystagmus, thereby greatly reducing the amount of data processing and improving data processing efficiency and accuracy.

[0140] About data optimization processing: Another embodiment of the present application, based on any of the above embodiments, the vestibular function data is the nystagmus information in each specific body position in the current vestibular function test, and after the nystagmus information in each specific body position is identified by the neural network model or other algorithms; it also includes: The nystagmus information in each specific position identified is subjected to any one or more of the following data optimization processes: information simplification, professional expression processing, and structured text processing, so as to generate vestibular function text data. The data optimization processing steps include, but are not limited to, any one or more of the following: information simplification (such as simplifying the data representation form, filtering invalid information, merging valid information, etc.), professional expression processing (such as feature concretization processing), and structured text processing, etc.

[0141] In an illustrative embodiment, the data processor is configured to perform any one or more of the following steps to simplify the data representation: (1) Classify and describe the duration of the nystagmus latency in the nystagmus information in each specific position, combined with the set nystagmus latency duration stages. Specifically, the time value is linked to the diagnosis, and it is no longer presented directly in the form of a value. Instead, it is modified according to the latency duration to include no latency, less than 10 seconds, less than 40 seconds, and more than 40 seconds.

[0142] (2) Classifying and describing the nystagmus duration in the nystagmus information of each specific body position identified, combined with the set duration stages of the nystagmus duration; for example, the nystagmus duration is divided into: less than half a minute, less than one minute, and more than one minute.

[0143] In an illustrative embodiment, at least one data processor is configured to perform invalid information filtering by performing any one or more of the following steps: (1) Nystagmus with intensity below the set threshold is uniformly described as no obvious nystagmus; (2) All eyes that maintain nystagmus throughout the test are uniformly described as spontaneous nystagmus; (3) Among the nystagmus recognition results based on the eye movement data in the horizontal, vertical and torsional directions, only the nystagmus recognition results in the corresponding directions whose nystagmus intensity reaches the set threshold are retained.

[0144] In an illustrative embodiment, at least one data processing is configured to perform feature visualization processing by executing the following steps: for nystagmus identified from horizontal eye movement trajectory data, a professional description of the direction of nystagmus is performed based on the current specific body position and the relationship between the direction of nystagmus and the ground.

[0145] Specifically, in the diagnosis of horizontal semicircular canal BPPV, the nystagmus is usually described as geotropic and apogeotropic, rather than horizontal left or right nystagmus, based on the relationship between the nystagmus direction and the ground, which is easier to associate with the diagnosis. In the right lateral decubitus position, horizontal right nystagmus is converted to geotropic nystagmus, and horizontal left nystagmus is apogeotropic nystagmus. In the left lateral decubitus position, the opposite is true.

[0146] In the Dix-Hallpike examination, if there is typical nystagmus when lying down and typical nystagmus when sitting up, it is described as the opposite direction of the nystagmus rather than directly described in terms of direction.

[0147] In one illustrative embodiment, at least one data processing module is configured to perform effective information merging by performing any one or more of the following steps: (1) The nystagmus induced by the same diagnostic factor in different body positions is combined and expressed. For example, in the supine position and the right posterior Dix-Hallpike position, an upward nystagmus with torsion to the right is observed, with a latency period of less than 10 seconds and a duration of less than half a minute.

[0148] (2) All diagnostic factors of the same diagnosis should be combined. For example, if a leftward nystagmus is observed in the left side lying position on the roll test and a rightward nystagmus is observed in the right side lying position on the roll test, then the combination should be "Bilateral geotropic nystagmus on the roll test" and other features can be described later.

[0149] (3) When there are multiple diagnostic factors with different diagnoses, they are described in order of the intensity of the diagnostic factors. For example, in the case of right posterior to right horizontal, the right posterior Dix-Hallpike test will have an upward nystagmus with torsional right nystagmus, and the right horizontal test will have horizontal left and right nystagmus.

[0150] (4) Combine and describe the nystagmus information detected in the same position at different times during the complete process of vestibular function testing. Specifically, establish a connection between the entire examination process. For example, when a specific nystagmus is observed during the first roll test, describe it. In the second examination, directly describe whether there is a characteristic change in the nystagmus, rather than directly repeating the description of the characteristics. If there is no change in the nystagmus, combine the descriptions of multiple examinations.

[0151] (5) Establish a relationship between diagnosis and treatment, and combine information on nystagmus characteristics before and after diagnosis and treatment. Specifically, if a patient undergoes a complete diagnosis process, then undergoes treatment, and finally undergoes a review, when combining valid information, summarize the information before treatment, summarize the characteristics during treatment, and compare the characteristics after treatment with the characteristics before treatment.

[0152] In one illustrative embodiment, structured text processing includes: Generate a segmented report of nystagmus for each position based on the set test type and position sequence; According to the segmented report of nystagmus corresponding to each body position and combined with the vestibular function assessment standards, a vestibular function assessment text is generated.

[0153] In this embodiment, through the above data optimization processing, the key information required for vestibular function diagnosis is output and invalid data is filtered out. Highlighting the core information most related to the diagnosis and filtering out the huge amount of complicated and invalid information can make the test results clearer and more conducive to the doctor's judgment; thereby improving the doctor's diagnosis and treatment efficiency and the accuracy of diagnosis and treatment, and effectively reducing the possibility of missed diagnosis and misdiagnosis.

[0154] The vestibular function data processing method embodiment of the present application corresponds to the vestibular function data processing system embodiment, and the technical details of the two can be referenced to each other. To reduce repetition, they will not be described in detail.

[0155] Another embodiment of the present application provides an electronic device, including a screen, a memory, one or more data processors, and one or more programs; wherein the one or more programs are stored in the memory; when the one or more data processors execute the one or more programs, the electronic device implements the vestibular function detection data processing method of the present application, and the details may refer to the steps of the vestibular function detection data processing method of any of the above embodiments.

[0156] Another embodiment of the present application discloses a storage medium storing computer executable instructions, which, when executed by a processor, causes the processor to execute the vestibular function data processing method of any of the above method embodiments or the steps that the data processor is configured to execute in any of the system embodiments. To reduce repetition, they are not described here.

[0157] Another embodiment of the present application further discloses a computer program product. When the computer program product is run on a computer, the computer executes the vestibular function detection data processing method of any embodiment of the present application.

[0158] It should be noted that the above embodiments can be freely combined as needed. The above are only preferred embodiments of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be regarded as the protection scope of the present invention.

Claims

1. A method for processing vestibular function detection data, characterized in that: include: Obtain the eye movement video of the subject during the vestibular function test; Extracting the eye movement trajectory of the subject from the eye movement video; the eye movement trajectory includes at least one of the following: a horizontal eye movement trajectory, a vertical eye movement trajectory, and a torsional eye movement trajectory; Acquiring auxiliary detection data of the subject; the auxiliary detection data includes body position data of the subject in the vestibular function test and / or nystagmus characteristic parameters acquired from the eye movement trajectory; The eye movement trajectory and the auxiliary detection data are input into a pre-trained neural network model, and the vestibular function data of the subject is output through the neural network model recognition; wherein: The neural network model includes: An auxiliary data processing network, used for processing the input auxiliary detection data and extracting auxiliary features; An eye movement data processing network is used to process the input eye movement trajectory and extract the eye movement trajectory features; A feature fusion layer, used for fusing the auxiliary features output by the auxiliary data processing network and the eye movement trajectory features output by the eye movement data processing network to obtain comprehensive feature data related to vestibular function; The output layer is used to generate corresponding vestibular function data according to the output data of the feature fusion layer.

2. The vestibular function detection data processing method according to claim 1, characterized in that: The auxiliary detection data input into the neural network model at least includes nystagmus characteristic parameters; The acquisition of the nystagmus characteristic parameters includes: Identify nystagmus information in eye movement trajectories in various directions and obtain corresponding nystagmus characteristic parameters; the nystagmus characteristic parameters include: any one or more of nystagmus slow phase angular velocity, nystagmus direction, nystagmus time, and nystagmus change trend.

3. The vestibular function detection data processing method according to claim 2, characterized in that: The method of identifying nystagmus information in the eye movement trajectory characteristics in each direction specifically includes: Obtain the motion characteristics of each data point in the eye movement trajectory in each direction; Determining whether there is periodic eye movement in the eye movement trajectory based on the movement characteristics; If there is periodic eye movement in the eye movement trajectory, the periodic eye movement in the eye movement trajectory is identified as nystagmus.

4. The vestibular function test data processing method according to claim 2, characterized in that: The method of identifying nystagmus information in eye movement trajectories in various directions specifically includes: Based on each data point in the eye movement trajectory in each direction, compare the position change trend of each data point with that of the previous data point, and find the data point with a changed change trend as the key data point; According to all the key data points found, the eye movement data in this direction is divided into N data segments; N is a positive integer greater than 1; Calculating the slope of each data segment, and identifying, based on the calculated slope, a first data segment having a slope greater than a first set value and a second data segment having a slope less than a second set value; wherein the first set value is greater than or equal to the second set value; All data segments are traversed to select target eye movement data including the first data segment and the second data segment that appear alternately as nystagmus; wherein the first data segment is the fast phase of nystagmus, used to indicate the direction of the nystagmus; and the second data segment is the slow phase of nystagmus, used to indicate the amplitude of the nystagmus.

5. The vestibular function test data processing method according to claim 1, characterized in that: The auxiliary detection data input into the neural network model includes the body position data or nystagmus characteristic parameters of the subject; the auxiliary data processing network of the neural network model includes a body position data processing network or a nystagmus characteristic processing network, wherein: The body position data processing network is used to process the input body position data and extract body position features; The nystagmus feature processing network is used to receive input nystagmus feature parameters.

6. The vestibular function test data processing method according to claim 1, characterized in that: The eye movement feature data input into the neural network model includes body position data and nystagmus feature parameters; the auxiliary data processing network of the neural network model includes a body position data processing network and a nystagmus feature processing network; wherein: The body position data processing network is used to process the input body position data and extract body position features; The nystagmus feature processing network is used to receive the input nystagmus feature parameters; The feature fusion layer is used to fuse the feature data output by the body position data processing network, the eye movement data processing network and the nystagmus feature processing network; and is also used to adjust the weights of different features by using a self-attention mechanism based on the fused feature vector; The output layer is used to identify and output the vestibular function data of the subject based on the data processed by the self-attention mechanism.

7. The vestibular function test data processing method according to claim 6, characterized in that: The feature fusion layer includes a first fusion layer and a second fusion layer; wherein: If the feature fusion layer adopts the first fusion architecture, then: The first fusion layer is used to perform feature fusion on the body position features output by the body position data processing network and the eye movement trajectory features output by the eye movement data processing network; The second fusion layer is used to fuse the nystagmus feature parameters output by the nystagmus feature processing network with the data output by the first fusion layer, and dynamically adjust the weights of different features by using a self-attention mechanism; If the feature fusion layer adopts the second fusion architecture, then: The first fusion layer is used to perform feature fusion on the eye movement trajectory features output by the eye movement data processing network and the nystagmus feature parameters output by the nystagmus feature processing network; The second fusion layer is used to perform feature fusion on the posture features output by the posture data processing network and the data output by the first fusion layer, and adopt a self-attention mechanism to dynamically adjust the weights of different features.

8. The vestibular function test data processing method according to claim 1, characterized in that: The vestibular function data output by the neural network model is the nystagmus information of the subject in each specific body position during the vestibular function test; The vestibular function detection data processing method also includes: Data optimization and structured text processing are performed on the nystagmus information in each specific body position to generate vestibular function text data.

9. The vestibular function test data processing method according to claim 8, characterized in that: The data optimization process includes any one or more of the following: Based on the preset screening rules, redundant information irrelevant to vestibular function diagnosis is removed; Based on the preset feature threshold, invalid data that does not meet the nystagmus characteristics are eliminated; According to the duration of the nystagmus latency period in each specific position identified, combined with several duration stages of the nystagmus latency period set, a corresponding classification description is generated; According to the duration of nystagmus in each specific position identified, combined with several duration stages of nystagmus duration set, a corresponding classification description is generated; According to the identified slow phase angular velocity of nystagmus in each specific body position, combined with the set minimum threshold of the slow phase angular velocity of nystagmus, a corresponding classification description is generated; The same type of nystagmus information is merged and processed.

10. An electronic device comprising a screen, a memory, one or more data processors, and one or more programs; wherein: The one or more programs are stored in the memory; it is characterized in that when the one or more data processors execute the one or more programs, the electronic device implements the vestibular function detection data processing method as described in any one of claims 1 to 9.

11. A computer program product, characterized in that When the computer program product is run on a computer, the computer is enabled to execute the vestibular function detection data processing method according to any one of claims 1 to 9.

12. A vestibular function detection system, characterized in that: include: Eye movement shooting module, eye movement trajectory acquisition module, auxiliary data acquisition module, intelligent recognition module; among which: The eye movement shooting module is equipped with at least one camera for collecting eye movement videos of the subject during the vestibular function test; An eye movement trajectory acquisition module is connected to the eye movement shooting module for receiving the eye movement video transmitted by the eye movement shooting module and extracting the eye movement trajectory of the subject from the eye movement video; the eye movement trajectory includes at least one of the following: horizontal eye movement trajectory, vertical eye movement trajectory, and torsional eye movement trajectory; The auxiliary data acquisition module is used to acquire the auxiliary detection data of the subject; the auxiliary detection data includes the body position data and / or nystagmus characteristic parameters of the subject; the auxiliary data acquisition module includes a body position acquisition submodule and / or an eye movement processing submodule; wherein: The body position acquisition submodule is used to acquire the body position data of the subject in the vestibular function test; The eye movement processing submodule is connected to the eye movement trajectory acquisition module for receiving the eye movement trajectory transmitted by the eye movement trajectory acquisition module and acquiring corresponding nystagmus characteristic parameters from the eye movement trajectory; The intelligent recognition module has a pre-trained neural network model built in, and the intelligent recognition module is respectively connected to the eye movement trajectory acquisition module and the auxiliary data acquisition module for inputting the eye movement trajectory and the auxiliary detection data into the neural network model to obtain the vestibular function data of the subject; wherein: The neural network model includes: An auxiliary data processing network, used for processing the input auxiliary detection data and extracting auxiliary features; An eye movement data processing network is used to process the input eye movement trajectory and extract the eye movement trajectory features; A feature fusion layer, the feature fusion layer is used to fuse the auxiliary features output by the auxiliary data processing network and the eye movement trajectory features output by the eye movement data processing network to obtain comprehensive feature data related to vestibular function; The output layer is used to generate corresponding vestibular function data according to the output data of the feature fusion layer.

13. The vestibular function detection system according to claim 12, characterized in that: The eye movement shooting module is arranged in an eye movement acquisition device, and the eye movement acquisition device adopts any product form among a wearable eye movement acquisition instrument, a head-mounted virtual reality or augmented reality device, and a non-wearable shooting device.

Images