A pupil tracking method and its BPPV intelligent auxiliary diagnosis method and system

Abstract

The present invention discloses a pupil tracking method and a BPPV intelligent auxiliary diagnosis method and system thereof. The pupil tracking method comprises the following steps: replacing the Xception model of DeepLabv3+ with a sub-network of Yolov5, extracting features of nystagmus video, obtaining a first effective feature layer of 1 / 8 size and a second effective feature layer of 1 / 16 size; performing a first convolution process on the first effective feature layer, sending the second effective feature layer to an ASPP enhanced feature extraction module for feature enhancement, fusing the first effective feature layer of the first convolution and the second effective feature layer of feature enhancement through an FFM module, obtaining a pupil segmentation result; and locating the pupil center to obtain the pupil boundary and center position. The present invention replaces the Xception model of DeepLabv3+ with a sub-network of Yolov5, and adds an FFM module for feature fusion, thereby achieving accurate and rapid pupil segmentation.

Description

Technical Field

[0001] The present invention relates to the technical field of BPPV intelligent virtual auxiliary diagnosis, and more specifically, to a pupil tracking method and BPPV intelligent auxiliary diagnosis method and system. Background Art

[0002] Benign paroxysmal vertigo (BPPV) is the most common type of clinical vertigo. The widespread use of videonystagmograph (VNG) makes the determination of nystagmus direction more objective and convenient. Some scholars have combined it with the diagnosis and treatment of BPPV, so that the nystagmus induced by patients in various body positions can be recorded and analyzed more intuitively and objectively.

[0003] In recent years, the localization diagnosis and reduction of BPPV are based on the nystagmus observed by the doctor, but the direction of rotational nystagmus is often difficult to distinguish, and some small nystagmus is easily ignored by the naked eye, which affects the localization diagnosis and subsequent reduction treatment of BPPV. In 2015, the comprehensive vertigo diagnosis and treatment system developed by Guoyi Huake (Tianjin) Medical Technology Group Co., Ltd. improved the nystagmus image acquisition system, realized the nystagmus axial recognition, and can capture complex nystagmus data such as axial rotation, which can effectively assist in the diagnosis and treatment of benign paroxysmal positional vertigo. However, the diagnosis and treatment of BPPV still have the following deficiencies:

[0004] (1) Lack of data on the spatial orientation of the semicircular canals. Both the diagnosis and treatment of BPPV are highly dependent on the knowledge of the spatial orientation of the semicircular canals. In particular, the judgment of the affected side of dorsal nystagmus in HSP-BPPV often requires secondary localization signs to make a judgment, and is highly dependent on the knowledge of the spatial orientation of the ampullary ridge. Its positioning judgment is even more controversial.

[0005] (2) Rationality of diagnostic techniques and repositioning methods. An analysis of the diagnostic techniques and repositioning methods for BPPV based on the semicircular canal model with spatial orientation revealed that many traditional diagnostic techniques and repositioning methods need to be improved or redesigned, the diagnostic strategy also needs to be redesigned, and even the significance of secondary localization signs needs to be re-identified.

[0006] (3) Limited diagnostic analysis. BPPV involves three pairs of semicircular canals on both sides, which can be divided into crista calculi and canal calculi. Stones can form in different locations on the semicircular canals, on different sides of the crista ampullaria, or even in multiple canals. Diagnostic tests have found that BPPV involving three semicircular canals can induce vertigo and nystagmus, making the diagnostic analysis quite complicated. The diagnostic tests need to be improved and redesigned.

[0007] Based on the above, BPPV diagnosis and treatment equipment mainly determines the location of the lesion through the characteristics of nystagmus. At the same time, the lesion location is currently mainly targeted at the detection of vestibular lesions, and the positioning accuracy needs to be improved. Therefore, how to accurately locate the BPPV lesion site and study intelligent virtual auxiliary diagnosis systems and methods are of great clinical significance for assisting clinicians to improve the diagnosis and treatment rate of BPPV. Summary of the invention

[0008] In view of the deficiencies existing in the above-mentioned technologies, the present invention provides a pupil tracking method and its BPPV intelligent auxiliary diagnosis method and system, which can realize accurate and rapid pupil segmentation and center positioning, provide accurate pupil position information for subsequent nystagmus information acquisition, and assist doctors in diagnosing BPPV patients.

[0009] In order to achieve these purposes and other advantages according to the present invention, the present invention is implemented by the following technical solutions:

[0010] An embodiment of the present invention provides a pupil tracking method, which comprises the steps of:

[0011] Pupil segmentation: Use a sub-network based on Yolov5 to replace the Xception model in DeepLabv3+, extract features from each frame of the nystagmus video, and obtain a first effective feature layer of 1 / 8 size and a second effective feature layer of 1 / 16 size; perform a first convolution on the first effective feature layer, and send the second effective feature layer to the ASPP enhanced feature extraction module for feature enhancement. The first effective feature layer after the first convolution and the second effective feature layer after the feature enhancement are fused through the FFM module to obtain the pupil segmentation result;

[0012] Pupil boundary and center positioning: performing pupil center positioning on the pupil segmentation result to obtain pupil boundary and pupil center position information.

[0013] Preferably, the second effective feature layer is sent to the ASPP enhanced feature extraction module for feature enhancement, comprising the following steps:

[0014] Performing five operations on the second effective feature layer to obtain five different outputs;

[0015] The five different outputs are sequentially subjected to a concat operation and a second convolution process to obtain an output with a step size of 16x;

[0016] Among them, the five operations include one third convolution process, three dilated convolution processes with different dilation rates, and one ImagePooling process.

[0017] Preferably, after the feature fusion, the method further comprises the following steps:

[0018] The result after feature fusion is adjusted through the fourth convolution, and the output layer size is adjusted by upsampling using resize to obtain the pupil segmentation result.

[0019] Preferably, performing pupil center positioning on the pupil segmentation result to obtain pupil boundary and pupil center position information comprises the steps of:

[0020] Obtaining a mask image of the pupil segmentation result;

[0021] The mask Figure 2 Value;

[0022] The mask image after binarization is fitted with an ellipse using the least squares method to obtain the pupil boundary and pupil center position information.

[0023] A BPPV intelligent auxiliary diagnosis method using a pupil tracking method comprises the following steps:

[0024] Get real-time nystagmus video;

[0025] Performing pupil segmentation and pupil center positioning on each frame of the nystagmus video to obtain pupil boundary and pupil center position information;

[0026] Extracting the pupil center coordinates of each frame image, analyzing the nystagmus trajectory according to the change of the pupil center coordinates in the nystagmus video, and obtaining real-time nystagmus information;

[0027] The pupil segmentation includes: replacing the Xception model in DeepLabv3+ with a sub-network based on Yolov5, performing feature extraction on each frame of the nystagmus video, and obtaining a first effective feature layer of 1 / 8 size and a second effective feature layer of 1 / 16 size; performing a first convolution process on the first effective feature layer, sending the second effective feature layer to the ASPP enhanced feature extraction module for feature enhancement, and completing feature fusion of the first effective feature layer after the first convolution process and the second effective feature layer after the feature enhancement process through the FFM module to obtain the pupil segmentation result.

[0028] Preferably, the BPPV intelligent auxiliary diagnosis method further comprises the following steps:

[0029] Obtain the actual position three-dimensional model of the carrier, and three-dimensionally display the actual position and posture information of the rotation process;

[0030] The universal semicircular canal space model is combined with the carrier head according to the actual position and posture information, the real-time posture information of the semicircular canals during the rotation is three-dimensionally displayed, the real-time nystagmus information is three-dimensionally displayed, and the real-time rotation of the three-dimensional model is controlled according to the real-time posture information of the semicircular canals and the actual position and posture information, and the navigation information for visually adjusting the actual position of the carrier is provided;

[0031] The attitude information includes attitude angle, acceleration in three directions and three-dimensional geomagnetic intensity; the attitude angle includes heading angle, pitch angle and roll angle.

[0032] A BPPV intelligent auxiliary diagnosis system using a pupil tracking method, comprising:

[0033] A nystagmus recording unit, which is used to obtain real-time nystagmus video;

[0034] A data processing unit, which is used to receive the real-time nystagmus video to perform pupil segmentation, pupil boundary and pupil center positioning information, analyze the nystagmus trajectory according to the change of the pupil center position in the real-time nystagmus video, and obtain real-time nystagmus information;

[0035] The pupil segmentation includes: replacing the Xception model in DeepLabv3+ with a sub-network based on Yolov5, performing feature extraction on each frame of the nystagmus video, and obtaining a first effective feature layer of 1 / 8 size and a second effective feature layer of 1 / 16 size; performing a first convolution process on the first effective feature layer, sending the second effective feature layer to the ASPP enhanced feature extraction module for feature enhancement, and completing feature fusion of the first effective feature layer after the first convolution process and the second effective feature layer after the feature enhancement process through the FFM module to obtain the pupil segmentation result.

[0036] Preferably, the BPPV intelligent auxiliary diagnosis system further comprises:

[0037] An orientation unit, which is used to obtain a three-dimensional model of the actual orientation of the carrier and three-dimensionally display the actual orientation and posture information of the rotation process;

[0038] An intelligent auxiliary diagnosis unit, which is used to combine the universal semicircular canal space model with the carrier head according to the actual position and posture information, three-dimensionally display the real-time posture information of the semicircular canals during the rotation, three-dimensionally display the real-time nystagmus information, and provide navigation information for visually adjusting the actual position of the carrier according to the real-time posture information of the semicircular canals and the actual position and posture information;

[0039] The attitude information includes attitude angle, acceleration in three directions and three-dimensional geomagnetic intensity; the attitude angle includes heading angle, pitch angle and roll angle.

[0040] Preferably, the nystagmus recording unit comprises:

[0041] A light source comprising four near-infrared light sources arranged in a matrix;

[0042] a dichroic mirror, which is tilted and positioned above the light source;

[0043] A camera module, which includes a lens, an image sensor, and a digital signal processor;

[0044] The lens is arranged in the middle of the four near-infrared light sources arranged in a matrix, and the four near-infrared light sources arranged in a matrix emit near-infrared light, which is reflected by the dichroic mirror and horizontally incident on the eye to form an image on the retina;

[0045] The lens receives infrared light reflected from the eyeball by the dichroic mirror to form an optical image, which is then projected onto the surface of the image sensor and converted into an electrical signal. The digital signal processor converts the electrical signal into a digital image signal of the pupil's real-time nystagmus video.

[0046] Preferably, the orientation unit comprises:

[0047] A three-axis gyroscope is used to measure the absolute velocity of the carrier in three directions and obtain the attitude angle of the actual orientation of the carrier;

[0048] A three-axis accelerometer, which is used to measure the acceleration of the carrier in three directions;

[0049] A three-axis magnetoresistive magnetometer for measuring three-dimensional geomagnetic intensity to provide initial alignment of the azimuth and to correct for heading drift;

[0050] A signal processor is respectively connected to the three-axis gyroscope, the three-axis accelerometer and the three-axis magnetoresistive magnetometer for communication, receives the attitude angle, the acceleration in the three directions and the three-dimensional geomagnetic intensity for signal processing, and displays the actual orientation attitude information in three dimensions.

[0051] The present invention has at least the following beneficial effects:

[0052] 1. The pupil tracking method provided by the present invention can achieve accurate and fast pupil segmentation through a new target segmentation network framework based on Yolov5 and improved Deeplabv3+ framework, replace the Xception model of Deeplabv3+ with Yolov5, and add the FFM module for feature fusion. In addition, the through-hole center position positioning processing is combined to provide accurate pupil position information for subsequent nystagmus information acquisition, and assist doctors in diagnosing BPPV patients;

[0053] 2. The BPPV intelligent auxiliary diagnosis method provided by the present invention uses a new target segmentation network framework based on Yolov5 and the improved Deeplabv3+ framework to perform pupil segmentation, pupil boundary and pupil center location for each frame of the nystagmus video, and further extract the pupil center coordinates of each frame. Then, according to the changes in the pupil center coordinates, the nystagmus trajectory is analyzed to obtain accurate real-time nystagmus information, which assists doctors in diagnosing BPPV patients.

[0054] 3. The BPPV intelligent auxiliary diagnosis method provided by the present invention also includes combining the universal semicircular canal space model with the carrier head according to the actual position posture information on the basis of obtaining the actual position three-dimensional model of the carrier, so as to achieve the effect of three-dimensionally displaying the real-time posture information of the semicircular canals during rotation and the real-time nystagmus information, and providing navigation information for visually adjusting the actual position of the carrier according to the real-time posture information and the actual position posture information of the semicircular canals, that is, on the one hand, real-time tracking of the three-dimensional spatial changes of the semicircular canals is realized, and intuitive and reliable real-time tracking change data of the three-dimensional spatial changes of the semicircular canals is obtained, which is very helpful for diagnosing otolithiasis, studying the pathogenesis of BPPV and improving innovative resetting techniques, and assisting diagnosis. It can also adjust the individual variation range of the semicircular canal spatial direction and analyze the influence of individual differences in the semicircular canal spatial direction on the resetting effect of different resetting methods; on the other hand, it can provide navigation information for visually adjusting the actual position of the carrier for the correction machine, which will greatly improve the diagnostic efficiency of clinicians.

[0055] 4. The BPPV intelligent auxiliary diagnosis system provided by the present invention is configured to obtain real-time nystagmus video and a data processing unit. The data processing unit builds a new target segmentation network framework of Yolov5 and an improved Deeplabv3+ framework to achieve pupil segmentation, pupil boundary and pupil center position positioning for each frame of the nystagmus video, and further extract the pupil center coordinates of each frame. Then, according to the changes in the pupil center coordinates, the nystagmus trajectory is analyzed to obtain accurate real-time nystagmus information to assist doctors in diagnosing BPPV patients.

[0056] 5. The BPPV intelligent auxiliary diagnosis system provided by the present invention is provided with an orientation unit and an intelligent auxiliary diagnosis unit. The actual orientation three-dimensional model of the carrier is obtained through the orientation unit. The universal semicircular canal space model is combined with the carrier head according to the actual orientation posture information through the intelligent auxiliary diagnosis unit, so as to achieve the effect of three-dimensional display of the real-time posture information of the semicircular canals during rotation and the real-time nystagmus information in three dimensions, without the need for naked eye observation by the doctor, and provide navigation information for visually adjusting the actual orientation of the carrier according to the real-time posture information and the actual orientation posture information of the semicircular canals, that is, on the one hand, real-time tracking of the three-dimensional spatial changes of the semicircular canals is achieved, and intuitive and reliable real-time tracking change data of the three-dimensional spatial changes of the semicircular canals is obtained, which is very helpful for diagnosing otolithiasis, studying the pathogenesis of BPPV and improving innovative resetting techniques, and assisting diagnosis. It can also adjust the individual variation range of the semicircular canal spatial direction and analyze the influence of individual differences in the semicircular canal spatial direction on the resetting effect of different resetting methods; on the other hand, it can provide navigation information for visually adjusting the actual orientation of the carrier for the correction machine, which will greatly improve the diagnostic efficiency of clinicians.

[0057] Other advantages, objectives and features of the present invention will be embodied in part through the following description, and in part will be understood by those skilled in the art through study and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0059] Figure 1 A flow chart of the pupil tracking method provided by the present invention;

[0060] Figure 2 A schematic diagram of the Yolov5-Deeplabv3+ semantic segmentation network framework provided by the present invention;

[0061] Figure 3 Schematic diagram of the Decoder module in the DeepLabv3+ network provided by the present invention;

[0062] Figure 4 Schematic diagram of the Encoder module in the DeepLabv3+ network provided by the present invention;

[0063] FIG5( a) is a mask diagram of the pupil segmentation result provided by the present invention;

[0064] FIG5( b ) is a binary image of the pupil segmentation result mask image provided by the present invention;

[0065] FIG5( c ) is a diagram showing the position information of the pupil boundary and pupil center obtained based on the least squares ellipse fitting provided by the present invention;

[0066] Figure 6 A flow chart of the BPPV intelligent auxiliary diagnosis method based on pupil tracking method provided by the present invention;

[0067] Figure 7 A flowchart of a method for setting a three-dimensional display-assisted diagnosis for the BPPV intelligent-assisted diagnosis method provided by the present invention;

[0068] Figure 8 The composition and communication diagram of the BPPV intelligent auxiliary diagnosis system provided by the present invention;

[0069] Fig. 9 A schematic diagram of the composition and communication of the nystagmus recording unit provided by the present invention;

[0070] Fig.10 A schematic diagram of a real-time nystagmus picture collected by the nystagmus recording unit provided by the present invention;

[0071] Fig.11 A schematic diagram of the composition and communication of the azimuth unit provided by the present invention;

[0072] In the figure:

[0073] 100-nystagmus recording unit;

[0074] 110-near infrared light source; 120-dichroic mirror; 130-camera module;

[0075] 131- lens; 132- image sensor; 133- digital signal processor;

[0076] 200-data processing unit;

[0077] 300-azimuth unit;

[0078] 310-three-axis gyroscope; 320-three-axis accelerometer; 330-three-axis magnetoresistive magnetometer; 340-signal processor; 350-temperature sensor;

[0079] 400-Intelligent auxiliary diagnosis unit. DETAILED DESCRIPTION

[0080] The technical solutions of the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0081] The terms such as “having”, “including” and “comprising” used in the different embodiments of the present invention described below do not exclude the existence or addition of one or more other elements or a combination thereof; the technical features involved can be combined with each other as long as there is no conflict between them.

[0082] <Implementation Method 1>

[0083] like Figure 1 and 2 As shown, an embodiment of the present invention provides a pupil tracking method, wherein the pupil segmentation network is constructed by integrating Yolov5 and a sub-network improved based on DeepLabv3+, specifically comprising the following steps:

[0084] S10, pupil segmentation:

[0085] S11, replace the Xception model in DeepLabv3+ with a sub-network based on Yolov5, extract features from each frame of the nystagmus video, and obtain the first effective feature layer of 1 / 8 size and the second effective feature layer of 1 / 16 size;

[0086] S12, performing a first convolution process on the first effective feature layer, sending the second effective feature layer to the ASPP enhanced feature extraction module for feature enhancement, and completing feature fusion of the first effective feature layer after the first convolution process and the second effective feature layer after the feature enhancement process through the FFM module to obtain a pupil segmentation result;

[0087] S20, pupil border and center positioning:

[0088] The pupil center is located on the pupil segmentation result to obtain the pupil boundary and pupil center position information.

[0089] In step S11, the Backbone used in the original DeepLabv3+ network is the Xception model, which makes extensive use of deep separable convolutions. Although this greatly improves the accuracy of semantic segmentation, it significantly reduces the calculation speed. Yolov5 has excellent performance in target detection. Its backbone uses a large number of CSP structures to solve the problem of gradient duplication in the optimization process. It supports feature propagation and feature reuse, thereby reducing parameters and calculations, improving the network's running speed, and is conducive to ensuring the network's real-time performance. Therefore, the Backbone of the Encoder module in the original DeepLabv3+ network is replaced with the Yolov5 subnetwork. For the improved DeepLabv3+ subnet, the size of the original DeepLabv3+ shallow feature layer is further modified, because the 1 / 4 size feature layer of Yolov5 is too shallow and has too few channels, only 64, while the feature layer obtained by DeepLabv3+ Backbone often has more than 256, and the 1 / 4 size feature layer is finally refined with 3*3, which is too computationally intensive. Therefore, in step S11, the first effective feature layer of 1 / 8 size and the second effective feature layer of 1 / 16 size are respectively taken, which is conducive to obtaining better segmentation results.

[0090] In step S12, a first convolution process is performed on the first effective feature layer to adjust the number of channels. Preferably, a 1×1 convolution is performed on the first effective feature layer to adjust the number of channels. ASPP (Atrous Spatial PyramidPooling), atrous spatial convolution pooling pyramid. Using the ASPP enhanced feature extraction module to enhance the features of the second effective feature layer of size 1 / 16 is to increase the receptive field of the network without downsampling and enhance the network's ability to obtain multi-scale context. The fusion method adopted by Deeplabv3+ is a simple stacking of two feature layers, and the FFM module replaces the original simple stacking. The FFM module fuses the features of the spatial path and the context path with good results. The spatial information captured by the spatial path encodes most of the rich detail information, and the output features of the context path mainly encode the context information, that is, the output features of the spatial path are low-level, while the output features of the context path are high-level. In step S12 of this embodiment, the second effective feature layer of 1 / 16 size obtained from the Encoder module contains sufficient context information and is a high-level feature. The first effective feature layer of 1 / 8 size obtained from the Yolov5 backbone contains most of the rich detail information and is a low-level feature. Therefore, Figure 3 As shown in the figure, using the FFM module to fuse these features can achieve better results than simply superimposing them.

[0091] In the above implementation, by using a new target segmentation network framework based on Yolov5 and an improved Deeplabv3+ framework, replacing the Xception model of Deeplabv3+ with Yolov5, and adding an FFM module for feature fusion, accurate and fast pupil segmentation can be achieved. Coupled with the through-hole center position positioning processing, it is beneficial to provide accurate pupil position information for subsequent nystagmus information acquisition, and assist doctors in diagnosing BPPV patients.

[0092] As a further preference, in step S12, the second effective feature layer is sent to the ASPP enhanced feature extraction module for feature enhancement, including the following steps:

[0093] Perform 5 operations on the second effective feature layer to obtain 5 different outputs;

[0094] The five different outputs are processed by a concat operation and a second convolution to obtain an output with a step size of 16x;

[0095] Among them, the five operations include one third convolution process, three dilated convolution processes with different dilation rates, and one ImagePooling process.

[0096] In this embodiment, if Figure 4 As shown, the ASPP enhanced feature extraction module is essentially composed of a 1×1 convolution + pooling pyramid + ASPP Pooling. Therefore, the third convolution process is a 1×1 convolution process, and the expansion factor of each layer of the pooling pyramid can be customized. Therefore, three different expansion rates can be customized, thereby realizing free multi-scale feature extraction. The second convolution is also preferably a 1×1 convolution process.

[0097] As a further preferred embodiment, in step S12, after feature fusion, the method further includes: performing channel adjustment on the result after feature fusion through a fourth convolution, and adjusting the output layer size through upsampling by resize to obtain a pupil segmentation result. In this implementation, the fourth convolution may be a 1×1 convolution.

[0098] As a further preferred embodiment, in step S20, pupil center positioning is performed on the pupil segmentation result to obtain pupil boundary and pupil center position information, including the steps of:

[0099] S21, obtaining a mask image of the pupil segmentation result;

[0100] S22, mask Figure 2 Value;

[0101] S23, performing ellipse fitting using the least squares method on the binarized mask image to obtain pupil boundary and pupil center position information.

[0102] In this embodiment, in step S21, after the pupil segmentation network, a mask image of the pupil segmentation result is first obtained, as shown in FIG5(a), and effective boundary information is extracted based on the mask image. Figure 2 The valorization is to remove various noise interferences of the original image from the mask image after pupil segmentation, as shown in Figure 5(b). In step S23, the ellipse fitting is based on the least squares method. Specifically, the second-order polynomial conic curve expression of the ellipse is: The constraints are: b 2 -4ac<0; where (x,y) are the coordinates of the input point, the polynomial Represents a point (x, y) to an ellipse The algebraic distance of The value of the coefficient in the equation minimizes the algebraic distance between the edge data points obtained and the fitted ellipse, and the edge data points obtained are processed by least squares, that is, The minimum value is determined The values ​​of the coefficients in the equation are used to determine the equation of the ellipse and the coordinates of the center of the ellipse, as shown in Figure 5(c). As can be seen from Figure 5(c), the fitting result contains accurate pupil boundary and center information.

[0103] In order to better illustrate the pupil segmentation effect of the embodiment, further experimental description is now provided.

[0104] The improved Yolov5-Deeplabv3+ semantic segmentation network is used to segment pupil data. For comparison, Deeplabv3+ is used as the basic network, and Yolov5 and FFM modules are added to it to study the impact of different modules on experimental performance. The fused semantic segmentation network performs well on the test set, with more accurate segmentation, smoother pupil boundaries, and most pupil pixels are segmented.

[0105] In addition, Table 1 lists the evaluation indicators of the semantic segmentation network after the fusion of Yolov5 and Deeplabv3+ and the semantic segmentation network of Deeplabv3+. It can be seen from the table that the accuracy index of the fused segmentation is higher than the accuracy index of the original Deeplabv3+ segmentation and the real-time performance is better.

[0106] Table 1 Comparison of Deeplabv3+, Yolov5-Deeplabv3+ and whether it uses the FFM module

[0107]

[0108] At the same time, this embodiment compares some semantic segmentation methods, such as UNET, PSPNet, BiSeNet, BiSeNetv, Yolov5-PSPNet, and Yolov5-BiSeNet. On the Dikablis dataset, the experimental results obtained by different semantic segmentation networks are shown in Table 2, among which Yolov5-Deeplabv3+ achieved 95.46%, 99.24%, 96.51%, and 76.3FPS in IOU, Precision, Recall, and FPS. It can be seen from Table 2 that compared with other classic semantic segmentation networks, the Yolov5-Deeplabv3+ network proposed in this embodiment has stronger segmentation ability and segmentation speed for tooth surface and pitting. In addition, in the experiments of adding Yolov5 feature extraction to PSPNet and BiSeNet, it can be seen that the accuracy of segmentation has been slightly improved, but the running speed has been greatly accelerated, which proves that the application of Yolov5 in the field of semantic segmentation can effectively improve the speed of segmentation. In the comparison of three different segmentation networks using Yolov5, it can be seen that Yolov5-Deeplabv3+ has the best segmentation accuracy, but the inference speed is slower than the other two. This is mainly because a large number of dilated convolutions are used in the ASPP feature enhancement module, which will significantly affect the inference speed of the model, but its improvement on the model segmentation performance is huge. Considering the needs in actual use, 76.3 FPS has met the need for real-time acquisition of nystagmus information in BPPV intelligent auxiliary diagnosis, so in pursuit of higher segmentation accuracy, this implementation method selects Yolov5-Deeplabv3+ as the pupil segmentation framework.

[0109] Table 2 Comparison of indicators with classic semantic segmentation networks

[0110]

[0111]

[0112] <Implementation Method 2>

[0113] This embodiment provides a BPPV intelligent auxiliary diagnosis method using the pupil tracking method proposed in embodiment 1, such as Figure 6 As shown, it includes the following steps:

[0114] S30, obtaining real-time nystagmus video;

[0115] S40, performing pupil segmentation and pupil center positioning on each frame of the nystagmus video to obtain pupil boundary and pupil center position information;

[0116] S50, extracting the pupil center coordinates of each frame image, analyzing the nystagmus trajectory according to the change of the pupil center coordinates in the nystagmus video, and obtaining real-time nystagmus information;

[0117] Among them, Figure 1 As shown in the figure, pupil segmentation includes: replacing the Xception model in DeepLabv3+ with a sub-network based on Yolov5, performing feature extraction on each frame of the nystagmus video, and obtaining a first effective feature layer of 1 / 8 size and a second effective feature layer of 1 / 16 size; performing a first convolution process on the first effective feature layer, sending the second effective feature layer to the ASPP enhanced feature extraction module for feature enhancement, and completing feature fusion of the first effective feature layer after the first convolution process and the second effective feature layer after the feature enhancement process through the FFM module to obtain the pupil segmentation result.

[0118] In this embodiment, the pupil segmentation, pupil boundary and pupil center position of each frame image in the nystagmus video are performed by using the pupil tracking method provided in Embodiment 1, and the pupil center coordinates of each frame image are further extracted, and then the nystagmus trajectory is analyzed according to the change of the pupil center coordinates to obtain real-time nystagmus information. Due to the accurate pupil segmentation in the pupil tracking method, accurate pupil position information is provided for the accurate acquisition of real-time nystagmus information, which assists doctors in diagnosing BPPV patients.

[0119] As a further preferred embodiment of the above embodiment, Figure 7 As shown, the BPPV intelligent auxiliary diagnosis method also includes the following steps:

[0120] S60, obtaining a three-dimensional model of the actual position of the carrier, and three-dimensionally displaying the actual position and posture information of the rotation process;

[0121] S70, combining the universal semicircular canal space model with the carrier head according to the actual position and posture information, three-dimensionally displaying the real-time posture information of the semicircular canals during rotation, three-dimensionally displaying the real-time nystagmus information, and providing visual navigation information for adjusting the actual position of the carrier according to the real-time posture information of the semicircular canals and the actual position and posture information; wherein the posture information includes the attitude angle, accelerations in three directions and the three-dimensional geomagnetic intensity; the attitude angle includes the heading angle, the pitch angle and the roll angle.

[0122] In this embodiment, on the basis of obtaining the actual position three-dimensional model of the carrier, the universal semicircular canal space model is combined with the carrier head according to the actual position posture information, so as to achieve the effect of three-dimensional display of the real-time posture information of the semicircular canals during rotation and the real-time nystagmus information, and provide navigation information for visually adjusting the actual position of the carrier according to the real-time posture information and the actual position posture information of the semicircular canals. That is, on the one hand, real-time tracking of the three-dimensional spatial changes of the semicircular canals is achieved, and intuitive and reliable real-time tracking change data of the three-dimensional space of the semicircular canals is obtained, which is very helpful for diagnosing otolithiasis, studying the pathogenesis of BPPV and improving innovative resetting techniques. It can also assist in diagnosis, adjust the individual variation range of the semicircular canal spatial direction, and analyze the influence of individual differences in the semicircular canal spatial direction on the resetting effect of different resetting methods; on the other hand, it can provide navigation information for visual adjustment of the actual position of the carrier for the correction machine, which will greatly improve the diagnostic efficiency of clinicians.

[0123] <Implementation Method 3>

[0124] Based on the pupil tracking method provided in Implementation 1 and the BPPV intelligent auxiliary diagnosis method provided in Implementation 2, this implementation provides a BPPV intelligent auxiliary diagnosis system, such as Figure 8 As shown, it includes a nystagmus recording unit 100 and a data processing unit 200. The nystagmus recording unit 100 is used to obtain real-time nystagmus video; the data processing unit 200 is used to receive the real-time nystagmus video to perform pupil segmentation, pupil boundary and pupil center positioning information, analyze the nystagmus trajectory according to the change of the pupil center position in the real-time nystagmus video, and obtain real-time nystagmus information; wherein, pupil segmentation includes: using a sub-network based on Yolov5 to replace the Xception model in DeepLabv3+, extracting features for each frame of the nystagmus video, and obtaining a first effective feature layer of 1 / 8 size and a second effective feature layer of 1 / 16 size; performing a first convolution process on the first effective feature layer, sending the second effective feature layer to the ASPP enhanced feature extraction module for feature enhancement, completing feature fusion of the first effective feature layer after the first convolution process and the second effective feature layer after the feature enhancement process through the FFM module, and obtaining the pupil segmentation result.

[0125] In this embodiment, the data processing unit 200 uses the pupil tracking method provided in embodiment 1 and the BPPV intelligent auxiliary diagnosis method provided in embodiment 2 to perform pupil segmentation, pupil boundary and pupil center position positioning for each frame image in the nystagmus video, and further extract the pupil center coordinates of each frame image, and then analyze the nystagmus trajectory according to the change of the pupil center coordinates to obtain real-time nystagmus information. Due to the precise pupil segmentation in the pupil tracking method, accurate pupil position information is provided for the precise acquisition of real-time nystagmus information, assisting doctors in diagnosing BPPV patients. It should be supplemented that, as for the structure of the nystagmus recording unit 100, it is preferred to use a wearable eye mask structure to facilitate wearing and obtaining real-time nystagmus video.

[0126] As a further preference of the above-mentioned embodiment, the BPPV intelligent auxiliary diagnosis system also includes an orientation unit 300 and an intelligent auxiliary diagnosis unit 400. The orientation unit 300 is used to obtain a three-dimensional model of the actual orientation of the carrier, and to display the actual orientation posture information of the rotation process in three dimensions. The intelligent auxiliary diagnosis unit 400 is used to combine the universal semicircular canal space model with the carrier head according to the actual orientation posture information, to display the real-time posture information of the semicircular canals during the rotation process in three dimensions, to display the real-time nystagmus information in three dimensions, and to control the real-time rotation of the three-dimensional model according to the real-time posture information of the semicircular canals and the actual orientation posture information, and to provide navigation information for visually adjusting the actual orientation of the carrier; wherein the posture information includes the attitude angle, acceleration in three directions, and three-dimensional geomagnetic intensity; the attitude angle includes heading angle, pitch angle, and roll angle.

[0127] In this embodiment, the actual orientation three-dimensional model of the carrier is obtained by the orientation unit 300. On this basis, the universal semicircular canal space model is combined with the carrier head according to the actual orientation posture information through the intelligent auxiliary diagnosis unit 400, so as to achieve the effect of three-dimensional display of the real-time posture information of the semicircular canals during rotation and the real-time nystagmus information. The doctor does not need to observe with the naked eye, and provides navigation information for visually adjusting the actual orientation of the carrier according to the real-time posture information and the actual orientation posture information of the semicircular canals. That is, on the one hand, real-time tracking of the three-dimensional spatial changes of the semicircular canals is achieved, and intuitive and reliable real-time tracking change data of the three-dimensional spatial changes of the semicircular canals is obtained, which is very helpful for diagnosing otolithiasis, studying the pathogenesis of BPPV and improving innovative reduction techniques. It can also assist in diagnosis, adjust the individual variation range of the semicircular canal spatial direction, and analyze the influence of individual differences in the semicircular canal spatial direction on the reduction effect of different reduction methods. On the other hand, it can also provide navigation information for visual adjustment of the actual orientation of the carrier for the correction machine, which will greatly improve the diagnostic efficiency of clinicians.

[0128] As a further preferred embodiment of the above embodiment, Fig. 9As shown, the nystagmus recording unit 100 includes a light source, a dichroic mirror 120, and a camera module 130. The light source includes four near-infrared light sources 110 arranged in a matrix, which are used to emit near-infrared light. The dichroic mirror 120 is tilted above the light source. The camera module 130 includes a lens 131, an image sensor 132, and a digital signal processor 133. Specifically, the lens 131 is arranged in the middle of the four near-infrared light sources 110 arranged in a matrix. The four near-infrared light sources 110 arranged in a matrix emit near-infrared light, which is reflected by the dichroic mirror 120 and horizontally captured by the eye to form an image on the retina; the lens 131 receives the infrared light reflected back from the eyeball through the dichroic mirror 120 to form an optical image, which is then projected on the surface of the image sensor 132 and converted into an electrical signal. The digital signal processor 133 converts the electrical signal into a digital image signal of the real-time nystagmus video for subsequent signal transmission and data processing.

[0129] In this embodiment, considering the imaging effect and the fact that there should be no harm to the human eye, the wavelength of the four near-infrared light sources 110 is selected to be 850nm, and the power is selected to be 50mw; the dichroic mirror 120 is selected to be 400-700nm for transmission, 800-950nm for reflection, and 50mm in diameter; the camera module 130 can use a camera with an acquisition frame rate of 1080p 30 frames (640*480 120 frames), a lens focal length of 12mm, a viewing angle of 30 degrees, a transmission interface of USB, and supports the UVC driver-free protocol, and transmits the collected digital image signal of the real-time nystagmus video to the intelligent auxiliary diagnosis unit 400, such as a computer, for further algorithm analysis. As a further preference, the angle between the incident light of the near-infrared light source 110 and the dichroic mirror 120 is set to 45 degrees, the angle between the reflected light of the eye and the dichroic mirror 120 is also set to 45 degrees, the axial distance between the lens 131 and the eye is set to 3 cm, and the vertical distance from the lens 131 to the dichroic mirror 120 is 5 cm, then the object distance of the camera is 3 cm + 5 cm = 8 cm, then the focal length of the lens 131 ensures that the image is clearest when the object distance is 8 cm, and the dark pupil required for the subsequent segmentation algorithm processing and the Purkinje spots used for nystagmus analysis calibration can be obtained, such as Fig.10 As further preferred, the image sensor 132 is preferably a CMOS image detector, and the digital signal processor 133 preferably includes an AD converter with analog-to-digital conversion function and a video signal acquisition chip, supporting wireless data transmission and reception.

[0130] As a further preferred embodiment of the above embodiment, Fig.11As shown, the orientation unit 300 includes a three-axis gyroscope 310, a three-axis accelerometer 320, a three-axis magnetoresistive magnetometer 330, and a signal processor 340. The three-axis gyroscope 310 is used to measure the absolute solution rate of the carrier in three directions, and the attitude angle of the actual orientation of the carrier is obtained by calculating the absolute solution rate in the three directions; the three-axis accelerometer 320 is used to measure the acceleration of the carrier in three directions; the three-axis magnetoresistive magnetometer 330 is used to measure the three-dimensional geomagnetic intensity to provide the initial alignment of the direction angle and correct the heading angle drift; the signal processor 340 is respectively connected to the three-axis gyroscope 310, the three-axis accelerometer 320, and the three-axis magnetoresistive magnetometer 330 for signal processing, and displays the actual orientation attitude information in three dimensions.

[0131] As a further preference, the orientation unit 300 also includes sensors for measuring other parameters of the carrier to provide data reference and support for auxiliary diagnosis, such as a temperature sensor 350. The temperature sensor 350 is communicatively connected to the signal processor 340, and the signal processor 340 controls the temperature sensor 350 to collect the real-time temperature of the carrier and display it in real time, which are not listed one by one in the present invention.

[0132] Obviously, the above embodiments are merely examples for clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from them are still within the protection scope of the invention.

Claims

1. A pupil tracking method, characterized in that: It includes the following steps: Pupil segmentation: Use a sub-network based on Yolov5 to replace the Xception model in DeepLabv3+, extract features from each frame of the nystagmus video, and obtain a first effective feature layer of 1 / 8 size and a second effective feature layer of 1 / 16 size; perform a first convolution on the first effective feature layer, and send the second effective feature layer to the ASPP enhanced feature extraction module for feature enhancement. The first effective feature layer after the first convolution and the second effective feature layer after the feature enhancement are fused through the FFM module to obtain the pupil segmentation result; Pupil boundary and center positioning: performing pupil center positioning on the pupil segmentation result to obtain pupil boundary and pupil center position information.

2. The pupil tracking method according to claim 1, wherein: The second effective feature layer is sent to the ASPP enhanced feature extraction module for feature enhancement, including the following steps: Performing five operations on the second effective feature layer to obtain five different outputs; The five different outputs are sequentially subjected to a concat operation and a second convolution process to obtain an output with a step size of 16x; Among them, the five operations include one third convolution process, three dilated convolution processes with different dilation rates, and one ImagePooling process.

3. The pupil tracking method according to claim 1, wherein: After feature fusion, the following steps are also included: The result after feature fusion is adjusted through the fourth convolution, and the output layer size is adjusted by upsampling using resize to obtain the pupil segmentation result.

4. The pupil tracking method according to claim 1, wherein: Performing pupil center positioning on the pupil segmentation result to obtain pupil boundary and pupil center position information includes the following steps: Obtaining a mask image of the pupil segmentation result; Binarizing the mask image; The mask image after binarization is fitted with an ellipse using the least squares method to obtain the pupil boundary and pupil center position information.

5. A BPPV intelligent auxiliary diagnosis method using the pupil tracking method according to claims 1-4, characterized in that: It includes the following steps: Get real-time nystagmus video; Performing pupil segmentation and pupil center positioning on each frame of the nystagmus video to obtain pupil boundary and pupil center position information; Extracting the pupil center coordinates of each frame image, analyzing the nystagmus trajectory according to the change of the pupil center coordinates in the nystagmus video, and obtaining real-time nystagmus information; The pupil segmentation includes: replacing the Xception model in DeepLabv3+ with a sub-network based on Yolov5, performing feature extraction on each frame of the nystagmus video, and obtaining a first effective feature layer of 1 / 8 size and a second effective feature layer of 1 / 16 size; performing a first convolution process on the first effective feature layer, sending the second effective feature layer to the ASPP enhanced feature extraction module for feature enhancement, and completing feature fusion of the first effective feature layer after the first convolution process and the second effective feature layer after the feature enhancement process through the FFM module to obtain the pupil segmentation result.

6. The BPPV intelligent auxiliary diagnosis method according to claim 5, characterized in that: It also includes the following steps: Obtain the actual position three-dimensional model of the carrier, and three-dimensionally display the actual position and posture information of the rotation process; The universal semicircular canal space model is combined with the carrier head according to the actual position and posture information, the real-time posture information of the semicircular canals during the rotation is three-dimensionally displayed, the real-time nystagmus information is three-dimensionally displayed, and the real-time rotation of the three-dimensional model is controlled according to the real-time posture information of the semicircular canals and the actual position and posture information, and the navigation information for visually adjusting the actual position of the carrier is provided; The attitude information includes attitude angle, acceleration in three directions and three-dimensional geomagnetic intensity; the attitude angle includes heading angle, pitch angle and roll angle.

7. A BPPV intelligent auxiliary diagnosis system using the pupil tracking method as described in claims 1-4, characterized in that: It includes: A nystagmus recording unit, which is used to obtain real-time nystagmus video; A data processing unit, which is used to receive the real-time nystagmus video, perform pupil segmentation and pupil center positioning to obtain pupil boundary and pupil center positioning information, analyze the nystagmus trajectory according to the change of the pupil center position in the real-time nystagmus video, and obtain real-time nystagmus information; The pupil segmentation includes: replacing the Xception model in DeepLabv3+ with a sub-network based on Yolov5, performing feature extraction on each frame of the nystagmus video, and obtaining a first effective feature layer of 1 / 8 size and a second effective feature layer of 1 / 16 size; performing a first convolution process on the first effective feature layer, sending the second effective feature layer to the ASPP enhanced feature extraction module for feature enhancement, and completing feature fusion of the first effective feature layer after the first convolution process and the second effective feature layer after the feature enhancement process through the FFM module to obtain the pupil segmentation result.

8. The BPPV intelligent auxiliary diagnosis system as claimed in claim 7, characterized in that: It also includes: An orientation unit, which is used to obtain a three-dimensional model of the actual orientation of the carrier and three-dimensionally display the actual orientation and posture information of the rotation process; An intelligent auxiliary diagnosis unit, which is used to combine the universal semicircular canal space model with the carrier head according to the actual position and posture information, three-dimensionally display the real-time posture information of the semicircular canals during the rotation, three-dimensionally display the real-time nystagmus information, and provide navigation information for visually adjusting the actual position of the carrier according to the real-time posture information of the semicircular canals and the actual position and posture information; The attitude information includes attitude angle, acceleration in three directions and three-dimensional geomagnetic intensity; the attitude angle includes heading angle, pitch angle and roll angle.

9. The BPPV intelligent auxiliary diagnosis system as claimed in claim 7, characterized in that: The nystagmus recording unit comprises: A light source comprising four near-infrared light sources arranged in a matrix; a dichroic mirror, which is tilted and positioned above the light source; A camera module, which includes a lens, an image sensor, and a digital signal processor; The lens is arranged in the middle of the four near-infrared light sources arranged in a matrix, and the four near-infrared light sources arranged in a matrix emit near-infrared light, which is reflected by the dichroic mirror and horizontally incident on the eye to form an image on the retina; The lens receives infrared light reflected from the eyeball by the dichroic mirror to form an optical image, which is then projected onto the surface of the image sensor and converted into an electrical signal. The digital signal processor converts the electrical signal into a digital image signal of the pupil's real-time nystagmus video.

10. The BPPV intelligent auxiliary diagnosis system as claimed in claim 8, characterized in that: The orientation unit comprises: A three-axis gyroscope is used to measure the absolute velocity of the carrier in three directions and obtain the attitude angle of the actual orientation of the carrier; A three-axis accelerometer, which is used to measure the acceleration of the carrier in three directions; A three-axis magnetoresistive magnetometer for measuring three-dimensional geomagnetic intensity to provide initial alignment of the azimuth and to correct for heading drift; A signal processor is respectively connected to the three-axis gyroscope, the three-axis accelerometer and the three-axis magnetoresistive magnetometer for communication, receives the attitude angle, the acceleration in the three directions and the three-dimensional geomagnetic intensity for signal processing, and displays the actual orientation attitude information in three dimensions.

Images