Eye muscle source compensatory head position auxiliary identification device based on instantaneous eye movement capture

Abstract

The invention provides an eye muscle derived compensatory head position auxiliary identification device based on instantaneous eye movement capture, and relates to the technical field of medical instruments, the eye muscle derived compensatory head position auxiliary identification device comprises a rotary video playing module, a skeletal muscle key point capture module, a capture module and a photo output module; the detected person watches the video, the video rotates in the reverse direction from the compensatory head position to the horizontal position, if the head position at the horizontal position is not improved, the video rotates towards the compensatory head position, in the rotating process, skeletal muscle key points capture the opposite sides of face and neck key points of the detected person and the head position is digitized, a capture module is started, photos are automatically captured, and eye position abnormity is analyzed through the photos; when eye position abnormity exists in the snapshot image, the snapshot image can be determined as an eye muscle source compensatory head position, and when the head position improvement is not obvious, one eye of the examiner is covered, and the other eye continues to watch the video. And the detected person watches the video, and after the video stops rotating, the head position and the fixation point are kept stable until the photo is automatically captured. When the head position of the snapshot image is improved, the head position can be clearly determined as the eye muscle-derived compensatory head position.

Description

Technical Field

[0001] The invention relates to the technical field of medical devices, and in particular to an eye muscle-derived compensatory head position auxiliary identification device based on instantaneous eye movement capture. Background Art

[0002] Torticollis is a common symptom in childhood, and 1 / 4 of the children have a compensatory head position caused by strabismus. The reason why these patients adopt a compensatory head position is that the visual problems under the normal head position include: abnormal eye position and nystagmus will be more prominent. Patients usually use head position compensation to obtain the best vision or maintain binocular single vision function, resulting in tilted head viewing, that is, torticollis. Ophthalmologists often encounter patients with torticollis in clinical work. These patients are often young and it is difficult for them to cooperate with the examination seriously, so that doctors cannot determine whether the cause is a compensatory head position of eye muscle origin. At present, the cause of compensatory head position is mainly determined clinically by correcting the patient's head position and observing the eye position, combined with complex eye muscle examinations. However, this process is complicated and young children cannot cooperate with the correction of the head position, and the examination is easily affected by the patient's cooperation. Therefore, a simple, fast, and easy-to-cooperate device is needed to assist in identifying whether the compensatory head position is caused by eye muscle origin, so as to facilitate further ophthalmic examinations. Summary of the invention

[0003] In view of the shortcomings of the prior art, the present invention provides an eye muscle-derived compensatory head position auxiliary identification device based on instantaneous eye movement capture, which fully offsets the compensation of head position by looking at the tilt of the object. When the skeleton key points capture the symmetry of the left and right facial data, the device instantly captures the eye position of both eyes with a photo.

[0004] An eye muscle-derived compensatory head position auxiliary identification device based on instantaneous eye movement capture, comprising: a rotating video playback module, a skeletal muscle key point capture module, a capture module, and a photo output module;

[0005] The rotating video playback module includes a display screen and a speaker. The display screen plays an animation and rotates a set degree at a set time through a video rotation algorithm. It first rotates in the opposite direction of the compensatory head position. When the head position is still not improved after rotating to the horizontal position, it rotates in the same direction of the compensatory head position to improve the head position.

[0006] The software part of the display screen includes three modules, namely, video rendering engine, video playback control, and user interaction platform; the video rendering engine is: when playing the video, the video rotation algorithm is used through the graphics library to change the vertex coordinates or use the shader to achieve the rotation effect, the video screen is rotated, the video stream is decoded into frames using the video decoding library, each frame is loaded into the Framebuffer as a texture, and the rotation matrix is ​​applied to display the rotated picture on the screen; the video playback control is: creating a user interface to allow the user to choose to control the playback, specifically including controlling the playback and pause of the video; the user interaction platform is: the user interacts during the video playback and adjusts the rotation angle of the video;

[0007] The skeletal muscle key point capture module includes a camera and an embedded processor. The camera is used to collect the patient's front image in real time, wherein the patient's front image resolution is not less than 1080P; data management and communication are performed on the collected data stream, and the collected patient's front image is transmitted to the embedded processor in real time through the WebSocket protocol; the embedded processor performs sensor data processing on the image, and then extracts the skeletal key point data through the skeletal key point detection algorithm; a skeletal muscle symmetry threshold is set, and when the coordinate deviation of the left and right corresponding points is within the threshold, it is determined that the head position is correct;

[0008] The skeleton key point detection algorithm locates anatomical landmarks from images through computer vision technology, and then constructs the human skeleton topology. Specifically, after inputting the patient's frontal image, a pre-trained CNN backbone network is used to extract multi-scale features; a key point heat map and a part association field are generated respectively through a dual-branch network; the Hungarian algorithm is used to complete key point matching, and the existing algorithms OpenPose's Part Affinity Fields and AlphaPose's RMPE framework are used to detect facial and neck key points.

[0009] The sensor data processing is responsible for the preprocessing and fusion of multimodal data. For the frontal image of the patient captured by the camera, white balance correction and non-uniform illumination compensation are first performed, and then Kalman filtering is applied to eliminate head micro-motion noise. At the same time, hardware timestamp synchronization technology is used to ensure that the video stream and the inertial measurement unit data are aligned in time domain, with a time error of less than 3ms.

[0010] The data management and communication are transmitted through real-time data streams, and the data frames are encoded using Protobuf; batch data is uploaded to the cloud, and a hybrid architecture is used for data storage: a time series database records the movement trajectory of key points, a relational database manages patient metadata, and Redis caches real-time detection results; security complies with HIPAA standards, uses AES-256-GCM to encrypt key point data, and protects patient privacy through facial feature desensitization.

[0011] The capture module is implemented by a zoom camera, an embedded processor, and a display screen. The zoom camera captures the patient's eye movement and eye position in real time, transmits them to the embedded processor, and visualizes them through the display screen. The eye tracking algorithm and calibration are performed. The eye gaze point position is extracted through the iris-corneal reflection method. A 9-point calibration is performed before the video is played to establish a screen-eye mapping relationship. The eye tracking data is aligned with the video frame timestamp, the eye movement events are calculated, and the eye movement event data is transmitted to the front-end web page in real time using the communication protocol WebSocket to realize the analysis and visualization of the eye movement events.

[0012] The eye tracking algorithm and calibration are optimized for children's adaptability based on the traditional iris-corneal reflection method. Specifically, a 9-point calibration guided by dynamic animation is adopted, and dynamic attention guidance is added. During the eye tracking process, when it is detected that the gaze point deviates from the center of the screen by more than 50% of the area for more than 5 seconds, a flashing effect is automatically superimposed in the animation to guide the line of sight back; in addition, the movement speed of all dynamic elements is ≤0.6m / s, and blink compensation is designed. When the pupil occlusion rate is detected to be >90%, data collection is suspended for 150ms, and historical data is used for interpolation and filling; head micro-motion correction is designed, and the nose tip coordinates of the skeleton key point data are combined to correct the error caused by head displacement in real time through affine transformation;

[0013] The analysis and visualization of the eye movement events are as follows: by processing the eye movement data stream in real time, combining multi-dimensional parameters to identify eye movement events, specifically including: when the eye movement speed lasts <30° / s and the duration is ≥100ms, it is determined as a fixation event; when the instantaneous eye speed is >80° / s, it is determined as a saccade event, and the amplitude and direction vector are calculated; when the pupil occlusion rate is >90% and lasts for 30-150ms, feature recognition determines it as a blink; real-time comparison of the ratio of the eye movement speed to the target movement speed determines the tracking movement event; when the retracement angle of the saccade path is >120°, it is marked as abnormal and returns to the saccade event; after capture, analyze and compare the eye movement event data of children of the same age, and visualize the abnormal indicators in the eye movement events;

[0014] The photo output module is implemented through a display screen. When the head position is corrected and the gaze point is stable, the current picture is automatically captured to generate a photo, which is used to visualize the eye position under the gaze point mapping after the head position is corrected; wherein the technology involves gaze point mapping and visualization;

[0015] When the skeletal muscle key point capture module performs head position correction judgment, if the head position cannot be corrected autonomously, monocular covering is performed, and the head position is evaluated under monocular gaze again by rotating the video playback module.

[0016] The beneficial effects of adopting the above technical solution are:

[0017] The present invention provides an eye muscle-induced compensatory head position auxiliary identification device based on instantaneous eye movement capture, the present invention provides a method for determining the cause of the compensatory head position by alleviating the eye muscle-induced compensatory head position by screen rotation, and according to the equipment requirements in the process of the method, a set of eye muscle-induced compensatory head position auxiliary identification device is invented. Compared with the traditional identification method, the device is simple to operate and can be completed by young children. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a flowchart of the auxiliary identification process of eye muscle-derived compensatory head position of the present invention;

[0019] Figure 2 It is a schematic diagram of the system structure of the present invention;

[0020] Figure 3 It is a schematic diagram of the external structure of the present invention;

[0021] Figure 4 It is a flow chart of the operation method of the eye muscle-derived compensatory head position auxiliary identification device in an embodiment of the present invention;

[0022] In the figure, 1-rotation video playback module, 2-skeletal muscle key point capture module, 3-capture module, 4-photo output module, 11-video rendering engine, 12-video rotation algorithm, 13-video playback control, 14-user interaction platform, 15-display screen, 16-speaker, 17-control play button, 21-skeletal key point detection algorithm, 22-sensor data processing, 23-data management and communication, 24-camera, 25-embedded processor, 31-eye tracking algorithm and calibration, 32-eye movement event analysis and visualization, 33-zoom camera, 41-gaze mapping and visualization. DETAILED DESCRIPTION

[0023] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0024] An auxiliary identification device for eye muscle-derived compensatory head position based on instantaneous eye movement capture, such as Figure 2 As shown, it includes: a rotating video playback module 1, a skeletal muscle key point capture module 2, a capture module 3, and a photo output module 4;

[0025] The rotating video playback module 1 is as follows Figure 3As shown, it includes a display screen 15 and a speaker 16, which are used to correct the patient's head position; the display screen 15 plays an animation, and rotates a set degree at a set time through a video rotation algorithm 12. In this embodiment, it rotates 5 degrees per minute, first rotating in the opposite direction of the compensatory head position, and when the head position is still not improved after rotating to the horizontal position, it rotates in the same direction of the compensatory head position to improve the head position; wherein the software part of the display screen 15 includes three modules, namely a video rendering engine 11, a video playback control 13, and a user interaction platform 14; the video rendering engine 11 is: when the video is played, through the graphics library (not limited to The video playback control 13 is to create a user interface to allow the user to select and control playback, including specifically controlling the playback and pausing of the video; the user interaction platform 14 is to allow the user to interact during video playback and adjust the rotation angle of the video;

[0026] The skeletal muscle key point capture module 2 includes a camera 24 and an embedded processor 25, which are used to determine whether the head position is correct; when the corresponding coordinate positions of the skeletal muscle nodes on the left and right sides of the face and neck are completely symmetrical and consistent, the head position is determined to be correct; the skeletal key point detection algorithm in computer vision technology, not limited to OpenPose or AlphaPose, is used to analyze the patient's facial and neck images to extract the key point coordinates of the left and right skeletal muscles; the patient's front image is collected in real time by the camera 24, wherein the patient's front image resolution is not less than 1080P; the collected patient's front image is transmitted to the embedded processor 25 in real time through the WebSocket protocol; the embedded processor 25 performs sensor data processing 22 on the image, and then implements the skeletal key point detection algorithm 21 through Python+OpenCV / TensorFlow to extract skeletal key point data; a skeletal muscle symmetry threshold is set, and when the coordinate deviation of the left and right corresponding points is within the threshold, the head position is determined to be correct;

[0027] The skeleton key point detection algorithm 21 uses computer vision technology to locate anatomical landmarks from the image and then construct the human skeleton topology; specifically: after inputting the patient's frontal image, a pre-trained CNN backbone network is used to extract multi-scale features; a dual-branch network is used to generate key point heat maps and part association fields respectively; the Hungarian algorithm is used to complete key point matching, and the existing algorithms OpenPose's Part Affinity Fields and AlphaPose's RMPE framework are used to detect 16 key points on the face and neck, and the coordinate deviation threshold is controlled within 2 mm.

[0028] The sensor data processing 22 is responsible for the preprocessing and fusion of multimodal data; for the frontal image of the patient captured by the camera 24, white balance correction and non-uniform illumination compensation are first performed, and then Kalman filtering is applied to eliminate head micro-motion noise. At the same time, hardware timestamp synchronization technology is used to ensure that the video stream and the inertial measurement unit data are aligned in time domain, and the time error is less than 3ms.

[0029] The data management and communication 23 adopts real-time data stream transmission, and the data frame is encoded using Protobuf, including key point information such as normalized coordinates and confidence; batch data is uploaded to the cloud, and the data storage adopts a hybrid architecture: the time series database records the movement trajectory of key points, the relational database manages patient metadata, and Redis caches real-time detection results (TTL 300 seconds); the security aspect complies with the HIPAA standard, uses AES-256-GCM to encrypt key point data, and protects patient privacy through facial feature desensitization.

[0030] The capture module 3 is implemented by a zoom camera 33, an embedded processor 25, and a display screen 15. The zoom camera 33 captures the patient's eye movement and eye position in real time, transmits them to the embedded processor 25, and visualizes them through the display screen 15. The eye tracking algorithm and calibration 31 are performed. The eye gaze point position is extracted through the iris-corneal reflection method, and a 9-point calibration is performed before the video is played to establish a screen-eye mapping relationship; the eye tracking data is aligned with the video frame timestamp, the eye movement events are calculated, and the eye movement event data is transmitted to the front-end web page in real time using WebSocket to realize the analysis and visualization of the eye movement events 32;

[0031] The eye tracking algorithm and calibration 31 are optimized for children's adaptability based on the traditional iris-corneal reflection method. Specifically, a 9-point calibration guided by dynamic animation (such as cartoon character movement calibration) is used to improve children's cooperation and increase dynamic attention guidance. During eye tracking, when it is detected that the gaze point deviates from the center of the screen by >50% for more than 5 seconds, a twinkling star effect is automatically superimposed in the animation to guide the line of sight back. In addition, the movement speed of all dynamic elements is ≤0.6m / s, which is in line with the smooth tracking ability range of children. During eye tracking, by fusing near-infrared imaging and visible light data, eyelash occlusion is eliminated, corneal reflection point positioning is improved, and blink compensation is designed. When the pupil occlusion rate is detected to be >90%, data collection is suspended for 150ms, and historical data is used for interpolation and filling. Head micro-motion correction is designed, and the nose tip coordinates of the skeleton key point data are combined to correct the error caused by head displacement in real time through affine transformation.

[0032] The analysis and visualization 32 of the eye movement events are as follows: by real-time processing of the eye movement data stream, combining multi-dimensional parameters to identify eye movement events, specifically including: when the eye movement speed lasts <30° / s and the duration is ≥100ms, it is determined to be a fixation event; when the instantaneous eye speed is >80° / s, it is determined to be a saccade event, and the amplitude (angle difference) and direction vector are calculated; when the pupil occlusion rate is >90% and lasts for 30-150ms, feature recognition determines it as a blink; real-time comparison of the ratio of the eye movement speed to the target movement speed determines the tracking motion event; when the retracement angle of the saccade path is >120°, it is marked as an abnormality and returns to the saccade event; after capture, analyze and compare the eye movement event data of children of the same age, and visualize the abnormal indicators in the eye movement events. If there is an abnormality in the gaze, it will prompt the gaze abnormality.

[0033] The photo output module 4 is implemented through the display screen 15. When the head position is corrected and the gaze point is stable, the current picture is automatically captured to generate a photo, which is used to visualize the eye position under the gaze point mapping after the head position is corrected 41; wherein the technology involves gaze point mapping and visualization 41;

[0034] When the skeletal muscle key point capture module 2 performs head position correction judgment, if the head position cannot be corrected autonomously, the single eye is covered for 10 minutes, and then the head position is evaluated again under single eye gaze by rotating the video playback module 1;

[0035] In this embodiment, based on the aforementioned eye muscle-derived compensatory head position auxiliary identification device, an operation method of the eye muscle-derived compensatory head position auxiliary identification device is realized, such as Figure 4 As shown, the following steps are included:

[0036] Step 1: Preparation stage;

[0037] The examinee sits in front of the display screen and adjusts the seat height so that the eyes are roughly level with the center of the screen; measure and record the distance from the patient's eyes to the screen, and adjust the camera zoom ratio;

[0038] According to the voice prompts, the patient looks at the 9 calibration points on the screen, each point for about 2 seconds.

[0039] Keep your head still during the calibration process and move your eyes to look at each point;

[0040] After successful calibration, the system automatically enters the video rotation mode;

[0041] Step 2: Video rotation test, such as Figure 1 As shown;

[0042] The subject watches the video, which first rotates in the opposite direction of the compensatory head position at 5 degrees per minute until it reaches the horizontal position. If the horizontal head position does not improve, it rotates in the direction of the compensatory head position until it is completed.

[0043] During the rotation, the skeletal muscle key points capture the subject, the face and neck key points are on the opposite side, the head position is dignified, the capture module is activated, and the system automatically captures the photo.

[0044] Analyze eye position abnormalities through photos. When there are eye position abnormalities in the captured images, it can be clearly identified as compensatory head position caused by eye muscles.

[0045] Step 3: Cover one eye;

[0046] If the head position does not improve significantly in step 2, cover one eye of the examiner (such as the left eye) and continue to watch the video with the other eye.

[0047] During the patching period, the patient should keep the patched eye completely closed and not peek. The default patching time is 10 minutes, during which the patient can close the eye to rest.

[0048] After 10 minutes, the subject stares at the video, and when the video stops rotating, keeps the head position and gaze point stable until the photo is automatically captured.

[0049] When the head position in the captured image is significantly improved, it can be determined to be a compensatory head position caused by eye muscles.

[0050] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) to form a technical solution.

Claims

1. An auxiliary identification device for eye muscle-derived compensatory head position based on instantaneous eye movement capture, characterized in that: include: Rotating video playback module, skeletal muscle key point capture module, capture module, photo output module; The rotating video playback module includes a display screen and a speaker. The display screen plays an animation and rotates a set degree at a set time through a video rotation algorithm. It first rotates in the opposite direction of the compensatory head position. When the head position is still not improved after rotating to the horizontal position, it rotates in the same direction of the compensatory head position to improve the head position. The skeletal muscle key point capture module includes a camera and an embedded processor, and the camera is used to collect the patient's front image in real time, wherein the patient's front image resolution is not less than 1080P; data management and communication are performed on the collected data stream, and the collected patient's front image is transmitted to the embedded processor in real time through the WebSocket protocol; The embedded processor processes the sensor data of the image, and then extracts the skeleton key point data through the skeleton key point detection algorithm; sets the skeletal muscle symmetry threshold, and when the coordinate deviation of the left and right corresponding points is within the threshold, it is determined that the head position is correct; The capture module is implemented by a zoom camera, an embedded processor, and a display screen. The zoom camera captures the patient's eye movement and eye position in real time, transmits them to the embedded processor, and visualizes them through the display screen. The eye tracking algorithm and calibration are performed. The eye gaze point position is extracted through the iris-corneal reflection method. A 9-point calibration is performed before the video is played to establish a screen-eye mapping relationship. The eye tracking data is aligned with the video frame timestamp, the eye movement events are calculated, and the eye movement event data is transmitted to the front-end web page in real time using the communication protocol WebSocket to realize the analysis and visualization of the eye movement events. The photo output module is implemented through a display screen. When the head position is corrected and the gaze point is stable, the current picture is automatically captured to generate a photo, which is used to visualize the eye position under the gaze point mapping after the head position is corrected; the technology involves gaze point mapping and visualization.

2. The device for assisting in distinguishing eye muscle-derived compensatory head position based on instantaneous eye movement capture according to claim 1, characterized in that: The software part of the display screen includes three modules, namely, video rendering engine, video playback control, and user interaction platform; the video rendering engine is: when the video is playing, the video rotation algorithm is used through the graphics library to change the vertex coordinates or use the shader to achieve the rotation effect, the video screen is rotated, the video stream is decoded into frames using the video decoding library, each frame is loaded into the Framebuffer as a texture, and the rotation matrix is ​​applied to display the rotated picture on the screen; the video playback control is: creating a user interface to allow the user to choose to control the playback, specifically including controlling the playback and pause of the video; the user interaction platform is: the user interacts during the video playback and adjusts the rotation angle of the video.

3. The device for assisting in distinguishing eye muscle-derived compensatory head position based on instantaneous eye movement capture according to claim 1, characterized in that: The skeleton key point detection algorithm locates anatomical landmarks from the image through computer vision technology, and then constructs the human skeleton topology; specifically, after inputting the patient's frontal image, a pre-trained CNN backbone network is used to extract multi-scale features; The key point heat map and part association field are generated respectively through a dual-branch network; the key point matching is completed using the Hungarian algorithm, and the existing algorithms OpenPose's Part Affinity Fields and AlphaPose's RMPE framework are used to detect facial and neck key points.

4. The device for assisting in distinguishing eye muscle-derived compensatory head position based on instantaneous eye movement capture according to claim 1, characterized in that: The sensor data processing is responsible for the preprocessing and fusion of multimodal data; for the frontal image of the patient captured by the camera, white balance correction and non-uniform illumination compensation are first performed, and then Kalman filtering is applied to eliminate head micro-motion noise. At the same time, hardware timestamp synchronization technology is used to ensure that the video stream is aligned in the time domain with the inertial measurement unit data, and the time error is less than 3ms.

5. The device for assisting in distinguishing eye muscle-derived compensatory head position based on instantaneous eye movement capture according to claim 1, characterized in that: The data management and communication are transmitted through real-time data streams, and the data frames are encoded using Protobuf; batch data is uploaded to the cloud, and a hybrid architecture is used for data storage: a time series database records the movement trajectory of key points, a relational database manages patient metadata, and Redis caches real-time detection results; security complies with HIPAA standards, uses AES-256-GCM to encrypt key point data, and protects patient privacy through facial feature desensitization.

6. The device for assisting in distinguishing eye muscle-derived compensatory head position based on instantaneous eye movement capture according to claim 1, characterized in that: The eye tracking algorithm and calibration are optimized for child adaptability based on the traditional iris-corneal reflection method. Specifically, a 9-point calibration guided by dynamic animation is adopted, and dynamic attention guidance is added. During the eye tracking process, when it is detected that the gaze point deviates from the center of the screen by >50% area for more than 5 seconds, a flashing effect is automatically superimposed in the animation to guide the line of sight back; in addition, the movement speed of all dynamic elements is ≤0.6m / s, and blink compensation is designed. When the pupil occlusion rate is detected to be >90%, data collection is suspended for 150ms, and historical data is used for interpolation and filling; a head micro-motion correction is designed, and the nose tip coordinates of the skeletal key point data are combined to correct the error caused by the head displacement in real time through affine transformation.

7. The device for assisting in distinguishing eye muscle-derived compensatory head position based on instantaneous eye movement capture according to claim 1, characterized in that: The analysis and visualization of the eye movement events are as follows: by real-time processing of the eye movement data stream, eye movement events are identified in combination with multi-dimensional parameters, specifically including: when the eye movement speed lasts <30° / s and the duration is ≥100ms, it is determined as a fixation event; when the instantaneous eye speed is >80° / s, it is determined as a saccade event, and the amplitude and direction vector are calculated; when the pupil occlusion rate is >90% and lasts for 30-150ms, feature recognition determines it as a blink; real-time comparison of the ratio of the eye movement speed to the target movement speed determines the tracking movement event; when the retracement angle of the saccade path is >120°, it is marked as abnormal and returns to the saccade event; after capture, the eye movement event data of children of the same age are analyzed and compared, and the abnormal indicators in the eye movement events are visualized.

8. The device for assisting in distinguishing eye muscle-derived compensatory head position based on instantaneous eye movement capture according to claim 1, characterized in that: When the skeletal muscle key point capture module performs head position correction judgment, if the head position cannot be corrected autonomously, monocular covering is performed, and the head position is evaluated under monocular gaze again by rotating the video playback module.

Images