An intelligent eye pain recognition human-computer interaction system and method

Abstract

The application discloses an eye pain intelligent recognition man-machine interaction system and method, which comprises a collection module for collecting facial images and depth data of a patient, a processing module and an interaction module; the collection module comprises a depth camera; the processing module comprises a processor, a dual-mode eye pain recognition function module and a pupil movement track recognition function module; the interaction module comprises a display for interacting with the patient and a voice interaction submodule; the system issues an eye movement guide instruction through the voice interaction submodule, and the pupil movement track recognition function module judges whether the eye movement of the patient meets the requirements; the system issues a dual-mode eye pain question through the voice interaction submodule, the dual-mode eye pain recognition function module recognizes the head movement of the patient, and feeds back two indexes of eye pain and retrobulbar pain. Compared with artificial detection and AI detection, the system has higher objectivity, real-time performance, accuracy and efficiency.

Description

Technical Field

[0001] This application relates to a human-computer interaction system and method for intelligent recognition of eye pain, particularly an intelligent human-computer interaction system for automatic judgment of eye pain indicators, belonging to the field of human-computer interaction technology for medical diagnosis. Background Technology

[0002] Thyroid-associated ophthalmopathy (TAO) is an eye disease related to the thyroid gland. Eye pain and retrobulbar pain are common symptoms of TAO and are two commonly used indicators and criteria for diagnosis in clinical practice. Eye pain is usually caused by inflammation of the extraocular muscles, while retrobulbar pain is caused by increased pressure within the orbit. Both types of pain can cause severe discomfort and significantly impact a patient's quality of life.

[0003] In my country, the clinical assessment and judgment of ocular pain and retrobulbar pain primarily rely on doctor's in-person examination and questioning. However, doctor's manual judgment is prone to subjectivity and lacks standardized procedures, leading to different doctors potentially arriving at different diagnoses for the same disease. Furthermore, as a common orbital disease, the large number of patients seeking treatment also results in a heavy workload for doctors.

[0004] With the development and popularization of artificial intelligence (AI) technology both domestically and internationally, and the implementation of various mature AI applications, medical AI has emerged as a new technological force in the medical field, addressing healthcare productivity issues. Applying medical AI technology to the diagnosis of ocular pain and retrobulbar pain can avoid the subjectivity of human diagnosis and standardize the diagnostic process. Furthermore, methods for recognizing 68 facial landmarks based on the dlib library and eye-tracking recognition based on the OpenCV library are already available. However, no AI methods such as facial landmark and eye-tracking recognition have yet been found to be applied to the identification and diagnosis of ocular pain and retrobulbar pain in TAO (tightening of the eye) ocular disease. Moreover, existing AI technologies suffer from the problems of large training data requirements, high system hardware performance requirements, and the inability to achieve real-time and rapid detection of these two pain indicators. Summary of the Invention

[0005] The technical problem this application aims to solve is that the current clinical assessment and judgment of patients' ocular pain and retrobulbar pain mainly relies on doctors' face-to-face consultations and inquiries, which makes it difficult to standardize the diagnostic process.

[0006] To address the aforementioned technical problems, the technical solution of this application is to provide a human-computer interaction system for intelligent recognition of eye pain, including an acquisition module, a processing module, and an interaction module for acquiring patient facial images and depth data;

[0007] The acquisition module includes a depth camera;

[0008] The processing module includes a processor, a dual-modal ocular pain recognition module, and a pupil movement trajectory recognition module;

[0009] The interaction module includes a display for patient interaction and a voice interaction submodule;

[0010] The system issues eye movement guidance commands through the voice interaction submodule, and the pupil movement trajectory recognition function module determines whether the patient's eye movements meet the requirements.

[0011] The system uses the voice interaction submodule to ask questions about bimodal eye pain. The bimodal eye pain recognition module identifies the patient's head movements and outputs two indicators: eye pain and retrobulbar pain.

[0012] Preferably, human-computer interaction includes a guidance component and an interaction component:

[0013] The guidance section involves the system placing the acquisition module into standby mode and calling the acquired images in real time. After the processor identifies 68 facial key points in the image, it calls the voice interaction submodule to send a voice preparation signal to the patient. After a system-defined delay, the voice interaction submodule issues an eye movement guidance command to guide the patient to look at the marked location. After the guidance section ends, the pupil movement trajectory recognition submodule executes the pupil movement trajectory recognition method to determine whether the patient has performed the correct eye movement and outputs the judgment result.

[0014] The interactive part involves the system asking the patient about their eye pain after they correctly complete the eye movements as guided. The first question, based on the eye movement results, asks the patient if they felt pain during the eye movements. The second question, based on the patient's own situation, asks if the patient experiences pain behind the eyeball in daily life. During the interaction, the bimodal eye pain recognition module uses a head movement recognition method to judge the patient's head movements, identifying nodding to indicate agreement and shaking the head to indicate disagreement, and outputting two indicators: eye pain and pain behind the eyeball.

[0015] Preferably, the pupil movement trajectory recognition method specifically includes:

[0016] Step 11: Segment the eye region and obtain the pupil coordinates;

[0017] Since the normal movements of human eyes are similar, it is only necessary to track the trajectory of one pupil.

[0018] Step 11.1 First, the image captured by the acquisition device is preprocessed. Gamma transform is used to enhance the image brightness and suppress the influence of other pixels in the acquisition device on the central pixel of the patient's pupil.

[0019]

[0020] Among them, S p This represents the grayscale value output after gamma transformation; C is the grayscale scaling factor, usually set to 1; R p γ represents the grayscale value of the input image; γ is the gamma factor, which varies depending on the lighting conditions of the scene.

[0021] Then, a face detector is used to determine the patient's facial location, extract the patient's facial ROI (Region of Interest) region as the first feature map, and an eye detector is used based on the first feature map to determine the patient's binocular ROI regions as the second feature map;

[0022] Step 11.2 Then, based on the second feature map, a pupil detector is used to determine the pupil coordinates in both eyes of the patient; by comparing the two pupil coordinates with the facial region coordinates in the first feature map, the pupil coordinates (x, y) of the right or left eye are obtained. p ,y p );

[0023] Step 12: Calculate relative coordinates;

[0024] The captured facial images were processed to obtain 68 facial key points; then, the coordinates of key points in the right or left eye region were extracted from these 68 key points, and the real-time center position (x, y) of the right or left eye was calculated from these coordinates. c ,y c );

[0025] center position (x) c ,y c The coordinates of the origin are used as the coordinates of the pupil; the relative coordinates (x, y) between the pupil and the center position are calculated according to the following formula. r ,y r and distance D r D r The projected distances to the x-axis and y-axis respectively are y r and x r ;

[0026]

[0027] Step 13: Eye movement trajectory determination;

[0028] The determination of a patient's eye movement trajectory is based on the origin coordinates (x, y, y). c ,yc ) and relative coordinates (x) r ,y r The calculation involves the vector formed by the connecting line and its properties; first, the angle θ between the connecting line and the x-axis is calculated according to the following formula. r :

[0029]

[0030] Then, using the obtained real-time relative coordinates (x... r ,y r ), distance D r included angle θ r and projection distance y r and x r Based on, where the included angle θ r As the primary feature criterion, x represents the direction of the patient's eye movement. r y r and D r The second feature criterion represents the relative distance and amplitude of the patient's eye movements, with coordinates (x, y, y) representing the distance and amplitude of the eye movements. r ,y r The quadrant in which the pupil is located serves as an auxiliary feature criterion, representing the patient's current pupil position.

[0031] By comparing the first feature criterion, the second feature criterion, and the auxiliary feature criterion with the system's preset threshold... The comparison is performed, and if the movement falls within the threshold range, it indicates that the patient has performed the correct eye movement. (Default) Threshold judgment can identify whether the trajectory of a patient's eye movement conforms to the upper right, upper left, lower left, and lower right corners, with the corresponding coordinates located in the first, second, third, and fourth quadrants, respectively. The judgment formula is as follows:

[0032]

[0033] Step 14: Output the judgment result. False means that the patient did not perform the correct eye movements.

[0034] Preferably, the dual-modal eye pain recognition function module executes a head action recognition method to recognize and process the captured facial image to obtain 68 facial key points. The coordinates of the key point at the tip of the nose are selected as the first feature criterion, and the depth data of the key point at the tip of the nose is used as the second feature criterion.

[0035] By analyzing and processing the degree of change of the first feature criterion and the second feature criterion over time, the patient's head movement feedback at this time is output.

[0036] The output patient head movement feedback includes two types: nodding to indicate agreement and shaking to indicate disagreement. The judgment is based on the amplitude of the shaking along the x and y axes of the patient's nasal tip key point within one second, combined with depth data at that moment, and calculated using standard deviation. Specifically:

[0037] The frame rate of the data acquisition device is N, and the set of key point coordinates of the patient's nasal tip position within one second is N. x N y And at this time, the coordinate depth set is N. D Each set consists of N continuously updated real-time data points; the standard deviation is calculated as follows:

[0038]

[0039] Where, x i Represents the i-th x-coordinate, S represents the average of N x-coordinates. x The x-axis represents the standard deviation; y i Represents the i-th ordinate. S represents the average of N ordinates. y D represents the standard deviation of the ordinate. i This represents the i-th depth data. S represents the average of N depth data points. D Indicates the standard deviation of depth data;

[0040] The calculated standard deviation is compared with the system preset threshold. In comparison, the formula for recognizing patient head movements is:

[0041]

[0042] Positive indicates agreement, Negative indicates negation, and None indicates meaningless action.

[0043] This application also provides a human-computer interaction method for intelligent recognition of eye pain, which uses the above-mentioned human-computer interaction system and includes the following interaction process:

[0044] Process 1: System Initialization

[0045] Step 1: The processor allocates the memory required for system operation and initializes the display;

[0046] Step 2: Initialize the depth camera and access live feeds;

[0047] Process Two: Interactive Eye Pain Recognition

[0048] Step 1: Identify and detect key facial points of the patient in the image, and obtain the coordinates of the key points needed from 68 facial key points;

[0049] Step 2: After obtaining stable key point coordinate data, the progress is displayed on the screen; the system calls the speaker to issue a voice to the patient, guiding the patient to look at the marked area, and at the same time the system begins to identify and detect the coordinates of the patient's eye center and pupil in the screen;

[0050] Step 3: After the patient completes the above actions, the system compares the feature criteria generated by the patient's eye movement with the preset threshold; if the trajectories are similar, the system will ask the patient again whether they felt pain during the eye movement; at the same time, the system will perform head movement recognition on the patient's face in the real-time image of the depth camera, and identify nodding to indicate agreement and shaking the head to indicate disagreement; if no corresponding action feedback is recognized from the patient or the eye movement trajectory is not similar, the system will remind the patient through voice.

[0051] Step 4: Based on the recognition results, output the eye pain index and display it on the monitor;

[0052] Process 3: Reciprocal pain after ball movement

[0053] Step 1: After outputting the eye pain index, the system calls the speaker to ask the patient whether they feel pain behind their eyeball in their daily life; at the same time, the system performs head movement recognition on the patient's face in the real-time image of the depth camera, recognizing nodding to indicate agreement and shaking the head to indicate disagreement; if no corresponding action feedback is recognized from the patient, the system will remind the patient through voice.

[0054] Step 2: Based on the recognition results, output the post-ball pain index and display it on the monitor;

[0055] Step 4: System operation ends

[0056] Step 1: End the screen display, turn off the depth camera and monitor;

[0057] Step 2: The processor releases the system memory it occupies.

[0058] This application proposes a bimodal ocular pain recognition and interactive system for detecting ocular and retrobulbar pain indicators in TAO (tightening of the eye) ocular disease. This system accurately and quickly obtains patient feedback regarding ocular and retrobulbar pain, reducing patient waiting time and improving hospital operational efficiency. Through interaction with patients, the system accurately obtains their feedback on ocular and retrobulbar pain, replacing traditional doctor-led consultations and offering objectivity and speed.

[0059] Currently, manual detection is subject to subjectivity, errors, and low efficiency; the system and method provided in this application have higher objectivity, real-time performance, accuracy, and efficiency compared to AI detection.

[0060] Furthermore, in response to existing algorithms that use machine learning and deep learning to identify the trajectory of patients' eye movements, this application proposes a pupil movement trajectory recognition method based on relative coordinates, which is based on real-time image acquisition for recognition and processing. Compared with machine learning or deep learning algorithms that require a large amount of data for training and have high deployment requirements, this method has the characteristics of low latency, real-time response and low memory usage. Attached Figure Description

[0061] Figure 1 This application presents a human-computer interaction system architecture diagram for intelligent eye pain recognition;

[0062] Figure 2 This is a diagram showing the center position of the right eye;

[0063] Figure 3 This is a schematic diagram showing the origin and pupil coordinates.

[0064] Figure 4 This is a schematic diagram showing the relative distance, included angle, and projection.

[0065] Figure 5 This is a flowchart illustrating the implementation process of the system in this application.

[0066] Figure 6 This is a schematic diagram of the pupil trajectory recognition process;

[0067] Figure 7 This is a schematic diagram of the head motion recognition process. Detailed Implementation

[0068] The present application will be further described in detail below with reference to the accompanying drawings.

[0069] After reading this specification, those skilled in the art may make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they fall within the scope of the claims of this application.

[0070] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0071] This application provides a human-computer interaction system for intelligent recognition of eye pain, including a data acquisition module, a processing module, and an interaction module. The data acquisition module includes a depth camera with spatial depth data acquisition capabilities. The processing module includes a processor, a dual-modal eye pain recognition module, and a pupil movement trajectory recognition module. The interaction module includes a display for patient interaction and a voice interaction submodule, which includes a speaker. In this system, the processor submodule issues eye movement guidance commands through the voice interaction submodule, then asks the patient two questions about eye pain and acquires the patient's head movements through the data acquisition module. The dual-modal eye pain recognition module processes the data and outputs two indicators: eye pain and retrobulbar pain. The pupil movement trajectory recognition module obtains the real-time pupil coordinates and the coordinates of the eye center, calculates the angle and projection distance of the line connecting the two points, and then obtains the patient's pupil movement trajectory. It compares the similarity with a preset trajectory equation to determine whether the patient's eye movements meet the requirements. This provides standardized indicators for the assessment and judgment of the patient's eye pain and retrobulbar pain.

[0072] In this embodiment, the system architecture diagram is shown below. Figure 1 The acquisition module uses a depth camera with spatial depth data acquisition capabilities to acquire two-dimensional images and depth data of the patient's face. The processing module processes the patient's facial data. The dual-modal eye pain recognition module and the pupil movement trajectory recognition module are two functional modules built into the processor, which execute corresponding methods to achieve patient head recognition, pupil detection, and trajectory recognition. The processing module is used to control voice interaction. The processing progress and other information are displayed on the monitor, and doctors can perform subsequent operations after the system outputs the results on the monitor.

[0073] In this embodiment of the application, the interaction module is used to interact with the patient through voice and video. The interaction process is divided into a guidance part and an interaction part. The system controls the voice interaction submodule through the processor submodule to convert text information into sound signals to achieve guidance or interaction with the patient.

[0074] The guidance section: After the system initialization is complete, the acquisition module is placed in standby mode and the depth camera image is called in real time. After the processor can identify 68 facial key points in the image based on the dlib library, the voice interaction submodule is called to send a voice preparation signal to the patient. Then, after the system-set delay, the voice interaction submodule sends an eye movement guidance command to guide the patient to look at the pre-set mark. After the guidance section is completed, the pupil movement trajectory recognition submodule executes the pupil movement trajectory recognition method to determine whether the patient has performed the correct eye movement and outputs the judgment result.

[0075] Interactive Section: The interactive section is divided into two parts: eye pain interaction and retrobulbar pain interaction. After the patient correctly completes the eye movements according to the guidance, the system asks the patient about their eye pain through the voice interaction submodule. The first question is based on the eye movement results, asking the patient: "Did you feel pain during the eye movements?" This question belongs to the eye pain interaction section. The second question is based on the patient's own situation, asking the patient: "Do you feel retrobulbar pain in your daily life?" This question belongs to the retrobulbar pain interaction section. During the interaction, the bimodal eye pain recognition function module executes the head movement recognition method to judge the patient's head movements, identify nodding (agreement) and shaking (disagreement) movements, and output the two indicators of eye pain and retrobulbar pain.

[0076] In this embodiment of the application, the pupil movement trajectory recognition method executed by the pupil movement trajectory recognition function module has two core parts: 1. pupil detection, 2. trajectory recognition;

[0077] Specifically, the following steps are included:

[0078] Step 11: Segment the eye region and obtain the pupil coordinates;

[0079] Pupil detection is implemented based on the face detector, eye detector, and pupil classifier in the existing OpenCV library. Since the normal movements of human eyes are similar, the method only needs to track the trajectory of one pupil. At the same time, this feature can also reduce the computational complexity of the method.

[0080] Take the right eye as an example;

[0081] Step 11.1 First, the image captured by the acquisition device is preprocessed. Gamma transform is used to enhance the image brightness and suppress the influence of other pixels in the acquisition device on the central pixel of the patient's pupil.

[0082]

[0083] Among them, S p This represents the grayscale value output after gamma transformation; C is the grayscale scaling factor, usually set to 1; R p γ represents the grayscale value of the input image; γ is the gamma factor, which varies depending on the lighting conditions of the scene.

[0084] Then, a face detector is used to determine the patient's facial location, extract the patient's facial ROI (Region of Interest) region as the first feature map, and an eye detector is used based on the first feature map to determine the patient's binocular ROI regions as the second feature map;

[0085] Step 11.2 Then, based on the second feature map, a pupil detector is used to determine the pupil coordinates in both eyes of the patient; by comparing the two pupil coordinates with the facial region coordinates in the first feature map, the pupil coordinates (x, y) of the right eye are obtained. p ,y p );

[0086] Step 12: Calculate relative coordinates;

[0087] The trajectory recognition employs existing facial landmark detection technology based on the dlib library. It processes facial images captured by a depth camera to obtain 68 facial landmarks. Then, it extracts the coordinates of landmarks 43-48 in the right eye region from these 68 landmarks (the same applies to the left eye). Specifically, it calculates the real-time center position (x) of the right eye by analyzing the 43rd, 45th, 46th, and 48th facial landmarks. c ,y c ),like Figure 2 As shown;

[0088] Let the coordinates of facial key point number 43 be (x 43 ,y 43 ), No. 45 is (x 45 ,y 45 ), No. 46 is (x 46 ,y 46 ), No. 48 is (x 48 ,y 48 If the coordinates of the center position of the right eye are (x, y), then the coordinates of the center position of the right eye are (x, y). c ,y c The calculation method for ) is shown in formula (1):

[0089]

[0090] After obtaining the coordinates of the center position of the right eye, use these coordinates as the origin, such as... Figure 3 As shown; calculate the relative coordinates (x, y) between the right pupil and the center of the right eye according to formula (2). r ,y r and distance D r D r The projected distances to the x-axis and y-axis respectively are y r and x r ;

[0091]

[0092] Step 13: Eye movement trajectory determination;

[0093] The principle for determining the patient's eye movement trajectory is based on the origin of the coordinate system (x). c ,y c ) and relative coordinates (x) r ,yr The calculation is performed on the vector formed by the connecting lines and its properties; such as Figure 4 First, calculate the angle θ between the connecting line and the x-axis according to formula (3). r :

[0094]

[0095] Then, using the obtained real-time relative coordinates (x... r ,y r ), distance D r included angle θ r and projection distance y r and x r Based on, where the included angle θ r As the primary feature criterion, x represents the direction of the patient's eye movement. r y r and D r The second feature criterion represents the relative distance and amplitude of the patient's eye movements, with coordinates (x, y, y) representing the distance and amplitude of the eye movements. r ,y r The quadrant in which the pupil is located serves as an auxiliary feature criterion, representing the patient's current pupil position.

[0096] By comparing the first feature criteria, the second feature criteria, and the auxiliary feature criteria with the system's preset threshold... The comparison is performed, and if the movement falls within the threshold range, it indicates that the patient has performed the correct eye movement. (Default) Threshold judgment can identify whether the trajectory of a patient's eye movement conforms to the upper right, upper left, lower left, and lower right corners, with the corresponding coordinates located in the first, second, third, and fourth quadrants, respectively. The judgment formula is as follows:

[0097]

[0098] Step 14: Output the judgment result. False means that the patient did not perform the correct eye movements.

[0099] By employing the aforementioned pupil movement trajectory recognition method, the system can acquire the patient's pupil coordinates in real time. Then, trajectory recognition is performed. By capturing the center position of the human eye in real time, the angle and projection distance of the patient's pupil coordinates relative to the line connecting the center position of the human eye are calculated. A similarity comparison is then performed with a preset threshold to determine whether the patient's eye movements meet the requirements. This trajectory recognition method is based on traditional image processing methods and, compared to machine learning and deep learning trajectory recognition methods, has the advantages of low memory usage, low latency, and real-time response.

[0100] In this embodiment, the head movement recognition method executed by the dual-modal ocular pain recognition function module, in the interaction part, after asking the patient a question, judges the patient's head movement, identifies nodding (agreement) and shaking (disagreement) movements, and outputs the judgment result; the head movement recognition method adopts the existing facial key point detection technology based on the dlib library. After receiving the system's call request, the head movement recognition method identifies 68 facial key points from the real-time captured image of the acquisition module, selects the coordinates of the key point at the tip of the nose as the first feature criterion, and uses the depth data about the key point at the tip of the nose generated by the acquisition module (depth camera) as the second feature criterion; by analyzing and processing the degree of change of the first feature criterion and the second feature criterion over time, the patient's head movement feedback at this time is output;

[0101] The output patient head movement feedback includes two types: nodding to indicate agreement and shaking to indicate disagreement. The judgment is based on the amplitude of the shaking along the x and y axes of the patient's nasal tip key point within one second, combined with depth data at that moment, and calculated using standard deviation. Specifically:

[0102] The frame rate of the data acquisition device is N (30fps), and the set of key point coordinates of the patient's nasal tip position within one second is N. x N y And at this time, the coordinate depth set is N. D Each set consists of N real-time data points that are constantly refreshed; the standard deviation is calculated as shown in formula (5):

[0103]

[0104] Where, x i Represents the i-th x-coordinate, S represents the average of N x-coordinates. x The x-axis represents the standard deviation; y i Represents the i-th ordinate. S represents the average of N ordinates. y D represents the standard deviation of the ordinate. i This represents the i-th depth data. S represents the average of N depth data points. D Indicates the standard deviation of depth data;

[0105] To eliminate differences between different acquisition devices, this method uses the standard deviation ratio for patient head movement recognition, comparing the calculated standard deviation with a system preset threshold. In comparison, the patient head movement recognition criterion formula is shown in formula (6):

[0106]

[0107] Positive indicates agreement, Negative indicates negation, and None indicates meaningless action.

[0108] By executing the head movement recognition method described above, this system outputs head movement feedback results. This method, based on standard deviation ratio, uses the key point coordinates of the patient's nose tip position as a foundation and employs classic image processing methods to determine the patient's head movement feedback in response to a question. The recognized content is agreement (nodding) and disagreement (shaking the head). The core step involves calculating the standard deviation of the x, y, and z-axis data of a frame, then performing a threshold judgment based on the ratio of the x and y-axis standard deviations, supplemented by the threshold judgment result of the z-axis standard deviation. If the judgment condition is met, the head movement feedback result is output. This recognition method improves real-time response speed and processing speed, achieving high accuracy while having low hardware performance requirements.

[0109] In this application, the assessment and judgment of two indicators—eye pain and retrobulbar pain—relies on standardized indicators provided by the bimodal eye pain recognition module and the pupil movement trajectory recognition module. The bimodal eye pain recognition module is the core module of this system, while the pupil movement trajectory recognition module is an auxiliary module, ultimately aiming to obtain accurate and standardized intelligent recognition results for eye pain. When the system interacts with the patient regarding the eye pain indicator, it needs to obtain feedback on whether the patient is experiencing eye pain through eye movements. The pupil trajectory recognition method is used to determine whether the patient has correctly performed eye movements according to voice guidance, thereby obtaining highly reliable feedback results. The system pre-calculates the changes in the angle and relative distance of the patient's pupil from the location of the acquisition module to the marker point as the judgment standard. Therefore, when the patient actually performs eye movements, the system compares the changes in angle and relative distance obtained from the real-time acquisition module with the preset judgment standard threshold. If the changes are within the threshold, it indicates that the patient has moved along the agreed path, and the feedback result obtained at this time is highly reliable; only then can accurate and standardized recognition results be obtained.

[0110] In this embodiment of the application, the interaction process of this system is as follows:

[0111] To obtain the results of ocular pain and retrobulbar pain in patients with thyroid-associated ophthalmopathy, facilitating diagnosis and analysis by doctors, this application provides a human-computer interaction system for intelligent ocular pain recognition. The entire process of using this system for patient ocular pain recognition interaction includes three main parts: system initialization and termination, ocular pain interaction, and retrobulbar pain interaction. Figure 5 As shown, the process for recognizing the patient's pupil movement trajectory is as follows: Figure 6 As shown, the patient head movement recognition process is as follows: Figure 7 As shown.

[0112] The specific interaction process is as follows:

[0113] Process 1: System Initialization

[0114] Step 1: The processor allocates the memory required for system operation and initializes the display;

[0115] Step 2: Initialize the depth camera and call up the live feed.

[0116] Process Two: Interactive Eye Pain Recognition

[0117] Step 1: Identify and detect key facial points of the patient in the image, and obtain the coordinates of the key points needed from 68 facial key points;

[0118] Step 2: After obtaining stable key point coordinate data, the progress is displayed on the screen; the system calls the speaker to issue a voice to the patient, guiding the patient to look at the marked area, and at the same time the system begins to identify and detect the coordinates of the patient's eye center and pupil in the screen;

[0119] Step 3: After the patient completes the above actions, the system compares the feature criteria generated by the patient's eye movement with a preset threshold, as shown in Formula 4. If the trajectories are similar, the system will again ask the patient via voice whether they felt pain during the eye movement. Simultaneously, the system will perform head movement recognition on the patient's face in real-time depth camera footage, identifying nodding (agreement) and shaking (disagreement) movements, as shown in Formula 5. If no corresponding action feedback is detected from the patient or the eye movement trajectories are dissimilar, the system will remind the patient via voice.

[0120] Step 4: Based on the recognition results, output the eye pain index and display it on the monitor.

[0121] Process 3: Reciprocal pain after ball movement

[0122] Step 1: After outputting the eye pain index, the system calls the speaker to ask the patient whether they feel pain behind their eyeball in their daily life; at the same time, the system performs head movement recognition on the patient's face in the real-time image of the depth camera, and recognizes nodding (agreement) and shaking (disagreement) actions, as shown in Formula 5; if no corresponding action feedback is recognized from the patient, the system will remind the patient through voice.

[0123] Step 2: Based on the recognition results, output the post-ball pain index and display it on the monitor.

[0124] Step 4: System operation ends

[0125] Step 1: End the screen display, turn off the depth camera and monitor;

[0126] Step 2: The processor releases the system memory it occupies.

Claims

1. A human-computer interaction system for intelligent recognition of eye pain, characterized in that, It includes a data acquisition module, a processing module, and an interaction module for acquiring patient facial images and depth data; The acquisition module includes a depth camera; The processing module includes a processor, a dual-modal ocular pain recognition module, and a pupil movement trajectory recognition module; The interaction module includes a display for patient interaction and a voice interaction submodule; The system issues eye movement guidance commands through the voice interaction submodule, and the pupil movement trajectory recognition function module determines whether the patient's eye movements meet the requirements. The system uses the voice interaction submodule to ask questions about bimodal eye pain. The bimodal eye pain recognition module recognizes the patient's head movements and outputs two indicators: eye pain and retrobulbar pain. Human-computer interaction includes a guidance component and an interaction component: The guidance section involves the system placing the acquisition module into standby mode and calling the acquired images in real time. After the processor identifies 68 facial key points in the image, it calls the voice interaction submodule to send a voice preparation signal to the patient. After a system-defined delay, the voice interaction submodule issues an eye movement guidance command to guide the patient to look at the marked location. After the guidance section ends, the pupil movement trajectory recognition submodule executes the pupil movement trajectory recognition method to determine whether the patient has performed the correct eye movement and outputs the judgment result. The interactive part involves the system asking the patient about their eye pain after they correctly complete the eye movements as guided. The first question asks the patient whether they felt pain during the eye movements, based on the results of the eye movements. The second question asks the patient whether they experience pain behind the eyeball in their daily life, based on their own situation. During the interaction, the bimodal eye pain recognition module uses a head movement recognition method to judge the patient's head movements, identifying whether nodding indicates agreement or shaking the head indicates disagreement, and outputting two indicators: eye pain and pain behind the eyeball. The pupil movement trajectory recognition method specifically includes: Step 11: Segment the eye region and obtain the pupil coordinates; Since the normal movements of human eyes are similar, it is only necessary to track the trajectory of one pupil. Step 11.1 First, the image captured by the acquisition device is preprocessed. Gamma transform is used to enhance the image brightness and suppress the influence of other pixels in the acquisition device on the central pixel of the patient's pupil: in, This represents the grayscale value output after gamma transformation; C is the grayscale scaling factor, which is usually set to 1. The input image grayscale value; This is the gamma factor, and its value varies depending on the lighting conditions of the scene. Then, a face detector is used to determine the patient's facial location, extract the patient's facial ROI region as the first feature map, and an eye detector is used based on the first feature map to determine the patient's bilateral ROI region as the second feature map. Step 11.2 Then, based on the second feature map, a pupil detector is used to determine the pupil coordinates in both eyes of the patient; by comparing the two pupil coordinates with the facial region coordinates in the first feature map, the pupil coordinates of the right or left eye are obtained. ; Step 12: Calculate relative coordinates; The captured facial images were processed to obtain 68 facial key points. Then, the coordinates of key points in the right or left eye region were extracted from these 68 key points, and the real-time center position of the right or left eye was calculated from these coordinates. ; center position The coordinates are taken as the origin coordinates; the relative coordinates between the pupil and the center position are calculated according to the following formula. and distance , Separation x axis, y The projected distance of the axis is and ; Step 13: Eye movement trajectory determination; The determination of a patient's eye movement trajectory is based on the origin coordinates. with relative coordinates The vector formed by the connecting lines and its properties are calculated; first, the following formula is used to calculate the vector formed by the connecting lines and their properties. x Angle between axes : Then, using the obtained real-time relative coordinates ,distance Angle and projection distance and Based on, where the included angle As the primary characteristic criterion, it represents the direction of the patient's eye movement. , and The second feature criterion represents the relative distance and amplitude of the patient's eye movements, in coordinates. The quadrant in which the pupil is located serves as an auxiliary feature criterion, representing the current position of the patient's pupil. By comparing the first feature criterion, the second feature criterion, and the auxiliary feature criterion with the system's preset threshold... 、 The comparison is performed, and if the movement falls within the threshold range, it indicates that the patient has performed the correct eye movement. (Default) The threshold judgment can identify whether the trajectory of the patient's eye movement conforms to the upper right, upper left, lower left, and lower right corners, with the corresponding coordinates located in the first, second, third, and fourth quadrants, respectively. The judgment formula is as follows: Step 14: Output the judgment result. This indicates that the patient did not perform correct eye movements; The dual-modal eye pain recognition module executes a head motion recognition method to recognize and process the captured facial images to obtain 68 facial key points. The coordinates of the key point at the tip of the nose are selected as the first feature criterion, and the depth data of the key point at the tip of the nose is used as the second feature criterion. By analyzing and processing the degree of change of the first feature criterion and the second feature criterion over time, the patient's head movement feedback at this time is output. The output of patient head movement feedback includes two types: nodding to indicate agreement and shaking to indicate disagreement, determined by the coordinates of a key point on the tip of the patient's nose. x shaft and y The amplitude of axis jitter within one second, combined with the depth data at that time, is used to calculate the standard deviation for judgment. Specifically: The frame rate of the data acquisition device is N, and the set of key point coordinates of the patient's nasal tip position within one second is as follows: And at this time, the coordinate depth set is Each set consists of N continuously updated real-time data points; the standard deviation is calculated as follows: in, Indicates the first i Each horizontal axis This represents the average of N horizontal axes. The x-axis represents the standard deviation; Indicates the first i One vertical axis, express N The average of the ordinates, The vertical axis represents the standard deviation. Indicates the first i A depth data, This represents the average of N depth data points. Indicates the standard deviation of depth data; The calculated standard deviation is compared with the system preset threshold. ， In comparison, the formula for recognizing patient head movements is: To express agreement To indicate negation, Indicates a meaningless action.

2. A human-computer interaction method for intelligent recognition of eye pain, employing the human-computer interaction system described in claim 1, characterized in that, The following interactive processes are included: Process 1: System Initialization Step 1: The processor allocates the memory required for system operation and initializes the display; Step 2: Initialize the depth camera and access live feeds; Process Two: Interactive Eye Pain Recognition Step 1: Identify and detect key facial points of the patient in the image, and obtain the coordinates of the key points needed from 68 facial key points; Step 2: After obtaining stable key point coordinate data, the progress is displayed on the screen; the system calls the speaker to issue a voice to the patient, guiding the patient to look at the marked area, and at the same time the system begins to identify and detect the coordinates of the patient's eye center and pupil in the screen; Step 3: After the patient completes the above actions, the system compares the feature criteria generated by the patient's eye movement with the preset threshold; if the trajectories are similar, the system will ask the patient again whether they felt pain during the eye movement; at the same time, the system will perform head movement recognition on the patient's face in the real-time image of the depth camera, and identify nodding to indicate agreement and shaking the head to indicate disagreement; if no corresponding action feedback is recognized from the patient or the eye movement trajectory is not similar, the system will remind the patient through voice. Step 4: Based on the recognition results, output the eye pain index and display it on the monitor; Process 3: Reciprocal pain after ball movement Step 1: After outputting the eye pain index, the system calls the speaker to ask the patient whether they feel pain behind their eyeball in their daily life; at the same time, the system performs head movement recognition on the patient's face in the real-time image of the depth camera, recognizing nodding to indicate agreement and shaking the head to indicate disagreement; if no corresponding action feedback is recognized from the patient, the system will remind the patient through voice. Step 2: Based on the recognition results, output the post-ball pain index and display it on the monitor; Step 4: System operation ends Step 1: End the screen display, turn off the depth camera and monitor; Step 2: The processor releases the system memory it occupies.

Images