A non-contact human-computer interaction system and method based on stereoscopic vision

Abstract

The present invention belongs to the field of non-contact human-computer interaction technology, and specifically relates to a non-contact human-computer interaction system and method based on stereoscopic vision. The invention, by combining the position of the pupil center and the corneal reflection point, can accurately locate the user's gaze point, improve the accuracy of human-computer interaction, and effectively eliminate the user's misoperation caused by eye fatigue, unexpected movement, etc. by detecting abnormal motion trajectories and generating compensation trajectories, improve robustness and stability, and can adaptively adjust the weight coefficient and compensation trajectory according to the user's motion characteristics, enhance adaptability to different users, and can perform human-computer interaction without the user having physical contact, providing a more natural and convenient operation experience, suitable for a variety of application scenarios, such as medical rehabilitation, smart home, etc.

Description

Technical Field

[0001] The present invention belongs to the technical field of non-contact human-computer interaction, and in particular relates to a non-contact human-computer interaction system and method based on stereoscopic vision. Background Art

[0002] At present, human-computer interaction technology is increasingly becoming a bridge connecting humans and the digital world. However, traditional human-computer interaction methods, such as keyboards, mice and other input devices, although they perform well in many application scenarios, may not be friendly or natural enough for some special groups, such as people with physical disabilities or users who need a highly immersive experience.

[0003] With the advancement of science and technology, especially the rapid development of computer vision and artificial intelligence, the demand for using natural human characteristics, such as eye movement, as an input method is growing. Eyes are one of the most important senses of human beings, and their movement contains rich information and intentions. Therefore, there is great potential for human-computer interaction through eyes.

[0004] In recent years, eye tracking technology has made some progress in practical applications, but existing technologies mainly focus on line of sight direction detection in a two-dimensional plane and lack accurate estimation of depth information. The user's eye movement may be affected by various factors, such as inattention, eye fatigue or external interference, which may cause abnormal eye movement trajectory. Summary of the invention

[0005] The purpose of the present invention is to provide a non-contact human-computer interaction method based on stereoscopic vision, which can accurately judge whether the user's gaze behavior is stable, and can also adaptively adjust the compensation trajectory and weight coefficient according to the user's actual eye movement, thereby improving the accuracy and stability of the interaction.

[0006] The technical solution adopted by the present invention is as follows:

[0007] A non-contact human-computer interaction method based on stereoscopic vision, comprising:

[0008] Acquire binocular images of the user, and extract eye feature points based on the binocular images, wherein the eye feature points include pupil center positions and corneal reflection point positions;

[0009] Obtain the gaze point position and the gaze object according to the pupil center position and the corneal reflection point position;

[0010] Acquire a motion image of the eye, acquire motion trajectory data of the eye according to the motion image data, and mark the data as a first motion trajectory;

[0011] Determine whether the first motion trajectory meets the first preset condition, and if not, mark it as an abnormal motion trajectory;

[0012] Obtaining a first weight coefficient according to the abnormal motion trajectory;

[0013] Constructing an evaluation period, obtaining eye movement trajectory data within the evaluation period, marking the data as a second movement trajectory, and obtaining an eye movement compensation coefficient according to the second movement trajectory;

[0014] Obtaining an eye movement compensation trajectory according to the abnormal movement trajectory, the first weight coefficient, and the eye movement compensation coefficient, and generating a predicted gaze point position according to the eye movement compensation trajectory;

[0015] It is determined whether the predicted gaze point position meets the second preset condition. If so, it is determined that the user's gaze behavior is stable and a corresponding control instruction is output.

[0016] In a preferred embodiment, the step of obtaining the gaze point position and the gaze object according to the pupil center position and the corneal reflection point position includes:

[0017] Obtain the gaze point based on the pupil center position and corneal reflection point position;

[0018] Get the human-computer interaction device in front of the user;

[0019] Obtain the gaze point position and the gaze object according to the gaze point and the human-computer interaction device.

[0020] In a preferred embodiment, the step of determining whether the first motion trajectory meets the first preset condition, and if not, marking it as an abnormal motion trajectory, comprises:

[0021] Obtain standard motion trajectory;

[0022] Obtaining trajectory similarity between the standard motion trajectory and the first motion trajectory;

[0023] Get trajectory similarity evaluation threshold;

[0024] Determine whether the trajectory similarity is lower than the trajectory similarity evaluation threshold;

[0025] If the trajectory similarity is lower than the trajectory similarity evaluation threshold, the binocular movement is judged to be abnormal and marked as an abnormal movement trajectory;

[0026] If the trajectory similarity is not lower than the trajectory similarity evaluation threshold, the binocular movement is judged to be normal.

[0027] In a preferred embodiment, the step of obtaining the trajectory similarity between the standard motion trajectory and the first motion trajectory includes:

[0028] Obtain a reference plane, and scale the first motion trajectory and the standard motion trajectory to the reference plane in equal proportion;

[0029] Establish a rectangular coordinate system on the reference plane;

[0030] Construct multiple trajectory evaluation segments according to the standard motion trajectory;

[0031] Obtaining the inflection point coordinates of the first motion trajectory and the standard motion trajectory in each trajectory evaluation segment;

[0032] Obtain the coordinate vector value of each inflection point corresponding to the first motion trajectory, and mark it as the first inflection point vector;

[0033] Obtain the coordinate vector value of each inflection point corresponding to the standard motion trajectory and mark it as the standard inflection point vector;

[0034] Get similarity function;

[0035] A plurality of first inflection point vectors and a plurality of standard inflection point vectors are input into a similarity function, and the output result is marked as trajectory similarity.

[0036] In a preferred embodiment, the step of obtaining a first weight coefficient according to the abnormal motion trajectory includes:

[0037] Obtaining the trajectory similarity corresponding to the abnormal motion trajectory and the trajectory similarity evaluation threshold;

[0038] Get the weight function;

[0039] The trajectory similarity and trajectory similarity evaluation threshold corresponding to the abnormal motion trajectory are input into the weight function, and the output result is marked as the first weight coefficient.

[0040] In a preferred embodiment, the steps of constructing an evaluation period, obtaining eye movement trajectory data within the evaluation period, marking the data as a second movement trajectory, and obtaining an eye movement compensation coefficient according to the second movement trajectory include:

[0041] Mark the time marked as the first motion track as the end time;

[0042] Get the standard assessment duration;

[0043] Get the start time based on the standard evaluation duration and end time;

[0044] Structuring evaluation periods based on start and end times;

[0045] Acquire the eye movement trajectory data during the evaluation period and mark it as the second movement trajectory;

[0046] Acquire similarities of a plurality of motion trajectories corresponding to the standard motion trajectory according to the plurality of second motion trajectories, and mark them as trajectory evaluation similarities;

[0047] Get the compensation function;

[0048] The similarities of multiple trajectory evaluations are input into the compensation function, and the output result is labeled as the eye movement compensation coefficient.

[0049] In a preferred embodiment, the step of obtaining an eye movement compensation trajectory according to the abnormal movement trajectory, the first weight coefficient and the eye movement compensation coefficient, and generating a predicted gaze point position according to the eye movement compensation trajectory includes:

[0050] Get standard functions;

[0051] Inputting the first weight coefficient and the eye movement compensation coefficient into the standard function, and marking the output result as the trajectory compensation value;

[0052] Obtaining a trajectory compensation table, wherein the trajectory compensation table includes a plurality of trajectory compensation interval values ​​and a trajectory compensation method corresponding to each trajectory compensation interval value;

[0053] Obtain the target trajectory compensation interval value according to the trajectory compensation value;

[0054] Obtain the corresponding trajectory compensation method from the trajectory compensation table according to the target trajectory compensation interval value;

[0055] Compensating the abnormal motion trajectory according to the trajectory compensation method to obtain the eye motion compensation trajectory;

[0056] And generate the predicted gaze point position based on the eye movement compensation trajectory.

[0057] In a preferred embodiment, the step of determining whether the predicted gaze point position meets the second preset condition, and if so, determining that the user's gaze behavior is stable and outputting a corresponding control instruction includes:

[0058] Establish a plane rectangular coordinate system;

[0059] Obtain the coordinate point of the standard gaze point position according to the plane rectangular coordinate system;

[0060] Obtain the coordinate point of the predicted gaze point position according to the plane rectangular coordinate system;

[0061] Get coordinate similarity function;

[0062] Input the coordinate point of the standard fixation point position and the coordinate point of the predicted fixation point position into the coordinate similarity function, and mark the output result as the fixation similarity value;

[0063] Get the gaze similarity evaluation interval value;

[0064] Determine whether the gaze similarity value is within the gaze similarity evaluation interval value;

[0065] If the gaze similarity value is within the gaze similarity evaluation interval value, the user's gaze behavior is determined to be stable, and a corresponding control instruction is output;

[0066] If the gaze similarity value is not within the gaze similarity evaluation interval value, it is determined that the user's gaze behavior is wrong.

[0067] The present invention also provides a non-contact human-computer interaction system based on stereoscopic vision, which is used in the above-mentioned non-contact human-computer interaction method based on stereoscopic vision, comprising:

[0068] A feature point module is used to obtain binocular images of the user and extract eye feature points based on the binocular images, wherein the eye feature points include the pupil center position and the corneal reflection point position;

[0069] A gaze module, used for obtaining the gaze point position and the gaze object according to the pupil center position and the corneal reflection point position;

[0070] A first trajectory module, used to obtain a motion image of the eye, obtain motion trajectory data of the eye according to the motion image data, and mark it as a first motion trajectory;

[0071] An abnormal trajectory module, used to determine whether the first motion trajectory meets the first preset condition, and if not, mark it as an abnormal motion trajectory;

[0072] A weight module, used for obtaining a first weight coefficient according to the abnormal motion trajectory;

[0073] A second trajectory module is used to construct an evaluation period, obtain eye movement trajectory data within the evaluation period, mark it as a second movement trajectory, and obtain an eye movement compensation coefficient according to the second movement trajectory;

[0074] A prediction module, used for obtaining an eye movement compensation trajectory according to the abnormal movement trajectory, the first weight coefficient and the eye movement compensation coefficient, and generating a predicted gaze point position according to the eye movement compensation trajectory;

[0075] The control instruction module is used to determine whether the predicted gaze point position meets the second preset condition. If so, it is determined that the user's gaze behavior is stable and output a corresponding control instruction.

[0076] And, a non-contact human-computer interaction terminal based on stereoscopic vision, comprising:

[0077] one or more processors;

[0078] a storage device having one or more programs stored thereon;

[0079] When the one or more programs are executed by the one or more processors, the one or more processors implement the non-contact human-computer interaction method based on stereoscopic vision.

[0080] The technical effects achieved by the present invention are:

[0081] The present invention can accurately locate the user's gaze point by combining the position of the pupil center and the corneal reflection point, thereby improving the accuracy of human-computer interaction. By detecting abnormal motion trajectories and generating compensation trajectories, it can effectively eliminate user misoperations caused by eye fatigue, accidental movement, etc., thereby improving robustness and stability. According to the user's motion characteristics, the weight coefficient and the compensation trajectory can be adaptively adjusted to enhance adaptability to different users. Human-computer interaction can be performed without the user having physical contact, thereby providing a more natural and convenient operating experience, and is suitable for a variety of application scenarios, such as medical rehabilitation, smart home, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 is a flow chart of the method provided by the present invention;

[0083] Figure 2 It is a system module diagram provided by the present invention. DETAILED DESCRIPTION

[0084] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0085] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0086] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure or characteristic that may be included in at least one implementation of the present invention. The phrase "in a preferred embodiment" that appears in different places in this specification does not refer to the same embodiment, nor is it a separate or selective embodiment that is mutually exclusive with other embodiments.

[0087] Secondly, the present invention is described in detail in conjunction with schematic diagrams. When describing the embodiments of the present invention in detail, for the convenience of explanation, the schematic diagrams are only examples and should not limit the scope of protection of the present invention.

[0088] Please refer to the attached Figure 1 As shown, a non-contact human-computer interaction method based on stereoscopic vision is provided, comprising:

[0089] S1. Obtain binocular images of the user, and extract eye feature points based on the binocular images, wherein the eye feature points include pupil center positions and corneal reflection point positions;

[0090] S2, obtaining the gaze point position and the gaze object according to the pupil center position and the corneal reflection point position;

[0091] S3, obtaining a motion image of the eye, obtaining motion trajectory data of the eye according to the motion image data, and marking it as a first motion trajectory;

[0092] S4, determining whether the first motion trajectory meets the first preset condition, and if not, marking it as an abnormal motion trajectory;

[0093] S5. Obtaining a first weight coefficient according to the abnormal motion trajectory;

[0094] S6, constructing an evaluation period, obtaining eye movement trajectory data within the evaluation period, marking the data as a second movement trajectory, and obtaining an eye movement compensation coefficient according to the second movement trajectory;

[0095] S7, obtaining an eye movement compensation trajectory according to the abnormal movement trajectory, the first weight coefficient and the eye movement compensation coefficient, and generating a predicted gaze point position according to the eye movement compensation trajectory;

[0096] S8. Determine whether the predicted gaze point position meets the second preset condition. If so, determine that the user's gaze behavior is stable and output a corresponding control instruction.

[0097] As in the above steps S1 to S8, first, a stereoscopic vision device (such as a dual camera) is used to capture the user's binocular images, and the pupil center position and corneal reflection point position are extracted by image processing technology to deduce the eye's gaze direction and gaze point. Based on the extracted pupil center and corneal reflection point positions, the user's gaze point and gaze object are determined by a geometric calculation method, which can be achieved through triangulation or other stereoscopic vision technologies. Continuous motion images of the eyes are obtained, and the eye's motion trajectory, i.e., the first motion trajectory, is generated based on these images. Some first preset conditions related to normal gaze behavior are pre-set. If the detected first motion trajectory does not meet these conditions, the motion trajectory is marked as an abnormal motion trajectory, which may indicate that the user is inattentive or accidentally disturbed. A first weight coefficient is calculated based on the detected abnormal motion trajectory, which can reflect the severity of the abnormal movement. During a certain evaluation period, the second motion trajectory of the eyes is continuously recorded. According to the second motion trajectory, the compensation coefficient of the eye movement is calculated, and the eye movement compensation trajectory is generated by combining the abnormal motion trajectory, the weight coefficient and the compensation coefficient, and the corrected predicted gaze point position is calculated to determine whether the predicted gaze point position meets the second preset condition. If so, the user's gaze behavior is considered to be stable, and corresponding control instructions will be generated for operating the device or triggering the interactive function. By combining the position of the pupil center and the corneal reflection point, the user's gaze point can be accurately located to improve the accuracy of human-computer interaction. By detecting abnormal motion trajectories and generating compensation trajectories, the user's misoperation caused by eye fatigue, accidental movement, etc. can be effectively eliminated, and the robustness and stability can be improved. According to the user's motion characteristics, the weight coefficient and the compensation trajectory can be adaptively adjusted to enhance the adaptability to different users. Human-computer interaction can be performed without physical contact between the user and the computer, providing a more natural and convenient operating experience, which is suitable for a variety of application scenarios, such as medical rehabilitation, smart home, etc.

[0098] In a preferred embodiment, the step of obtaining the gaze point position and the gaze object according to the pupil center position and the corneal reflection point position includes:

[0099] S201, obtaining a gaze point according to the pupil center position and the corneal reflection point position;

[0100] S202, obtaining a human-computer interaction device in front of the user;

[0101] S203, obtaining the gaze point position and the gaze object according to the gaze point and the human-computer interaction device.

[0102] As in the above steps S201 to S203, the user's gaze point is determined by the geometric relationship between the pupil center position and the corneal reflection point position. The gaze point reflects the user's gaze direction and the specific position of the gaze. The interactive device in front of the user is detected or identified. The device may include a screen, a control panel, a smart home device, etc., and the user's possible interaction target is understood. The user's gaze point and the detected human-computer interaction device are combined. Through a spatial mapping algorithm (such as projecting the gaze point to the interaction area of ​​the device), the specific position of the user's gaze is inferred, and the object or interface that the user is concerned about is identified. Through the precise combination of the pupil and the corneal reflection point, the user's gaze point can be determined with high accuracy, ensuring the accuracy of human-computer interaction and reducing misjudgment. Various devices in front of the user can be identified, which is suitable for a variety of interaction scenarios and enhances the applicability and flexibility in different environments. Based on the automatic identification of the gaze point and the determination of the gaze object, a seamless and natural user interaction experience is achieved, and the intuitiveness and convenience of the operation are improved. By automatically identifying the device and object the user is looking at, the user's operation needs can be quickly responded to, reducing the necessity of manual input and intervention.

[0103] In a preferred embodiment, the step of determining whether the first motion trajectory meets the first preset condition, and if not, marking it as an abnormal motion trajectory, comprises:

[0104] S401, obtaining a standard motion trajectory;

[0105] S402, obtaining the trajectory similarity between the standard motion trajectory and the first motion trajectory;

[0106] S403, obtaining a trajectory similarity evaluation threshold;

[0107] S404, determining whether the trajectory similarity is lower than a trajectory similarity evaluation threshold;

[0108] If the trajectory similarity is lower than the trajectory similarity evaluation threshold, the binocular movement is judged to be abnormal and marked as an abnormal movement trajectory;

[0109] If the trajectory similarity is not lower than the trajectory similarity evaluation threshold, the binocular movement is judged to be normal.

[0110] As in the above steps S401 to S404, first, a standard motion trajectory model based on normal eye movement is established. The model may be pre-set or dynamically generated based on the user's historical motion data. The first motion trajectory actually detected is compared with the standard motion trajectory. The trajectory similarity between the two is calculated using algorithms such as dynamic time warping (DTW). A similarity evaluation threshold is set to measure the minimum standard for trajectory similarity. The threshold can be adjusted according to different application scenarios or user requirements. Whether the eye movement is abnormal is determined by comparing the trajectory similarity and the similarity evaluation threshold. If the similarity is lower than the set threshold, the binocular movement is determined to be abnormal, and the motion trajectory is considered abnormal. The trajectory is marked as an abnormal trajectory; otherwise, the binocular movement is regarded as normal. By establishing a standard movement trajectory and calculating the trajectory similarity, the user's abnormal eye movement can be detected quickly and efficiently, providing a stable foundation for subsequent human-computer interaction. The trajectory similarity evaluation threshold can be adjusted according to the user's personalized needs and can adapt to the eye movement habits of different users, thereby improving the flexibility of interaction. By accurately calculating the trajectory similarity, misjudgment can be reduced, and the accidental deviation of the eyeball can be distinguished from the real abnormal movement, thereby improving the accuracy and reliability of abnormality detection. Even if the user's eye movement has a short-term irregular fluctuation, the judgment can still be dynamically adjusted according to the similarity threshold to ensure the stability of the overall interaction.

[0111] In a preferred embodiment, the step of obtaining the trajectory similarity between the standard motion trajectory and the first motion trajectory includes:

[0112] S4021, obtaining a reference plane, and scaling the first motion trajectory and the standard motion trajectory to the reference plane in equal proportion;

[0113] S4022. Establish a rectangular coordinate system on the reference plane;

[0114] S4023, constructing multiple trajectory evaluation segments according to the standard motion trajectory;

[0115] S4024, obtaining the inflection point coordinates of the first motion trajectory and the standard motion trajectory in each trajectory evaluation segment;

[0116] S4025, obtaining a coordinate vector value of each inflection point corresponding to the first motion trajectory, and marking it as a first inflection point vector;

[0117] S4026, obtaining the coordinate vector value of each inflection point corresponding to the standard motion trajectory, and marking it as a standard inflection point vector;

[0118] S4027, obtaining a similarity function;

[0119] S4028. Input the plurality of first inflection point vectors and the plurality of standard inflection point vectors into a similarity function, and output the result marked as trajectory similarity.

[0120] As in the above steps S4021 to S4028, the first motion trajectory and the standard motion trajectory are proportionally scaled to a unified reference plane to ensure that the trajectories can be compared within the same spatial scale to avoid errors caused by different scales or sizes. A rectangular coordinate system is established on the reference plane to standardize the trajectory analysis. This coordinate system provides a unified reference framework so that each inflection point of the trajectory can be represented by an accurate coordinate value. The standard motion trajectory is divided into multiple trajectory evaluation segments. The inflection point coordinates in each trajectory evaluation segment are obtained and converted into vectors. The inflection point vector of the first motion trajectory is called the first inflection point vector, and the inflection point vector of the standard motion trajectory is called the standard inflection point vector. These vectors provide the motion direction and amplitude of each trajectory segment. The similarity between multiple first inflection point vectors and multiple standard inflection point vectors is calculated through a similarity function. The final output result is the overall trajectory similarity, which is used to evaluate the degree of fit between two trajectories. The similarity function is: , where X represents the trajectory similarity, i represents the number of the first inflection point vector and the number of the standard inflection point vector, and n represents the total number of the first inflection point vector and the total number of the standard inflection point vector. Represented as the i-th first inflection point vector, It is expressed as the i-th standard inflection point vector. By scaling and establishing a coordinate system on a unified reference plane, high-precision trajectory comparison can be achieved to avoid comparison deviations caused by scale differences. The trajectory is divided into multiple evaluation segments, which can be used for detailed analysis of complex motion trajectories and improve the precision and accuracy of trajectory similarity calculation. By vectorizing the inflection point coordinates, the motion changes of the trajectory can be captured more intuitively. The mathematical relationship between vectors is used to calculate the similarity, which ensures the scientificity and effectiveness of the trajectory comparison. The similarity function is used to calculate the inflection point vector, which can flexibly cope with various complex motion trajectory forms, thereby improving the ability to handle irregular trajectories.

[0121] In a preferred embodiment, the step of obtaining the first weight coefficient according to the abnormal motion trajectory includes:

[0122] S501, obtaining the trajectory similarity corresponding to the abnormal motion trajectory and the trajectory similarity evaluation threshold;

[0123] S502, obtaining a weight function;

[0124] S503: Input the trajectory similarity and trajectory similarity evaluation threshold corresponding to the abnormal motion trajectory into a weight function, and output the result marked as a first weight coefficient.

[0125] As in the above steps S501 to S503, first, the similarity of the abnormal motion trajectory and the preset trajectory similarity evaluation threshold are obtained. The trajectory similarity reflects the matching degree between the abnormal motion trajectory and the standard motion trajectory. The evaluation threshold is used to determine the reasonable range of similarity. The obtained trajectory similarity and evaluation threshold are input into the weight function. The weight function dynamically adjusts the weight coefficient according to the level of trajectory similarity. The weight function is , where D is the first weight coefficient, Y is the trajectory similarity evaluation threshold, and X is the trajectory similarity. By inputting the trajectory similarity and the evaluation threshold into the weight function, different degrees of weight coefficients can be dynamically allocated to accurately reflect the degree of abnormal motion and improve the recognition accuracy of abnormal motion. The introduction of weight coefficients can flexibly adjust the processing intensity of abnormal motion. For example, when the degree of abnormality is relatively light, the weight coefficient is low and the reaction is relatively mild; when the degree of abnormality is relatively serious, the weight coefficient is high, and stronger compensation or adjustment will be made. The influence of abnormal motion is balanced by the weight coefficient to ensure adaptive adjustment when facing different degrees of abnormality, thereby improving the stability and robustness of human-computer interaction.

[0126] In a preferred embodiment, the steps of constructing an evaluation period, obtaining eye movement trajectory data within the evaluation period, marking the data as a second movement trajectory, and obtaining an eye movement compensation coefficient according to the second movement trajectory include:

[0127] S601, marking the time marked as the first motion track as the end time;

[0128] S602, obtaining standard evaluation duration;

[0129] S603, obtaining the start time according to the standard evaluation duration and the end time;

[0130] S604, constructing an evaluation period according to the start time and the end time;

[0131] S605, obtaining eye movement trajectory data during the evaluation period, and marking it as a second movement trajectory;

[0132] S606, obtaining similarities of multiple motion trajectories corresponding to the standard motion trajectory according to the multiple second motion trajectories, and marking them as trajectory evaluation similarities;

[0133] S607, obtaining a compensation function;

[0134] S608 , inputting the multiple trajectory evaluation similarities into the compensation function, and marking the output result as the eye movement compensation coefficient.

[0135] As in the above steps S601 to S608, the time marked as the first motion trajectory is set as the end time, and then, based on the preset standard evaluation duration, the start time is calculated, and an evaluation period is constructed. The motion data in the evaluation period is used for subsequent compensation analysis, and the eye motion trajectory data in the evaluation period is recorded, and these data are marked as a second motion trajectory. The second motion trajectory represents the actual movement of the user's eyes in a specific period of time. The trajectory evaluation similarity is obtained by calculating the similarity between multiple second motion trajectories and the standard motion trajectory. The similarity can be calculated through a similarity function. The similarity is used to judge the stability and consistency of the user's eye movement in different time periods. The multiple trajectory evaluation similarities are input into the compensation function, and the eye movement compensation coefficient is output. The compensation coefficient is used to adjust the deviation of the eye movement to ensure that the response can compensate and correct the abnormality in the user's eye movement. The compensation function is , where B represents the eye movement compensation coefficient, r represents the number of multiple trajectory similarity evaluations, and m represents the total number of multiple trajectory similarity evaluations. It is expressed as the similarity evaluation of the r-th trajectory, It is expressed as the similarity of the r-1th trajectory evaluation. By setting the evaluation period, the real-time data of eye movement can be flexibly analyzed to ensure that the compensation mechanism is based on the latest motion information, which improves adaptability. By calculating the trajectory evaluation similarity and generating the compensation coefficient, it can automatically adjust the response to the user's eye movement, optimize the compensation effect, and reduce misoperation. The introduction of the compensation coefficient can smooth out abnormal fluctuations in eye movement, improve the stability of human-computer interaction, and ensure reliable response under different motion conditions. Through the comparative analysis of multiple second motion trajectories, the user's motion pattern can be deeply understood, providing more refined data support for compensation calculations.

[0136] In a preferred embodiment, the step of obtaining the eye movement compensation trajectory according to the abnormal movement trajectory, the first weight coefficient and the eye movement compensation coefficient, and generating a predicted gaze point position according to the eye movement compensation trajectory includes:

[0137] S701, obtaining a standard function;

[0138] S702, inputting the first weight coefficient and the eye movement compensation coefficient into the standard function, and marking the output result as the trajectory compensation value;

[0139] S703, obtaining a trajectory compensation table, wherein the trajectory compensation table includes a plurality of trajectory compensation interval values ​​and a trajectory compensation method corresponding to each trajectory compensation interval value;

[0140] S704, obtaining a target trajectory compensation interval value according to the trajectory compensation value;

[0141] S705, obtaining a corresponding trajectory compensation method from a trajectory compensation table according to the target trajectory compensation interval value;

[0142] S706, compensating the abnormal motion trajectory according to the trajectory compensation method to obtain an eye motion compensation trajectory;

[0143] S707: Generate a predicted gaze point position according to the eye movement compensation trajectory.

[0144] As in the above steps S701 to S707, a standard function is obtained, and the first weight coefficient and the eye movement compensation coefficient are input into the function to calculate the trajectory compensation value. The standard function is used to combine the weight coefficient and the compensation coefficient to obtain a compensation value for adjusting the trajectory. The standard function is A=D*B, where A represents the trajectory compensation value, D represents the first weight coefficient, and B represents the eye movement compensation coefficient. A trajectory compensation table including multiple trajectory compensation interval values ​​and their corresponding trajectory compensation methods is obtained. According to the calculated trajectory compensation value, the target trajectory compensation interval value is searched from the compensation table to determine the appropriate compensation method. According to the target trajectory compensation interval value, the corresponding trajectory compensation method is obtained from the trajectory compensation table. These compensation methods may include different adjustment algorithms or compensation strategies for processing trajectory anomalies within a specific range. The abnormal motion is processed according to the selected trajectory compensation method. The eye movement trajectory is compensated to generate an eye movement compensation trajectory. The compensated trajectory corrects the abnormal movement and improves the prediction accuracy. The predicted gaze point position is generated according to the compensated eye movement trajectory. The predicted gaze point position represents the actual gaze point after compensation, which provides accurate input for subsequent interactive operations. The trajectory compensation value is calculated by the standard function, and the appropriate compensation method is selected according to the compensation table. The eye movement trajectory can be accurately corrected to improve the prediction accuracy of the user's gaze point. The various compensation methods in the trajectory compensation table allow flexible adjustment according to different abnormal situations to ensure effective compensation for various abnormal movements. The compensated eye movement trajectory is closer to the user's actual gaze point, which enhances the ability to predict user intentions, improves the experience of human-computer interaction, and can adapt to different degrees of abnormal movements. The compensation mechanism reduces the errors caused by abnormalities and improves the overall stability and reliability.

[0145] In a preferred embodiment, the step of determining whether the predicted gaze point position meets the second preset condition, and if so, determining that the user's gaze behavior is stable and outputting a corresponding control instruction includes:

[0146] S801, establishing a plane rectangular coordinate system;

[0147] S802, obtaining the coordinate point of the standard gaze point position according to the plane rectangular coordinate system;

[0148] S803, obtaining a coordinate point of a predicted gaze point position according to a plane rectangular coordinate system;

[0149] S804, obtaining a coordinate similarity function;

[0150] S805, inputting the coordinate point of the standard gaze point position and the coordinate point of the predicted gaze point position into a coordinate similarity function, and marking the output result as a gaze similarity value;

[0151] S806, obtaining a gaze similarity evaluation interval value;

[0152] S807, determining whether the gaze similarity value is within the gaze similarity evaluation interval value;

[0153] If the gaze similarity value is within the gaze similarity evaluation interval value, the user's gaze behavior is determined to be stable, and a corresponding control instruction is output;

[0154] If the gaze similarity value is not within the gaze similarity evaluation interval value, it is determined that the user's gaze behavior is wrong.

[0155] As in the above steps S801 to S807, a plane rectangular coordinate system is established to provide a unified reference frame for the standard gaze point and the predicted gaze point, ensuring that the subsequent coordinate point comparison is performed on the same basis to avoid errors caused by different coordinate systems. The coordinate points of the standard gaze point position and the predicted gaze point position are obtained respectively, and a coordinate similarity function is obtained. The standard gaze point coordinates and the predicted gaze point coordinates are input into the function, and the gaze similarity value is calculated. The gaze similarity value represents the degree of difference between the standard gaze point and the predicted gaze point. The coordinate similarity function is: , where H represents the gaze similarity value, x1 represents the x-axis coordinate point of the standard gaze point position, y1 represents the y-axis coordinate point of the standard gaze point position, x2 represents the x-axis coordinate point of the predicted gaze point position, and y2 represents the y-axis coordinate point of the predicted gaze point position. The gaze similarity evaluation interval value is obtained. The interval value is used to determine whether the gaze similarity value is within a preset reasonable range and whether the calculated gaze similarity value is within the evaluation interval value. If the gaze similarity value is within the gaze similarity evaluation interval value, the user's gaze behavior is determined to be stable and the corresponding control instruction is output; if it is not within the evaluation interval value, the gaze behavior is determined to be abnormal or wrong, and the user's binocular images need to be re-acquired. By establishing a standardized coordinate system and calculating the gaze similarity value, it is possible to accurately evaluate whether the user's gaze behavior is stable and provide reliable feedback. It is possible to effectively detect whether the user's gaze behavior meets expectations, thereby ensuring the correct understanding and response to the user's intention in actual operation. The gaze similarity evaluation interval value can be adjusted according to specific applications or user needs, and can adapt to different interactive environments and accuracy requirements. By accurately judging the stability of the gaze behavior, more accurate control instructions can be provided when the user interacts, thereby improving the user experience.

[0156] Please refer to the attached Figure 2 As shown, the present invention also provides a non-contact human-computer interaction system based on stereoscopic vision, which is used for the above-mentioned non-contact human-computer interaction method based on stereoscopic vision, comprising:

[0157] A feature point module is used to obtain binocular images of the user and extract eye feature points based on the binocular images, wherein the eye feature points include the pupil center position and the corneal reflection point position;

[0158] A gaze module, used for obtaining the gaze point position and the gaze object according to the pupil center position and the corneal reflection point position;

[0159] A first trajectory module, used for acquiring a motion image of the eye, acquiring motion trajectory data of the eye according to the motion image data, and marking the data as a first motion trajectory;

[0160] An abnormal trajectory module, used to determine whether the first motion trajectory meets the first preset condition, and if not, mark it as an abnormal motion trajectory;

[0161] A weight module, used for obtaining a first weight coefficient according to the abnormal motion trajectory;

[0162] A second trajectory module is used to construct an evaluation period, obtain eye movement trajectory data within the evaluation period, mark it as a second movement trajectory, and obtain an eye movement compensation coefficient according to the second movement trajectory;

[0163] A prediction module, used for obtaining an eye movement compensation trajectory according to the abnormal movement trajectory, the first weight coefficient and the eye movement compensation coefficient, and generating a predicted gaze point position according to the eye movement compensation trajectory;

[0164] The control instruction module is used to determine whether the predicted gaze point position meets the second preset condition. If so, it is determined that the user's gaze behavior is stable and output a corresponding control instruction.

[0165] In the above, the feature point module obtains the user's binocular images and extracts eye feature points, such as the pupil center position and the corneal reflection point position. The gaze module calculates the gaze point position and the gaze object based on the pupil center position and the corneal reflection point position. The first trajectory module obtains the eye movement image and extracts the motion trajectory data, which is marked as the first motion trajectory. The abnormal trajectory module determines whether the first motion trajectory meets the first preset condition. If not, it is marked as an abnormal motion trajectory. The weight module calculates the first weight coefficient based on the abnormal motion trajectory to quantify the impact of the abnormal motion. The second trajectory module constructs an evaluation period, obtains the eye motion trajectory data within the period, marks it as the second motion trajectory, and calculates the eye motion compensation coefficient. The prediction module obtains the eye motion compensation trajectory based on the abnormal motion trajectory, the first weight coefficient and the eye motion compensation coefficient, and generates a predicted gaze point position to correct the user's eyes. Motion trajectory, improve the accuracy of the predicted gaze point position, the control instruction module determines whether the predicted gaze point position meets the second preset condition, if yes, outputs the corresponding control instruction, outputs the control instruction according to the stable gaze behavior, ensures the accurate response and effective interaction of the system, the feature point module and the gaze module ensure that the system can accurately track the movement of the user's eyes and the gaze target, and provide accurate interactive input, the abnormal trajectory module and the weight module can effectively identify abnormal movement, and calculate the weight coefficient, provide the necessary information for the compensation module, and optimize the system response, the second trajectory module and the prediction module can dynamically adjust the system response, by calculating the compensation coefficient and predicting the gaze point position, improve the stability and accuracy of user interaction, the control instruction module ensures the accurate response of the system to the user input by judging the stability of the gaze behavior, and improves the user's operating experience and the overall reliability of the system.

[0166] And, a non-contact human-computer interaction terminal based on stereoscopic vision, comprising:

[0167] one or more processors;

[0168] a storage device having one or more programs stored thereon;

[0169] When the one or more programs are executed by the one or more processors, the one or more processors implement the non-contact human-computer interaction method based on stereoscopic vision.

[0170] The above is only a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention. The structures, devices and operating methods not specifically described and explained in the present invention shall be implemented according to the conventional means in the art unless otherwise specified and limited.

Claims

1. A non-contact human-computer interaction method based on stereoscopic vision, characterized in that: include: Acquire binocular images of the user, and extract eye feature points based on the binocular images, wherein the eye feature points include pupil center positions and corneal reflection point positions; Obtain the gaze point position and the gaze object according to the pupil center position and the corneal reflection point position; Acquire a motion image of the eye, acquire motion trajectory data of the eye according to the motion image data, and mark the data as a first motion trajectory; Determine whether the first motion trajectory meets the first preset condition, and if not, mark it as an abnormal motion trajectory; Obtaining a first weight coefficient according to the abnormal motion trajectory; Constructing an evaluation period, obtaining eye movement trajectory data within the evaluation period, marking the data as a second movement trajectory, and obtaining an eye movement compensation coefficient according to the second movement trajectory; Obtaining an eye movement compensation trajectory according to the abnormal movement trajectory, the first weight coefficient, and the eye movement compensation coefficient, and generating a predicted gaze point position according to the eye movement compensation trajectory; Determine whether the predicted gaze point position meets the second preset condition, and if so, determine that the user's gaze behavior is stable, and output a corresponding control instruction; The step of obtaining an eye movement compensation trajectory according to the abnormal movement trajectory, the first weight coefficient and the eye movement compensation coefficient, and generating a predicted gaze point position according to the eye movement compensation trajectory includes: Get standard functions; Inputting the first weight coefficient and the eye movement compensation coefficient into the standard function, and marking the output result as the trajectory compensation value; Obtaining a trajectory compensation table, wherein the trajectory compensation table includes a plurality of trajectory compensation interval values ​​and a trajectory compensation method corresponding to each trajectory compensation interval value; Obtain the target trajectory compensation interval value according to the trajectory compensation value; Obtain the corresponding trajectory compensation method from the trajectory compensation table according to the target trajectory compensation interval value; Compensating the abnormal motion trajectory according to the trajectory compensation method to obtain the eye motion compensation trajectory; And generate the predicted gaze point position based on the eye movement compensation trajectory.

2. The non-contact human-computer interaction method based on stereoscopic vision according to claim 1, characterized in that: The step of obtaining the gaze point position and the gaze object according to the pupil center position and the corneal reflection point position comprises: Obtain the gaze point based on the pupil center position and corneal reflection point position; Get the human-computer interaction device in front of the user; Obtain the gaze point position and the gaze object according to the gaze point and the human-computer interaction device.

3. The non-contact human-computer interaction method based on stereoscopic vision according to claim 1, characterized in that: The step of determining whether the first motion trajectory meets the first preset condition, and if not, marking it as an abnormal motion trajectory, comprises: Obtain standard motion trajectory; Obtaining trajectory similarity between the standard motion trajectory and the first motion trajectory; Get trajectory similarity evaluation threshold; Determine whether the trajectory similarity is lower than the trajectory similarity evaluation threshold; If the trajectory similarity is lower than the trajectory similarity evaluation threshold, the binocular movement is judged to be abnormal and marked as an abnormal movement trajectory; If the trajectory similarity is not lower than the trajectory similarity evaluation threshold, the binocular movement is judged to be normal.

4. The non-contact human-computer interaction method based on stereoscopic vision according to claim 3, characterized in that: The step of obtaining the trajectory similarity between the standard motion trajectory and the first motion trajectory comprises: Obtain a reference plane, and scale the first motion trajectory and the standard motion trajectory to the reference plane in equal proportion; Establish a rectangular coordinate system on the reference plane; Construct multiple trajectory evaluation segments according to the standard motion trajectory; Obtaining the inflection point coordinates of the first motion trajectory and the standard motion trajectory in each trajectory evaluation segment; Obtain the coordinate vector value of each inflection point corresponding to the first motion trajectory, and mark it as the first inflection point vector; Obtain the coordinate vector value of each inflection point corresponding to the standard motion trajectory and mark it as the standard inflection point vector; Get similarity function; A plurality of first inflection point vectors and a plurality of standard inflection point vectors are input into a similarity function, and the output result is marked as trajectory similarity.

5. The non-contact human-computer interaction method based on stereoscopic vision according to claim 3, characterized in that: The step of obtaining a first weight coefficient according to the abnormal motion trajectory comprises: Obtaining the trajectory similarity corresponding to the abnormal motion trajectory and the trajectory similarity evaluation threshold; Get the weight function; The trajectory similarity and trajectory similarity evaluation threshold corresponding to the abnormal motion trajectory are input into the weight function, and the output result is marked as the first weight coefficient.

6. The non-contact human-computer interaction method based on stereoscopic vision according to claim 3, characterized in that: The steps of constructing an evaluation period, obtaining eye movement trajectory data within the evaluation period, marking the data as a second movement trajectory, and obtaining an eye movement compensation coefficient according to the second movement trajectory include: Mark the time marked as the first motion track as the end time; Get the standard assessment duration; Get the start time based on the standard evaluation duration and end time; Structuring evaluation periods based on start and end times; Acquire the eye movement trajectory data during the evaluation period and mark it as the second movement trajectory; Acquire similarities of a plurality of motion trajectories corresponding to the standard motion trajectory according to the plurality of second motion trajectories, and mark them as trajectory evaluation similarities; Get the compensation function; The similarities of multiple trajectory evaluations are input into the compensation function, and the output result is labeled as the eye movement compensation coefficient.

7. The non-contact human-computer interaction method based on stereoscopic vision according to claim 1, characterized in that: The step of judging whether the predicted gaze point position meets the second preset condition, and if so, judging that the user's gaze behavior is stable and outputting a corresponding control instruction comprises: Establish a plane rectangular coordinate system; Obtain the coordinate point of the standard gaze point position according to the plane rectangular coordinate system; Obtain the coordinate point of the predicted gaze point position according to the plane rectangular coordinate system; Get coordinate similarity function; Input the coordinate point of the standard fixation point position and the coordinate point of the predicted fixation point position into the coordinate similarity function, and mark the output result as the fixation similarity value; Get the gaze similarity evaluation interval value; Determine whether the gaze similarity value is within the gaze similarity evaluation interval value; If the gaze similarity value is within the gaze similarity evaluation interval value, the user's gaze behavior is determined to be stable, and a corresponding control instruction is output; If the gaze similarity value is not within the gaze similarity evaluation interval value, it is determined that the user's gaze behavior is wrong.

8. A non-contact human-computer interaction system based on stereoscopic vision, applied to the non-contact human-computer interaction method based on stereoscopic vision according to any one of claims 1 to 7, characterized in that: include: A feature point module is used to obtain binocular images of the user and extract eye feature points based on the binocular images, wherein the eye feature points include the pupil center position and the corneal reflection point position; A gaze module, used for obtaining the gaze point position and the gaze object according to the pupil center position and the corneal reflection point position; A first trajectory module, used to obtain a motion image of the eye, obtain motion trajectory data of the eye according to the motion image data, and mark it as a first motion trajectory; An abnormal trajectory module, used to determine whether the first motion trajectory meets the first preset condition, and if not, mark it as an abnormal motion trajectory; A weight module, used for obtaining a first weight coefficient according to the abnormal motion trajectory; A second trajectory module is used to construct an evaluation period, obtain eye movement trajectory data within the evaluation period, mark it as a second movement trajectory, and obtain an eye movement compensation coefficient according to the second movement trajectory; A prediction module, used for obtaining an eye movement compensation trajectory according to the abnormal movement trajectory, the first weight coefficient and the eye movement compensation coefficient, and generating a predicted gaze point position according to the eye movement compensation trajectory; The control instruction module is used to determine whether the predicted gaze point position meets the second preset condition. If so, it is determined that the user's gaze behavior is stable and output a corresponding control instruction.

9. A non-contact human-computer interaction terminal based on stereoscopic vision, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the non-contact human-computer interaction method based on stereoscopic vision as described in any one of claims 1 to 7.

Images