A human-computer interaction perception method and system based on hand-eye tracking

Abstract

The present invention discloses a human-computer interaction perception method and system based on hand-eye tracking, belonging to the technical field of human-computer interaction, including obtaining the position coordinates and direction of the operator's eyes in a virtual space when performing interactive operations, calculating the precise position of gaze pointing, and judging whether a virtual target is within a trigger threshold; simulating two-handed operations similar to force feedback of extending and pressing the arms and twisting the wrists, increasing the effect of force feedback through six servos with different torques, and implementing quantitative control of force feedback intensity by adjusting the pulse width of the servos, with the magnitude of the force increasing or decreasing in a corresponding proportion according to the change in the pulse width. The present invention generates corresponding visual feedback in the process of perceiving a virtual target by tracking the operator's eyes, thereby improving interaction efficiency and accuracy, and improving the sense of reality and immersion in the interaction process.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction technology, and in particular to a human-computer interaction perception method and system based on hand-eye tracking. Background Art

[0002] With the development of virtual reality technology, the efficiency of human-computer interaction, entertainment, safety and other aspects have been improved. Among them, virtual reality technology supports the simulation of a complete virtual three-dimensional interactive environment, presenting users with a realistic multi-sensory experience. Augmented reality supports the expansion of the real world in multiple dimensions, and interacts with users based on the superposition of virtual objects in the real world. Mixed reality is a technology that combines VR and AR. By introducing real environmental information into the virtual world, a mixed virtual and real world is built. The above three technologies are collectively referred to as extended reality technology, which helps users complete daily tasks and improve the efficiency of interaction.

[0003] Although XR devices have significantly improved in terms of lightness, compactness, and user experience, common XR devices still have some limitations in terms of interaction methods. Current devices use virtual hands, which is suitable for interactions at close range, such as touchable user interfaces, but cannot intuitively respond to long-distance interaction needs. Although virtual pointers increase the ability to interact with objects at long distances, they usually require remote controls to achieve this, and studies have shown that users are not very willing to use controllers.

[0004] Through the analysis of the current status of eye tracking and two-hand force feedback devices, it is found that the existing spatial interactions applied to virtual reality, augmented reality or mixed reality are mostly through single input to interact with the target in the scene, lacking multimodal research feedback. Users cannot feel additional feedback clues, which reduces the interactive experience. Summary of the invention

[0005] In view of the above-mentioned prior art, the present invention provides a human-computer interaction perception method and system based on hand-eye tracking, which mainly solves the technical problems existing in the above-mentioned background technology.

[0006] To achieve the above object, the technical solution of the embodiment of the present invention is implemented as follows:

[0007] On the one hand, the present invention provides a human-computer interaction perception method based on hand-eye tracking, and the method includes the following steps: step S1: presenting visual information of the eye movement and force interaction interface and human body perception mechanical feedback information through the interaction and perception unit; step S2: detecting the input information data of the eye operation through the eye tracking motion detection unit; step S3: detecting the input information data of the two-hand operation through the two-hand force tracking motion detection unit; step S4: processing the data detected by the eye tracking motion detection unit and the data detected by the two-hand force tracking motion detection unit through the main processing unit to obtain corresponding virtual target response data; step S5: the visual feedback unit receives and decodes the virtual target response data from the main processing unit, generates a corresponding visual feedback signal, and transmits it to the interaction and perception unit, and the operator performs the next operation on the virtual target according to the perceived visual feedback, and the main processing unit calls and executes the function corresponding to the virtual target according to the operator's further operation.

[0008] Specifically, the step S1 specifically includes: step S11: the visual information of the eye tracking operation interface is presented by the visual information of the interaction and perception unit, specifically including the position and icon of the gaze point in the eye tracking, and the position of the eyes in the virtual space; step S12: the visual information of the virtual target through the two-hand force operation interface is presented by the visual information of the interaction and perception unit, specifically including the position, size, graphic outline and shape change of the two-hand force virtual target.

[0009] Specifically, step S2 specifically includes: the eye tracking motion detection unit detects the movement of the operator's eyes in the interaction and perception unit, and establishes a three-dimensional coordinate system in the virtual space with the X-axis as the horizontal direction, the Y-axis as the vertical direction, and the Z-axis as the depth direction, to obtain the position of the eyes in the virtual space and the position of the virtual target, and transmit them to the main processing unit as input information data.

[0010] Specifically, the step S3 specifically includes: detecting the movement of the operator's hands on the device through a two-hand force tracking motion detection unit, obtaining the movement input information data of the operator's hands on the device, obtaining the current status information of the six rudders, and transmitting it to the main processing unit.

[0011] Specifically, the step S4 specifically includes: step S41: the main processor unit obtains the position coordinates of the eyes in the virtual space at this time according to the eye operation input information data from the eye tracking motion detection unit, and transmits them to the interaction and perception unit; step S42: the signal processing module of the main processor unit obtains the current position of the virtual target and calculates the distance and angular distance to the eye position according to the eye operation input information data from the eye tracking motion detection unit; step S43: the main processor unit activates the main processing module according to the activation signal of the two-hand force tracking motion detection unit from the eye tracking motion detection unit, and determines the different states of different numbered servos according to the signal from the two-hand force tracking motion detection unit; step S44: the servo state information determined by the main processing unit and the positions of the manipulator and the locator are transmitted to the visual feedback unit.

[0012] Specifically, the step S5 specifically includes: step S51: the operator's eyes track the visual feedback in the virtual space, and the operator's direction and position in the virtual space are represented by the gaze point;

[0013] Step S52: visual feedback of the operator's two-handed operation in the virtual space, representing the operator's two-handed action on the executed object through the deformation of the virtual object, and representing the corresponding action and amplitude on the virtual target in the virtual space;

[0014] Step S53: The operator can change the operation mode, cancel the trigger target through eye tracking, or the main processing unit can change other operation modes according to the operator's further operation call and execute the function corresponding to the virtual target through two-handed operation.

[0015] On the other hand, the present invention also provides a human-computer interaction perception system based on hand-eye tracking, which is applied to the method as described above, and the system includes a hardware module and a software module, the software module includes an interaction and perception unit, an eye tracking motion detection unit, a main processing unit, and a visual feedback unit, and the hardware module includes a two-hand force tracking motion detection unit; the interaction and perception unit is used to provide eye tracking information and visual information of the interactive interface, and provide corresponding virtual target feedback according to the operator's eye operation and two-hand operation; the eye tracking motion detection unit is used to provide eye interaction data, and provide corresponding virtual space target response data according to the operator's eye gaze operation data;

[0016] The two-hand force tracking motion detection unit is used to obtain the operator's two hands' response data to the triggered target in the eye tracking unit in real time, obtain the information data of the two-hand manipulation input, and transmit the two-hand manipulation input information data to the main processing unit; the main processing unit is used to receive the two-hand manipulation input information data of the two-hand force tracking motion detection unit, and perform visual feedback on the virtual objects corresponding to the two-hand operation and the corresponding operation range according to the received two-hand manipulation input information data, and complete the function call and execution of the virtual target at the same time; the visual feedback unit is used to receive the visual function call information of the main processing unit, generate the corresponding visual drive signal and send it to the interaction and perception unit, generate a virtual force feedback feeling, and realize visual interaction with the operator.

[0017] Specifically, the interaction and perception unit includes an eye tracking information presentation module and a force feedback unit visual information presentation module, wherein the eye tracking information presentation module is used to present eye tracking information and visual information of the interactive interface, and the force feedback unit visual information presentation module is used to produce corresponding virtual target feedback according to the operator's eye operation and hand operation; the eye tracking motion detection unit includes an eye movement detection module, wherein the eye movement detection module obtains eye movement interaction data and generates corresponding virtual space target response data according to the operator's eye gaze operation data; the main processing unit includes a force information encoding module, a force sensor signal processing module, and a function call execution module, wherein the force information encoding module receives and encodes the input information data of the pressure sensor, wherein the force sensor signal processing module analyzes and processes the encoded input information data of the pressure sensor and converts it into a corresponding control instruction, wherein the function call execution module performs corresponding function call and execution according to the corresponding control instruction; the visual feedback unit includes a visual information receiving and calling module and a visual drive signal generating module, wherein the visual information receiving and calling module receives and decodes the visual function calling information sent from the function calling execution module, and wherein the visual drive signal generating module generates a virtual force feedback feeling according to the visual function calling information.

[0018] Specifically, the two-hand force tracking motion detection unit includes a pair of bending servos, three groups of stretching servos, a torsion servos, a pair of control rods, and a pair of pressure sensors. The pair of bending servos are respectively located at both ends of the three groups of stretching servos, and the pair of bending servos and the three groups of stretching servos are located on the same vertical plane. The pair of bending servos and the three groups of stretching servos are connected by connecting blocks, and the pair of bending servos and the three groups of stretching servos connected by the connecting blocks can rotate radially, one of the bending servos can rotate axially, and the three groups of stretching servos are also connected internally by connecting blocks. A tracker is installed on the side of one group of stretching servos, the torsion servo is installed near the axially rotating bending servo, a pair of pressure sensors are installed near the other bending servo, and a pair of control rods are respectively installed on a pair of bending servos and can rotate radially.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] 1. Through the perception of virtual objects based on eye tracking control, users do not have to endure the physical exertion and low efficiency caused by using their hands to participate in selection. They can select virtual targets accurately and quickly, improving interaction efficiency and accuracy.

[0021] 2. Through the force feedback perception of virtual objects based on two-handed operation, it can provide real force perception, so that the operator can get a more realistic operation experience in the virtual environment;

[0022] 3. Through visual feedback based on eye tracking and force feedback of virtual objects operated by both hands, eye tracking and force feedback can be integrated together, increasing the visual and cognitive experience during the interaction process, allowing the operator to maximize the interactive experience and enhance the operator's sense of immersion. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A flowchart of the steps of a human-computer interaction perception method based on hand-eye tracking provided by the present invention;

[0024] Figure 2 A structural flow chart of a human-computer interaction perception system based on hand-eye tracking provided by the present invention;

[0025] Figure 3 This is a schematic structural diagram of the two-hand force tracking motion detection unit provided by the present invention.

[0026] In the figure: 100, bending servo; 200, stretching servo; 300, twisting servo; 400, control rod 400. DETAILED DESCRIPTION

[0027] The technical solution of the present invention is further elaborated in detail below in conjunction with the drawings and specific embodiments of the specification. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention. In the following description, the expression "some embodiments" is related to a subset of all possible embodiments, but it should be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0028] In the following description, a large number of specific details are provided to provide a more thorough understanding of the present invention. However, it is apparent to those skilled in the art that the present invention can be implemented without one or more of these details. In other examples, in order to avoid confusion with the present invention, some technical features well known in the art are not described.

[0029] It should be understood that the present invention can be implemented in different forms and should not be interpreted as being limited to the embodiments proposed herein. On the contrary, providing these embodiments will make the disclosure thorough and complete, and the scope of the present invention will be fully conveyed to those skilled in the art. And the purpose of the terms used herein is only to describe specific embodiments and is not intended to be a limitation of the present invention. When used herein, the singular forms of "one", "one" and "said / the" are also intended to include plural forms, unless the context clearly indicates another way. It should also be understood that the terms "compose" and / or "include" when used in this specification determine the presence of the features, integers, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or groups. When used herein, the term "and / or" includes any and all combinations of the relevant listed items.

[0030] It should also be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "inside", "outside", "left", "right" and similar expressions used herein are for illustrative purposes only and are not intended to be the only implementation method.

[0031] In order to fully understand the present invention, a detailed structure will be proposed in the following description to illustrate the technical solution proposed by the present invention. The optional embodiments of the present invention are described in detail as follows, but in addition to these detailed descriptions, the present invention may also have other implementations.

[0032] Please refer to the attached Figure 1 On the one hand, the present invention provides a human-computer interaction perception method based on hand-eye tracking, and the method comprises the following steps: step S1: presenting visual information of the eye movement and force interaction interface and human body perception mechanical feedback information through the interaction and perception unit; step S2: detecting the input information data of the eye operation through the eye tracking motion detection unit; step S3: detecting the input information data of the two-hand operation through the two-hand force tracking motion detection unit; step S4: processing the data detected by the eye tracking motion detection unit and the data detected by the two-hand force tracking motion detection unit through the main processing unit to obtain corresponding virtual target response data; step S5: the visual feedback unit receives and decodes the virtual target response data from the main processing unit, generates a corresponding visual feedback signal, and transmits it to the interaction and perception unit, and the operator performs the next step operation on the virtual target according to the perceived visual feedback, and the main processing unit calls and executes the function corresponding to the virtual target according to the operator's further operation.

[0033] Exemplarily, step S1 presents the operator with a visual interactive interface through the interaction and perception unit. These interfaces contain visual information virtual targets and mechanical feedback information, providing the operator with intuitive operation experience and feedback. Step S2 identifies the operator's gaze point or virtual target of interest by detecting the operator's eye movement through the eye tracking motion detection unit. Analyze data such as the speed, direction and dwell time of eyeball rotation, and convert them into input information to ensure that the system understands the operator's attention. Step S3 detects the strength and dynamic movement of the operator's hands through the two-hand force tracking motion detection unit. It can capture the force and action applied by the operator in the virtual interaction, such as pressing, dragging or grabbing, and provide corresponding hand operation data for it. Step S4 comprehensively analyzes and processes the input data from eye tracking and hand tracking through the main processing unit. To determine the operator's intention and generate corresponding virtual target response data. These response data contain the specific parameters of the operation instructions that the operator wants to execute or interact with the virtual target. Step S5 receives and decodes the virtual target response data of the main processing unit through the visual feedback unit, generates a corresponding visual feedback signal (for example, changes the color, size or shape of the target), and then sends it back to the interaction and perception unit. The operator perceives these feedbacks to determine whether his operation is correct and gradually completes the target interaction. If the operator needs to continue the operation, the main processing unit will call and execute the function corresponding to the virtual target according to the observed further operation.

[0034] Specifically, the step S1 specifically includes: step S11: visual information of the eye tracking operation interface is presented by visual information of the interaction and perception unit, specifically including the position and icon of the gaze point in the eye tracking, and the position of the eyes in the virtual space;

[0035] Exemplarily, in this step, the interaction and perception unit begins to collect and display visual information of the eye tracking interface. The system will display the specific location of the operator's gaze point (gaze position) in the virtual space in real time. The "gaze point" viewed by the operator will be displayed in the form of an icon to clearly indicate to the operator where his or her current gaze is focused.

[0036] The position of the icon will change dynamically with the operator's eye movement, providing real-time feedback to help the operator understand the current interaction status more intuitively. This enhances the operator's immersion in the virtual environment and allows them to easily track their gaze in the virtual space. Step S12: The visual information of the virtual target through the two-hand force operation interface is presented by the visual information of the interaction and perception unit, specifically including the position, size, graphic outline and shape change of the two-hand force virtual target.

[0037] Exemplarily, in this step, the interaction and perception unit presents visual information of the two-hand force operation interface. These virtual targets will be displayed in front of the operator. As the operator's hand movements change, the shape and appearance of the virtual targets will also change accordingly.

[0038] Specifically, step S2 specifically includes: the eye tracking motion detection unit detects the movement of the operator's eyes in the interaction and perception unit, and establishes a three-dimensional coordinate system in the virtual space with the X-axis as the horizontal direction, the Y-axis as the vertical direction, and the Z-axis as the depth direction, to obtain the position of the eyes in the virtual space and the position of the virtual target, and transmit them to the main processing unit as input information data.

[0039] Exemplarily, in this step, the eye tracking motion detection unit monitors each movement of the operator's eyes in the interaction and perception unit, and converts the motion data into a digital signal. A three-dimensional coordinate system is established in the virtual space, and the three-dimensional position of the operator's eyes in the virtual space can be accurately obtained.

[0040] In addition, the position of the virtual target in this three-dimensional coordinate system is recorded and this information is passed to the main processing unit as input data. The operator's eye movement data combined with the position of the virtual target provides the basis for subsequent interaction.

[0041] Specifically, the step S3 specifically includes: detecting the movement of the operator's hands on the device through the two-hand force tracking motion detection unit, obtaining the movement input information data of the operator's hands on the device, and obtaining the current six steering gears Status information ,in, Indicates the angle of the servo number position at time, Indicates the angular velocity of the servo number position at time, T represents the transpose, and two force sensors The output voltage of the sensor , Represents the number and is passed to the main processing unit.

[0042] Specifically, the step S4 specifically includes: step S41: the main processor unit obtains the position coordinates of the eyes in the virtual space at this time according to the eye operation input information data from the eye tracking motion detection unit, and transmits it to the interaction and perception unit;

[0043] Step S42: The signal processing module of the main processor unit obtains the current position of the virtual target according to the eye operation input information data from the eye tracking motion detection unit. And calculate the distance and angle distance from the eye position; as follows:

[0044]

[0045] in, Indicates the distance between the current position of the virtual target and the eye position, Indicates the X-axis coordinate point of the virtual target at the current moment. Indicates the Y-axis coordinate point of the virtual target at the current moment. Indicates the Z-axis coordinate point of the virtual target at the current moment. Indicates the X-axis coordinate point of the eye at the current moment, Indicates the Y-axis coordinate point of the eye at the current moment, Indicates the Z-axis coordinate point of the eye at the current moment. in, Indicates the position of the eyes, The angular distance between the virtual target and the line of sight is compared with the trigger threshold distance. If it is within the trigger threshold, it is sent to the main processor unit and the two-hand force tracking motion detection unit. Otherwise, the information data of the eye tracking motion detection unit continues to be collected; step S43: the main processor unit activates the main processing module according to the two-hand force tracking motion detection unit activation signal from the eye tracking motion detection unit, and determines the different states of the servos with different numbers according to the activation signal from the two-hand force tracking motion detection unit; step S44: the servo state information determined by the main processing unit and the positions of the manipulator and the locator are transmitted to the visual feedback unit.

[0046] Exemplarily, the different situations corresponding to step S43 specifically include:

[0047] a: If the servo number Corresponding servo Signal input, there are three servos that will release the set resistance value to the operator, pressure sensor number Compatible pressure sensor Signal input, output voltage If the information output is 0V~10V, it is judged as stretched state, otherwise the output voltage If the information output is -10V~0V, it is judged as pressed state;

[0048] b: If the servo number Corresponding servo Signal input, there is a total of one servo that will release the set resistance value to the operator, pressure sensor number Compatible pressure sensor Signal input, output voltage If the information output is 0V~10V, it is judged as clockwise twisting state, otherwise the output voltage If the information output is -10V~0V, it is judged as a counterclockwise twisting state;

[0049] c: If the servo number Corresponding servo Signal input, there are two servos that will release the set resistance value to the operator, pressure sensor number Compressive stress sensor Signal input, output voltage If the information output is 0V~10V, it is judged as the bending deformation recovery state, otherwise the output voltage If the information output is -10V~0V, it is judged to be in a bent state.

[0050] Specifically, the step S5 specifically includes: step S51: the operator's eyes track the visual feedback in the virtual space, and the operator's direction and position in the virtual space are represented by the gaze point;

[0051] Step S52: visual feedback of the operator's two-handed operation in the virtual space, representing the operator's two-handed action on the executed object through the deformation of the virtual object, and representing the corresponding action and amplitude on the virtual target in the virtual space;

[0052] Step S53: The operator can change the operation mode, cancel the trigger target through eye tracking, or the main processing unit can change other operation modes according to the operator's further operation call and execute the function corresponding to the virtual target through two-handed operation.

[0053] On the other hand, the present invention also provides a human-computer interaction perception system based on hand-eye tracking, see Figure 2, the system is applied to the method as described above, the system includes a hardware module and a software module, the software module includes an interaction and perception unit, an eye tracking motion detection unit, a main processing unit, and a visual feedback unit, the hardware module includes a two-hand force tracking motion detection unit; the interaction and perception unit is used to provide eye tracking information and visual information of the interactive interface, and provide corresponding virtual target feedback according to the operator's eye operation and two-hand operation; the eye tracking motion detection unit is used to provide eye interaction data, and provide corresponding virtual space target response data according to the operator's eye gaze operation data;

[0054] The two-hand force tracking motion detection unit is used to obtain the operator's two hands' response data to the triggered target in the eye tracking unit in real time, obtain the information data of the two-hand manipulation input, and transmit the two-hand manipulation input information data to the main processing unit; the main processing unit is used to receive the two-hand manipulation input information data of the two-hand force tracking motion detection unit, and perform visual feedback on the virtual objects corresponding to the two-hand operation and the corresponding operation range according to the received two-hand manipulation input information data, and complete the function call and execution of the virtual target at the same time; the visual feedback unit is used to receive the visual function call information of the main processing unit, generate the corresponding visual drive signal and send it to the interaction and perception unit, generate a virtual force feedback feeling, and realize visual interaction with the operator.

[0055] Specifically, the interaction and perception unit includes an eye tracking information presentation module and a force feedback unit visual information presentation module, wherein the eye tracking information presentation module is used to present eye tracking information and visual information of the interactive interface, and the force feedback unit visual information presentation module is used to produce corresponding virtual target feedback according to the operator's eye operation and hand operation; the eye tracking motion detection unit includes an eye movement detection module, wherein the eye movement detection module obtains eye movement interaction data and generates corresponding virtual space target response data according to the operator's eye gaze operation data; the main processing unit includes a force information encoding module, a force sensor signal processing module, and a function call execution module, wherein the force information encoding module receives and encodes the input information data of the pressure sensor, wherein the force sensor signal processing module analyzes and processes the encoded input information data of the pressure sensor and converts it into a corresponding control instruction, wherein the function call execution module performs corresponding function call and execution according to the corresponding control instruction; the visual feedback unit includes a visual information receiving and calling module and a visual drive signal generating module, wherein the visual information receiving and calling module receives and decodes the visual function calling information sent from the function calling execution module, and wherein the visual drive signal generating module generates a virtual force feedback feeling according to the visual function calling information.

[0056] For details, see Figure 3The two-hand force tracking motion detection unit includes a pair of bending servos 100, three groups of stretching servos 200, a torsion servo 300, a pair of control rods 400, and a pair of pressure sensors. The pair of bending servos 100 are respectively located at both ends of the three groups of stretching servos 200, and the pair of bending servos 100 and the three groups of stretching servos 200 are located on the same vertical plane. The pair of bending servos 100 and the three groups of stretching servos 200 are connected by a connecting block. The pair of bending servos 100 and the three groups of stretching servos 200 connected by the connecting block can rotate radially, one of the bending servos 100 can rotate axially, and the three groups of stretching servos 200 are also connected internally by a connecting block. A tracker is installed on the side of one group of stretching servos 200, the torsion servo 300 is installed at a position close to the axially rotating bending servo 100, a pair of pressure sensors are installed at a position close to another bending servo 100, and a pair of control rods 400 are respectively installed on the pair of bending servos 100 and can rotate radially.

[0057] Exemplarily, when the operator uses both hands to operate, the bending servo 100 will perform radial and axial rotations according to the natural movement of the hand, and transmit the hand posture information in real time. The activities of the three sets of stretching servos 200 are adapted to the operator's finger movements, and can capture the stretching and contraction information of the hand under different circumstances. The torsion servo 300 can measure the twisting movement of the wrist, convert it into a corresponding electrical signal, and combine it with the gesture information to ensure that the system can fully process the operator's input. The grip information provided by the pressure sensor is used to reflect the operator's control strength over the virtual target, and can generate different feedback signals for different pressure changes, deepening the operator's interaction with the virtual environment. The design of the joystick 400 allows the operator to input more detailed instructions through direct physical operations, thereby enhancing the flexibility of the operator's interaction.

[0058] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. The protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A human-computer interaction perception method based on hand-eye tracking, characterized in that: The method comprises the following steps: step S1: presenting visual information of the eye movement and force interaction interface and human body perception mechanical feedback information through the interaction and perception unit; step S2: detecting input information data of eye operation through the eye movement tracking motion detection unit; step S3: detecting input information data of two-hand operation through the two-hand force tracking motion detection unit; the two-hand force tracking motion detection unit comprises a pair of bending servos, three groups of stretching servos, a torsion servo, a pair of control rods, and a pair of pressure sensors, wherein the pair of bending servos are respectively located at the two ends of the three groups of stretching servos, the pair of bending servos and the three groups of stretching servos are located on the same vertical plane, and the pair of bending servos and the three groups of stretching servos are connected to each other. A pair of bending servos and three groups of stretching servos are connected by a connecting block, and the pair of bending servos and three groups of stretching servos connected by the connecting block can rotate radially, one of the bending servos can rotate axially, and the three groups of stretching servos are also connected internally by a connecting block, a tracker is installed on the side of one group of stretching servos, a torsion servo is installed at a position close to the axially rotating bending servo, a pair of pressure sensors are installed at a position close to another bending servo, and a pair of control rods are respectively installed on a pair of bending servos and can rotate radially; Step S4: The data detected by the eye tracking motion detection unit and the data detected by the two-hand force tracking motion detection unit are processed by the main processing unit to obtain corresponding virtual target response data; Step S5: The visual feedback unit receives and decodes the virtual target response data from the main processing unit, generates a corresponding visual feedback signal, and transmits it to the interaction and perception unit, and the operator performs the next step of operation on the virtual target according to the perceived visual feedback, and the main processing unit calls and executes the function corresponding to the virtual target according to the operator's further operation.

2. The human-computer interaction perception method based on hand-eye tracking according to claim 1, characterized in that: The step S1 specifically includes: step S11: the visual information of the eye tracking operation interface is presented and displayed by the visual information of the interaction and perception unit, specifically including the position and icon of the gaze point in the eye tracking, and the position of the eyes in the virtual space; step S12: the visual information of the virtual target through the two-hand force operation interface is presented and displayed by the visual information of the interaction and perception unit, specifically including the position, size, graphic outline and shape change of the two-hand force virtual target.

3. The human-computer interaction perception method based on hand-eye tracking according to claim 2 is characterized in that: The step S2 specifically includes: the eye tracking motion detection unit detects the movement of the operator's eyes in the interaction and perception unit, and establishes a three-dimensional coordinate system in the virtual space with the X-axis as the horizontal direction, the Y-axis as the vertical direction, and the Z-axis as the depth direction, to obtain the position of the eyes in the virtual space and the position of the virtual target, and transmit them to the main processing unit as input information data.

4. The human-computer interaction perception method based on hand-eye tracking according to claim 3 is characterized in that: The step S3 specifically includes: detecting the movement of the operator's hands on the device through the two-hand force tracking motion detection unit, obtaining the movement input information data of the operator's hands on the device, obtaining the current status information of the six rudders, and transmitting it to the main processing unit.

5. The human-computer interaction perception method based on hand-eye tracking according to claim 4 is characterized in that: The step S4 specifically includes: step S41: the main processor unit obtains the position coordinates of the eyes in the virtual space at this time according to the eye operation input information data from the eye tracking motion detection unit, and transmits them to the interaction and perception unit; step S42: the signal processing module of the main processor unit obtains the position of the virtual target at the current moment and calculates the distance and angular distance to the eye position according to the eye operation input information data from the eye tracking motion detection unit; step S43: the main processor unit activates the main processing module according to the activation signal of the two-hand force tracking motion detection unit from the eye tracking motion detection unit, and determines the different states that the servos with different numbers should respond to according to the signal from the two-hand force tracking motion detection unit; step S44: the servo state information determined by the main processing unit and the positions of the manipulator and the locator are transmitted to the visual feedback unit.

6. The human-computer interaction perception method based on hand-eye tracking according to claim 5 is characterized in that: The step S5 specifically includes: step S51: visual feedback of the operator's eye tracking in the virtual space, indicating the operator's direction and position in the virtual space through the gaze point; Step S52: providing visual feedback of the operator's two-handed operation in the virtual space, representing the operator's two-handed action on the executed object through the deformation of the virtual object, and representing the corresponding action and amplitude on the virtual target in the virtual space; Step S53: The operator can change the operation mode, cancel the trigger target through eye tracking, or the main processing unit can call and execute the function corresponding to the virtual target according to the operator's further operation to change other operation modes through two-handed operation.

7. A human-computer interaction perception system based on hand-eye tracking, the system being applied to the method according to any one of claims 1 to 6, characterized in that: The system includes a hardware module and a software module. The software module includes an interaction and perception unit, an eye tracking motion detection unit, a main processing unit, and a visual feedback unit. The hardware module includes a two-hand force tracking motion detection unit. The interaction and perception unit is used to provide eye tracking information and visual information of the interactive interface, and provide corresponding virtual target feedback according to the operator's eye operation and two-hand operation. The eye tracking motion detection unit is used to provide eye movement interaction data, and provide corresponding virtual space target response data according to the operator's eye gaze operation data. The two-hand force tracking motion detection unit is used to obtain the operator's two-hand response data to the triggered target in the eye tracking unit in real time, obtain the information data of the two-hand control input, and transmit the two-hand control input information data to the main processing unit; the main processing unit is used to receive the two-hand control input information data of the two-hand force tracking motion detection unit, and provide visual feedback to the virtual objects corresponding to the operation of the two hands and the corresponding operation range according to the received two-hand control input information data, and complete the function call and execution of the virtual target at the same time; The visual feedback unit is used to receive the visual function call information of the main processing unit, generate a corresponding visual drive signal and send it to the interaction and perception unit, generate a virtual force feedback feeling, and realize visual interaction with the operator.

8. The human-computer interaction perception system based on hand-eye tracking according to claim 7, characterized in that: The interaction and perception unit includes an eye tracking information presentation module and a force feedback unit visual information presentation module. The eye tracking information presentation module is used to present eye tracking information and visual information of the interactive interface. The force feedback unit visual information presentation module is used to produce corresponding virtual target feedback according to the operator's eye operation and hand operation; the eye tracking motion detection unit includes an eye movement detection module, which obtains eye movement interaction data and generates corresponding virtual space target response data according to the operator's eye gaze operation data; the main processing unit includes a force information encoding module, a force sensor signal processing module, and a function call execution module. The force information encoding module receives and encodes the input information data of the pressure sensor, the force sensor signal processing module analyzes and processes the encoded input information data of the pressure sensor, and converts it into a corresponding control instruction, and the function call execution module performs corresponding function call and execution according to the corresponding control instruction; the visual feedback unit includes a visual information receiving and calling module and a visual drive signal generating module. The visual information receiving and calling module receives and decodes the visual function call information sent from the function call execution module, and the visual drive signal generating module generates a virtual force feedback feeling according to the visual function call information.

Images