Gesture interactive data processing method and device based on AR glasses

Abstract

The invention provides a gesture interactive data processing method and device based on AR glasses, and relates to the field of data processing. The method comprises the following steps: acquiring eye movement tracking data and hand action data which are sent by AR glasses and aim at a user; determining sight line direction data of the user according to the eye movement tracking data; determining a gesture feature vector of the user according to the sight line direction data and the hand action data; matching a virtual force field for a virtual object corresponding to the user based on the gesture feature vector; and carrying out dynamic fusion display on the virtual force field and the virtual object through AR glasses. By implementing the technical scheme provided by the invention, the immersion of user operation and the interaction intuition are improved.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and particularly relates to a gesture interactive data processing method and device based on an AR glasses. Background Art

[0002] With the continuous development of augmented reality technology, gesture-based interaction has become an important means for users to directly operate virtual content.

[0003] However, most current AR gesture interaction systems mainly adopt discrete click or drag operations. This method usually only converts the user's hand movements into simple instructions to directly trigger preset actions, lacking effective simulation of physical feedback in the real world. Due to the lack of continuity and real physical response, when users drag objects in the virtual space, they often feel a rigid and unnatural operation experience, unable to present smooth and gradual displacement and motion transitions like real objects, resulting in an unsmooth user operation experience and reducing the immersion and interaction intuitiveness.

[0004] Therefore, there is an urgent need for a gesture interactive data processing method and device based on an AR glasses. Summary of the Invention

[0005] This application provides a gesture interactive data processing method and device based on an AR glasses, which is convenient for improving the immersion and interaction intuitiveness of user operations.

[0006] In the first aspect of this application, a gesture interactive data processing method based on an AR glasses is provided. The method includes: obtaining eye movement tracking data and hand movement data of the user sent by the AR glasses; determining the line-of-sight direction data of the user according to the eye movement tracking data; determining the gesture feature vector of the user according to the line-of-sight direction data and the hand movement data; matching a virtual force field for the virtual object corresponding to the user based on the gesture feature vector; and dynamically fusing and displaying the virtual force field and the virtual object through the AR glasses.

[0007] By adopting the above technical solution, traditional gesture interaction systems often rely solely on hand movements and are prone to accidental triggering. This method determines the user's line of sight direction through eye-tracking data, combines hand movement analysis with interaction intentions, ensures that only the objects the user is truly focused on will be operated, thereby improving accuracy and reducing misoperations. Traditional methods may simply detect grasping or releasing gestures while ignoring more complex interaction patterns. This method comprehensively analyzes information such as gesture shape, speed, and direction through gesture feature vectors to distinguish different interaction methods, enabling the system to intelligently understand the user's intentions. Most traditional AR gesture interactions use direct displacement methods, that is, the virtual object moves synchronously by the same amount as the hand moves, resulting in a rigid movement of the object, lacking inertia and feedback, and reducing the sense of immersion. This method matches the virtual object through a virtual force field to make the interaction more in line with physical laws. This method enables smooth movement of the virtual object through dynamic fusion display, keeping it synchronized with the hand movement. When the user drags an object, the object does not teleport abruptly but gradually adjusts its position and rotation angle under the action of the virtual force field, forming a smooth transition with the user's operation and making the experience more immersive. Therefore, it is convenient to improve the sense of immersion and interaction intuitiveness of user operations.

[0008] Optionally, the obtaining of the eye-tracking data and hand movement data of the user sent by the AR glasses specifically includes: receiving the original eye movement data of the user sent by the eye camera located on the AR glasses; performing image processing on the original eye movement data using a convolutional neural network to obtain the eye-tracking data; receiving the original action data of the user during hand movement sent by the AR glasses; and performing data processing on the original action data to obtain the hand movement data, where the data processing includes denoising, filtering, and normalization processing.

[0009] By adopting the above technical solutions, image processing of the original eye movement data using a convolutional neural network can effectively extract the key features of the user's eye movement and improve the accuracy of eye movement tracking. Traditional methods may rely on rule matching or simple geometric calculations, while CNN can optimize the recognition effect through a deep learning model, reduce the influence of noise, and improve the accuracy of calculating the user's fixation point. By receiving the original data of hand movement and performing data denoising, filtering, and normalization, measurement noise can be effectively removed, jitter errors can be reduced, and gesture data under different users and different environments can be made more stable. Abnormal data caused by interference such as device jitter and light changes is removed to improve the data purity. Low-pass filtering is used to smooth the data and reduce the jitter of the hand trajectory, making gesture recognition smoother. The hand movement data under different users and different devices is standardized to make the model more adaptable and improve the stability of gesture recognition. Using CNN for eye movement tracking calculation and preprocessing the gesture data at the same time can ensure the efficient operation of data processing and reduce system latency. Traditional methods may have problems such as large computational complexity and slow data processing, while this method accelerates the real-time processing of eye movement and gesture data through parallel computing and data optimization to ensure a low-latency experience during user interaction.

[0010] Optionally, determining the gesture feature vector of the user according to the line-of-sight direction data and the hand movement data specifically includes: determining an eye movement vector according to the line-of-sight direction data; according to the hand movement data, calculating the coordinates of each joint of the hand using a key point detection algorithm, where the coordinates include the three-dimensional coordinates and two-dimensional coordinates of the hand; performing feature fusion on the eye movement vector and the coordinates to obtain the gesture feature vector.

[0011] By adopting the above technical solutions, traditional gesture recognition systems only rely on hand movement data and are easily affected by background interference, occlusion problems, or confusion of similar gestures, resulting in recognition errors. This method introduces an eye movement vector to assist in judging the interaction object through the user's line-of-sight direction, reducing the probability of gesture misrecognition. Traditional gesture recognition may only rely on bone point regression or simple hand edge detection, while this method uses a key point detection algorithm to accurately locate the coordinates of each joint of the hand, improving the recognition accuracy and interaction experience. Directly using hand coordinates for gesture judgment may result in rigid and discontinuous recognition. This method constructs a gesture feature vector that conforms more to the natural interaction logic of the human body through the fusion of the eye movement vector and gesture key point features, making gesture recognition more stable and smooth, and enabling the AR system to better simulate hand interactions in the real world. For example, when the user's gesture dragging speed is different, the system can dynamically adjust the movement inertia of the object.

[0012] Optionally, matching a virtual force field for the virtual object corresponding to the user based on the gesture feature vector specifically includes: obtaining the virtual three-dimensional coordinates and the current velocity vector of the virtual object; calculating a displacement vector between the hand and the virtual object according to the virtual three-dimensional coordinates and the hand three-dimensional coordinates; calculating a first product of the displacement vector and the elastic coefficient, and taking the negative of the first product to obtain an elastic force value; calculating a second product of the current velocity vector and the damping coefficient, and taking the negative of the second product to obtain a damping force value; generating the virtual force field based on the elastic force value and the damping force value.

[0013] By adopting the above technical solution, the method makes the movement of the virtual object more in line with real physical laws by constructing a virtual force field, improving the naturalness and immersion of the interaction. It calculates the elastic force (simulating the elastic recovery trend when an object is pulled) and the damping force (simulating the speed decay effect during movement) using the displacement vector between the hand and the virtual object, so that the object does not show a rigid instantaneous movement during the gesture operation, but presents dynamic effects similar to smooth dragging, inertial sliding, and buffered pauses in the real world. Through this mechanism, users can more intuitively control the virtual object and obtain more real tactile feedback. Even in remote interaction or virtual design environments, a smooth and expected interaction experience can be achieved.

[0014] Optionally, dynamically fusing and displaying the virtual force field with the virtual object through the AR glasses specifically includes: summing the elastic force value and the damping force value to obtain a total acting force; using the total acting force to control the dynamic fusion of the virtual force field and the virtual object to obtain a picture to be displayed; displaying the picture to be displayed to the user through the AR glasses.

[0015] By adopting the above technical solution, the method makes the movement of the object in the AR interaction smoother and more natural through the dynamic fusion display of the virtual force field and the virtual object, improving the user's immersion and manipulation accuracy. It calculates the total acting force by summing the elastic force and the damping force, ensuring that the movement of the virtual object follows physical laws, avoiding rigid jumps or abrupt stops, and making the interaction more in line with the real feel. In addition, the method can render the virtual object after being stressed in real time in the AR glasses, providing intuitive visual feedback, enabling users to clearly perceive the movement trend of the object during operation, improving the interaction fluency and experience comfort, and is especially suitable for virtual design, remote control, and immersive application scenarios.

[0016] Optionally, the method further includes: during the dynamic fusion display of the virtual force field and the virtual object, adding visual animations to obtain a target display animation, where the visual animations include fade-in animations and deformation animations; if it is determined that the absolute value of the displacement vector is greater than or equal to a preset threshold, adding the target display animation to the virtual force field to obtain a target force field; and dynamically fusing and displaying the target force field and the virtual object through the AR glasses.

[0017] By adopting the above technical solution, the method enhances the dynamic feedback effect of the virtual object by introducing visual animations (fade-in animations + deformation animations), making the interaction more intuitive and vivid. By detecting the absolute value of the displacement vector, it intelligently judges the user's operation intensity and timely adds a target display animation to the virtual force field, such as the deformation, elastic feedback, or fade-in effect of an object, to enhance the visual hierarchy and the realism of the operation. This not only optimizes the motion transition effect of the object, avoiding a jerky switch, but also provides a more delicate haptic simulation feedback, enabling the user to more intuitively perceive physical properties such as force, direction, and resistance in the AR environment, thereby enhancing the immersion and control experience of the operation. It is particularly suitable for scenarios such as virtual design, remote control, and 3D modeling.

[0018] Optionally, the method further includes: obtaining a target instruction input by the user, where the target instruction includes a target gesture action; and performing a follow-up virtual integration on the virtual object based on the target gesture action.

[0019] By adopting the above technical solution, the method drives the follow-up virtual integration through the target gesture action, enabling the user to real-time control the changes and integration of the virtual object through simple and intuitive gesture operations. This gesture-based interaction method improves the naturalness and efficiency of the operation, enabling the user to achieve precise virtual object operations without complex input devices. Through this intuitive gesture control, the sense of interaction between the virtual and the real world is enhanced, making the virtual integration process more smooth and flexible, and improving the user experience. It is particularly suitable for scenarios such as virtual design and augmented reality applications.

[0020] In a second aspect of the present application, a gesture interactive data processing device based on AR glasses is provided, characterized in that the device includes an acquisition module and a processing module, wherein the acquisition module is used to acquire eye tracking data and hand motion data for the user sent by the AR glasses; the processing module is used to determine the user's line of sight direction data based on the eye tracking data; the processing module is also used to determine the user's gesture feature vector based on the line of sight direction data and the hand motion data; the processing module is also used to match a virtual force field for the virtual object corresponding to the user based on the gesture feature vector; the processing module is also used to dynamically fuse the virtual force field with the virtual object for display through the AR glasses.

[0021] In the third aspect of the present application, an electronic device is provided, which includes a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device performs the method described above.

[0022] In a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores instructions, and when the instructions are executed, the method described above is executed.

[0023] In summary, one or more technical solutions provided in this application have at least the following technical effects or advantages: Traditional gesture interaction systems often rely only on hand movements and are prone to false triggering. This method uses eye tracking data to determine the user's line of sight and combines hand movements to analyze the interaction intention to ensure that only the object that the user is really paying attention to will be operated, thereby improving accuracy and reducing false operations. Traditional methods may simply detect grabbing or releasing gestures, while ignoring more complex interaction modes. This method uses gesture feature vectors to comprehensively analyze gesture shape, speed, direction and other information to distinguish different interaction methods, allowing the system to intelligently understand user intentions. Traditional AR gesture interactions mostly use direct displacement methods, that is, the virtual object moves synchronously with the hand movement, resulting in stiff movement of objects, lack of inertia and feedback, and reduced immersion. This method matches virtual objects through virtual force fields to make the interaction more in line with physical laws. This method uses dynamic fusion display to make the movement process of virtual objects smooth and synchronized with hand movements. When the user drags an object, the object will not teleport abruptly, but will gradually adjust its position and rotation angle with the action of the virtual force field, forming a smooth transition with the user's operation, making the experience more immersive. Therefore, it is easy to improve the immersion of user operations and the intuitiveness of interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 Schematic flowchart of a gesture interactive data processing method based on an AR glasses provided by an embodiment of the present application; Figure 2 Another schematic flowchart of a gesture interactive data processing method based on an AR glasses provided by an embodiment of the present application; Figure 3 Schematic module diagram of a gesture interactive data processing device based on an AR glasses provided by an embodiment of the present application; Figure 4 Schematic structural diagram of an electronic device provided by an embodiment of the present application.

[0025] Explanation of reference numerals: 31, acquisition module; 32, processing module; 41, processor; 42, communication bus; 43, user interface; 44, network interface; 45, memory. Detailed implementation manners

[0026] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments.

[0027] In the description of the embodiments of the present application, words such as "for example" or "for illustration" are used to indicate examples, illustrations or explanations. Any embodiment or design solution described as "for example" or "for illustration" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Exactly speaking, the use of words such as "for example" or "for illustration" is intended to present related concepts in a specific manner.

[0028] In the description of the embodiments of the present application, the meaning of the term "a plurality of" refers to two or more. For example, a plurality of systems refers to two or more systems, and a plurality of screen terminals refers to two or more screen terminals. In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the technical features indicated. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. The terms "include", "comprise", "have" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0029] With the rapid development of augmented reality technology, gesture-based interaction methods have become an important means for users to interact with virtual content.

[0030] However, most current AR gesture interaction systems still adopt discrete click or drag operations. These systems usually simplify the user's hand movements into basic instructions and trigger preset actions, lacking effective simulation of real-world physical feedback. Due to the lack of continuity and real physical response, when users drag objects in a virtual environment, they often feel that the operations are rigid and unnatural, and the objects cannot exhibit smooth and gradual motion transitions like real objects. This experience makes the user's operations less fluent, thus weakening the immersion and intuitive nature of the interaction.

[0031] To solve the above technical problems, this application provides a gesture interactive data processing method based on AR glasses. Referring to Figure 1 , Figure 1 is a schematic flowchart of a gesture interactive data processing method based on AR glasses provided by an embodiment of this application. This gesture interactive data processing method is applied to a server and includes steps S110 to S150. The above steps are as follows: S110. Obtain the eye movement tracking data and hand movement data of the user sent by the AR glasses.

[0032] Specifically, the AR glasses are equipped with a variety of sensors, such as an eye movement camera and a hand movement sensor. The eye movement camera records the user's line of sight position, line of sight direction, and eye movement trajectory, while the hand movement sensor captures the dynamic data of the user's gestures and hand movements. The collected original eye movement data and hand movement data are preprocessed (such as noise reduction, filtering, normalization) and then sent to the central server in real time through a wireless network (such as Wi-Fi, Bluetooth, or 5G). After receiving these data, the server will further analyze and integrate them to support subsequent application functions, such as judging the user's intention, updating the virtual scene in real time, or achieving more accurate gesture interaction.

[0033] For example, assume that an engineer is using AR glasses for the assembly design of virtual equipment. When the engineer gazes at a virtual part, the eye movement camera of the AR glasses captures his line of sight direction; at the same time, when he makes a gesture of grasping or dragging the part, the hand movement sensor records the specific gesture and movement trajectory. These eye movement data and gesture data will be transmitted to the server in real time. The server combines these two types of data to judge the engineer's operation intention (such as selecting and moving a certain part), and then feeds back the processing result to the AR glasses, enabling the virtual part to smoothly move to a new position under the engineer's operation.

[0034] In a possible implementation, obtaining the eye movement tracking data and hand movement data for the user sent by the AR glasses specifically includes: receiving the original eye movement data of the user sent by the eye movement camera located on the AR glasses; performing image processing on the original eye movement data using a convolutional neural network to obtain the eye movement tracking data; receiving the original movement data of the user during hand movement sent by the AR glasses; and performing data processing on the original movement data to obtain the hand movement data, where the data processing includes denoising, filtering, and normalization processing.

[0035] Specifically, the eye movement camera built into the AR glasses is responsible for capturing the original image data of the user's eye movement. This data may include information such as the user's line of sight direction, fixation point, and blink state. The convolutional neural network is used to process the original eye movement images. Through image recognition and feature extraction, the original data with more blur or noise is converted into accurate eye movement tracking data, which is convenient for determining the specific fixation direction of the user. At the same time, the AR glasses also capture the original data of the user's hand movement, which may include the trajectory of the hand during movement and the position changes of key joints. A series of data processing operations are performed on the captured original hand data, including denoising, filtering, and normalization processing, to eliminate noise and irregular fluctuations, thereby obtaining stable and standardized hand movement data.

[0036] For example, assume that the user is using the AR glasses for virtual interaction, such as selecting an icon on a virtual interface: When the user fixes their gaze on the icon on the screen, the eye movement camera of the AR glasses captures the movement image of the user's eyes. The server analyzes these images through CNN and accurately determines the position where the user is fixating on the icon. At the same time, when the user's hand moves to perform the selection action, the built-in sensors record the movement trajectory and joint positions of the hand. After denoising and filtering processing, these data become smoother and more accurate. The server combines the accurate eye movement tracking data with the processed hand movement data to determine whether the user is performing a click or grab operation, and thus accurately responds to the user's instructions, such as highlighting or activating the icon. Through this data collection and preprocessing method, the server can capture the user's true intention more accurately and quickly, ensure that the interaction operation is smoother and more natural, and improve the overall AR experience.

[0037] S120. Determine the line of sight direction data of the user according to the eye movement tracking data.

[0038] Specifically, the eye movement camera built into the AR glasses captures image data of the user's eye movements. After preprocessing and analysis by a convolutional neural network, this data forms precise eye movement tracking data. This data includes information such as the user's eye position, pupil center, and eye movement trajectory. After receiving this processed data, the server uses key features therein to calculate a line-of-sight vector. This vector describes the direction from the user's eyes, including three-dimensional coordinate information, and is used to represent the direction the user is gazing at. After obtaining the line-of-sight direction data, the server can use it to determine the specific object or area the user is gazing at in a virtual or real environment. For example, this data can help the server highlight the virtual object the user is gazing at or trigger corresponding interaction operations.

[0039] S130. Determine the gesture feature vector of the user according to the line-of-sight direction data and the hand movement data.

[0040] Specifically, the line-of-sight direction data is provided by the eye movement tracking server built into the AR glasses, and the formed line-of-sight vector can represent the direction or area the user is gazing at. The hand movement data comes from the hand movement capture server, and by capturing the two-dimensional or three-dimensional coordinates of key hand points (such as fingers, palms, etc.), it describes the position, posture, and dynamic trajectory of the hand during movement. The server fuses the line-of-sight direction data and the hand movement data. A common method is to splice these two parts of data or perform feature fusion through a deep learning model to generate a gesture feature vector. This vector contains comprehensive information on the user's eye movement and hand movement and can more comprehensively reflect the user's interaction intention. For example, if the user is looking at a virtual object while making a grasping gesture, the fused feature vector can indicate that the user intends to perform a grasping operation on that object.

[0041] In a possible implementation manner, determining the gesture feature vector of the user according to the line-of-sight direction data and the hand movement data specifically includes: determining an eye movement vector according to the line-of-sight direction data; according to the hand movement data, using a key point detection algorithm to calculate the coordinates of each joint of the hand, and the coordinates include the three-dimensional coordinates and two-dimensional coordinates of the hand; performing feature fusion on the eye movement vector and the coordinates to obtain the gesture feature vector.

[0042] Specifically, based on the line-of-sight direction data collected by the AR glasses, the server first extracts the information of the user's eye movement and converts it into an eye movement vector. At the same time, the server utilizes the hand movement data and employs key-point detection algorithms (such as OpenPose, MediaPipe, etc.) to analyze the images or video frames of the user's hand. This algorithm will detect the coordinates of each key part of the hand (such as finger joints, fingertips, palm center, wrist, etc.). These coordinates include both two-dimensional coordinates and three-dimensional coordinates, thus comprehensively reflecting the position and posture of the hand in space. Finally, the server fuses the extracted eye movement vector with the key-point coordinates of the hand. This fusion can adopt simple concatenation or more complex deep learning methods to integrate the information of the two types of data into a unified feature vector. The resulting gesture feature vector not only contains the position where the user is looking but also reflects the specific actions and position changes of the hand, which provides richer feature information for subsequent gesture recognition and intention determination.

[0043] For example, assume that a user is using AR glasses to control a button on a virtual interface: when the user looks at this button, the eye movement tracking server captures his line-of-sight direction and generates an eye movement vector, such as g = [0.5, 0.2, 0.8] (indicating that the user is looking in the direction where the button is located). At the same time, the user makes a clicking action with his hand, and the camera of the AR glasses records this hand movement. Through the key-point detection algorithm, the server extracts the two-dimensional and three-dimensional coordinates of multiple joints of the user's fingers, palm, etc. The server fuses the eye movement vector and the hand coordinates to generate a comprehensive feature vector containing the user's gaze and gesture information. Based on this feature vector, the server can determine the user's operation intention, that is, the user intends to click the button, thereby triggering corresponding virtual operations or feedback. This method can more accurately capture the user's true intention by fusing eye movement data and gesture data, improving the naturalness and accuracy of gesture interaction in the AR server, and making the user's operation more intuitive and efficient.

[0044] S140. Based on the gesture feature vector, match a virtual force field for the virtual object corresponding to the user.

[0045] Specifically, by fusing the user's eye movement and hand movement to obtain the gesture feature vector, the server can accurately determine which virtual object the user has operated on and how the user intends to operate the object (such as grasping, dragging, rotating, etc.). Based on the information contained in the gesture feature vector (such as the speed, direction, and strength of the gesture), the server configures a virtual force field for the corresponding virtual object. This virtual force field is usually composed of an elastic force (simulating the pulling force of an object) and a damping force (simulating the motion attenuation of an object), which can make the object produce a smooth, continuous, and natural motion response when subjected to user operations. By matching a virtual force field to the virtual object, the server can provide physical feedback during user operations. For example, when dragging an object, the object will gradually approach as the hand moves rather than move instantaneously, thus simulating the motion trajectory of an object when subjected to a force in reality and making the interaction experience more real and smooth.

[0046] In a possible implementation manner, based on the gesture feature vector, matching a virtual force field for the virtual object corresponding to the user specifically includes: obtaining the virtual three-dimensional coordinates and the current velocity vector of the virtual object; calculating the displacement vector between the hand and the virtual object according to the virtual three-dimensional coordinates and the hand three-dimensional coordinates; calculating the first product between the displacement vector and the elastic coefficient, and taking the negative of the first product to obtain the elastic force value; calculating the second product between the current velocity vector and the damping coefficient, and taking the negative of the second product to obtain the damping force value; generating a virtual force field based on the elastic force value and the damping force value.

[0047] Specifically, first, the server will obtain the three-dimensional coordinates of the current virtual object (i.e., its position in the virtual space) and its current velocity vector (indicating the direction and speed of the object's movement). By obtaining the three-dimensional coordinates of the hand (obtained from gesture data) and the three-dimensional coordinates of the virtual object, a displacement vector can be calculated. This vector reflects the relative position difference between the hand and the virtual object, indicating in which direction there is a distance difference between the hand and the object. Multiply the above displacement vector by a preset elastic coefficient to obtain a force value. Here, the elastic coefficient determines the strength of the "pull back" or "push away" when the distance between the hand and the object deviates from the expected value. Take the negative of this product to obtain the elastic force value, and the acting direction is to pull the object towards the hand (or back to the expected state). At the same time, the server will take the velocity vector of the current virtual object and multiply it by a preset damping coefficient to obtain another force value. After taking the negative of this product, a damping force value is formed, and its function is to reduce or alleviate the movement of the object (preventing it from moving too fast or oscillating) to make the movement smoother. Finally, combine the elastic force and the damping force to form the total acting force. This total force constitutes the "virtual force field", which will affect the virtual object, so that the object shows inertia, buffering and rebound effects similar to real objects when responding to the user's gesture operations. Through this method, the virtual force field can make the virtual object show natural physical feedback during the interaction process, improving the user's immersion and operation experience.

[0048] Therefore, this method makes the movement of the virtual object more in line with real physical laws by constructing a virtual force field, improving the naturalness and immersion of the interaction. It uses the displacement vector between the hand and the virtual object to calculate the elastic force (simulating the elastic recovery trend when the object is pulled) and the damping force (simulating the speed decay effect during movement), so that the object does not show a rigid instantaneous movement during the gesture operation, but presents dynamic effects similar to smooth dragging, inertial sliding, and buffered pauses in the real world. Through this mechanism, users can more intuitively control the virtual object and obtain more realistic tactile feedback. Even in remote interactions or virtual design environments, a smooth and expected interaction experience can be achieved.

[0049] S150. Dynamically fuse and display the virtual force field with the virtual object through the AR glasses.

[0050] Specifically, the server uses AR glasses to apply the previously calculated "virtual force field" effect to virtual objects, and updates and displays it to users in real time, so that virtual objects can show dynamic responses similar to real objects under physical effects during the interaction process. The server continuously receives data such as user gestures and eye movements, and calculates a virtual force field based on these data. This virtual force field contains effects such as "attraction" and "damping" to simulate the natural movement of real objects when subjected to external forces. The server applies this force field effect to virtual objects, that is, "integrates" these physical effects into the motion control of virtual objects. In this way, when users observe through AR glasses, virtual objects will not only move according to the user's operation, but also show natural physical responses such as smooth acceleration, deceleration, and buffering. The whole process is dynamic, which means that whenever the user's gesture or line of sight changes, the server will recalculate the force field effect and update the state of the virtual object so that the display effect is always consistent with the user's operation.

[0051] For example, imagine an AR game scene where the user sees a virtual bouncy ball through AR glasses. When the user reaches out to grab the ball, the server first detects the user's hand movements and gaze direction, and then calculates a force field effect. For example, when the user approaches the sphere, the sphere will produce an "attraction" effect, while being "damped" during movement. Therefore, when the user drags the hand, the sphere will not instantly follow the movement of the hand, but will have a natural slow acceleration process, and will slightly rebound or pause when approaching the hand, just like the movement trajectory of a real object after being subjected to force. In the end, in the picture seen by the user, the sphere presents a very smooth and natural movement state under the action of the constantly changing force field, making the entire interactive experience more realistic and immersive.

[0052] Therefore, traditional gesture interaction servers often rely only on hand movements and are prone to accidental triggering. This method determines the user's line of sight direction through eye tracking data and combines hand movement analysis to understand the interaction intention, ensuring that only the objects the user truly focuses on are operated, thereby improving accuracy and reducing misoperations. Traditional methods may simply detect grasping or releasing gestures and ignore more complex interaction patterns. This method comprehensively analyzes information such as gesture shape, speed, and direction through gesture feature vectors to distinguish different interaction methods, enabling the server to intelligently understand the user's intention. Most traditional AR gesture interactions use the direct displacement method, that is, the virtual object moves synchronously by the same amount as the hand moves, resulting in a rigid movement of the object, lacking inertia and feedback, and reducing the sense of immersion. This method matches the virtual object through a virtual force field to make the interaction more in line with physical laws. This method enables smooth movement of the virtual object through dynamic fusion display and keeps it synchronized with hand movements. When the user drags an object, the object does not teleport abruptly but gradually adjusts its position and rotation angle under the action of the virtual force field, forming a smooth transition with the user's operation and making the experience more immersive. Therefore, it is convenient to improve the immersion of user operations and the intuitiveness of interaction.

[0053] In a possible implementation, the virtual force field and the virtual object are dynamically fused and displayed through AR glasses, which specifically includes: summing the elastic force value and the damping force value to obtain the total acting force; using the total acting force to control the dynamic fusion of the virtual force field and the virtual object to obtain the picture to be displayed; and displaying the picture to be displayed to the user through AR glasses.

[0054] Specifically, this passage describes a process of dynamically fusing and displaying the virtual force field effect and the motion state of the virtual object through AR glasses. Further, first, the server superimposes the two forces currently acting on the virtual object, namely the elastic force and the damping force, to obtain a total acting force that reflects the comprehensive influence of the object's current motion. Then, the server uses this total acting force to real-time control the dynamic response of the virtual object, enabling the motion state of the object to naturally and smoothly reflect the force field effect (such as smooth acceleration, slow deceleration, and slight rebound effects). Finally, the picture after dynamic update and fusion is displayed to the user through AR glasses, allowing the user to see a virtual object similar to the actual physical motion.

[0055] For example, imagine that in an AR design software, the user is dragging a virtual cube with their hand. As the user moves their hand, the server simultaneously measures the relative position change between the user's hand and the cube (generating an elastic force) and the current motion state of the cube (generating a damping force). The combination of these two forces forms a total acting force, which determines how smoothly the cube moves towards the user's hand without sudden jumps or oscillations during the movement. After this dynamic fusion process, the AR glasses will display a very natural dragging effect in real time, as if the user is manipulating an object with real elastic and inertial properties.

[0056] In a possible implementation, during the process of dynamically fusing and displaying the virtual force field and the virtual object, visual animations are added to obtain a target display animation. The visual animations include fade-in animations and deformation animations. If it is determined that the absolute value of the displacement vector is greater than or equal to a preset threshold, the target display animation is added to the virtual force field to obtain a target force field. The target force field and the virtual object are dynamically fused and displayed through the AR glasses.

[0057] Specifically, when combining and displaying the virtual force field and the virtual object, not only physical force simulation is relied on, but also visual effects are added. These visual animations include fade-in animations (smooth transitions of color or brightness) and deformation animations (dynamic changes in the shape of the object). These animations together constitute the "target display animation", making the movement and changes of the object more visually impactful and realistic. The server monitors the displacement vector generated during the user's operation (i.e., the relative distance change between the hand position and the virtual object). When the absolute value of this displacement is greater than or equal to a preset threshold, it indicates that the user's operation force is relatively large or the movement amplitude is relatively large. At this time, the server adds the previously mentioned target display animation to the virtual force field. This process generates a "target force field", that is, a virtual force field with special visual animation effects. Finally, after the target force field with the target display animation is dynamically fused with the virtual object, it is displayed to the user in real time through the AR glasses. In this way, the user can not only feel the physical force effects (such as smooth dragging and buffering effects), but also see visual changes, such as fade-in or deformation effects, further enhancing the immersion and intuitiveness of the interaction.

[0058] For example, imagine an AR interactive application where a user is dragging a virtual balloon with gestures. When the user moves his hand slowly, the virtual balloon may only show a basic follow-up effect, moving slightly and slowing down slowly, giving a smooth and natural feeling of movement. But if the user suddenly drags his hand significantly (at this time, the displacement between the hand and the balloon becomes very large, exceeding the preset threshold), the server will add visual animations to the calculated virtual force field: Gradient animation: The color or brightness of the balloon may change gradually, prompting the user that the balloon is being acted upon by a large force; Deformation animation: The balloon may be stretched or compressed briefly, similar to the deformation of a real rubber object when it is subjected to force. This target force field with visual animation will be dynamically integrated with the movement state of the balloon and displayed in the AR glasses. The user can not only see the balloon moving smoothly, but also intuitively feel the force of the operation and the physical feedback of the object, making the interactive experience more realistic and immersive. Through this mechanism, the server can automatically adjust the display effect according to the force of the user's operation, so that the feedback of the virtual object is more in line with the user's expectations, further improving the overall interactive effect.

[0059] In one possible implementation, refer to Figure 2 , Figure 2 Another flow chart of a gesture interactive data processing method based on AR glasses provided in an embodiment of the present application includes steps S210 to S220, and the above steps are as follows: S210, obtaining a target instruction input by a user, where the target instruction includes a target gesture action; S220, performing a follow-up virtual integration of a virtual object based on the target gesture action.

[0060] Specifically, the server receives the instructions input by the user in the AR glasses, which are mainly completed through gestures, rather than traditional buttons or voice commands. The core part of the target instruction is the "target gesture action", that is, the specific gesture performed by the user in space, such as grabbing, rotating, dragging, etc. The server uses computer vision algorithms (such as key point detection, convolutional neural network, etc.) to parse the user's hand movements, extract feature vectors, and identify the gesture type. For example, if the user makes a "grab" gesture, the server will recognize that the instruction corresponding to this gesture is to "select" or "attach" a virtual object. Follow-up virtual integration means that the user's gesture can directly affect the movement of the virtual object, so that it keeps pace with the user's operation, rather than discrete single-step operations. The server calculates how the virtual object should respond based on the user's gesture data. For example: the drag gesture indicates that the virtual object moves with the hand, the rotation gesture indicates that the virtual object rotates according to the gesture, and the grab indicates that the virtual object is selected and then attached to a target position. This mechanism ensures that the movement of the virtual object can remain natural and smooth, rather than a single click or jump operation.

[0061] For example, Example 1: Manipulation of virtual objects in AR design software. Suppose users are using AR glasses for indoor decoration design and they want to place a virtual lamp on the ceiling: The users make a "grasping" gesture towards the lamp, and the server recognizes this gesture and "selects" the lamp. Based on the gesture feature vector of the users, the server makes the lamp "follow" and move in the direction of the users' hand movement. The users raise their hands to the ceiling position and make a "releasing" gesture by spreading their fingers. The server recognizes this instruction and ensures that the lamp is stably located on the ceiling. In this process, the users do not need to click buttons or use menus to select "move" or "place" operations, but directly use gestures for smooth interaction to achieve natural virtual object manipulation.

[0062] Example 2: Prop interaction in AR games. In an AR game, players need to manipulate an energy ball: The players extend their palms and make a "grasping" gesture, and the energy ball is immediately "selected" and adsorbed to the position of the players' palms. The server continuously tracks the players' gestures and makes the energy ball move smoothly following the hand movement. When the players make a "throwing" gesture (pushing the arm forward and spreading the fingers), the server calculates the hand speed vector and makes the energy ball fly out in this direction, while adjusting its movement trajectory to conform to physical laws. In this way, players can intuitively manipulate virtual objects through natural gestures, making their behavior similar to real objects, thereby enhancing the immersion and interaction experience.

[0063] Therefore, the server obtains the target gesture actions of the users and parses them into specific interaction instructions. By calculating the spatial coordinates and movement directions of the users' hands, the server makes the virtual objects respond to the users' operations in a "follow-up" manner instead of discrete single-step operations. This method makes AR interaction more intuitive and smooth. Users can interact with virtual content as if they were operating real objects, enhancing the immersion and usage experience. This follow-up type of virtual integration is particularly suitable for fields such as AR design, games, and industrial training, enabling users to manipulate virtual content in a natural way and achieving a more efficient and intuitive interaction experience.

[0064] This application also provides a gesture interactive data processing device based on AR glasses. Refer to Figure 3 , Figure 3It is a module schematic diagram of a gesture interactive data processing device provided by an embodiment of the present application. The gesture interactive data processing device is a server, and the server includes an acquisition module 31 and a processing module 32. The acquisition module 31 acquires eye movement tracking data and hand movement data of a user sent by an AR glasses. The processing module 32 determines the line-of-sight direction data of the user according to the eye movement tracking data. The processing module 32 determines the gesture feature vector of the user according to the line-of-sight direction data and the hand movement data. The processing module 32 matches a virtual force field for the virtual object corresponding to the user based on the gesture feature vector. The processing module 32 dynamically fuses and displays the virtual force field and the virtual object through the AR glasses.

[0065] In a possible implementation manner, the acquisition module 31 acquires the eye movement tracking data and the hand movement data of the user sent by the AR glasses, specifically including: the acquisition module 31 receives the original eye movement data of the user sent by an eye movement camera located on the AR glasses. The processing module 32 performs image processing on the original eye movement data by using a convolutional neural network to obtain the eye movement tracking data. The acquisition module 31 receives the original movement data of the user during the hand movement process sent by the AR glasses. The processing module 32 performs data processing on the original movement data to obtain the hand movement data, and the data processing includes denoising, filtering, and normalization processing.

[0066] In a possible implementation manner, the processing module 32 determines the gesture feature vector of the user according to the line-of-sight direction data and the hand movement data, specifically including: the processing module 32 determines an eye movement vector according to the line-of-sight direction data. The processing module 32 calculates the coordinates of each joint of the hand by using a key point detection algorithm according to the hand movement data, and the coordinates include the three-dimensional coordinates of the hand and the two-dimensional coordinates of the hand. The processing module 32 performs feature fusion on the eye movement vector and the coordinates to obtain the gesture feature vector.

[0067] In a possible implementation manner, the processing module 32 matches a virtual force field for the virtual object corresponding to the user based on the gesture feature vector, specifically including: the acquisition module 31 acquires the virtual three-dimensional coordinates and the current velocity vector of the virtual object. The processing module 32 calculates the displacement vector between the hand and the virtual object according to the virtual three-dimensional coordinates and the three-dimensional coordinates of the hand. The processing module 32 calculates the first product between the displacement vector and the elastic coefficient, and takes the negative of the first product to obtain the elastic force value. The processing module 32 calculates the second product between the current velocity vector and the damping coefficient, and takes the negative of the second product to obtain the damping force value. The processing module 32 generates a virtual force field based on the elastic force value and the damping force value.

[0068] In a possible implementation, the processing module 32 dynamically fuses and displays a virtual force field and a virtual object through an AR glasses, which specifically includes: the processing module 32 sums the elastic force value and the damping force value to obtain a total acting force; the processing module 32 uses the total acting force to control the dynamic fusion of the virtual force field and the virtual object to obtain a to-be-displayed picture; the processing module 32 displays the to-be-displayed picture to the user through the AR glasses.

[0069] In a possible implementation, during the process of dynamically fusing and displaying the virtual force field and the virtual object, the processing module 32 adds a visual animation to obtain a target display animation, and the visual animation includes a fade-in animation and a deformation animation; if the processing module 32 determines that the absolute value of the displacement vector is greater than or equal to a preset threshold, the target display animation is added to the virtual force field to obtain a target force field; the processing module 32 dynamically fuses and displays the target force field and the virtual object through the AR glasses.

[0070] In a possible implementation, the acquisition module 31 acquires a target instruction input by the user, and the target instruction includes a target gesture action; the processing module 32 performs a follow-up virtual integration on the virtual object based on the target gesture action.

[0071] It should be noted that: when the device provided in the above embodiment realizes its functions, only the above-mentioned division of each functional module is used for illustration. In actual application, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be seen in the method embodiment, which will not be elaborated here.

[0072] This application also provides an electronic device, refer to Figure 4 , Figure 4 which is a schematic structural diagram of an electronic device provided in an embodiment of this application. The electronic device may include: at least one processor 41, at least one network interface 44, a user interface 43, a memory 45, and at least one communication bus 42.

[0073] Among them, the communication bus 42 is used to realize the connection and communication between these components.

[0074] Among them, the user interface 43 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 43 may further include a standard wired interface and a wireless interface.

[0075] Among them, the network interface 44 may optionally include a standard wired interface and a wireless interface (such as a Wi-Fi interface).

[0076] Among them, the processor 41 may include one or more processing cores. The processor 41 connects various parts within the entire server through various interfaces and lines. By running or executing instructions, programs, code sets, or instruction sets stored in the memory 45, and by calling the data stored in the memory 45, it performs various functions of the server and processes data. Optionally, the processor 41 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). The processor 41 may integrate a combination of one or several of a central processing unit (CPU), a graphics processing unit (GPU), and a modem, etc. Among them, the CPU mainly processes the operating system, user interface, application programs, etc.; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communication. It can be understood that the above-mentioned modem may not be integrated into the processor 41 and may be implemented separately through a single chip.

[0077] Among them, the memory 45 may include random access memory (RAM) and may also include read-only memory. Optionally, the memory 45 includes a non-transitory computer-readable storage medium. The memory 45 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 45 may include a program storage area and a data storage area. Among them, the program storage area may store instructions for implementing the operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-mentioned method embodiments, etc.; the data storage area may store the data involved in the above-mentioned method embodiments. Optionally, the memory 45 may also be at least one storage device located far from the aforementioned processor 41. As Figure 4 shown, the memory 45, as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an application program of a gesture interactive data processing method based on AR glasses.

[0078] In Figure 4In the electronic device shown, the user interface 43 is mainly used to provide an interface for the user to input and obtain the data input by the user; while the processor 41 can be used to call an application program stored in the memory 45 for a gesture interactive data processing method based on AR glasses. When executed by one or more processors, the electronic device executes one or more of the methods in the above embodiments.

[0079] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0080] The present application also provides a computer-readable storage medium storing instructions. When executed by one or more processors, the electronic device executes one or more of the methods described in the above embodiments.

[0081] In the above embodiments, the descriptions of the various embodiments have their own emphases. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0082] In several embodiments provided by the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some service interfaces. The indirect couplings or communication connections of the device or unit can be in electrical or other forms.

[0083] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0084] In addition, in each embodiment of the present application, the functional units can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0085] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned memory includes: various media such as USB flash drives, mobile hard disks, magnetic disks, or optical discs that can store program codes.

[0086] The above are only exemplary embodiments of the present disclosure, and the scope of the present disclosure cannot be limited thereby. That is, any equivalent changes and modifications made in accordance with the teachings of the present disclosure still fall within the scope covered by the present disclosure. Those skilled in the art will readily think of other implementation manners of the present disclosure after considering the specification and the practice of the disclosure. This application aims to cover any variations, uses, or adaptive changes of the present disclosure, and these variations, uses, or adaptive changes follow the general principles of the present disclosure and include common general knowledge or conventional technical means in the technical field not recorded in the present disclosure. The specification and the embodiments are only regarded as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A gesture interactive data processing method based on an AR glasses, characterized in that The method includes: Obtaining eye movement tracking data and hand movement data for the user sent by the AR glasses; Determining the line-of-sight direction data of the user according to the eye movement tracking data; Determining the gesture feature vector of the user according to the line-of-sight direction data and the hand movement data; Based on the gesture feature vector, matching a virtual force field for the virtual object corresponding to the user; Dynamically fusing and displaying the virtual force field and the virtual object through the AR glasses.

2. The gesture interactive data processing method based on an AR glasses according to claim 1, wherein The obtaining of the eye movement tracking data and hand movement data for the user sent by the AR glasses specifically includes: Receiving the original eye movement data of the user sent by the eye movement camera on the AR glasses; Performing image processing on the original eye movement data by using a convolutional neural network to obtain the eye movement tracking data; Receiving the original movement data of the user during hand movement sent by the AR glasses; Performing data processing on the original movement data to obtain the hand movement data, and the data processing includes denoising, filtering, and normalization processing.

3. The gesture interactive data processing method based on an AR glasses according to claim 1, wherein The determining of the gesture feature vector of the user according to the line-of-sight direction data and the hand movement data specifically includes: Determining an eye movement vector according to the line-of-sight direction data; According to the hand movement data, calculating the coordinates of each joint of the hand by using a key point detection algorithm, and the coordinates include three-dimensional coordinates and two-dimensional coordinates of the hand; Performing feature fusion on the eye movement vector and the coordinates to obtain the gesture feature vector.

4. The gesture interactive data processing method based on an AR glasses according to claim 3, wherein The matching of the virtual force field for the virtual object corresponding to the user based on the gesture feature vector specifically includes: Obtaining the virtual three-dimensional coordinates and the current velocity vector of the virtual object; Calculating the displacement vector between the hand and the virtual object according to the virtual three-dimensional coordinates and the three-dimensional coordinates of the hand; Calculating the first product between the displacement vector and the elastic coefficient, and taking the negative of the first product to obtain the elastic force value; Calculating the second product between the current velocity vector and the damping coefficient, and taking the negative of the second product to obtain the damping force value; Generating the virtual force field based on the elastic force value and the damping force value.

5. The gesture interactive data processing method based on an AR glasses according to claim 4, characterized in that The dynamically fusing and displaying the virtual force field and the virtual object through the AR glasses specifically includes: Summing the elastic force value and the damping force value to obtain the total acting force; Using the total acting force to control the dynamic fusion of the virtual force field and the virtual object to obtain the picture to be displayed; Displaying the picture to be displayed to the user through the AR glasses.

6. The gesture interactive data processing method based on an AR glasses according to claim 4, wherein The method further includes: Adding visual animations during the process of dynamically fusing and displaying the virtual force field and the virtual object to obtain a target display animation, and the visual animations include fade-in animations and deformation animations; If it is determined that the absolute value of the displacement vector is greater than or equal to a preset threshold, then adding the target display animation to the virtual force field to obtain a target force field; Dynamically fusing and displaying the target force field and the virtual object through the AR glasses.

7. The gesture interactive data processing method based on an AR glasses according to claim 1, wherein The method further includes: Obtain the target instruction input by the user, where the target instruction includes a target gesture action; Based on the target gesture action, perform a follow-up virtual integration on the virtual object.

8. A gesture interactive data processing device based on an AR glasses, characterized in that, The device includes an acquisition module (31) and a processing module (32), where, The acquisition module (31) is configured to acquire eye movement tracking data and hand movement data for the user sent by the AR glasses; The processing module (32) is configured to determine the user's line-of-sight direction data according to the eye movement tracking data; The processing module (32) is further configured to determine the user's gesture feature vector according to the line-of-sight direction data and the hand movement data; The processing module (32) is further configured to match a virtual force field for the virtual object corresponding to the user based on the gesture feature vector; The processing module (32) is further configured to dynamically fuse and display the virtual force field and the virtual object through the AR glasses.

9. An electronic device, characterized in that, The electronic device includes a processor (41), a memory (45), a user interface (43), and a network interface (44). The memory (45) is used to store instructions. Both the user interface (43) and the network interface (44) are used to communicate with other devices. The processor (41) is used to execute the instructions stored in the memory (45) so that the electronic device executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions, and when the instructions are executed, the method according to any one of claims 1 to 7 is executed.

Images