Virtual interaction method and system based on AR glasses

Abstract

The invention discloses a virtual interaction method and system based on AR glasses, and relates to the technical field of virtual interaction methods, in a kart game scene, an attack instruction and a movement instruction of a kart are determined according to head actions of a player; determining a first action instruction of the kart according to the voice information of the player; the head actions and the voice information of the player are fully utilized, and the corresponding actions of the kart in the kart game scene are realized. Therefore, the target locking instruction and the dynamic feedback instruction of the kart are determined according to the eyeball information of the player; the second corresponding action instruction of the kart is determined according to the gesture information of the player, so that the head action, the voice information, the eyeball information and the gesture information of the player are controlled in multiple dimensions, and the game effect of the player in the kart game scene is achieved.

Description

Technical Field

[0001] The present invention relates to the field of virtual interaction technology, and in particular to a virtual interaction method and system based on AR glasses. Background Art

[0002] With the development of science and technology, AR glasses are gradually applied to people's lives and worn on the players' heads. Players wear AR glasses and enter the kart game scene under the action of AR glasses. In the existing technology summary, AR glasses collect the player's voice information and determine the corresponding actions based on the analysis of the voice information. Other functions in the kart game scene need to be presented through the combination of the player and the handle, which affects the player's game effect in the kart game scene. Summary of the Invention

[0003] The purpose of the present invention is to overcome the deficiencies of the prior art, and the present invention provides a virtual interaction method and system based on AR glasses.

[0004] An embodiment of the present invention provides a virtual interaction method based on AR glasses, comprising:

[0005] When the player wears AR glasses, the dynamic data of the player's head is mapped to the kart game scene;

[0006] In a kart game scenario, the kart's attack and movement commands are determined based on the player's head movements;

[0007] The AR glasses collect the player's voice information and determine the first action instruction of the kart based on the player's voice information; the first corresponding action instruction includes a skill activation instruction and a prop call instruction;

[0008] AR glasses collect the player's eye information and determine the kart's target locking instructions and dynamic feedback instructions based on the player's eye information;

[0009] The AR glasses collect the player's gesture information and determine the second corresponding action instructions of the kart based on the player's gesture information; the second action instructions include prop pickup instructions and melee attack instructions.

[0010] An embodiment of the present invention provides a virtual interaction system based on AR glasses, which is applied to the above-mentioned virtual interaction method based on AR glasses. The virtual interaction system based on AR glasses includes:

[0011] The scene module is used to map the player's head dynamic data to the kart game scene when the player wears AR glasses;

[0012] The head action module is used to determine the kart's attack and movement instructions based on the player's head movements in the kart game scene;

[0013] A voice information module is used for the AR glasses to collect the player's voice information and determine the first action instruction of the kart based on the player's voice information; the first corresponding action instruction includes a skill activation instruction and a prop call instruction;

[0014] The eye information module is used by AR glasses to collect the player's eye information and determine the kart's target locking instructions and dynamic feedback instructions based on the player's eye information;

[0015] The gesture information module is used by the AR glasses to collect the player's gesture information and determine the second corresponding action instructions of the kart based on the player's gesture information; the second action instructions include prop pickup instructions and melee attack instructions.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] In an embodiment of the present invention, through the method in the embodiment of the present invention, when the player wears AR glasses, the dynamic data of the player's head is mapped to the kart game scene; in the kart game scene, the attack instructions and movement instructions of the kart are determined according to the player's head movements; the AR glasses collect the player's voice information and determine the first action instructions of the kart according to the player's voice information; the first corresponding action instructions include skill activation instructions and prop call instructions, which make full use of the player's head movements and voice information, and realize the corresponding actions of the kart in the kart game scene.

[0018] Therefore, the AR glasses collect the player's eye information and determine the kart's target locking instructions and dynamic feedback instructions based on the player's eye information; the AR glasses collect the player's gesture information and determine the kart's second corresponding action instructions based on the player's gesture information; the second action instructions include prop pickup instructions and melee attack instructions, collecting the player's eye information and gesture information, so as to facilitate multi-dimensional control of the player's head movement, voice information, eye information and gesture information, thereby realizing the player's game effect in the kart game scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flowchart of a virtual interaction method based on AR glasses in an embodiment of the present invention;

[0020] Figure 2 1 is a flow chart of step S11 in the virtual interaction method based on AR glasses in an embodiment of the present invention;

[0021] Figure 3is a flow chart of step S12 in the virtual interaction method based on AR glasses in an embodiment of the present invention;

[0022] Figure 4 is a flow chart of step S13 in the virtual interaction method based on AR glasses in an embodiment of the present invention;

[0023] Figure 5 1 is a flow chart of step S14 in the virtual interaction method based on AR glasses in an embodiment of the present invention;

[0024] Figure 6 is a flow chart of step S15 in the virtual interaction method based on AR glasses in an embodiment of the present invention;

[0025] Figure 7 It is a schematic diagram of the structural composition of a virtual interaction system based on AR glasses in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention.

[0027] See also Figures 1 to 7 A virtual interaction method based on AR glasses is applied to a virtual interaction scene based on AR glasses; the virtual interaction method based on AR glasses includes:

[0028] Step S11: When the player wears the AR glasses, the dynamic data of the player's head is mapped to the kart game scene;

[0029] Step S12: In the kart game scene, determining the kart's attack and movement instructions based on the player's head movements;

[0030] Step S13: The AR glasses collect the player's voice information and determine a first action instruction for the kart based on the player's voice information; the first corresponding action instruction includes a skill activation instruction and a prop call instruction;

[0031] Step S14: The AR glasses collect the player's eye information and determine the kart's target locking instruction and dynamic feedback instruction based on the player's eye information;

[0032] Step S15: The AR glasses collect the player's gesture information and determine a second corresponding action instruction for the kart based on the player's gesture information; the second action instruction includes a prop pickup instruction and a melee attack instruction;

[0033] refer to Figure 2 ,In step S11, when the player wears AR glasses, the dynamic data of the player's head is mapped to the kart game scene;

[0034] In the specific implementation process of the present invention, the specific steps are:

[0035] S111: The AR glasses are worn by the player, the AR glasses are in working state, and the player enters the kart game scene as a virtual character; the AR glasses monitor the player's head in real time;

[0036] S112: Collect dynamic data of the player's head; construct a corresponding three-dimensional coordinate system based on the dynamic data of the head, and map the three-dimensional coordinate system to the kart game scene; at this time, the rotation angle of the player's head is captured in real time through the built-in gyroscope and accelerometer of the AR glasses.

[0037] In an embodiment of the present application, the AR glasses are worn by the player, the AR glasses are in working state, and the player enters the kart game scene as a virtual character; the AR glasses monitor the player's head in real time, realizing the real-time monitoring of the player's head by the AR glasses and fully controlling the movement of the player's head.

[0038] At this point, the player correctly wears the AR glasses, ensuring the lenses are aligned with their eyes and the straps or brackets are securely supported. AR glasses are typically designed with ergonomics in mind to ensure comfort during extended wear. Alternatively, Xiao Ming prepares to start a go-kart race. He picks up the AR glasses, aligns the lenses with his eyes, and adjusts the straps to fit his head. After putting on the glasses, Xiao Ming can clearly see the world in front of him, and the display lights up, ready to receive and display game content.

[0039] When the AR glasses are worn correctly, the sensors and processors inside the glasses start working; the player sees the game interface through the display screen of the glasses, and at the same time the camera or infrared sensor of the glasses starts to capture the player's movements and environmental information; the game system generates a virtual character based on the player's identity information (such as login account) and places the character in the karting game scene; optionally, after Xiao Ming puts on the AR glasses, the system inside the glasses starts to start up; he sees a sci-fi karting game interface through the transparent display screen of the glasses; the system recognizes Xiao Ming's login account and generates a personalized virtual character for him - a cartoon image wearing a cool racing suit; Xiao Ming can now see his virtual character standing on a futuristic racing track, surrounded by various high-tech facilities and spectator seats.

[0040] The sensors built into the AR glasses (such as gyroscopes, accelerometers, cameras, etc.) begin to monitor the player's head position and movements in real time; these sensors can capture the position adjustments of the player's head, such as rotation and tilt, and transmit these data to the game system in real time; the game system adjusts the perspective or vehicle posture in the game according to these data to achieve interaction between the player and the game; optionally, after Xiao Ming puts on the AR glasses, his head movements are captured in real time by the sensors built into the glasses; when he turns his head slightly to the left, the perspective in the game also rotates to the left accordingly, allowing him to see the spectators and facilities on the left side of the racetrack; when he tilts his head forward, the virtual character in the game also tilts forward, as if preparing to accelerate and sprint; this real-time head movement monitoring and feedback makes Xiao Ming feel as if he is really on the racetrack, integrated with the virtual character.

[0041] Specifically, after Xiao Ming put on the AR glasses, the glasses immediately started working; he saw a sci-fi kart game interface through the display screen of the glasses, and saw his virtual character standing on the racetrack; Xiao Ming began to turn his head, and found that the perspective in the game also rotated, as if his eyes were the camera in the game; when he tilted his head forward, the virtual character in the game also tilted forward, preparing to accelerate and sprint; this real-time head movement monitoring and feedback made Xiao Ming feel very excited and immersed, he felt as if he was really driving a kart on the racetrack.

[0042] Therefore, the dynamic data of the player's head is collected; a corresponding three-dimensional coordinate system is constructed based on the dynamic data of the head, and the three-dimensional coordinate system is mapped to the karting game scene; at this time, the gyroscope and accelerometer built into the AR glasses are used to capture the rotation angle of the player's head in real time, introduce the karting game scene, and perform corresponding operations based on the dynamic data of the head in the karting game scene.

[0043] At this time, when the player wears AR glasses and enters the game scene, the built-in sensors in the glasses (such as cameras, infrared sensors, gyroscopes and accelerometers, etc.) start working to collect the player's head dynamic data in real time; these data may include the position of the head, rotation angle, tilt degree, movement speed, etc.; these data are the basis for the subsequent construction of a three-dimensional coordinate system and mapping to the game scene; at the same time, the camera may be used to capture the overall position of the head and the spatial relationship relative to the environment; the gyroscope is used to measure the rotation speed and angle change of the head; the accelerometer is used to measure the linear acceleration and tilt degree of the head.

[0044] Based on the collected head dynamic data, the game system constructs a three-dimensional coordinate system corresponding to the player's head movements; this coordinate system usually includes the X-axis (left and right direction), the Y-axis (up and down direction) and the Z-axis (front and back direction), which are used to describe the spatial position and posture of the player's head; the system converts this data into coordinate points in the coordinate system through an algorithm, thereby achieving an accurate description of the player's head movements; optionally, the system may use Kalman filtering or other data fusion algorithms to improve the accuracy and stability of the data; the coordinate system may be aligned with the global coordinate system of the game world to correctly map the player's head movements to the game scene.

[0045] At the same time, the constructed three-dimensional coordinate system is mapped to the kart game scene; this means that the player's head movement will directly affect the perspective or vehicle posture in the game; the system uses an algorithm to convert the coordinate points in the coordinate system into corresponding positions and directions in the game scene, thereby realizing the interaction between the player and the game; optionally, the mapping process may involve perspective transformation, projection, and rendering technologies; the system uses technologies such as frustum clipping to ensure that only the game content within the player's field of view is displayed.

[0046] After the three-dimensional coordinate system is constructed and mapped to the game scene, the gyroscope and accelerometer built into the AR glasses continue to capture the player's head rotation angle in real time; this data is used to update the coordinate points in the three-dimensional coordinate system, thereby ensuring that the in-game perspective or vehicle posture is synchronized with the player's head movement; optionally, the data from the gyroscope and accelerometer are fused to provide more accurate head rotation information; the system uses a low-pass filter to reduce the impact of noise and jitter on the measurement of head rotation angle.

[0047] Specifically, player Xiao Ming is using AR glasses to play a kart racing game; when she puts on the glasses and enters the game scene, the built-in sensors in the glasses begin to collect dynamic data of her head; Xiao Ming turns her head slightly to the left, and the gyroscope and accelerometer of the glasses capture this action and send the data to the game system; the system constructs a three-dimensional coordinate system based on this data and maps the angle of Xiao Ming's head rotation to the game scene; at this time, Xiao Ming finds that the perspective in the game also rotates to the left accordingly, as if she is controlling the direction of the car with her eyes; when she tilts her head forward, the virtual car in the game also tilts forward, preparing to accelerate and sprint; this real-time head motion capture and mapping makes Xiao Ming feel as if she is really driving a kart on the track, providing a very immersive gaming experience.

[0048] refer to Figure 3 ,In step S12, in the kart game scene, the attack command and movement command of the kart are determined according to the player's head movement;

[0049] In the specific implementation process of the present invention, the specific steps are:

[0050] S121: collecting the player's head movements, and determining nodding movements and shaking movements based on the analysis of the player's head movements;

[0051] S122: After marking the target to be attacked in the kart game scene, determine the corresponding attack instruction based on the action information of the nodding action; and determine the corresponding movement instruction based on the action information of the shaking action.

[0052] In an embodiment of the present application, the player's head movements are collected, and the nodding movement and the shaking movement are determined based on the analysis of the player's head movements. The nodding movement and the shaking movement are introduced to facilitate subsequent command control of the nodding movement and the shaking movement.

[0053] At this time, the sensors built into the AR glasses (such as gyroscopes, accelerometers, cameras, etc.) are responsible for collecting the player's head movement data; this data may include changes in the position, angle, speed, and other information of the head; the sensor continuously monitors the player's head and captures its tiny movement changes in real time; optionally, the gyroscope is used to measure the rotation angle and speed of the head, and can accurately capture the turning and tilting movements of the player's head; the accelerometer is used to measure the linear acceleration of the head, and can detect rapid movement or vibration of the head; the camera may be used to capture the overall position of the head and the spatial relationship relative to the environment, to assist in identifying movements such as nodding and shaking the head.

[0054] The collected head movement data is transmitted to the processing unit of the game system, which analyzes and interprets the data through an algorithm; the algorithm determines whether the player's head movement meets the characteristics of nodding or shaking the head based on a preset head movement recognition model; these characteristics may include the head's movement trajectory, angle change range, speed, etc.; optionally, a recognition model for nodding and shaking movements is established by training a large amount of head movement data; during the recognition process, continuous head movements, as well as the time intervals and correlations between movements are taken into account to improve the accuracy and stability of recognition; once a nodding or shaking movement is recognized, the system will convert the movement into corresponding instructions within the game according to the preset mapping relationship.

[0055] Specifically, suppose player Xiao Ming is using AR glasses to play a racing game; in the game, he needs to control the acceleration and deceleration of the car by nodding and shaking his head; after Xiao Ming puts on the AR glasses, the built-in sensors in the glasses begin to collect his head movement data in real time; when Xiao Ming wants to speed up, he nods gently; the sensor captures this movement and records the angle change and speed information of the head.

[0056] The collected data is transmitted to the processing unit of the game system; the algorithm analyzes the data to determine whether Xiao Ming's head movement meets the characteristics of nodding; after calculation, the algorithm determines that Xiao Ming did nod; once the nodding action is recognized, the system converts this action into an acceleration instruction in the game according to the preset mapping relationship; then, Xiao Ming's car starts to accelerate in the game; similarly, if Xiao Ming wants to slow down, he will do so by shaking his head; after the system recognizes the shaking action, it will convert this action into a deceleration instruction in the game; in this way, Xiao Ming can control the acceleration and deceleration of the car in the game through simple nodding and shaking of the head, realizing real-time interaction with the game.

[0057] Furthermore, after marking the target to be attacked in the kart game scene, the corresponding attack instruction is determined based on the action information of the nodding action; the corresponding movement instruction is determined based on the action information of the shaking action, and the attack instruction or movement instruction is introduced to control the kart at the attack level and the movement level.

[0058] At this point, the game system first needs to mark the target to be attacked in the karting game scene; this target can be an enemy vehicle, an obstacle, or other object that requires player interaction. The marking process may involve technologies such as target recognition, tracking, and positioning. The game system uses the real-time image data provided by the AR glasses, combined with computer vision algorithms, to accurately mark the target to be attacked in the game scene. Optionally, the game system uses an image recognition algorithm to identify the target object from the real-time image provided by the AR glasses. This may involve steps such as feature extraction and classifier training. Once the target is identified, the game system will continuously track its position and movement trajectory to ensure the target is accurately represented in the game scene.

[0059] After marking the target to be attacked, the game system begins to monitor the player's nodding action; when the player nods, the built-in sensor of the AR glasses will capture this action and convert it into a digital signal and transmit it to the game system; the game system analyzes these signals to determine whether the player has nodded, and according to the preset mapping relationship, converts the nodding action into the corresponding attack command.

[0060] Optionally, the built-in sensors in the AR glasses (such as gyroscopes, accelerometers, etc.) are responsible for capturing the player's nodding movements; these sensors can measure the angular changes and acceleration of the head, thereby identifying the nodding movements; the game system presets an action-command mapping table for mapping the nodding movements to attack commands; when the system recognizes the nodding movement, it will look up the mapping table and trigger the corresponding attack command.

[0061] At the same time, similar to the nodding action, the game system will also monitor the player's head shaking action; when the player shakes his head, the sensor of the AR glasses will capture this action and convert it into a digital signal; the game system parses these signals to determine whether the player has shaken his head, and converts the head shaking action into a corresponding movement instruction according to the preset mapping relationship; optionally, like the nodding action, the capture and analysis of the head shaking action also depends on the built-in sensors of the AR glasses and the algorithm of the game system; the game system also presets an action-instruction mapping table for mapping the head shaking action into a movement instruction; when the system recognizes the head shaking action, it will look up the mapping table and trigger the corresponding movement instruction (such as turn left, turn right, accelerate, decelerate, etc.) according to the direction and amplitude of the head shaking.

[0062] Specifically, suppose player Xiao Ming is using AR glasses to play a kart game; in the game, she needs to drive the kart to attack enemy vehicles and avoid obstacles; the game system successfully identifies and marks the enemy vehicle in front as a target to be attacked through the real-time image data provided by the AR glasses.

[0063] Nodding attack: When Xiao Ming wants to attack an enemy vehicle, she nods slightly; the sensor of the AR glasses captures this action and converts it into a digital signal that is transmitted to the game system; after analyzing the signal, the game system determines that Xiao Ming nodded and triggers the corresponding attack command; then, Xiao Ming's kart launches a missile at the enemy vehicle in front, successfully destroying it.

[0064] Headshake Movement: To avoid obstacles, Xiao Ming controls the kart's direction by shaking her head. When she shakes her head to the left, the game system recognizes this movement and triggers a left turn. Similarly, when she shakes her head to the right, the game system triggers a right turn. Through continuous headshaking, Xiao Ming successfully steers the kart to avoid the obstacles ahead. In this way, Xiao Ming can control the kart's attacks and movements in the game with simple nodding and shaking movements, achieving real-time interaction and an immersive experience.

[0065] In one embodiment of the present application, the game system presets an action-command matching table for mapping the player's head movements (such as nodding and shaking the head) to corresponding commands in the game (such as attacking and moving); the action-command matching table is shown in Table 1:

[0066] Table 1 Action-instruction matching table:

[0067] Head movements Action feature description Game Instructions nod The head moves forward and downward quickly and then returns to its original position Attack Command Slightly shake your head Small movements of the head from side to side, no more than 15 degrees Fine-tuning movement instructions Obvious shaking of the head Violent side-to-side head movements, exceeding 15 degrees but not exceeding 45 degrees Turn movement instructions Continuous rapid shaking of the head The head moves rapidly and continuously from side to side, and the speed exceeds a certain threshold Emergency hedging instructions

[0068] After the player marked an enemy vehicle as a target to be attacked in the game, he nodded; the game system recognized the attack command corresponding to the nodding action based on the matching table, triggered the missile launch function, and successfully hit the enemy vehicle; when the player needed to fine-tune the vehicle's position to avoid an obstacle, he shook his head slightly; the game system recognized the fine-tuning movement command corresponding to the slight shake of the head based on the matching table, and adjusted the vehicle's position so that the player could pass the obstacle smoothly.

[0069] refer to Figure 4 In step S13, the AR glasses collect the player's voice information and determine the first action instruction of the kart according to the player's voice information; the first corresponding action instruction includes a skill activation instruction and a prop call instruction;

[0070] In the specific implementation process of the present invention, the specific steps are:

[0071] S131: collecting the player's voice information, dividing the voice information into multiple segments based on the detection of the player's voice information, and determining corresponding action keywords and corresponding action-related words based on the recognition of the multiple voice segments; at this time, while collecting the player's voice information, noise reduction processing is performed on the ambient noise;

[0072] S132: Determine a first action instruction for the kart based on the action keyword and the corresponding time-related word; the first corresponding action instruction includes a skill activation instruction and a prop call instruction;

[0073] S133: When the action keyword contains a corresponding skill word, determining a skill activation instruction based on the skill word and the action associated word of the action keyword;

[0074] S134: When the action keyword contains a corresponding prop word, a prop calling instruction is determined according to the prop word of the action keyword and the action associated word.

[0075] In an embodiment of the present application, the player's voice information is collected, and multiple voice segments are divided based on the detection of the player's voice information, and corresponding action keywords and corresponding action-related words are determined according to the recognition of multiple voice segments; at this time, when collecting the player's voice information, noise reduction processing is performed on the environmental noise, action keywords and corresponding action-related words are introduced, and command control is performed through the action keywords and corresponding action-related words.

[0076] At this time, the game system collects the player's voice information through a microphone or other voice collection device; this voice information may be commands issued by the player to control the game character or perform specific game actions; in order to ensure that the collected voice information is clear and accurate, the system usually uses a high-quality microphone and may integrate noise suppression technology to reduce the interference of background noise; optionally, the game system may use a high-sensitivity microphone to ensure that it can capture subtle voice changes of the player; the system may set specific parameters such as sampling rate, bit depth and number of channels to optimize the collection quality of voice information.

[0077] After collecting the player's voice information, the system needs to segment this information and divide it into multiple voice segments; these voice segments usually correspond to single words or phrases uttered by the player; the process of dividing the voice segments may involve voice signal processing technology, such as End-Point Detection (EPD), which is used to determine the starting and ending points of the voice signal; optionally, the system may adopt an energy-based endpoint detection algorithm to determine the boundaries of the voice segments by analyzing the energy changes of the voice signal; in addition to endpoint detection, the system may also use VAD technology to distinguish between voice signals and non-voice signals (such as background noise), thereby further improving the accuracy of voice segment division.

[0078] After dividing the speech segments, the system needs to identify each speech segment to determine its corresponding action keywords and action-related words; this process usually involves automatic speech recognition (ASR) technology to convert speech signals into text information; the system may use pre-trained speech recognition models that can recognize various words and phrases that players may say; optionally, the system may adopt deep learning-based speech recognition models, such as recurrent neural networks (RNN) or long short-term memory networks (LSTM), to improve recognition accuracy and robustness; to improve recognition accuracy, the system may use specific game-related vocabularies and language models; these vocabularies and models can better match the terms and expressions that players may use in the game.

[0079] At the same time, when collecting the player's voice information, the system needs to perform noise reduction processing on the environmental noise; this process usually involves signal processing techniques such as filtering, spectral subtraction or noise suppression algorithms; the purpose of noise reduction processing is to reduce the interference of background noise on the voice signal, thereby improving the accuracy and clarity of voice recognition; optionally, the system may use bandpass filters or notch filters to remove noise within a specific frequency range; spectral subtraction estimates and removes background noise by analyzing the spectral characteristics of the voice signal; in addition to traditional signal processing techniques, the system may also use noise suppression algorithms based on machine learning, which can adaptively adjust noise reduction parameters to cope with different noise environments.

[0080] Specifically, suppose a player is using a kart game application based on AR technology; in the game, the player needs to control the kart's movements through voice commands; the player says a voice command: "Speed up and turn right!" The game system collects this voice information through the microphone; the system uses endpoint detection technology and VAD technology to divide this voice command into two voice segments: "Speed up" and "Turn right!".

[0081] The system automatically performs speech recognition on each speech segment and converts it into text: "accelerate" and "turn right" (note: due to the limitations of speech recognition technology, the actual recognition results may contain certain errors or ambiguities). Based on the recognized text information, the system determines the action keywords "accelerate" and "turn", as well as the action-related words "and" and "to the right"; these keywords and related words are used to generate in-game action commands. Through this process, the game system can accurately recognize the player's voice commands and generate corresponding game action commands based on the command content, thereby achieving real-time control of the kart.

[0082] Furthermore, the first action instruction of the kart is determined based on the action keywords and the corresponding time-related words; the first corresponding action instruction includes a skill activation instruction and a prop call instruction, which is compatible with the overall consideration of the action keywords and the corresponding time-related words, ensures the accuracy of the first action instruction of the kart, makes full use of the player's head movements and voice information, and realizes the corresponding action of the kart in the kart game scene.

[0083] At this time, the game system needs to accurately identify the action keywords and time-related words in the player's voice commands; action keywords usually correspond to specific actions or skills in the game, such as "accelerate", "drift", "use props", etc.; time-related words are used to specify the time or order of action execution, such as "immediately", "later", "after...", etc.; the system extracts these keywords by parsing the voice commands issued by the player, providing a basis for subsequent action command generation; optionally, the system may use NLP technology to parse and understand the player's voice commands; NLP technology can identify and analyze the grammatical, semantic and contextual information in the text, so as to accurately extract action keywords and time-related words; in addition to NLP technology, the system may also use dictionary matching and rule parsing methods to identify keywords; the system maintains a dictionary containing in-game action and skill vocabulary, and determines action keywords by matching the vocabulary in the player's voice commands; at the same time, the system also defines a set of rules to parse time-related words, such as determining the time of action execution by identifying words such as "immediately" and "later".

[0084] After identifying the action keywords and time-related words, the system needs to determine the first action instruction of the kart based on this information; this instruction may be a skill activation instruction, used to activate specific skills in the game, such as acceleration, drifting, etc.; it may also be a prop call instruction, used to call specific props in the game, such as missiles, acceleration props, etc.; the system converts the action keywords and time-related words into corresponding game action instructions according to preset mapping relationships or rules; optionally, the system may maintain an action instruction mapping table, which maps action keywords and time-related words to specific game action instructions; when the system identifies the action keywords and time-related words, it will look up the mapping table and generate the corresponding action instructions; in addition to the mapping table, the system may also use instruction generation rules to dynamically generate action instructions; these rules may be designed based on the logic of the game and the player's behavior patterns to ensure that the generated instructions meet the requirements of the game and the player's expectations.

[0085] Specifically, suppose a player is using a karting game application based on AR technology and issues a voice command: "Use the acceleration props and speed up immediately!"; action keywords: "use acceleration props", "accelerate"; time-related words: "immediately"; the system accurately identifies these keywords through NLP technology or dictionary matching and rule parsing methods.

[0086] Based on the identified action keywords and time-related words, the system searches the action instruction mapping table or applies the instruction generation rules to generate the following two action instructions:

[0087] Prop call command: "Use the acceleration prop immediately";

[0088] Skill activation command: "Speed up immediately";

[0089] These two instructions will be executed almost simultaneously or continuously as required by the time associated word "immediately" to achieve the game effect expected by the player;

[0090] Through this process, the game system can accurately understand the player's voice commands and generate corresponding game action commands based on the command content, thereby realizing real-time control of the kart; this interactive method not only improves the fun and immersion of the game, but also reduces the difficulty and complexity of the player's operation of the game.

[0091] Therefore, when an action keyword contains a corresponding skill word, the skill activation instruction is determined based on the skill word of the action keyword and the action-associated word, and the skill activation instruction is introduced.

[0092] At this time, the game system needs to detect whether the action keywords in the player's voice command contain specific skill words; these skill words usually correspond to preset skills or actions in the game, such as "acceleration", "drift", "jump", etc.; the system determines whether there are matching skill words by comparing the player's voice command with the preset skill word library; optionally, the system maintains a skill word library containing all skill or action words in the game; this skill word library exists in the form of a list, dictionary or database for storing and retrieving skill words; the system uses a string matching algorithm to detect whether the player's voice command contains words in the skill word library; these algorithms may include simple exact matching, fuzzy matching (such as Levenshtein distance), regular expression matching, etc.

[0093] After detecting the skill words, the system needs to further parse the action-related words associated with the skill words; these related words may be used to specify the method, scope, target, etc. of skill execution, such as "accelerate forward", "use missiles on enemies on the left", etc.; the system obtains more detailed skill execution information by parsing these related words; optionally, the system may use grammatical analysis technology to parse the player's voice instructions and determine the grammatical relationship between skill words and related words; this helps the system accurately understand the player's intentions and generate skill activation instructions that conform to the game logic; the system may use grammatical analysis technology to parse the player's voice instructions and determine the grammatical relationship between skill words and related words; this helps the system accurately understand the player's intentions and generate skill activation instructions that conform to the game logic.

[0094] After parsing the skill terms and associated words, the system needs to generate a skill activation instruction based on this information; this instruction will be used to activate a specific skill in the game and may contain specific parameters for skill execution, such as skill name, execution method, target object, etc.; the system passes the skill activation instruction to the game system by calling the game engine or related API to realize the execution of the skill; optionally, the system needs to define a unified instruction format to represent the skill activation instruction; this format may include instruction type, skill name, parameter list and other parts to ensure the accuracy and readability of the instruction; the system passes the generated skill activation instruction to the game system by calling the interface or API provided by the game engine; the game engine is responsible for parsing and executing these instructions to realize the skill effect in the game.

[0095] Specifically, suppose a player is using a karting game application based on AR technology and issues a voice command: "Accelerate forward and drift through the corner!"; the system detects that the voice command contains the skill words "accelerate" and "drift".

[0096] The system parses the associated word "forward" associated with "accelerate," indicating the direction of acceleration; the system parses the associated word "corner" associated with "drift," indicating the purpose or scenario of drifting; based on the parsed skill words and associated words, the system generates the following skill activation instructions:

[0097] Acceleration command: "Accelerate forward", means the kart will accelerate forward;

[0098] Drift command: "Drift cornering" means the kart will drift when cornering;

[0099] The system passes these instructions to the game engine, which is responsible for parsing and executing these instructions to achieve the game effect expected by the player. Through this process, the game system can accurately understand the player's voice commands and generate corresponding skill activation instructions based on the content of the instructions, thereby achieving real-time control of the kart skills. This interactive method not only increases the fun and immersion of the game, but also reduces the difficulty and complexity of the player's operation of the game.

[0100] At the same time, when the action keyword contains the corresponding prop word, the prop call instruction is determined according to the prop word of the action keyword and the action associated word, and the prop call instruction is introduced.

[0101] At this time, the game system needs to detect whether the action keywords in the player's voice command contain specific prop words; these prop words usually correspond to preset props or items in the game, such as "missiles", "speed props", "shields", etc.; the system determines whether there are matching prop words by comparing the player's voice command with the preset prop word library; optionally, similar to the skill word library, the system maintains a prop word library that contains all props or item words in the game; this library is used to store and retrieve prop words for matching when the player issues a voice command; the system uses a string matching algorithm to detect whether the player's voice command contains words in the prop word library; these algorithms may include exact matching, fuzzy matching, regular expression matching, etc. to ensure that prop words can be accurately identified.

[0102] After detecting the prop word, the system needs to further parse the action-related words and possible prop parameters associated with the prop word; these related words and parameters may be used to specify the use method, target, quantity, etc. of the prop, such as "use missiles on the enemy in front", "use three acceleration props", etc.; the system obtains more detailed prop usage information by parsing these related words and parameters; optionally, the system uses grammatical analysis technology to parse the player's voice instructions and determine the grammatical relationship between the prop words and related words; at the same time, the system may also use semantic understanding technology to parse the specific meaning of related words and parameters to ensure accurate understanding of the player's command intentions; for voice instructions containing prop parameters, the system needs to extract these parameters and perform appropriate processing; for example, if the player specifies the number of props to be used, the system needs to convert this number into an integer type for use in subsequent game logic.

[0103] After parsing out the prop words, associated words and parameters, the system needs to generate a prop call instruction based on this information; this instruction will be used to call specific props in the game, and may contain specific parameters for the use of props, such as prop name, usage method, target object, quantity, etc.; the system passes the prop call instruction to the game system by calling the game engine or related API to realize the use of props; optionally, similar to the skill activation instruction, the system needs to define a unified instruction format to represent the prop call instruction; this format should include parts such as instruction type, prop name, parameter list, etc. to ensure the accuracy and readability of the instruction; the system passes the generated prop call instruction to the game system by calling the interface or API provided by the game engine; the game engine is responsible for parsing and executing these instructions to realize the prop effects in the game.

[0104] Specifically, suppose a player is using an AR-based kart racing app and issues a voice command: "Fire three missiles at the enemy on the left!" The system detects that the voice command contains the prop word "missile"; the system parses the associated word "missile" and extracts the associated word "fire at the enemy on the left," indicating how to use the missile and the target; the system also extracts the prop parameter "three," indicating the number of missiles; based on the parsed prop word, associated words, and parameters, the system generates the following prop call command:

[0105] Missile launch command: "Launch three missiles at the enemy on the left" means that the kart will launch three missiles at the enemy on the left;

[0106] The system passes these instructions to the game engine, which is responsible for parsing and executing these instructions to achieve the game effect expected by the player. Through this process, the game system can accurately understand the player's voice instructions and generate corresponding prop call instructions based on the content of the instructions, thereby realizing real-time call of kart props. This interactive method not only increases the fun and immersion of the game, but also provides players with a more convenient and intuitive operation method.

[0107] refer to Figure 5 In step S14, the AR glasses collect the player's eye information and determine the target locking instruction and dynamic feedback instruction of the kart according to the player's eye information;

[0108] In the specific implementation process of the present invention, the specific steps are:

[0109] S141: The infrared camera of the AR glasses tracks eye movements and determines a corresponding eye trajectory map based on the eye movements. The player's eye information is determined based on the recognition of the eye trajectory map. The player's eye information includes the coordinates of the eye gaze point and the eye gaze duration. If the eye gaze duration exceeds a preset gaze duration threshold, the gazed target in the kart game scene is locked, and a corresponding collaborative action is output for the gazed target.

[0110] S142: If the coordinates of the eyeball gaze point are frequently adjusted, a corresponding prop calling instruction is determined according to the adjustment of the coordinates of the eyeball gaze point.

[0111] In an embodiment of the present application, the infrared camera of the AR glasses tracks eye movements, determines a corresponding eye trajectory diagram based on the eye movements, and determines the player's eye information based on the recognition of the eye trajectory diagram; the player's eye information covers the coordinates of the eye gaze point and the eye gaze duration; if the eye gaze duration exceeds the preset gaze duration threshold, the target being gazed at in the kart game scene is locked, and the corresponding collaborative action is output to the target being gazed at, thereby introducing the player's eye information and realizing eye tracking.

[0112] At this time, AR glasses are equipped with a high-precision infrared camera that can capture and analyze the player's eye movements; the infrared camera uses infrared light technology to determine the position and movement trajectory of the eye by detecting the reflected light on the surface of the eyeball; this technology has the characteristics of high sensitivity and high precision, and can track the tiny movements of the eyeball in real time; optionally, the infrared camera is usually equipped with a special filter that only allows infrared light to pass through to reduce interference from ambient light; the camera determines the direction and speed of the eyeball's movement by continuously taking images of the eyeball and comparing the differences between adjacent images.

[0113] The system uses image processing algorithms to convert eye movement data captured by the infrared camera into an eye trajectory map; the eye trajectory map is a two-dimensional or three-dimensional graphical representation that shows the movement path of the eye over a certain period of time; through the eye trajectory map, the system can intuitively understand the player's eye movement pattern; optionally, the image processing algorithm may include edge detection, corner detection, optical flow method, etc., which are used to extract key feature points in the eye image; the system connects these feature points to form the eye movement trajectory and draws it into an eye trajectory map.

[0114] The system extracts the player's eye information by identifying the eye trajectory graph, including the coordinates of the eye gaze point and the duration of eye gaze; the coordinates of the eye gaze point represent the focus position of the player's current line of sight, while the duration of eye gaze represents the length of time the player stays at that position; optionally, the system may use a machine learning algorithm to identify specific patterns in the eye trajectory graph to determine the eye gaze point and duration; the determination of the gaze point coordinates may require correction in combination with the physiological structure of the eye and the viewing angle of the camera.

[0115] The system presets a gaze duration threshold to determine whether the player's gaze behavior constitutes an effective target lock; when the eye gaze duration exceeds this threshold, the system considers that the player has maintained attention on a target and automatically locks on the target; after locking, the system will output corresponding collaborative actions according to the target type, such as attack, pick up, defense, etc.; optionally, the setting of the gaze duration threshold may need to be adjusted according to factors such as game type and player habits; the output of collaborative actions may need to be decided in combination with game logic and player character status.

[0116] Specifically, suppose a player is using a karting game application based on AR technology and wearing AR glasses to play the game; after the player starts the game, the infrared camera of the AR glasses begins to track the player's eye movements; the system draws an eye trajectory map based on the eye movement data captured by the camera; the system identifies the gaze point coordinates and gaze duration in the eye trajectory map; for example, the player stares at an acceleration prop box in front of the track for a long time; when the gaze duration exceeds the preset gaze duration threshold (for example, 3 seconds), the system automatically locks the acceleration prop box as the target and outputs a pick-up collaborative action; at this time, the player's kart will automatically navigate to the acceleration prop box position and pick up the props, thereby obtaining an acceleration effect; through this process, the eye tracking function of the AR glasses not only improves the interactivity of the game, but also enables players to control game characters and props through natural eye movements, thereby obtaining a more immersive gaming experience.

[0117] Furthermore, if the coordinates of the eyeball gaze point are frequently adjusted, the corresponding prop call instruction is determined according to the adjustment of the coordinates of the eyeball gaze point, and the prop call instruction is introduced.

[0118] At this time, the system continuously monitors the coordinates of the player's eye gaze point and detects whether these coordinates are frequently adjusted in a short period of time; frequent adjustments mean that the player's line of sight is moving quickly, possibly searching for a target in the scene or making a decision; optionally, the system captures eye movement data through the infrared camera of the AR glasses and calculates the coordinates of the eye gaze point in real time; the system sets a time window (such as 0.5 seconds or 1 second) and counts the number of changes in the eye gaze point coordinates within the time window; if the number of changes exceeds a preset threshold (such as 5 times or more), it is determined to be a frequent adjustment.

[0119] When the system detects frequent adjustments in the coordinates of the eye gaze point, it will determine the corresponding prop call instructions based on the patterns and trends of these adjustments; prop call instructions may include operations such as attack, defense, acceleration, deceleration, and steering, depending on the game logic and the player's current game status; optionally, the system analyzes the adjustment trajectory of the eye gaze point coordinates to determine whether the player's line of sight is focused on a specific area or target; if the line of sight is focused on the enemy, it may trigger the call instruction of the attack prop; if the line of sight switches quickly between multiple targets, it may mean that the player is looking for the best attack or defense position, and the system may automatically select the most suitable prop according to the current situation; the system also needs to consider the current status of the player character (such as health value, energy value, etc.) and the use restrictions of the props (such as cooling time, number of uses, etc.) to ensure the rationality and effectiveness of the prop call.

[0120] Specifically, suppose a player is using a kart racing game application based on AR technology and wearing AR glasses to play the game; the player drives a kart in the game, and the system continuously monitors the coordinates of the player's eye gaze point; during the race, the player finds that there are multiple bends and obstacles ahead, and there are other players racing at the same time; in order to avoid obstacles and overtake opponents, the player's line of sight quickly switches between the track ahead, obstacles, and opponent vehicles.

[0121] The system detected that the coordinates of the player's eye gaze point adjusted frequently in a short period of time, and determined that the player was making an urgent decision; the system analyzed the player's line of sight adjustment trajectory and found that the player was mainly paying attention to the curve and opponent's vehicle in front; in order to maintain speed and overtake the opponent, the system decided to call the "accelerator" and "nitrogen acceleration" commands; the player's kart automatically used the acceleration props and nitrogen acceleration, instantly increasing the speed, successfully avoiding obstacles and overtaking the opponent in front; through this process, the eye tracking function of AR glasses not only enhances the interactivity and immersion of the game, but also enables players to control the prop calls in the game through natural eye movements, so as to more flexibly respond to various challenges in the game.

[0122] In one embodiment of the present application, the prop call instruction matching table is shown in Table 2:

[0123] Table 2 Prop call instruction matching table

[0124] Eye gaze point coordinate adjustment mode Prop call instructions Quickly scan left and right Use speed boosters Move up and down quickly Launching missiles Rapid shift after prolonged fixation Shield on

[0125] When driving a kart, the player finds that there are multiple bends and obstacles ahead. To meet these challenges, the player's eyes quickly scan left and right on the screen to find the best driving route. The system detects that the coordinates of the player's eye gaze point show an adjustment mode of rapid left and right scanning, which matches the "rapid left and right scanning" mode in the matching table. Therefore, the system triggers the "use acceleration props" command, and the player's kart automatically accelerates to help the player pass the bends and obstacles smoothly.

[0126] refer to Figure 6 In step S15, the AR glasses collect the player's gesture information and determine the second corresponding action instruction of the kart according to the player's gesture information; the second action instruction includes a prop pickup instruction and a melee attack instruction;

[0127] In the specific implementation process of the present invention, the specific steps are:

[0128] S151: collecting a player's gesture image, determining the player's gesture information based on the recognition of the player's gesture image; and determining a gesture type based on the player's gesture information;

[0129] S152: If the gesture type is a pickup type, an item pickup instruction is determined according to the gesture type; if the gesture type is an attack type, a melee attack instruction is determined according to the gesture type.

[0130] In an embodiment of the present application, a player's gesture image is collected, and the player's gesture information is determined based on the recognition of the player's gesture image; a gesture type is determined based on the player's gesture information, and a gesture type is introduced.

[0131] At this time, the system captures the player's hand movements in real time through AR glasses or other camera devices; these camera devices are usually equipped with highly sensitive image sensors and image processing chips, which can capture subtle hand movements and posture changes; when collecting gesture images, the system needs to ensure that the image quality is clear and stable, and contains sufficient information for subsequent gesture recognition algorithms; optionally, the camera device may use infrared technology or visible light technology to capture hand images; the image sensor needs to have sufficient resolution and frame rate to capture subtle hand movements; the image processing chip may include image preprocessing algorithms, such as denoising, contrast enhancement, etc., to improve image quality.

[0132] Perform feature extraction and pattern recognition on the collected gesture images; in the feature extraction stage, the system will extract key features such as the hand contour, shape, color, texture, etc. from the image; in the pattern recognition stage, the system will match the extracted features with predefined gesture templates to determine the player's gesture information; optionally, the feature extraction algorithm may include edge detection, contour extraction, shape analysis, etc.; the pattern recognition algorithm may use machine learning models such as convolutional neural networks (CNN), support vector machines (SVM), etc. These models need to be trained in advance to learn the features of the gesture templates; the gesture templates include sample images of multiple gestures, and these sample images need to be representative and able to cover various gestures that the player may make.

[0133] The system classifies the recognized gesture information into one of the predefined gesture types based on the information; gesture types may include picking, attacking, defending, navigating, interacting, and other types, depending on the game settings and requirements; in order to determine the gesture type, the system may need to further process and analyze the gesture information, such as calculating the direction, speed, strength and other parameters of the gesture; optionally, the determination of the gesture type may require a comprehensive judgment based on multiple gesture features; the system may need to set gesture type thresholds or rules to distinguish different gesture types; the result of the gesture type determination will directly affect the subsequent game logic and the behavior of the player character.

[0134] Specifically, suppose a player is using an adventure game application based on AR technology and wearing AR glasses to play the game; the player makes a gesture of opening the palm and facing a certain direction in the game; the camera device of the AR glasses captures the image of this gesture in real time and transmits it to the system for processing; the system performs feature extraction and pattern recognition on the collected gesture image; through edge detection and contour extraction algorithms, the system extracts the contour features of the hand and matches it with the predefined gesture template; after matching, the system recognizes the player's gesture information as "open palm and facing a certain direction".

[0135] Based on the recognized gesture information, the system classifies it as a "pick up" type gesture; because in this game, the "open palm and face in a certain direction" gesture is set as an instruction to pick up props; the system further calculates the direction of the gesture, determines the location of the prop the player wants to pick up, and triggers the corresponding game logic, so that the player's character automatically moves to the prop location and picks up the prop; through this process, the gesture recognition function of AR glasses not only enhances the interactivity and immersion of the game, but also enables players to control game characters and props through natural gesture movements, so that they can more flexibly respond to various challenges in the game.

[0136] Furthermore, if the gesture type is a picking type, the item picking instruction is determined according to the gesture type; if the gesture type is an attack type, the melee attack instruction is determined according to the gesture type. Melee attack instructions and item picking instructions are introduced. At the same time, the player's eye information and gesture information are collected to facilitate multi-dimensional control of the player's head movement, voice information, eye information and gesture information, thereby realizing the player's game effect in the karting game scene.

[0137] At this time, the system first determines whether the current gesture type is a picking type based on the gesture information determined in step S151; the picking type gesture is usually associated with the player's behavior of wanting to obtain a prop or item in the game; optionally, the system presets a set of mapping relationships between gesture types and corresponding behaviors. When gesture information is recognized, the gesture type is determined by looking up the mapping relationship; the picking type gesture may have specific characteristics, such as an open palm, facing a specific direction, etc., and the system needs to be able to recognize these characteristics.

[0138] If the system determines that the gesture type is a picking type, it will determine the prop picking instruction based on the specific information of the gesture (such as the direction and position of the gesture); the prop picking instruction will guide the character or object in the game to perform the picking action; optionally, the system may need to calculate the relative relationship between the direction of the gesture and the position of the prop in the game to determine the target of the picking action; the picking instruction may include the character's movement instruction, the triggering instruction of the picking animation, etc.

[0139] If the system determines that the gesture is not a pickup type, it will further check whether it is an attack type; attack type gestures are usually associated with the player's behavior of wanting to attack enemies or objects in the game; attack type gestures may have different characteristics, such as clenched fists, rapid swings, etc., and the system needs to be able to distinguish these characteristics; the system may need to consider parameters such as the strength and speed of the attack gesture to determine the intensity and effect of the attack action.

[0140] If the system determines that the gesture type is an attack type, it will determine the melee attack command based on the specific information of the gesture (such as the strength and direction of the gesture); the melee attack command will guide the character in the game to perform the attack action; optionally, the system needs to calculate the relative relationship between the direction of the attack gesture and the enemy position in the game to determine the target of the attack action; the attack command may include the character's attack animation trigger, damage calculation, enemy hit feedback, etc.

[0141] Specifically, suppose that the player is using a role-playing game application based on AR technology and wearing AR glasses to play the game; the player sees a healing potion item in front of him in the game, so he makes a gesture with his palm open and facing the healing potion; the system recognizes the gesture information and determines that the gesture type is a picking type; the system determines the item picking instruction based on the specific information of the picking type gesture (the gesture is facing the healing potion); the instruction includes guiding the player character to move to the location of the healing potion and execute the picking animation; the player character successfully picks up the healing potion and restores a certain amount of health.

[0142] Later, the player encounters an enemy in the game, so he makes a gesture of clenching his fist and quickly swinging it at the enemy; the system recognizes the gesture information and determines that the gesture type is an attack type; the system determines the melee attack command based on the specific information of the attack type gesture (gesture strength, direction towards the enemy); the command includes triggering the player character's attack animation, calculating the damage value to the enemy, and displaying the feedback effect of the enemy being hit; the player character successfully hits the enemy, and the enemy's health value decreases; through this process, the gesture recognition function of AR glasses not only enhances the interactivity and immersion of the game, but also enables players to control the game character to perform actions such as picking up and attacking through natural gesture movements, so as to more flexibly respond to various challenges in the game.

[0143] See also Figure 7 , Figure 7 : is a schematic diagram of the structure of a virtual interaction system based on AR glasses in an embodiment of the present invention; the virtual interaction system based on AR glasses includes:

[0144] The scene module 21 is used to map the dynamic data of the player's head to the kart game scene when the player wears AR glasses;

[0145] The head action module 22 is used to determine the attack and movement instructions of the kart according to the player's head action in the kart game scene;

[0146] The voice information module 23 is used for the AR glasses to collect the player's voice information and determine the first action instruction of the kart according to the player's voice information; the first corresponding action instruction includes a skill activation instruction and a prop call instruction;

[0147] The eye information module 24 is used by the AR glasses to collect the player's eye information and determine the target locking instruction and dynamic feedback instruction of the kart based on the player's eye information;

[0148] The gesture information module 25 is used by the AR glasses to collect the player's gesture information and determine the second corresponding action instructions of the kart according to the player's gesture information; the second action instructions include prop pickup instructions and melee attack instructions.

[0149] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A virtual interaction method based on AR glasses, characterized in that: include: When the player wears AR glasses, the dynamic data of the player's head is mapped to the kart game scene; In the kart game scene, the kart's attack and movement instructions are determined based on the player's head movements.

2. The virtual interaction method based on AR glasses according to claim 1, characterized in that: When the player wears AR glasses, the dynamic data of the player's head is mapped to the kart game scene, including: Collect dynamic data of the player's head; construct a corresponding three-dimensional coordinate system based on the dynamic data of the head, and map the three-dimensional coordinate system to the kart game scene; use the built-in gyroscope and accelerometer of the AR glasses to capture the rotation angle of the player's head in real time.

3. The virtual interaction method based on AR glasses according to claim 1, characterized in that: In the kart game scene, determining the kart's attack and movement instructions based on the player's head movements includes: Collect the player's head movements and determine nodding and shaking movements based on the analysis of the player's head movements; After marking the target to be attacked in the kart game scene, the corresponding attack instruction is determined based on the action information of the nodding action; and the corresponding movement instruction is determined based on the action information of the shaking action.

4. The virtual interaction method based on AR glasses according to claim 1, characterized in that: The virtual interaction method based on AR glasses further includes: collecting voice information of the player, and determining a first action instruction of the kart according to the voice information of the player; the first corresponding action instruction includes a skill activation instruction and a prop call instruction; The collecting of the player's voice information and determining the first action instruction of the kart according to the player's voice information; the first corresponding action instruction includes a skill activation instruction and a prop call instruction, including: Collecting the player's voice information, dividing the voice information into multiple segments based on the detection of the player's voice information, and determining corresponding action keywords and corresponding action-related words based on the recognition of the multiple voice segments; at this time, when collecting the player's voice information, noise reduction processing is performed on the ambient noise; The first action instruction of the kart is determined according to the action keyword and the corresponding time-related word; the first corresponding action instruction includes a skill activation instruction and a prop calling instruction.

5. The virtual interaction method based on AR glasses according to claim 4, characterized in that: The collecting of the player's voice information and determining the first action instruction of the kart according to the player's voice information; The first corresponding action instruction includes a skill activation instruction and an item call instruction, and also includes: When an action keyword contains a corresponding skill word, the skill activation instruction is determined based on the skill word and action-related words in the action keyword; When the action keyword contains a corresponding prop word, the prop call instruction is determined according to the prop word of the action keyword and the action associated word.

6. The virtual interaction method based on AR glasses according to claim 1, characterized in that: The virtual interaction method based on AR glasses further includes: collecting eye information of the player, and determining a target locking instruction and a dynamic feedback instruction of the kart according to the eye information of the player; The collecting of the player's eye information and determining the target locking instruction and dynamic feedback instruction of the kart according to the player's eye information include: The infrared camera of the AR glasses tracks eye movements, determines the corresponding eye trajectory map based on the eye movements, and determines the player's eye information based on the recognition of the eye trajectory map; the player's eye information covers the coordinates of the eye gaze point and the eye gaze duration; if the eye gaze duration exceeds the preset gaze duration threshold, the target being gazed at in the kart game scene will be locked, and the corresponding collaborative action will be output to the target being gazed at.

7. The virtual interaction method based on AR glasses according to claim 6, characterized in that: The method of collecting the player's eye information and determining the target locking instruction and dynamic feedback instruction of the kart according to the player's eye information also includes: If the coordinates of the eyeball gaze point are frequently adjusted, the corresponding prop calling instruction is determined according to the adjustment of the coordinates of the eyeball gaze point.

8. The virtual interaction method based on AR glasses according to claim 1, characterized in that: The virtual interaction method based on AR glasses further includes: collecting gesture information of the player, and determining a second corresponding action instruction of the kart according to the gesture information of the player; the second action instruction includes a prop pickup instruction and a melee attack instruction; The collecting of the player's gesture information and determining a second corresponding action instruction of the kart according to the player's gesture information; the second action instruction including a prop pickup instruction and a melee attack instruction, including: The player's gesture image is collected, and the player's gesture information is determined based on the recognition of the player's gesture image; and the gesture type is determined based on the player's gesture information.

9. The virtual interaction method based on AR glasses according to claim 8, characterized in that: The method collects the player's gesture information and determines a second corresponding action instruction of the kart according to the player's gesture information; the second action instruction includes a prop pickup instruction and a melee attack instruction, and further includes: If the gesture type is a pickup type, the item pickup instruction is determined based on the gesture type; if the gesture type is an attack type, the melee attack instruction is determined based on the gesture type.

10. A virtual interaction system based on AR glasses, characterized in that: The AR glasses-based virtual interaction system is applied to the AR glasses-based virtual interaction method according to any one of claims 1 to 9, and the AR glasses-based virtual interaction system includes: The scene module is used to map the player's head dynamic data to the kart game scene when the player wears AR glasses; The head action module is used to determine the kart's attack and movement instructions based on the player's head movements in the kart game scene; A voice information module is used for the AR glasses to collect the player's voice information and determine the first action instruction of the kart based on the player's voice information; the first corresponding action instruction includes a skill activation instruction and a prop call instruction; The eye information module is used by AR glasses to collect the player's eye information and determine the kart's target locking instructions and dynamic feedback instructions based on the player's eye information; The gesture information module is used by the AR glasses to collect the player's gesture information and determine the second corresponding action instructions of the kart based on the player's gesture information; the second action instructions include prop pickup instructions and melee attack instructions.

Images